ListGrid: updateCaches doesn't rearrange records if it has no full cache. What to do?

[d.cavestro]

Calling DataSource.updateCaches() doesn't rearrange ListGrid records when the update changes the value for a field on which the grid bound to the DataSource has sorting enabled AND the ListGrid has no full cache.

I can reproduce it on a 50 records clientonly datasource and a ListGrid with no full cache (ListGrid.drawAllMaxCells() disabled, ListGrid.drawAheadRatio() set to 1 and data page size reduced to 25) when I directly scroll to the table bottom (hence skipping the fetch of some records, so that the cache is not complete) and then I update the cache for record 45

See attached screenshot and code for a complete standalone example.

To reproduce the issue on the standalone example you should:
* load the page in dev mode
* press the scroll to row button or scroll to the record that you are going to update (by default it's 45, just to skip fetching some records)
* press the update cahches button

At this point the name field changes so that it should be repositioned to the top of the table (as per sorting defined on that field), while it remains on the original position.

I guess this happens when the local cache is not full, hence the ListGrid doesn't do a local sort. In fact the logs say

Global Log Priorities updated: Logging messages at priority 'Debug' and above for category 'ResultSet'.
Global Log Priorities updated: Logging messages at priority 'Debug' and above for category 'ListGrid'.
17:59:14.700:MUP9:DEBUG:ResultSet:isc_ResultSet_0 (created by: isc_ListGrid_0):dataSource data changed firing
17:59:14.701:MUP9:INFO:ResultSet:isc_ResultSet_0 (created by: isc_ListGrid_0):updating cache in place after operationType: update, cached rows: 44, total rows: 50
17:59:14.703:MUP9:INFO:ResultSet:isc_ResultSet_0 (created by: isc_ListGrid_0):Updating cache: operationType 'update' (no componentID) ,1 rows update data:
[
{__ref: {GWT Java Obj},
id: 45,
name: "aaarow 45 with changed value!!!"}
]
17:59:14.707:MUP9:DEBUG:ResultSet:isc_ResultSet_0 (created by: isc_ListGrid_0):updated cache: 0 row(s) added, 1 row(s) updated, 0 row(s) removed.
17:59:14.708:MUP9:DEBUG:ResultSet:isc_ResultSet_0 (created by: isc_ListGrid_0):full length set to: 50
17:59:14.736:TMR3:DEBUG:ResultSet:isc_ResultSet_0 (created by: isc_ListGrid_0):getRange(38, 50) satisfied from cache

when updating row 45, while updating row 35 it says

17:59:55.822:MUP5:DEBUG:ResultSet:isc_ResultSet_0 (created by: isc_ListGrid_0):dataSource data changed firing
17:59:55.823:MUP5:INFO:ResultSet:isc_ResultSet_0 (created by: isc_ListGrid_0):updating cache in place after operationType: update, allMatchingRowsCached true
17:59:55.825:MUP5:INFO:ResultSet:isc_ResultSet_0 (created by: isc_ListGrid_0):Updating cache: operationType 'update' (no componentID) ,1 rows update data:
[
{__ref: {GWT Java Obj},
id: 35,
name: "aaarow 35 with changed value!!!"}
]
17:59:55.831:MUP5:DEBUG:ResultSet:isc_ResultSet_0 (created by: isc_ListGrid_0):updated cache: 0 row(s) added, 1 row(s) updated, 0 row(s) removed.
17:59:55.837:MUP5:INFO:ResultSet:isc_ResultSet_0 (created by: isc_ListGrid_0):$391: sorting on properties [name] : directions [true] : full cache allows local sort
17:59:55.868:TMR7:DEBUG:ResultSet:isc_ResultSet_0 (created by: isc_ListGrid_0):getRange(30, 42) satisfied from cache

I guess this behavior is by design... how could the client sort data if the entire dataset is not available?
So if there's no way to update an incomplete cache and still obtain valid record sorting should systematically invoke a fetch after ListGrid.updateCaches ()? This would waste the cache update... or should instead try to find a way to detect if the cache is full (hence allowing local sort): in this case proceed with the cache update, otherwise falling into a fetch?

Reproduced with SmartGWT 3.1 LGPL [SmartClient Version: v8.3_2012-11-20/LGPL Development Only (built 2012-11-20)]

[Isomorphic]

You are correct, this behavior is by design.

From a usability perspective, records are typically updated by some user action, and having the record suddenly sort out of the viewport or having the entire list reload are both very bad. So the grid goes into an "unsorted" state and various interactions (such as scrolling such that new records need to be loaded) will then drop the cache and fetch fresh rows.

[d.cavestro]

...the grid goes into an "unsorted" state and various interactions (such as scrolling such that new records need to be loaded) will then drop the cache and fetch fresh rows.

In fact only scrolling to an area never drawn I see

12:12:31.306:SCR8:INFO:ResultSet:isc_ResultSet_0 (created by: isc_ListGrid_0):getRange(15, 27) will fetch from 13 to 38
12:12:31.701:TMR0:INFO:ResultSet:isc_ResultSet_0 (created by: isc_ListGrid_0):fetching rows 13,38 from server
12:12:31.706:TMR0:INFO:ResultSet:isc_ResultSet_0 (created by: isc_ListGrid_0):invalidating rows on fetch due to 'add'/'update' operation with updatePartialCache

Clear... only one doubt: isn't it a bit strange having to scroll to an area/page not cached yet in order to force the drop of the cache?
I mean usually the user is really not aware of this hidden dynamics and will get new records (hence refreshing removing the "unsorted" state effects) on a pure fortuitous basis (scrolling up and down just to search some data).

For certain scenarios, during a one-row cache update I would implement this (for every ListGrid bound to that datasource):
* save the current selection (tipically on the record that changed)
* detect if that updateCaches carries a change on a field implied on some sorting
* if so, at least invalidate the cache, or at best try to see if the change on sorted data can be satisfied reordering available records within the partial cache (though I guess potentially it has too many corner cases)
* restore the saved selection

Do you think I could implement this behavior? Are there any particular advice against implementing it? Maybe I could wrap the one-record cache update just to implement the above pseudo-code, provided that I have a way to enumerate ListGrids bound to the datasource (that's really needed, cause I also have some code that updates the datasource and doesn't know the grid at all).

[Isomorphic]

Scrolling down to reveal more data would, in general, show the user new records. Nothing is different here - the user's expectations are being met perfectly.

No, we wouldn't recommend implementing your proposal - it seems just worse, and we don't really understand what perceived problem you're trying to solve.

[d.cavestro]

Scrolling down to reveal more data would, in general, show the user new records. Nothing is different here - the user's expectations are being met perfectly.
...we don't really understand what perceived problem you're trying to solve.

I was thinking about this scenario: the user scrolls directly to the end of the ListGrid to edit a record (let's say he skips caching some data pages).
It changes the value for a field on which the grid is sorting, hence he would expect the record to be moved (say however remaining within the current grid viewport): this doesn't happens cause the client has no full cache, hence the grid moves to "unsorted" state.
For what we said above at this point the user has to scroll up enough to trigger a fetch and then scroll down again to re-fetch updated data and see the record moved at the position he expected.
But the user doesn't know enough about caching/sorting internals to adopt this tricks.

Moreover I guess the worst case would be when the client cache has almost all data but is not full... in this case the user would have to scroll to the last missing page (wherever it may be) in order to trigger the re-fetch and get the record moved at the right position.
Al these considerations led me to propose the described behavior, but I'm really interested on why you consider it worse than the depicted scenario.

[Isomorphic]

What the user actually cares about is that the row he has just edited doesn't vanish out of the visible area of the grid.

We've never encountered a user who is excited to see the record disappear, so, there is no motivation for the user to start scrolling around to try to make it happen.