Unexpected ListGrid record selection behavior with filters when data is updated

SmartClient Version: v9.1p_2014-07-13/LGPL Development Only (built 2014-07-13)

I have a case where I am getting selection behavior in the grid that I did not expect. I don't know if this is a defect or if my expectations are incorrect or if there are settings that might allow me to control this behavior.

Sample code is attached. You should be able to load this control into a page and see the behavior.

I have a grid with a filter attached to it. I also have asynchronous server events causing updates to the data source backing the grid.

If a record is selected in the grid and I update the data for that record such that it no longer matches the filter, the record disappears from the grid, but if there is a record after it in the grid, that record becomes selected.

Using the sample code, select the record in the grid with ID "0" (the sample code selects this record automatically for you), the press the "Update selected record button" (this simulates my background server events that update the data source). This will modify the grid data such that record "0" no longer matches the criteria. Upon doing this, you will see record "0" removed from the grid display, and now record "1" is selected. I was expecting that no records would be selected after this and that a selection change event would fire.

If you select record "1" instead you will see similar behavior, but record ID "2" will be selected after the update.

If a selected record in the grid gets filtered out after an update, I would expect it to drop from the grid's selected record list without any new records being selected and a selection change event to be fired. It does not seem like correct behavior to me that filtering a record on update would cause a new record in the grid to get selected when the user did not select that record.

[Isomorphic]

We've made a change to address this issue
Please try the next nightly build dated Aug 8 or above.

Regards
Isomorphic Software

Thanks for the fix. I finally got around to downloading (8/14 build) and trying it. When I ran the sample code that I sent you, the problem was fixed. However, when I used it in our product we are still having the same issue. In our product our data source is hooked up to a REST service (unlike the sample where it is a client side data source), so that my be the issue or it may be something else we are doing in our product that is affecting the behavior. It will be a little while before I have time to try to analyze the differences between the sample and our product, but I thought I let you know anyway. Maybe there are other pathways in your code that have the same issue as the one you did fix? Just a thought.

[Isomorphic]

Nothing springs to mind, unfortunately, so I think we'll have to wait for a test case.
It ought to be pretty easy to expand this test case to use a REST rather than a client-only DS - you can simply point the fetch and the update requests to different static files so you always get the same result that you're getting in your real app.

Regards
Isomorphic Software

I'm still having problems getting things to work correctly when I want to dynamically update my grid data. I think that I am going to need to put together a more comprehensive example for you to demonstrate the issues. However, this is a significant undertaking. Before I go to all that effort, I would like to get some feedback from you about whether or not I am even going about the updates correctly.

What we are trying to do is associate a ListGrid with a DataSource that we retrieve initially with a REST call. After we have the data, we get events from our server about updates to the data. Based upon these update we need to handle the following types of changes:

1. An existing record has some fields whose value changed.
2. A new record is added.
3. An existing record is deleted.

In addition, the grid usually has filters applied which should case a record that has been added to not show, or a record that has been updated to transition from visible to hidden.

The grid may also be sorted (and the updates may affect sort fields).

The grid may also be grouped (and the updates may affect the group-by fields).

The grid may have records that are selected and the updates may affect records that are currently selected.

We want the grid to maintain selection, filter records in and out as necessary, maintain the records in sorted order, and move records between groups as necessary.

We are attempting to do this by calling updateCaches on our data source (see attached sample code). We really do not know if this is the best/recommended way to do this. Guidance here would be greatly appreciated.

We also need these updates/adds to be fast when there could be 2500 entries in the grid (including adjustments if a filter, sort, or group by field changes).

We seem to need to force a full resort of the grid when a sort field changes which seems really slow when there are a 1000 records in the grid and seems overkill when just one record potentially needs to move. However the re-ordering of the updated record based upon the sort does not seem to happen automatically when we update the record.

So, before I try to work up a sample outside of our full app, can you indicate whether or not our approach is even correct? If not, can you please explain the recommended approach?

[Isomorphic]

From your description it sounds like
- Your REST DS does an initial fetch against a web service.
- You then get information from the server via some parallel mechanism (other RPC requests, etc), notifying you of server side changes, which you need the app to reflect. These changes are basically standard CRUD operations - deletions, additions and modifications.

The DataSource.updateCaches() mechanism is designed for handling this. Essentially what you do is assemble a DSResponse yourself which reflects the operation type and what has changed, and pass it to the dataSource. So for a modification of some field value for some record, you'd set the operationType to "update", and the data to the modified record (including the primary key of course), and pass that to the DataSource.
The framework will then handle notifying any components which are displaying the data from that dataSource and causing them to refresh, etc.

This appears to be what your code sample is doing - though you're also calling "markForRedraw" on a grid (which shouldn't be necessary - the DataSource will cause things to refresh if they need).

Note that this should handle filtering, maintaining selection, regrouping, etc.
Also, for large grids with paged ResultSets, the changes should be integrated into the grids data fine.

So - in short, your approach seems fine and we may well need to see a sample to comment further.

Regards
Isomorphic Software

OK, here's a starting point. Code attached.

What I'm seeing is that when adding a record, the sort, filter, selection, and group by are all being honored.

If a select a record and use one of the other buttons at the bottom of the window to update the record then...

The selection are being honored in all cases.

If I update the content field (the update is implemented to change the field in such a way that the record will be filtered out), then the filter is being honored.

However, the following cases do not seem to be working properly.

If I press the button to update the sort field, the record does not move to the new position. Note the update button uses a random number generator to pick a new sort value, so it may not change and you will need to press the button multiple times until it does change.

If I group by the status column and open all of the groups, and then I press the button to update the status field, the record is not moved into the correct group. Note the update button uses a random number generator to pick a new status value, so it may not change and you will need to press the button multiple times until it does change.

The other thing I am noticing is that if (in the data source class) I increase the number of records to 2500, the adds take significantly longer (running in production mode, and everything seems to take a really long time when in development mode, even just loading the grid the first time).

[Isomorphic]

We'll take a look at your test cases and follow up when we have more information

Regards
Isomorphic Software

[Isomorphic]

We've spent some time analyzing this situation and have made some changes to ensure things work as they should (present in the latest nightly build - 9.1, 10.0 branches, and above).

Here's a summary of what we've found / what's changed:
----
Issue 1:

If I press the button to update the sort field, the record does not move to the new position.

When the grid's ResultSet has a full cache of matching rows, then the record will move to the new position because client-side sorting should be enabled.

When the grid's ResultSet does not have a full cache (which can happen with data paging involved- see ListGrid.setDataFetchMode()), behavior depends on the value of ResultSet.updatePartialCache.

If this property is true (the default), then the record will remain at its current row numb rather than being sorted into place. This behavior is actually documented under the 'updatePartialCache' attribute documentation:

... updated rows will remain in their current position. No attempt will be made to sort them into a new position even if the sort field was updated.

However there was a bug here - the sort icon continues to show for the field. We've made a change to address this - the grid will now appear "unsorted" after the update in this case.

If ResultSet.updatePartialCache is false, then the partial cache is invalidated whenever an update occurs. This will cause the grid to fetch the visible window of data again, and the grid will remain sorted.

The default of updatePartialCache is true. To use updatePartialCache:false, the dataProperties of the grid can be set:

        ResultSet dataProperties = new ResultSet();
        dataProperties.setUpdatePartialCache(false);
        grid.setDataProperties(dataProperties);

Note: updatePartialCache:false implies that a fetch will occur against the dataSource when the 'updateCaches' API runs.
In normal usage, if you call "updateCaches" you're essentially notifying the client of a change which has already occurred to the server-side data, so the fetch will of course pick up the same updated record.

In sample code, where we're working with a client-only dataSource, you'd want to mimic this by directly updating the record in the clientOnly dataSource's cache before you call the updateCaches API - something like this:

    // helper to find a record by ID from an array
    public static <R extends Record> R find(R[] records, String property, Object value) {
        if (records == null) return null;
        for (R record : records) {
            if (record == null) continue;
            Object recordValue = record.getAttributeAsObject(property);
            if ((value == null && recordValue == null) ||
                (value != null && value.equals(recordValue)))
            {
                return record;
            }
        }
        return null;
    }

//...


        button = new Button("Update sort field of selected");
        button.addClickHandler(new ClickHandler() {
            @Override
            public void onClick(ClickEvent event) {
                Record[] origData = ds.getCacheData();
                for (ListGridRecord record: grid.getSelectedRecords()) {
                    Record updatedRecord = ds.copyRecord(record);
                    updatedRecord.setAttribute("sortField", FullGridSampleDataSource.getSortValue());
                    // Modify the record in the DataSource's cache-data - 
                    // equivalent to a server-side transaction occurring 
                    // outside the app in a "real life" scenario.
                    Record origRecord = find(origData, "idField", record.getAttributeAsInt("idField"));
                    origRecord.setAttribute("sortField", updatedRecord.getAttributeAsInt("sortField"));
                    // fire the updateCache API to cause dataBound components
                    // to react to the update
                    updateCache(grid, updatedRecord, DSOperationType.UPDATE);
                }
            }
        });

----
Issue 2:

If I group by the status column and open all of the groups, and then I press the button to update the status field, the record is not moved into the correct group.

The problem is that your logic is directly modifying the result of ListGrid.getSelectedRecords().
The records returned by getSelectedRecords() are the underlying record objects displayed in the grid, and should be treated as read-only.
Your code currently modifies the records directly, and then passes those modified records to the updateCaches method.
There is logic in the ListGrid to check for changes between the current data objects and the updated values -- since you've already applied the changes directly to the records, this is detecting no change and re-grouping isn't occurring.

One solution is to call DataSource.copyRecord() to copy the record before you modify it and pass the modified version to updateCaches():

        button = new Button("Update status field of selected");
        button.addClickHandler(new ClickHandler() {
            @Override
            public void onClick(ClickEvent event) {
                Record[] origData = ds.getCacheData();
                for (ListGridRecord record: grid.getSelectedRecords()) {
                    String currentStatus = record.getAttribute("statusField");
                    // Duplicate the record so we don't directly modify
                    // data objects within the grid.
                    Record updatedRecord = ds.copyRecord(record);
                    updatedRecord.setAttribute("statusField", FullGridSampleDataSource.getDifferentStatusValue(currentStatus));

                    // Modify the record in the DataSource's cache-data - 
                    // equivalent to a server-side transaction occurring 
                    // outside the app in a "real life" scenario.
                    Record origRecord = find(origData, "idField", record.getAttributeAsInt("idField"));
                    origRecord.setAttribute("sortField", updatedRecord.getAttributeAsInt("sortField"));
                    // fire the updateCache API to cause dataBound components
                    // to react to the update
                    updateCache(grid, updatedRecord, DSOperationType.UPDATE);
                }
            }
        });

----
Issue 3:

The other thing I am noticing is that if (in the data source class) I increase the number of records to 2500, the adds take significantly longer

This is expected when showAllRecords is enabled. In this mode the grid is loading, and drawing every record in the dataSet. This negates the effects of both data paging (reducing the amount of data which must be loaded at once) and incremental rendering (reducing the overhead generating thousands of cells' worth of HTML).

Regards
Isomorphic Software

Thank you for all of the help/info. I have a few clarification questions:

Note: Unlike the sample code, my app does not really use a client-only data source. It actually uses REST to get the data from the server, but it requests that all of the records be fetched up front.

1. On issue 1, you mentioned the use of updatePartialCache, but it seems from your description that this is only relevant if the ResultSet does not have a full cache. My settings on the grid/dataSource cause it to fetch all of the data up front. If I understand correctly, that means that this setting will not apply in my case. Is that correct?

2. On issue 2, you mentioned copying the "selected record" that I found. Unfortunately, this is another case where I did something different to create a simpler example. My actual app does something more like this:

Record record = null;
ResultSet resultSet = grid.getOriginalResultSet();
if (resultSet != null) {
  record = resultSet.find("ID", value);
}
return record;

I assume from your description that this is a case where I should copy the record as well. Is that correct?

3. If I load all of the data up front, but don't set "showAllRecords" to true, it will slow down scrolling through the data, but it will not re-fetch data from the server - correct?

Sorry, another question about issue #1. Why is it not working in my sample code. This is a client-only data source. Doesn't it have a full cache? It sounds like it should have resorted automatically on update since it had all of the data, but it did not resort. Is there some reason it thinks that it doesn't have all of the data?

[Isomorphic]

1. On issue 1, you mentioned the use of updatePartialCache, but it seems from your description that this is only relevant if the ResultSet does not have a full cache. My settings on the grid/dataSource cause it to fetch all of the data up front. If I understand correctly, that means that this setting will not apply in my case. Is that correct?

There are 2 caches we're talking about here. One is the ResultSet cache and one is the DataSource cache.
To put this another way
- You have a ResultSet which requests data from its DataSource. Depending on the specified fetchMode (which as we noted can be set via 'dataFetchMode' on the ListGrid), it may request just "windows" of data (the first 75 rows for example, and then ask for more as the user scrolls), or it may request a complete filtered dataSet (everything that matches the current criteria) -- or it may request a complete dataSet and then perform filtering entirely within the ResultSet itself.
- You also have your dataSource which fulfills these requests. In your case it's a REST dataSource so it is issuing requests to the server and processing REST format responses.
A totally standard dataSource would simply pass the requests from the client to the server for each fetch.
However if you have DataSource.cacheAllData set (or autoCacheAllData), then instead of doing this, it'll issue a one-time fetch request to the server for the full set of data, and then provide responses to requests from that cache.

The updatePartialCache attribute is a ResultSet setting - it impacts the case where your ResultSet's fetchMode is "paged" [IE it is picking up windows of data from the DataSource], and it hasn't yet filled all the rows, so can't do certain things like sort or filter without referring back to the DataSource.

If you have ResultSet.dataFetchMode set to "local" or "basic" then updatePartialCache has no effect.

----

2. On issue 2, you mentioned copying the "selected record" that I found. Unfortunately, this is another case where I did something different to create a simpler example. My actual app does something more like this:
_________
Code:
Record record = null;
ResultSet resultSet = grid.getOriginalResultSet();
if (resultSet != null) {
record = resultSet.find("ID", value);
}
return record;
__________

I assume from your description that this is a case where I should copy the record as well. Is that correct?

Yes - in this case you're getting hold of the record directly from the ResultSet - the underlying data object as far as the ResultSet (and the ListGrid) are concerned. If you modify this directly, and then tell the DataSource to "updateCaches" with the modified record, the ResultSet logic is going to compare the updated record with the underlying record. Since you've modified that underlying record already it won't realize things have changed and thus regroup. You could hack around this by explicitly calling "regroup" or similar, or you could simply copy the record before manipulating it.

----

3. If I load all of the data up front, but don't set "showAllRecords" to true, it will slow down scrolling through the data, but it will not re-fetch data from the server - correct?

Define "slow" in this context! :)
Your basic premise is right, I think.
If showAllRecords is false, the grid will render incrementally, meaning that it'll redraw as the user scrolls. this means that if they drag the scrollthumb over a previously unseen area, there can be a visible redraw flash, rather than having all the HTML already rendered.
We work hard to minimize such effects, and there are some configuration settings you can play with, such as the drawAheadRatio as well.

On the other hand, if you set showAllRecords to true, the grid will create and render out HTML for every cell in the grid on every redraw (happens in a number of cases you may not even think about, like sort, resizing columns, reacting to data changes, etc). For a large number of rows and columns this can be significantly slow.

This is all somewhat independent from whether the data is all fetched and cached by the resultSet. Obviously if showAllRecords is true the ResultSet has to load all the visible rows so it can render them, making "paged" mode unsupported.
However you can have your ResultSet with "local" or "basic" fetch mode, and still have incremental rendering [IE have showAllRecords be false].

Probably best to experiment with these settings in your app to get the user-experience you feel is best.

Regards
Isomorphic Software

I think I followed your recommendations in my sample code (see updated files attached). Things seem much better.

However, when I try my GroupBy scenario, I lose my record selection. The record moves to the new group, but it is no longer selected. I have not added any code to see if the grid is firing a selection changed event in this case.

Also, the bug fix for the sort indicator that you referred to - is that in the current builds available for download, or do I need to wait for the next set of builds to get it?

I am currently using:
SmartClient Version: v9.1p_2014-09-10/LGPL Development Only (built 2014-09-10)

UPDATE: I added an selectionUpdatedHandler and even though the selection went away in the grid after the update, this event did not fire.

Here's the latest version of my sample code. With this, in general things seem to be working very well (including performance).

However, I am still seeing some issues when I am in group by mode. At this point, it seems that any update causes the grid to re-group (which takes a while with 2500 records). And during the re-grouping it loses any selection in the grid.

Maybe it's really necessary, but I do not understand why updates to fields that are not part of the "group by" set (e.g., i group on my status column, but update my sort or content columns) causes/requires the grid to be regrouped. I am also not certain why it loses selection when automatic re-grouping takes place.

Can you please verify what I am seeing and whether or not this is correct behavior? If it is correct behavior, I would appreciate it if you could explain why?