Filtered ListGrid: new matching records don't appear after DataSource.updateCaches

[d.cavestro]

I have some lists grid showing "pre-filtered data views" based on a particular field value. When some data changes arrive (triggered by events external from SC datasources), I use DataSource.updateCaches with update operation type to integrate them.
The problem is that when the changes would "move" a record from a list grid to another (since the "discriminator field" value has changed) the record doesn't appear in the grid where the filter should match.

Could you please give it a look?
The grids loads all data, hence filtering them at client side...

I've attached a complete example, however the relevant bits are:

private void simulateCustomServerRountripAndUpdate (final Record record) {
    final DSRequest dsRequest = new DSRequest ();
    dsRequest.setOperationType (DSOperationType.UPDATE);
    dsRequest.setDataSource (mainDataSource.getID ());
    final DSResponse dsResponse = new DSResponse ();
    dsResponse.setData (new Record[]{record});
    mainDataSource.updateCaches (dsResponse, dsRequest);
}

where the grids are created using

    final ListGrid listGrid = new ListGrid ();
    listGrid.setDataFetchMode (FetchMode.LOCAL);
    listGrid.setAutoFetchData (true);
    listGrid.setDataSource (getMainDataSource ());
    listGrid.setShowFilterEditor (true);
    listGrid.setHeight (130);
    listGrid.setSelectionType (SelectionStyle.SINGLE);

    if (optionFilter != null) {
        listGrid.setInitialCriteria (new Criteria ("option_id", optionFilter.toString ()));
    }

    final ListGridField idField = new ListGridField ("id");
    final ListGridField nameField = new ListGridField ("name");
    final ListGridField optionField = new ListGridField ("option_id");
    optionField.setOptionDataSource (getOptionDataSource ());
    optionField.setValueField ("id");
    optionField.setDisplayField ("name");

From developer console I see

12:14:48.829:MUP0:INFO:ResultSet:isc_ResultSet_0 (created by: isc_ListGrid_0):updating cache in place after operationType: update, allMatchingRowsCached true
12:14:48.829:MUP0:INFO:ResultSet:isc_ResultSet_0 (created by: isc_ListGrid_0):Updating cache: operationType 'update' (no componentID) ,1 rows update data

and that allMatchingRowsCached true doesn't sound so good...

Using SmartClient Version: SC_SNAPSHOT-2010-12-31/LGPL Development Only (built 2010-12-31)

[Isomorphic]

This has a lot of extra stuff going on, including simulated save. Try simplifying to a clientOnly DataSource and a programmatically issued update - you should find that this works, then you can revisit what's wrong with your large test case.

[d.cavestro]

This has a lot of extra stuff going on, including simulated save. Try simplifying to a clientOnly DataSource and a programmatically issued update - you should find that this works, then you can revisit what's wrong with your large test case.

Ok, I'll try to simplify the test, but please keep in mind I need to use updateCaches in the real world case (the data changed at server-side with other mechanisms, and I am notified only after the update happened). I have no doubt ait works if I use DataSource.updateData()

Cheers
Davide

[d.cavestro]

The simplified example still reproduces the issue

This has a lot of extra stuff going on, including simulated save. Try simplifying to a clientOnly DataSource and a programmatically issued update - you should find that this works, then you can revisit what's wrong with your large test case.

Hi Isomorphic,
I've changed the example to use updateData() on a clientOnly DataSource instead of updateCaches(), and removed some extra stuff (like the option data source, the select items and so on).
Strangely the issue remains reproducible: if you change the value for the "option_id" field, that record does not appear on the listgrid related to the new value. Could you please tell me where is the problem within my code?

Cheers
Davide

UPDATE: I've updated the sample to let the SelectItem use a ValueMap (it doesn't affect the relevant issue)

[Isomorphic]

We're not quite sure what you're getting at now - you eliminated updateCaches(), so are you trying to report cache sync doesn't work in general (obviously not true)?

Why does this still involve a RestDataSource? Cache sync uses the data returned from the update operation, so it can't be simulated with a static response. With a static response, whatever value exists in the response will always be used regardless of what value was being saved.

[d.cavestro]

We're not quite sure what you're getting at now - you eliminated updateCaches(), so are you trying to report cache sync doesn't work in general (obviously not true)?

Not at all

Why does this still involve a RestDataSource? Cache sync uses the data returned from the update operation, so it can't be simulated with a static response. With a static response, whatever value exists in the response will always be used regardless of what value was being saved.

The example still involves a RestDataSource for this reason: if I use a DataSource filled with setTestData then when I call updateData your observation applies, but if I use a clientOnly RestDataSource automatically populated with data from a json file (cacheAllData is true) then calling updateData() the response brings the exact data I have just updated, hence updating local data cache the right way (peeking into response.getData () on the updateData callback I see it is right).
But the problem is that a listGrid having an initialCriteria doesn't automatically show pre-existent cached rows that match the criteria only after data modifications while the rows that after the update don't match the criteria (no more) correctly "disappear" (decimation case works but not the opposite case).
This happens both using updateData and also simulating an update through updateCaches.

But maybe I've just found a workaround: calling ListGrid.filterByEditor () on the updateData callback then the row that matches the filter criteria "magically appears". Using filterByEditor the behavior remains correct even if I set some other filtering criteria on the editor.
I think the solution is a bit tricky cause this way I have to "know" that a ListGrid could need a forced filtering after some data changes (usually a shared data source hides these details).
However I'm going to attach an updated version of test code and data (you can see the workaround commenting out the following code).

for (final ListGrid grid : filteredGrids) {
//  grid.filterByEditor ();//WORKAROUND: FORCES FILTERING BY EDITOR CRITERIA
}

What do you think? (Please tell me if I'm trying to do something really stupid)

Cheers
Davide

[Isomorphic]

Sorry, we're still not following your explanation of what you think should happen - fundamentally, we're not sure what you're trying to achieve with a clientOnly RestDataSource, which seems a bizarre combination of settings. If you want a clientOnly DataSource populated by setTestData()/setCacheData(), there's no need for it to be a RestDataSource. If you want a RestDataSource to cache all results, you can setCacheAllData() on it.

[d.cavestro]

Sorry, we're still not following your explanation of what you think should happen - fundamentally, we're not sure what you're trying to achieve with a clientOnly RestDataSource, which seems a bizarre combination of settings. If you want a clientOnly DataSource populated by setTestData()/setCacheData(), there's no need for it to be a RestDataSource. If you want a RestDataSource to cache all results, you can setCacheAllData() on it.

Sorry, clearly I have not explained myself well.
Put it this way: the combination of clientOnly DataSource and setCacheAllData() is only for testing purposes: I want to simulate an update operation on a datasource, provided that the test has no server counterpart.
I tried using setTestData/setCacheData on a client only simple DataSource instance, but then calling updateData on the datasource no longer worked. That's surely my fault, but I thought it was due to the fact that I've never used them. On the other hand my real-world code uses a RestDataSource with a real server and calling updateData triggers the issue so - as per Client Only DataSources documentation saying

If you specify your DataSource as clientOnly: true, omit testData entirely, and provide either a dataURL or a testFileName, the DataSource will lazily make a one-time fetch against the specified data file the first time an operation is called on it. From then on, the DataSource will work against the local cache created from this initial request. This is a quick way to prototype against some test data that may eventually be returned from an arbitrary back-end.

I opted to configure the datasource this way

mainDataSource.setDataFormat (DSDataFormat.JSON);
mainDataSource.setDataURL ("data/mainds_fetch.js");
mainDataSource.setCacheAllData (true);
mainDataSource.setClientOnly (true);

but I repeat myself: these are only implementation details of the test case.
The whole point of the test is to demonstrate that when I call DataSource.updateData () on a data source used by a list grid having a certain initial criteria AND the change makes the criteria match a row previously not matched, then that row doesn't appear while it disappears if criteria no more match.

Follows a visual explanation of the issue

Original state
[ATTACH]2953[/ATTACH]

After data update
[ATTACH]2955[/ATTACH]

Thank you for dedicating your time to my explanation attempts :-)
Cheers
Davide

[Isomorphic]

We get what the behavior was (understood it from the beginning). The problem is, your test case is not valid - it's not just implementation details, it's that the test case is muddled and doesn't seem to indicate a bug.

Whatever is happening in your real world app is likely unrelated to what's happening in this test case.

If you can show a test case based on a clientOnly DataSource that is not a RestDataSource and does not need to access external files to fetch or update, that would be a valid bug report.

Alternatively, if you think the real problem is related to a RestDataSource, then you should be able to show a test case that doesn't involve also using clientOnly (which is a weird set of settings).

[d.cavestro]

Ok, thank you for speaking so clear :-)

I don't know if the problem is somewhat related to RestDataSource, but as you said, it's really easy to convert my previous example to avoid clientOnly.

So I'm going to attach a further simplified, limited example for a RestDataSource that still reproduces the behavior I complained.

Now that I have a workaround the whole thing is not pressing at all, nonetheless if it is a bug then a fix can be useful to other people.

Cheers
Davide