DataSource client cache invalidation and ListGridField optionDataSource

[agalataud]

Hi,

We're using SmartGWT 4.1 Power Edition (v9.1p_2014-06-04).
I'm facing a problem regarding DataSources with cacheAllData set to true and ListGridField using such data sources as optionDataSource.

Scenario:
- 2 users connected to the application.
- 1st one performs an update operation on cached data source.
- 2nd user client is notified of operation (with our own push system), which triggers call to DataSource.invalidateCache()
- 2nd user refreshes a grid with a field having this DS set as optionDataSource
=> Change isn't taken into account.

This problem seems to affect only ListGridFields: performing a refresh on a ListGrid that uses this DS, or displaying a SelectItem with this same DS triggers the following request:

=== 2014-06-05 14:39:11,325 [l0-6] DEBUG RPCManager - Request #1 (DSRequest) payload: {
    criteria:{
    },
    operationConfig:{
        dataSource:"businessUnit",
        operationType:"fetch",
        textMatchStyle:"exact"
    },
    componentId:"(cacheAllData fetch)",
    appID:"builtinApplication",
    operation:"loadCache",
    oldValues:null
}

When DS is used only as optionDataSource on a ListGridField, refreshing the parent ListGrid doesn't trigger cache reload.

Issue can be reproduced with all operations (add/update/remove).

DataSource definition:

<DataSource xmlns:fmt="WEB-INF/"
    ID="businessUnit"
	serverType="sql"
	tableName="SEC_BUSINESS_UNIT"
	serverConstructor="com.fircosoft.cdb.server.administration.BusinessUnitDataSource"
	audited="true"
	insertAuditEvent="2701"
	updateAuditEvent="2702"
	removeAuditEvent="2703"
	titleField="NAME"
	fireEventOnDataChanged="true"
	cacheAllData="true"
	cacheAllOperationId="loadCache"
	cacheAcrossOperationIds="false"
>

...

    <operationBinding operationType="fetch" operationId="loadCache">
	        <selectClause>SEC_BUSINESS_UNIT.ID, SEC_BUSINESS_UNIT.NAME</selectClause>
	        <whereClause>SEC_BUSINESS_UNIT.DELETED = 0 AND SEC_BUSINESS_UNIT.ID > 0</whereClause>
	    </operationBinding>
    </operationBindings>
</DataSource>

Client code that handles data changes notifications:

  final String id = ds.getID();
			
  //Listener for main data source
  EventManager.register(new DataChangedEventListener(id) {
  @Override
  public void notifyDataChanged(DataChangedEvent event) {
	String dsId = id;
	if (DataSource.get(dsId) != null && DataSource.get(dsId).getCacheAllData())
		DataSource.get(dsId).invalidateCache();
        }
}, DataChangedEvent.class);

Did I miss something in client-side cache configuration? Or is there a special setup for ListGridField?

Thanks in advance for your help
Antoine

[Isomorphic]

ListGrid.invalidateCache() would not be expected to clear caches related to listGridField.optionDataSource, since it's a separate DataSource, so there's no indication that the cached data has become invalid.

Use DataSource.updateCaches() to publish updates to the data in the optionDataSource, inclusive of the ability to just invalidate all caches as describes in the docs for that method.

[agalataud]

Thank you for this quick answer.

For the record, I said I was using DataSource.invalidateCache and not ListGrid.invalidateCache. Doc of this method says it can be used to invalidate cache when cacheAllData is set to true.

I've just tried DataSource.updateCaches, as following:

DSResponse rep = new DSResponse(id);
rep.setInvalidateCache(true);
rep.setOperationType(DSOperationType.UPDATE);

//Needed?
Record r = new Record();
r.setAttribute("ID", event.getId());
rep.setData(r);

DataSource.get(id).updateCaches(rep);

I tried with and without DSResponse.setData(record);
I still have the same issue with ListGridField.optionDataSource.

[agalataud]

Hi,

Do you have any progress status of this investigation?

To summarize the problem is: DataSource.invalidateCache() or DataSource.updateCaches() has the expected behaviour on ListGrid or SelectItem that uses this data source (updateCaches immediately triggers re-fetch, invalidateCache triggers refetch only when the cache is requested by user interaction or else).

The problem is only with ListGridField that uses datasource with setOptionDataSource and the cache operation Id

field.setOptionOperationId(DataSource.get("myDS").getCacheAllOperationId());.

When cache is updated or invalidated by an external event (different browser than the one on which add/update/remove operation happened), and user reloads grid for which such field is set, the wrong values are shown (in case of update) or value is shown instead of display value (add).

I would expect, like for ListGrid and SelectItem, that DS cache invalidation triggers refetch, using the cacheAllOperationId.
This isn't the case, and the logs validate this assertion (no request on DS with that operation Id).

This is an important issue for our product, as we are using this mechanism quite intensively.

Regards
Antoine

[Isomorphic]

We're looking into this right now - we'll update later today

[Isomorphic]

We've not been able to reproduce this in our tests here, and we can't use the snippets you posted. If you can provide a minimal, runnable sample, we can take a deeper look.

[agalataud]

By not able to reproduce, do you mean that on your side, when you test with 2 different browsers and your own push system that notifies 2nd client of changes made on 1st client, the ListGridField cached values are invalidated?

As you can imagine, it's difficult to build such simple test case as it requires to have a push mechanism. We have our own, built on top of Atmosphere. You have your proprietary solution, which I guess you can use to build the test.
Once push is setup, test scenario is easy to build: update operation on a DS in 1st client, listen to push and invalidate cache on 2nd client. With a ListGrid configured as I described, reload it after update operation and see if data is re-fetched or not.

Testing only with one browser isn't relevant at all, as data sources are updated directly on user intervention.

Regards
Antoine

[Isomorphic]

What we tested was whether a call to DataSource.invalidateCache() will cause the expected invalidation of cached datasets created due to the listGridField.optionDataSource setting, with the additional settings you described.

There is no plausible way that presence or absence of a server push system could affect this behavior.

However, perhaps you are taking a series of actions (to take a random example - perhaps you also have a call to ListGrid.setFields() happening at around the same time), and this is the actual cause.

If it's something like this, you should be able to create a test case that reproduces it and does not involve a server push system.

And, for completeness, it would obviously be a waste of time for us to attempt a test case that happened to involve server push, as it would be no more or less likely to uncover an issue with a particular series of method calls than trying random calls in random orders.

[agalataud]

Ok push system isn't necessary here, but I wanted to highlight the fact that update operation must take place in a different browser than the one where invalidateCache is called.

I will build a test case with such grid and a button that calls DataSource.invalidateCache.

[agalataud]

Here is the test case.

@Override
public void onModuleLoad() {
	
	DataSource.load(new String [] {"operator","businessUnit"}, new Function() {
		
		@Override
		public void execute() {
			
			final ListGrid grid = new ListGrid(DataSource.get("operator"));
			
			ListGridField cachedField = new ListGridField("BUNIT_ID");
			cachedField.setOptionDataSource(DataSource.get("businessUnit"));
			cachedField.setValueField("ID");
			cachedField.setDisplayField("NAME");
			cachedField.setOptionOperationId("loadCache");
			
			grid.setFields(cachedField, new ListGridField("ID"));
			grid.setAutoFetchData(true);
			
			DynamicForm f = new DynamicForm();
			SelectItem i = new SelectItem("t","BU");
			i.setOptionDataSource(DataSource.get("businessUnit"));
			i.setValueField("ID");
			i.setDisplayField("NAME");
			i.setOptionOperationId("loadCache");
			f.setItems(i);
			
			Button clearCache = new Button("Clear cache");
			clearCache.addClickHandler(new ClickHandler() {
				@Override
				public void onClick(ClickEvent event) {
					DSResponse rep = new DSResponse("businessUnit", DSOperationType.UPDATE);
					rep.setInvalidateCache(true);
					DataSource.get("businessUnit").updateCaches(rep);
				}
			});
			Button refreshGrid = new Button("Refresh grid");
			refreshGrid.addClickHandler(new ClickHandler() {
				@Override
				public void onClick(ClickEvent event) {
					grid.invalidateCache();
				}
			});
			
			VLayout l = new VLayout();
			l.setWidth("500");
			l.setHeight("500");
			l.setMembers(grid,f,clearCache,refreshGrid);
			l.draw();
		}
	}, false);
}

operator.ds.xml

<DataSource
    ID="operator"
	serverType="sql"
	tableName="SEC_OPERATOR"
	useAnsiJoins="true"
>
	<fields>
    	<field name="ID" type="sequence" sequenceName="SQ_SEC_OPERATOR" hidden="true" primaryKey="true"/>
    	<field name="BUNIT_ID" type="integer" foreignKey="businessUnit.ID" hidden="true" joinType="outer"/>
	</fields>
</DataSource>

businessUnit.ds.xml

<DataSource 
    ID="businessUnit"
	serverType="sql"
	tableName="SEC_BUSINESS_UNIT"
	cacheAllData="true"
	cacheAllOperationId="loadCache"
	cacheAcrossOperationIds="false"
	
	<fields>
    	<field name="ID" type="sequence" sequenceName="SQ_SEC_BUSINESS_UNIT" hidden="true" primaryKey="true"/>
		<field name="NAME" type="text" title="$name" escapeHTML="true" required="true" length="32">
	</fields>
	
	<operationBindings>
		<operationBinding operationType="fetch" operationId="loadCache">
	        <selectClause>SEC_BUSINESS_UNIT.ID, SEC_BUSINESS_UNIT.NAME</selectClause>
	        <whereClause>SEC_BUSINESS_UNIT.ID > 0</whereClause>
	    </operationBinding>
    </operationBindings>
</DataSource>

Scenario: display screen, update value in DB, click clear cache then refresh grid: business unit display value has changed in SelectItem, not in ListGrid.

Screenshots attached (tested with 'TEST8').

[Isomorphic]

Thanks for the test case - we had to fix a couple of bugs in the DataSources, but once we got it going, we were able to see your issue, and it's now been fixed.

Please retest with a build dated June 14 or later.

[agalataud]

Hi,

I've just tested the fix with nightly build 2014-06-14.
The cache is correctly invalidated and re-fetched as expected.

However, testing with the test case I've provided, and looking at RPC calls performed when DataSource.updateCaches is executed, I've found 2 important problems:

Problem 1: 27 RPC calls are executed (traced in Network tab of browser dev tools). Following is an example of RPC call:

=== 2014-06-16 13:57:23,876 [0-18] DEBUG RPCManager - Request #1 (DSRequest) payload: {
    criteria:{
    },
    operationConfig:{
        dataSource:"businessUnit",
        operationType:"fetch",
        textMatchStyle:"exact"
    },
    startRow:297,
    endRow:371,
    componentId:"(created directly)",
    useStrictJSON:true,
    appID:"builtinApplication",
    operation:"businessUnit_fetch",
    oldValues:{
    }
}
[...]
=== 2014-06-16 13:57:23,881 [0-18] DEBUG SQLDataSource - [builtinApplication.businessUnit_fetch] SQL windowed select rows 7->371, result size 364. Query: SELECT * FROM (SELECT *, ROW_NUMBER() OVER (ORDER BY ID) AS rowID FROM (SELECT TOP 100 PERCENT  SEC_BUSINESS_UNIT.ACTIVE, SEC_BUSINESS_UNIT.CODE, SEC_BUSINESS_UNIT.CREATED, SEC_BUSINESS_UNIT.CREATED_BY, SEC_BUSINESS_UNIT.DELETED, SEC_BUSINESS_UNIT.DESCRIPTION, SEC_BUSINESS_UNIT.ID, SEC_BUSINESS_UNIT.NAME, SEC_BUSINESS_UNIT.UPDATED, SEC_BUSINESS_UNIT.UPDATED_BY FROM SEC_BUSINESS_UNIT WHERE 
		    	('1'='1') AND SEC_BUSINESS_UNIT.DELETED = 0 AND SEC_BUSINESS_UNIT.ID > 0 
		    	AND (
		    		ID IN (SELECT BUNIT_ID FROM SEC_ASSIGNED_DOMAIN, SEC_MEMBERSHIP WHERE SEC_MEMBERSHIP.OPERATOR_ID = 2 AND SEC_MEMBERSHIP.GROUP_ID = SEC_ASSIGNED_DOMAIN.GROUP_ID)
		    		OR UPDATED_BY = 2
		    	)
		    ) x) y WHERE y.rowID BETWEEN 8 AND 371
=== 2014-06-16 13:57:23,882 [0-18] INFO  DSResponse - [builtinApplication.businessUnit_fetch] DSResponse: List with 0 items

This is a major performance problem. It looks like it does a kind of paging, but here there are only 10 rows in the table. Moreover, I've seen better paging strategies in SmartGWT than this one.


Problem 2: when setting a specific operation id to update caches, it isn't taken into account. See above logs and following code sample:

DataSource.get("businessUnit").updateCaches(rep, new DSRequest(DSOperationType.FETCH, "loadCache"));

A last observation: DataSource.updateCaches triggers automatic data-bound component refresh (e.g. adding a row will trigger ListGrid refresh).
While I think this can be a good idea for low volume / low frequency updates, this can clearly be unmanageable for end-user if lot of cache updated are done in a short period of time.
Would it be possible to have a behaviour similar to what we observe with DataSource.invalidateCache, that's to say silently invalidating the cache and re-fetching values only when a call is performed on the relevant method (invalidateCache, fetchData, ...) of data-bound component?
I've tried already with DataSource.invalidateCache, it doesn't seem to work for ListGridField with option data source or SelectItem.

Regards
Antoine

[Isomorphic]

We've made changes to address the multiple unnecessary ranged fetches you were seeing, and probably the other issue you mention.

Please retest with a build dated June 19 or later and let us know if you still see issues.

[agalataud]

First issue with multiple RPC calls sent on updateCaches is solved. Thanks

However the second issue is still present. This is critical for us as we absolutely need to use a special operation for cache loading.

I reproduce the problem with the test case I already provided.
Wrong RPC payload (I expect operation to be 'loadCache' - what is done when grid is first instantiated):

=== 2014-06-20 15:31:40,673 [0-19] DEBUG RPCManager - Request #1 (DSRequest) payload: {
    criteria:{
    },
    operationConfig:{
        dataSource:"businessUnit",
        operationType:"fetch",
        textMatchStyle:"exact"
    },
    startRow:0,
    endRow:75,
    componentId:"(created directly)",
    useStrictJSON:true,
    appID:"builtinApplication",
    operation:"businessUnit_fetch",
    oldValues:{
    }
}

I tried with and without providing an optional DSRequest to DataSource.updateCaches, as following:

DataSource.get("businessUnit").updateCaches(rep, new DSRequest(DSOperationType.FETCH, "loadCache"));

[Isomorphic]

Ok, we see the issue where one of the fetches is not being executed with the correct operationId, and we have somebody looking into it. We'll update here when we have a fix.