[agalataud]

I confirm the issue is now fully fixed. Thanks.

[Isomorphic]

This should be fixed in builds dated June 26 and later.

[Isomorphic]

Ok, we see the issue where one of the fetches is not being executed with the correct operationId, and we have somebody looking into it. We'll update here when we have a fix.

[agalataud]

First issue with multiple RPC calls sent on updateCaches is solved. Thanks

However the second issue is still present. This is critical for us as we absolutely need to use a special operation for cache loading.

I reproduce the problem with the test case I already provided.
Wrong RPC payload (I expect operation to be 'loadCache' - what is done when grid is first instantiated):

=== 2014-06-20 15:31:40,673 [0-19] DEBUG RPCManager - Request #1 (DSRequest) payload: {
    criteria:{
    },
    operationConfig:{
        dataSource:"businessUnit",
        operationType:"fetch",
        textMatchStyle:"exact"
    },
    startRow:0,
    endRow:75,
    componentId:"(created directly)",
    useStrictJSON:true,
    appID:"builtinApplication",
    operation:"businessUnit_fetch",
    oldValues:{
    }
}

I tried with and without providing an optional DSRequest to DataSource.updateCaches, as following:

DataSource.get("businessUnit").updateCaches(rep, new DSRequest(DSOperationType.FETCH, "loadCache"));

[Isomorphic]

We've made changes to address the multiple unnecessary ranged fetches you were seeing, and probably the other issue you mention.

Please retest with a build dated June 19 or later and let us know if you still see issues.

[agalataud]

Hi,

I've just tested the fix with nightly build 2014-06-14.
The cache is correctly invalidated and re-fetched as expected.

However, testing with the test case I've provided, and looking at RPC calls performed when DataSource.updateCaches is executed, I've found 2 important problems:

Problem 1: 27 RPC calls are executed (traced in Network tab of browser dev tools). Following is an example of RPC call:

=== 2014-06-16 13:57:23,876 [0-18] DEBUG RPCManager - Request #1 (DSRequest) payload: {
    criteria:{
    },
    operationConfig:{
        dataSource:"businessUnit",
        operationType:"fetch",
        textMatchStyle:"exact"
    },
    startRow:297,
    endRow:371,
    componentId:"(created directly)",
    useStrictJSON:true,
    appID:"builtinApplication",
    operation:"businessUnit_fetch",
    oldValues:{
    }
}
[...]
=== 2014-06-16 13:57:23,881 [0-18] DEBUG SQLDataSource - [builtinApplication.businessUnit_fetch] SQL windowed select rows 7->371, result size 364. Query: SELECT * FROM (SELECT *, ROW_NUMBER() OVER (ORDER BY ID) AS rowID FROM (SELECT TOP 100 PERCENT  SEC_BUSINESS_UNIT.ACTIVE, SEC_BUSINESS_UNIT.CODE, SEC_BUSINESS_UNIT.CREATED, SEC_BUSINESS_UNIT.CREATED_BY, SEC_BUSINESS_UNIT.DELETED, SEC_BUSINESS_UNIT.DESCRIPTION, SEC_BUSINESS_UNIT.ID, SEC_BUSINESS_UNIT.NAME, SEC_BUSINESS_UNIT.UPDATED, SEC_BUSINESS_UNIT.UPDATED_BY FROM SEC_BUSINESS_UNIT WHERE 
		    	('1'='1') AND SEC_BUSINESS_UNIT.DELETED = 0 AND SEC_BUSINESS_UNIT.ID > 0 
		    	AND (
		    		ID IN (SELECT BUNIT_ID FROM SEC_ASSIGNED_DOMAIN, SEC_MEMBERSHIP WHERE SEC_MEMBERSHIP.OPERATOR_ID = 2 AND SEC_MEMBERSHIP.GROUP_ID = SEC_ASSIGNED_DOMAIN.GROUP_ID)
		    		OR UPDATED_BY = 2
		    	)
		    ) x) y WHERE y.rowID BETWEEN 8 AND 371
=== 2014-06-16 13:57:23,882 [0-18] INFO  DSResponse - [builtinApplication.businessUnit_fetch] DSResponse: List with 0 items

This is a major performance problem. It looks like it does a kind of paging, but here there are only 10 rows in the table. Moreover, I've seen better paging strategies in SmartGWT than this one.


Problem 2: when setting a specific operation id to update caches, it isn't taken into account. See above logs and following code sample:

DataSource.get("businessUnit").updateCaches(rep, new DSRequest(DSOperationType.FETCH, "loadCache"));

A last observation: DataSource.updateCaches triggers automatic data-bound component refresh (e.g. adding a row will trigger ListGrid refresh).
While I think this can be a good idea for low volume / low frequency updates, this can clearly be unmanageable for end-user if lot of cache updated are done in a short period of time.
Would it be possible to have a behaviour similar to what we observe with DataSource.invalidateCache, that's to say silently invalidating the cache and re-fetching values only when a call is performed on the relevant method (invalidateCache, fetchData, ...) of data-bound component?
I've tried already with DataSource.invalidateCache, it doesn't seem to work for ListGridField with option data source or SelectItem.

Regards
Antoine

[Isomorphic]

Thanks for the test case - we had to fix a couple of bugs in the DataSources, but once we got it going, we were able to see your issue, and it's now been fixed.

Please retest with a build dated June 14 or later.

[agalataud]

Here is the test case.

@Override
public void onModuleLoad() {
	
	DataSource.load(new String [] {"operator","businessUnit"}, new Function() {
		
		@Override
		public void execute() {
			
			final ListGrid grid = new ListGrid(DataSource.get("operator"));
			
			ListGridField cachedField = new ListGridField("BUNIT_ID");
			cachedField.setOptionDataSource(DataSource.get("businessUnit"));
			cachedField.setValueField("ID");
			cachedField.setDisplayField("NAME");
			cachedField.setOptionOperationId("loadCache");
			
			grid.setFields(cachedField, new ListGridField("ID"));
			grid.setAutoFetchData(true);
			
			DynamicForm f = new DynamicForm();
			SelectItem i = new SelectItem("t","BU");
			i.setOptionDataSource(DataSource.get("businessUnit"));
			i.setValueField("ID");
			i.setDisplayField("NAME");
			i.setOptionOperationId("loadCache");
			f.setItems(i);
			
			Button clearCache = new Button("Clear cache");
			clearCache.addClickHandler(new ClickHandler() {
				@Override
				public void onClick(ClickEvent event) {
					DSResponse rep = new DSResponse("businessUnit", DSOperationType.UPDATE);
					rep.setInvalidateCache(true);
					DataSource.get("businessUnit").updateCaches(rep);
				}
			});
			Button refreshGrid = new Button("Refresh grid");
			refreshGrid.addClickHandler(new ClickHandler() {
				@Override
				public void onClick(ClickEvent event) {
					grid.invalidateCache();
				}
			});
			
			VLayout l = new VLayout();
			l.setWidth("500");
			l.setHeight("500");
			l.setMembers(grid,f,clearCache,refreshGrid);
			l.draw();
		}
	}, false);
}

operator.ds.xml

<DataSource
    ID="operator"
	serverType="sql"
	tableName="SEC_OPERATOR"
	useAnsiJoins="true"
>
	<fields>
    	<field name="ID" type="sequence" sequenceName="SQ_SEC_OPERATOR" hidden="true" primaryKey="true"/>
    	<field name="BUNIT_ID" type="integer" foreignKey="businessUnit.ID" hidden="true" joinType="outer"/>
	</fields>
</DataSource>

businessUnit.ds.xml

<DataSource 
    ID="businessUnit"
	serverType="sql"
	tableName="SEC_BUSINESS_UNIT"
	cacheAllData="true"
	cacheAllOperationId="loadCache"
	cacheAcrossOperationIds="false"
	
	<fields>
    	<field name="ID" type="sequence" sequenceName="SQ_SEC_BUSINESS_UNIT" hidden="true" primaryKey="true"/>
		<field name="NAME" type="text" title="$name" escapeHTML="true" required="true" length="32">
	</fields>
	
	<operationBindings>
		<operationBinding operationType="fetch" operationId="loadCache">
	        <selectClause>SEC_BUSINESS_UNIT.ID, SEC_BUSINESS_UNIT.NAME</selectClause>
	        <whereClause>SEC_BUSINESS_UNIT.ID > 0</whereClause>
	    </operationBinding>
    </operationBindings>
</DataSource>

Scenario: display screen, update value in DB, click clear cache then refresh grid: business unit display value has changed in SelectItem, not in ListGrid.

Screenshots attached (tested with 'TEST8').

[agalataud]

Ok push system isn't necessary here, but I wanted to highlight the fact that update operation must take place in a different browser than the one where invalidateCache is called.

I will build a test case with such grid and a button that calls DataSource.invalidateCache.

[Isomorphic]

What we tested was whether a call to DataSource.invalidateCache() will cause the expected invalidation of cached datasets created due to the listGridField.optionDataSource setting, with the additional settings you described.

There is no plausible way that presence or absence of a server push system could affect this behavior.

However, perhaps you are taking a series of actions (to take a random example - perhaps you also have a call to ListGrid.setFields() happening at around the same time), and this is the actual cause.

If it's something like this, you should be able to create a test case that reproduces it and does not involve a server push system.

And, for completeness, it would obviously be a waste of time for us to attempt a test case that happened to involve server push, as it would be no more or less likely to uncover an issue with a particular series of method calls than trying random calls in random orders.

[agalataud]

By not able to reproduce, do you mean that on your side, when you test with 2 different browsers and your own push system that notifies 2nd client of changes made on 1st client, the ListGridField cached values are invalidated?

As you can imagine, it's difficult to build such simple test case as it requires to have a push mechanism. We have our own, built on top of Atmosphere. You have your proprietary solution, which I guess you can use to build the test.
Once push is setup, test scenario is easy to build: update operation on a DS in 1st client, listen to push and invalidate cache on 2nd client. With a ListGrid configured as I described, reload it after update operation and see if data is re-fetched or not.

Testing only with one browser isn't relevant at all, as data sources are updated directly on user intervention.

Regards
Antoine

[Isomorphic]

We've not been able to reproduce this in our tests here, and we can't use the snippets you posted. If you can provide a minimal, runnable sample, we can take a deeper look.

[Isomorphic]

We're looking into this right now - we'll update later today

[agalataud]

Hi,

Do you have any progress status of this investigation?

To summarize the problem is: DataSource.invalidateCache() or DataSource.updateCaches() has the expected behaviour on ListGrid or SelectItem that uses this data source (updateCaches immediately triggers re-fetch, invalidateCache triggers refetch only when the cache is requested by user interaction or else).

The problem is only with ListGridField that uses datasource with setOptionDataSource and the cache operation Id

field.setOptionOperationId(DataSource.get("myDS").getCacheAllOperationId());.

When cache is updated or invalidated by an external event (different browser than the one on which add/update/remove operation happened), and user reloads grid for which such field is set, the wrong values are shown (in case of update) or value is shown instead of display value (add).

I would expect, like for ListGrid and SelectItem, that DS cache invalidation triggers refetch, using the cacheAllOperationId.
This isn't the case, and the logs validate this assertion (no request on DS with that operation Id).

This is an important issue for our product, as we are using this mechanism quite intensively.

Regards
Antoine

[agalataud]

Thank you for this quick answer.

For the record, I said I was using DataSource.invalidateCache and not ListGrid.invalidateCache. Doc of this method says it can be used to invalidate cache when cacheAllData is set to true.

I've just tried DataSource.updateCaches, as following:

DSResponse rep = new DSResponse(id);
rep.setInvalidateCache(true);
rep.setOperationType(DSOperationType.UPDATE);

//Needed?
Record r = new Record();
r.setAttribute("ID", event.getId());
rep.setData(r);

DataSource.get(id).updateCaches(rep);

I tried with and without DSResponse.setData(record);
I still have the same issue with ListGridField.optionDataSource.