Need help loading list grid data incrementally with filterdata

[mathias]

SmartGWT 6.0-p20171202

I have this use case:

I want to have a list grid where I do not load any data initially (i.e. autofetchdata = false). Then at a later time, when the user does a certain thing, I want to fetch data from the server with an integer as a filter criteria.

I have done this previously in many places, but with autoFetchData on and an "initialcriteria", which I later change depending on what the user does.

However, in this case, i.e. i have auto fetch off, and I later perform the fetch like this:

Criteria filter = new Criteria();
filter.addCriteria(NubaClientServerConstants.FIELD_LOCATIONID, locationId);
listGrid.setData(Constants.EMPTY_DATA);//in case fetch goes wrong I want the old data gone
listGrid.filterData(filter);

I noticed something that feels strange:

On the server, I see that the first X rows are fetched as normal, depending on the height of the list grid:
ObserverImageWebService;fetch, locid: 4, start: 0, end: 123

But then, new requests arrive right away, which usually doesn't happen until the user scrolls down!:

ObserverImageWebService;fetch, locid: 4, start: 123, end: 198
ObserverImageWebService;fetch, locid: 4, start: 198, end: 273


Perhaps there is something simple I'm missing or not understanding, and I really hope you can help me out. How can I get incremental, "scrolling-based" data fetches without autofetch and using filterdata? Is this even possible?

Thankful for pointers.

[Isomorphic]

That’s not normal and you can see dozens of samples where we use code like you’re using and this doesn’t happen.

Best guess you have some code that is meant to traverse a big range of loaded rows of the grid after data comes back. You’ve reordered operations such that this code now fires when the grid hasn’t loaded data.

Your code can check resultSet.lengthIsKnown() to see whether data is loaded. You can also use the “fetchTrace” log category to see stack traces that may show what is initiating those extra fetches.

[mathias]

OK, I have now done some more in-depth debugging, and this is the strange thing:

 ListGrid grid = new ListGrid();
//ListGridField field = new ListGridField("createdOn");
//grid.setFields(field);
grid.setDataSource(DataSource.get("image"));

If I create the list grid as per above, very basic, it only fetches the first 75 rows. If I uncomment the two rows where I set what fields the grid should contain, it continues with the additional fetches as I described above.

This is about as a narrow test-case as I can make it, and yet I can't see why it behaves this way. Again, 6.0-p20171202

Any ideas?

[mathias]

I also did the "fetchTrace" debugging as you requested. I did it with and without setting the list grid fields. I compared the stack trace output in both requests and the only difference was that if I DO NOT set the fields, as expected some parameters related to fields were different:

if I DO NOT set:
fields: Array[3],
completeFields: Array[3],
defaultFieldState: "[{name:"locationId"},{name:"note"},{name..."[54],

if I set the fields to the single field:
fields: Array[1],
originalFields: Array[0],
defaultFieldState: "[{name:"id"}]",

Other than that all is the same, except for that there are more requests after the initial identical one when I did set the fields in the grid:

15:37:00.249:TMR7:WARN:ResultSet:isc_ResultSet_0 (dataSource: image, created by: ddid):ResultSet server fetch with server criteria: {
"locationId":4
}
ResultSet.fetchRemoteData(_1=>Obj, _2=>75, _3=>150)
ResultSet.$39w()
[c]Class.fireCallback(_1=>"$39w", _2=>null, _3=>null, _4=>[ResultSet ID:isc_ResultSet_0 (dataSource: image, created by: ddid)], _5=>undef)
[c]Class.$cm()
** recursed on [c]Class.fireCallback

15:37:00.310:TMR6:WARN:ResultSet:isc_ResultSet_0 (dataSource: image, created by: ddid):ResultSet server fetch with server criteria: {
"locationId":4
}
ResultSet.fetchRemoteData(_1=>Obj, _2=>150, _3=>225)
ResultSet.$39w()
[c]Class.fireCallback(_1=>"$39w", _2=>null, _3=>null, _4=>[ResultSet ID:isc_ResultSet_0 (dataSource: image, created by: ddid)], _5=>undef)
[c]Class.$cm()
** recursed on [c]Class.fireCallback

[mathias]

Finally, I also enabled the "resultset" debug output and think I found something interesting. The first request is exactly the same:

16:08:33.881:MUP5:INFO:ResultSet:isc_ListGrid_0:Creating new isc.ResultSet for operation 'image_fetch' with filterValues: {
"locationId":4
}
16:08:33.886:MUP5:INFO:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):setCriteria: filter criteria changed, invalidating cache
16:08:33.887:MUP5:INFO:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):Invalidating cache
16:08:33.892:MUP5:DEBUG:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):getRange(0,53), cache check: 0,63 firstMissingRow: 0 lastMissingRow: 63
16:08:33.892:MUP5:DEBUG:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):getRange: guessing forward scrolling
16:08:33.893:MUP5:INFO:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):getRange(0, 53) will fetch from 0 to 75
16:08:33.894:MUP5:INFO:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):fetching rows 0,75 from server
16:08:33.916:TMR9:DEBUG:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):getRange(0, 53) satisfied from cache
16:08:33.929:XRP0:INFO:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):Received 75 records from server
16:08:33.929:XRP0:DEBUG:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):full length set to: 150
16:08:33.930:XRP0:DEBUG:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):integrating 75 rows into cache at position 0
16:08:33.931:XRP0:INFO:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):cached 75 rows, from 0 to 75 (150 total rows, 75 cached)

HOWEVER, then it differs. If I don't set the fields, there is only one more row in the log:

16:08:33.940:TMR3:DEBUG:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):getRange(0, 53) satisfied from cache

But when I do set the listgridfields, there are more rows I don't know what to make of:

16:02:20.355:TMR1:DEBUG:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):getRange: no scrolling direction detected

16:02:20.356:TMR1:INFO:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):getRange(0, 150) will fetch from 75 to 150
16:02:20.409:TMR3:INFO:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):fetching rows 75,150 from server
16:02:20.430:XRP8:INFO:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):Received 75 records from server
16:02:20.430:XRP8:DEBUG:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):full length set to: 225
16:02:20.430:XRP8:DEBUG:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):integrating 75 rows into cache at position 75
16:02:20.430:XRP8:INFO:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):cached 75 rows, from 75 to 150 (225 total rows, 150 cached)
16:02:20.433:TMR0:DEBUG:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):getRange(0,225), cache check: 150,224 firstMissingRow: 150 lastMissingRow: 224
16:02:20.433:TMR0:DEBUG:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):getRange: no scrolling direction detected
16:02:20.434:TMR0:INFO:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):getRange(0, 225) will fetch from 150 to 225
16:02:20.478:TMR2:INFO:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):fetching rows 150,225 from server
16:02:20.498:XRP9:INFO:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):Received 64 records from server
16:02:20.499:XRP9:DEBUG:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):full length set to: 214
16:02:20.500:XRP9:DEBUG:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):integrating 64 rows into cache at position 150
16:02:20.500:XRP9:INFO:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):Fetch request returned range 150,214 differs from requested range 150,225. Assuming client/server batch size mismatch and clearing loading markers greater than 214
16:02:20.501:XRP9:INFO:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):cached 64 rows, from 150 to 214 (214 total rows, 214 cached)
16:02:20.501:XRP9:INFO:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):Cache for current criteria complete
16:02:20.504:TMR1:DEBUG:ResultSet:isc_ResultSet_0 (dataSource: image, created by: isc_ListGrid_0):getRange(0, 214) satisfied from cache

I do hope these logs can help you to be able to tell me what's wrong :) It feels super weird that setting the listgridfields manually should cause all these changes in behavior.

[Isomorphic]

We just belatedly noticed that you’re more than two years out of date on patches. That means this is probably a long since fixed bug related to drawAllMaxCells.

Staying up to date can avoid time wasting deep dives like this.

[mathias]

As always, thanks for quick response.

I am sorry but I don't think that's it. I tried with 12.1-p20190820 on Saturday already (we have purchased that license too), and it still showed the same behavior.

I have done some more digging, and I have nailed it down to this interesting case:

Again if I create the listgrid per "normal":

 ListGrid grid = new ListGrid();
grid.setDataSource(DataSource.get("image"));

it works as expected. However, if I add ONE field only via "setFields", it will load all rows until the rows returned are less than the "end-start" rows, as I have described:

 ListGridField field = new ListGridField("createdOn");
grid.setFields(field);

Now comes the interesting part: BUT, as soon as I include more than one field in the "setFields", it works as normal again! (i.e. only the first 75 rows are fetched) It does not matter if the criteria field I'm using is included in the fields or not.

 ListGridField field1 = new ListGridField("createdOn");
 ListGridField field2 = new ListGridField("note");
grid.setFields(field1, field2);

This is very strange, but it's what happens.

[Isomorphic]

As they say, extraordinary claims require extraordinary evidence :) This is a very common case where you're claiming obviously wrong behavior.

You could try setting drawAllMaxCells to 0, disabling the feature, and if that works, you should look very closely at whether you are still somehow using old libraries (since you are reproducing a long-since patched issue).

Other than that, since the reproduction conditions seem so simple, just let us know how we can get the same thing to happen by modifying a Showcase sample.