potential defect in ListGrid configured for paging?

[JLivermore]

SmartClient Version: v9.1p_2014-10-20/PowerEdition Deployment (built 2014-10-20)
GWT 2.6.1
IE-11

I have a ListGrid setup for paging and progressive loading. Looking in the SmartClient Developer's Console's RPC tab, I see the first DSRequest go out OK. Then I start to scroll down, and when the ListGrid makes the 2nd DSRequest for more records, I see 2 DSRequests going out that are identical that are 1.8 seconds apart when there should only be 1 DSRequest going out to fetch more records? At this point, the ListGrid no longer makes requests to load more records, even though I know there are more to be fetched.

The first DSResponse looks like this:

        endRow:99, 
        invalidateCache:false, 
        isDSResponse:true, 
        queueStatus:0, 
        startRow:0, 
        status:0, 
        totalRows:199

The second double DSResponses (from the double DSRequest from the grid) look like this:

        endRow:199, 
        invalidateCache:false, 
        isDSResponse:true, 
        queueStatus:0, 
        startRow:100, 
        status:0, 
        totalRows:200

Is this a known bug that has been fixed?

I would only expect 1 DSRequest to go out on the 2nd fetch, not two. Here are the double DSRequests going out:
1

{
    dataSource:"pivot_table_flat_fact_POF_384_682", 
    operationType:"fetch", 
    componentId:"isc_ListGrid_4", 
    data:{
        spc:[
            "1", 
            "2013-10-01", 
            "2013-12-31", 
            "POF", 
            "70:2^682|16:2^Y|1:2", 
            "10:2^|12:2^|1:2^|6:2^", 
            "5|6|21|29|36|24|25|28|26|30", 
            "false", 
            "100", 
            "199", 
            "5", 
            "", 
            "", 
            "", 
            "384", 
            "2285", 
            "682", 
            "postTradeEquitiesFactDataSource"
        ]
    }, 
    startRow:99, 
    endRow:199, 
    textMatchStyle:"exact", 
    resultSet:[ResultSet ID:isc_ResultSet_3 (dataSource: pivot_table_flat_fact_POF_384_682, created by: isc_ListGrid_4)], 
    callback:{
        caller:[ResultSet ID:isc_ResultSet_3 (dataSource: pivot_table_flat_fact_POF_384_682, created by: isc_ListGrid_4)], 
        methodName:"fetchRemoteDataReply"
    }, 
    willHandleError:true, 
    showPrompt:false, 
    prompt:"Loading...", 
    oldValues:{
        spc:[
            "1", 
            "2013-10-01", 
            "2013-12-31", 
            "POF", 
            "70:2^682|16:2^Y|1:2", 
            "10:2^|12:2^|1:2^|6:2^", 
            "5|6|21|29|36|24|25|28|26|30", 
            "false", 
            "100", 
            "199", 
            "5", 
            "", 
            "", 
            "", 
            "384", 
            "2285", 
            "682", 
            "postTradeEquitiesFactDataSource"
        ]
    }, 
    requestId:"pivot_table_flat_fact_POF_384_682$62723", 
    internalClientContext:{
        requestIndex:2
    }, 
    useStrictJSON:true, 
    fallbackToEval:true, 
    dataPageSize:100, 
    dataFetchMode:"paged", 
    lastClientEventThreadCode:"TMR2", 
    bypassCache:true
}

2

{
    dataSource:"pivot_table_flat_fact_POF_384_682", 
    operationType:"fetch", 
    componentId:"isc_ListGrid_4", 
    data:{
        spc:[
            "1", 
            "2013-10-01", 
            "2013-12-31", 
            "POF", 
            "70:2^682|16:2^Y|1:2", 
            "10:2^|12:2^|1:2^|6:2^", 
            "5|6|21|29|36|24|25|28|26|30", 
            "false", 
            "100", 
            "199", 
            "5", 
            "", 
            "", 
            "", 
            "384", 
            "2285", 
            "682", 
            "postTradeEquitiesFactDataSource"
        ]
    }, 
    startRow:49, 
    endRow:149, 
    textMatchStyle:"exact", 
    resultSet:[ResultSet ID:isc_ResultSet_3 (dataSource: pivot_table_flat_fact_POF_384_682, created by: isc_ListGrid_4)], 
    callback:{
        caller:[ResultSet ID:isc_ResultSet_3 (dataSource: pivot_table_flat_fact_POF_384_682, created by: isc_ListGrid_4)], 
        methodName:"fetchRemoteDataReply"
    }, 
    willHandleError:true, 
    showPrompt:false, 
    prompt:"Loading...", 
    oldValues:{
        spc:[
            "1", 
            "2013-10-01", 
            "2013-12-31", 
            "POF", 
            "70:2^682|16:2^Y|1:2", 
            "10:2^|12:2^|1:2^|6:2^", 
            "5|6|21|29|36|24|25|28|26|30", 
            "false", 
            "100", 
            "199", 
            "5", 
            "", 
            "", 
            "", 
            "384", 
            "2285", 
            "682", 
            "postTradeEquitiesFactDataSource"
        ]
    }, 
    requestId:"pivot_table_flat_fact_POF_384_682$62724", 
    internalClientContext:{
        requestIndex:3
    }, 
    useStrictJSON:true, 
    fallbackToEval:true, 
    dataPageSize:100, 
    dataFetchMode:"paged", 
    lastClientEventThreadCode:"TMR0", 
    bypassCache:true
}

In both DSRequests, the startRow and endRow values are different?

[Isomorphic]

Sorry, that is way way too little information. You haven't mentioned anything about how the grid is configured, row ranges of the request and responses, etc.

[Isomorphic]

It looks like you have started to provide the information by editing the post. Please post anew when you think you have provided enough.

Also, in general, if you are providing more information, please create a new post. Only new posts produce notifications to Support. For the same reason, it's a bad idea to post incomplete information that Support can't act on, even if you plan to amend it later.

[JLivermore]

Sorry about that. Here is how my ListGrid is configured:

listGrid = new ListGrid();

        listGrid.setCanSort(Boolean.FALSE);
        listGrid.setDataFetchMode(FetchMode.PAGED);
        listGrid.setDataPageSize(PivotTableFactFlatQuerySPContext.pageSize);
        listGrid.setShowAllRecords(Boolean.FALSE);

[JLivermore]

Am I suppose to get the startRow and endRow values from the DSRequest? I noticed the double DSRequests going out have different values for startRow and endRow?

Because, I am arbitrarily setting the rows requested from a counter I have.

[Isomorphic]

Because, I am arbitrarily setting the rows requested from a counter I have.

You'd need to elaborate on this quite a bit to enable us to help - we can't even tell if this means client or server side or what?

To give a general hint, some common ways you can create "extra" requests:

1. set a dataPageSize which fetches too few rows to fill the visible area of the ListGrid + drawAhead (see listGrid.drawAheadRatio for a discussion)

2. have some logic that blindly loops calling getRecord(index) up to the current totalRows, without trying to figure out if records are actually loaded

3. have flawed off-by-one server logic which doesn't actually return the full requested row range

[JLivermore]

Another thing I noticed is when the 2nd DSRequest is made automatically by scrolling, when the data comes back into the grid, there is a blank record from the first batch of records loaded and the 2nd batch of records loaded, even though there is no break in the primary key value? It is a sequence of integers starting with 1.

[Isomorphic]

By the way, you can enable the "fetchTrace" log category in the Developer Console to get a stack trace logged every time the ListGrid's ResultSet decides it needs to go to the server.

This can often give a very clear indication of the problem, especially if the cause is #2.

[JLivermore]

By the way, you can enable the "fetchTrace" log category in the Developer Console to get a stack trace logged every time the ListGrid's ResultSet decides it needs to go to the server.

This can often give a very clear indication of the problem, especially if the cause is #2.

thanks...let me see what it says...

[Isomorphic]

Another thing I noticed is when the 2nd DSRequest is made automatically by scrolling, when the data comes back into the grid, there is a blank record from the first batch of records loaded and the 2nd batch of records loaded, even though there is no break in the primary key value? It is a sequence of integers starting with 1.

Sounds very much like you have an off-by-one error handling startRow and endRow.

Note that startRow and endRow are zero-based, inclusive at the beginning and exclusive at the end (like substring), so startRow: 0, endRow: 1 is a request for the first record.

[JLivermore]

You'd need to elaborate on this quite a bit to enable us to help - we can't even tell if this means client or server side or what?

The counter is on the client side, I'm just keeping a startRow and endRow counter for my interface to my DB stored proc. Would be be better to get these values from the DSRequest and not have to track these counters myself?

[JLivermore]

thanks...let me see what it says...

Here is the log with fetchTrace enabled going through the same scenario again:
1. Open the grid.
2. Scroll down to produce an automatic fetch (in my case it generates 2 fetches).

Global Log Priorities updated: Logging messages at priority 'Debug' and above for category 'fetchTrace'.
15:20:05.069:TMR5[E]:WARN:ResultSet:isc_ResultSet_3 (dataSource pivot_table_time_frame, created by: isc_PickListMenu_3):ResultSet server fetch with server criteria: {
    "clientID":"682", 
    "factTableID":"POF", 
    "lastDataPoint":"2013-10-31", 
    "dataSource":"postTradeEquitiesFactDataSource", 
    "dimensionID":"1"
}
    ResultSet.fetchRemoteData(_1=>Obj, _2=>0, _3=>75)
    ResultSet.$39w()
    ResultSet.getRangePaged(_1=>0, _2=>1, _3=>undef, _4=>undef)
    ResultSet.$39u(_1=>0, _2=>1, _3=>undef, _4=>undef)
    ResultSet.getRange(_1=>0, _2=>1, _3=>undef, _4=>undef)
    ListGrid.requestVisibleRows()
    Canvas.filterWithCriteria(_1=>Obj, _2=>Obj{ID:pivot_table_time_frame_fetch}, _3=>Obj)
    Canvas.$wo(_1=>"filter", _2=>Obj, _3=>Obj, _4=>Obj)
    [a]MathFunction.invokeSuper(_1=>null, _2=>"$wo", _3=>undef, _4=>undef, _5=>undef, _6=>undef, _7=>undef, _8=>undef, _9=>undef, _10=>undef, _11=>undef, _12=>undef, _13=>undef)
    [a]MathFunction.Super(_1=>"$wo", _2=>Array[4], _3=>[object Arguments])
    ListGrid.$wo(_1=>"filter", _2=>Obj, _3=>Obj, _4=>Obj, _5=>undef)
    Canvas.filterData(_1=>Obj, _2=>Obj, _3=>Obj)
    ** recursed on [a]MathFunction.invokeSuper

15:20:06.354:TMR9:WARN:fetchMissingValues:isc_SelectItem_3[timeFrame]:FetchMissingValues - filterLocally is false yet optionDataSource fetch included records that do not match our current data value. Ignoring these values.
15:20:06.542:WARN:ResultSet:isc_ResultSet_4 (dataSource pivot_table_flat_fact_POF_384_682, created by: isc_ListGrid_0):ResultSet server fetch with server criteria: {
}
    ResultSet.fetchRemoteData(_1=>Obj, _2=>0, _3=>100)
    ResultSet.$39w()
    ResultSet.getRangePaged(_1=>0, _2=>37, _3=>undef, _4=>undef)
    ResultSet.$39u(_1=>0, _2=>37, _3=>undef, _4=>undef)
    ResultSet.getRange(_1=>0, _2=>37, _3=>undef, _4=>undef)
    ListGrid.requestVisibleRows()
    Canvas.filterWithCriteria(_1=>Obj, _2=>Obj{ID:pivot_table_flat_fact_POF_384_682_fetch}, _3=>Obj)
    Canvas.$wo(_1=>"fetch", _2=>Obj, _3=>undef, _4=>Obj)
    [a]MathFunction.invokeSuper(_1=>null, _2=>"$wo", _3=>undef, _4=>undef, _5=>undef, _6=>undef, _7=>undef, _8=>undef, _9=>undef, _10=>undef, _11=>undef, _12=>undef, _13=>undef)
    [a]MathFunction.Super(_1=>"$wo", _2=>Array[4], _3=>[object Arguments])
    ListGrid.$wo(_1=>"fetch", _2=>Obj, _3=>undef, _4=>Obj, _5=>undef)
    Canvas.fetchData(_1=>Obj, _2=>undef, _3=>undef)
    ** recursed on [a]MathFunction.invokeSuper

15:20:32.435:TMR5:WARN:ResultSet:isc_ResultSet_4 (dataSource pivot_table_flat_fact_POF_384_682, created by: isc_ListGrid_0):ResultSet server fetch with server criteria: {
}
    ResultSet.fetchRemoteData(_1=>Obj, _2=>99, _3=>199)
    ResultSet.$39w()
    [c]Class.fireCallback(_1=>"$39w", _2=>undef, _3=>Array[0], _4=>[ResultSet ID:isc_ResultSet_4 (dataSource: pivot_table_flat_fact_POF_384_682, created by: isc_ListGrid_0)], _5=>undef) on [Class Class]
    [c]Class.$cm()
    ** recursed on [c]Class.fireCallback

15:20:34.309:TMR3:WARN:ResultSet:isc_ResultSet_4 (dataSource pivot_table_flat_fact_POF_384_682, created by: isc_ListGrid_0):ResultSet server fetch with server criteria: {
}
    ResultSet.fetchRemoteData(_1=>Obj, _2=>49, _3=>149)
    ResultSet.$39w()
    [c]Class.fireCallback(_1=>"$39w", _2=>undef, _3=>Array[0], _4=>[ResultSet ID:isc_ResultSet_4 (dataSource: pivot_table_flat_fact_POF_384_682, created by: isc_ListGrid_0)], _5=>undef) on [Class Class]
    [c]Class.$cm()
    ** recursed on [c]Class.fireCallback

[Isomorphic]

The counter is on the client side, I'm just keeping a startRow and endRow counter for my interface to my DB stored proc. Would be be better to get these values from the DSRequest and not have to track these counters myself?

You still haven't given any indication of what this counter is used for, so we still can't comment.

Unfortunately the "fetchTrace" log in this case is inconclusive - it doesn't really distinguish between the possibilities we laid out. You should check into each of those causes, and let us know if you see something that suggests a framework bug.

[JLivermore]

OK, I think I fixed the DSResponse issue with startRow beginning with zero and not 1. But I am still seeing a blank row after the 2nd DSResponse comes back, and I'm seeing 2 DSResponses going out for the first scrolled request. Here are the details:
SmartDevConsole Log:

15:36:25.171:INFO:Log:initialized
15:36:26.307:INFO:Log:isc.Page is loaded
15:37:08.995:TMR7[E]:WARN:ResultSet:isc_ResultSet_0 (dataSource pivot_table_time_frame, created by: isc_PickListMenu_0):ResultSet server fetch with server criteria: {
    "clientID":"682", 
    "factTableID":"POF", 
    "lastDataPoint":"2013-10-31", 
    "dataSource":"postTradeEquitiesFactDataSource", 
    "dimensionID":"1"
}
    ResultSet.fetchRemoteData(_1=>Obj, _2=>0, _3=>75)
    ResultSet.$39w()
    ResultSet.getRangePaged(_1=>0, _2=>1, _3=>undef, _4=>undef)
    ResultSet.$39u(_1=>0, _2=>1, _3=>undef, _4=>undef)
    ResultSet.getRange(_1=>0, _2=>1, _3=>undef, _4=>undef)
    ListGrid.requestVisibleRows()
    Canvas.filterWithCriteria(_1=>Obj, _2=>Obj{ID:pivot_table_time_frame_fetch}, _3=>Obj)
    Canvas.$wo(_1=>"filter", _2=>Obj, _3=>Obj, _4=>Obj)
    [a]MathFunction.invokeSuper(_1=>null, _2=>"$wo", _3=>undef, _4=>undef, _5=>undef, _6=>undef, _7=>undef, _8=>undef, _9=>undef, _10=>undef, _11=>undef, _12=>undef, _13=>undef)
    [a]MathFunction.Super(_1=>"$wo", _2=>Array[4], _3=>[object Arguments])
    ListGrid.$wo(_1=>"filter", _2=>Obj, _3=>Obj, _4=>Obj, _5=>undef)
    Canvas.filterData(_1=>Obj, _2=>Obj, _3=>Obj)
    ** recursed on [a]MathFunction.invokeSuper

15:37:10.450:TMR3:WARN:fetchMissingValues:isc_SelectItem_0[timeFrame]:FetchMissingValues - filterLocally is false yet optionDataSource fetch included records that do not match our current data value. Ignoring these values.
15:37:10.566:WARN:ResultSet:isc_ResultSet_1 (dataSource pivot_table_flat_fact_POF_384_682, created by: isc_ListGrid_4):ResultSet server fetch with server criteria: {
}
    ResultSet.fetchRemoteData(_1=>Obj, _2=>0, _3=>100)
    ResultSet.$39w()
    ResultSet.getRangePaged(_1=>0, _2=>37, _3=>undef, _4=>undef)
    ResultSet.$39u(_1=>0, _2=>37, _3=>undef, _4=>undef)
    ResultSet.getRange(_1=>0, _2=>37, _3=>undef, _4=>undef)
    ListGrid.requestVisibleRows()
    Canvas.filterWithCriteria(_1=>Obj, _2=>Obj{ID:pivot_table_flat_fact_POF_384_682_fetch}, _3=>Obj)
    Canvas.$wo(_1=>"fetch", _2=>Obj, _3=>undef, _4=>Obj)
    [a]MathFunction.invokeSuper(_1=>null, _2=>"$wo", _3=>undef, _4=>undef, _5=>undef, _6=>undef, _7=>undef, _8=>undef, _9=>undef, _10=>undef, _11=>undef, _12=>undef, _13=>undef)
    [a]MathFunction.Super(_1=>"$wo", _2=>Array[4], _3=>[object Arguments])
    ListGrid.$wo(_1=>"fetch", _2=>Obj, _3=>undef, _4=>Obj, _5=>undef)
    Canvas.fetchData(_1=>Obj, _2=>undef, _3=>undef)
    ** recursed on [a]MathFunction.invokeSuper

15:37:33.198:TMR5:WARN:ResultSet:isc_ResultSet_1 (dataSource pivot_table_flat_fact_POF_384_682, created by: isc_ListGrid_4):ResultSet server fetch with server criteria: {
}
    ResultSet.fetchRemoteData(_1=>Obj, _2=>100, _3=>200)
    ResultSet.$39w()
    [c]Class.fireCallback(_1=>"$39w", _2=>undef, _3=>Array[0], _4=>[ResultSet ID:isc_ResultSet_1 (dataSource: pivot_table_flat_fact_POF_384_682, created by: isc_ListGrid_4)], _5=>undef) on [Class Class]
    [c]Class.$cm()
    ** recursed on [c]Class.fireCallback

15:37:35.098:TMR3:WARN:ResultSet:isc_ResultSet_1 (dataSource pivot_table_flat_fact_POF_384_682, created by: isc_ListGrid_4):ResultSet server fetch with server criteria: {
}
    ResultSet.fetchRemoteData(_1=>Obj, _2=>50, _3=>150)
    ResultSet.$39w()
    [c]Class.fireCallback(_1=>"$39w", _2=>undef, _3=>Array[0], _4=>[ResultSet ID:isc_ResultSet_1 (dataSource: pivot_table_flat_fact_POF_384_682, created by: isc_ListGrid_4)], _5=>undef) on [Class Class]
    [c]Class.$cm()
    ** recursed on [c]Class.fireCallback

First DSResponse:

[
    {
        affectedRows:0, 
        data:[
            -- data snipped --
        ], 
        endRow:100, 
        invalidateCache:false, 
        isDSResponse:true, 
        queueStatus:0, 
        startRow:0, 
        status:0, 
        totalRows:200
    }
]

Scroll Initiated DSRequest 1

{
    dataSource:"pivot_table_flat_fact_POF_384_682", 
    operationType:"fetch", 
    componentId:"isc_ListGrid_4", 
    data:{
        spc:[
            "1", 
            "2013-10-01", 
            "2013-12-31", 
            "POF", 
            "70:2^682|16:2^Y|1:2", 
            "10:2^|12:2^|1:2^|6:2^", 
            "5|6|21|29|36|24|25|28|26|30", 
            "false", 
            "101", 
            "200", 
            "5", 
            "", 
            "", 
            "", 
            "384", 
            "2285", 
            "682", 
            "postTradeEquitiesFactDataSource"
        ]
    }, 
    startRow:100, 
    endRow:200, 
    textMatchStyle:"exact", 
    resultSet:[ResultSet ID:isc_ResultSet_1 (dataSource: pivot_table_flat_fact_POF_384_682, created by: isc_ListGrid_4)], 
    callback:{
        caller:[ResultSet ID:isc_ResultSet_1 (dataSource: pivot_table_flat_fact_POF_384_682, created by: isc_ListGrid_4)], 
        methodName:"fetchRemoteDataReply"
    }, 
    willHandleError:true, 
    showPrompt:false, 
    prompt:"Loading...", 
    oldValues:{
        spc:[
            "1", 
            "2013-10-01", 
            "2013-12-31", 
            "POF", 
            "70:2^682|16:2^Y|1:2", 
            "10:2^|12:2^|1:2^|6:2^", 
            "5|6|21|29|36|24|25|28|26|30", 
            "false", 
            "101", 
            "200", 
            "5", 
            "", 
            "", 
            "", 
            "384", 
            "2285", 
            "682", 
            "postTradeEquitiesFactDataSource"
        ]
    }, 
    requestId:"pivot_table_flat_fact_POF_384_682$6277", 
    internalClientContext:{
        requestIndex:2
    }, 
    useStrictJSON:true, 
    fallbackToEval:true, 
    dataPageSize:100, 
    dataFetchMode:"paged", 
    lastClientEventThreadCode:"TMR5", 
    bypassCache:true
}

Scroll Initiated DSRequest 2

{
    dataSource:"pivot_table_flat_fact_POF_384_682", 
    operationType:"fetch", 
    componentId:"isc_ListGrid_4", 
    data:{
        spc:[
            "1", 
            "2013-10-01", 
            "2013-12-31", 
            "POF", 
            "70:2^682|16:2^Y|1:2", 
            "10:2^|12:2^|1:2^|6:2^", 
            "5|6|21|29|36|24|25|28|26|30", 
            "false", 
            "101", 
            "200", 
            "5", 
            "", 
            "", 
            "", 
            "384", 
            "2285", 
            "682", 
            "postTradeEquitiesFactDataSource"
        ]
    }, 
    startRow:50, 
    endRow:150, 
    textMatchStyle:"exact", 
    resultSet:[ResultSet ID:isc_ResultSet_1 (dataSource: pivot_table_flat_fact_POF_384_682, created by: isc_ListGrid_4)], 
    callback:{
        caller:[ResultSet ID:isc_ResultSet_1 (dataSource: pivot_table_flat_fact_POF_384_682, created by: isc_ListGrid_4)], 
        methodName:"fetchRemoteDataReply"
    }, 
    willHandleError:true, 
    showPrompt:false, 
    prompt:"Loading...", 
    oldValues:{
        spc:[
            "1", 
            "2013-10-01", 
            "2013-12-31", 
            "POF", 
            "70:2^682|16:2^Y|1:2", 
            "10:2^|12:2^|1:2^|6:2^", 
            "5|6|21|29|36|24|25|28|26|30", 
            "false", 
            "101", 
            "200", 
            "5", 
            "", 
            "", 
            "", 
            "384", 
            "2285", 
            "682", 
            "postTradeEquitiesFactDataSource"
        ]
    }, 
    requestId:"pivot_table_flat_fact_POF_384_682$6278", 
    internalClientContext:{
        requestIndex:3
    }, 
    useStrictJSON:true, 
    fallbackToEval:true, 
    dataPageSize:100, 
    dataFetchMode:"paged", 
    lastClientEventThreadCode:"TMR3", 
    bypassCache:true
}

The two Scroll Initiated DSRequest above went out, when I was only expecting 1 to go out.

[Isomorphic]

Without a test case and with such partial information, all we can say is that a blank row very strongly suggests either that the startRow/endRow numbers are still wrong, or that you are actually returning a blank record in your dataset.