big data fetch getTotalRows() problem.

[jarekevi]

SmartClient Version: v9.1p_2014-03-04/PowerEdition Deployment (built 2014-03-04)
Firefox 25

DSRequest req = new DSRequest();
		req.setStartRow(0);
		req.setEndRow(1);
		presenter.getView().getMessageGrid().getDataSource().fetchData(crit, new DSCallback() {
			@Override
			public void execute(DSResponse dsResponse, Object data, DSRequest dsRequest) {
				dsResponse.getTotalRows();
			}
		}, req);

Hi,
I want to check the the number of records that fit the given criteria. So i fetch data with Request range 1. Everything works fine on small data sets, but when the fetch finds more than 200k (or so, the problem occurs for sure for over 200k, might work for smaller sets) records the

dsResponse.getTotalRows()

method allways returns 21 !?!
Works fine for smaller data sets.

[Isomorphic]

Can you confirm - what kind of DataSource is this (IE what is the storage)? Is this a standard SmartGWT SQL dataSource with a very large table in the database or something else? Any custom server logic, etc?

If you open the developer console and enable RPC request tracking on the "RPC" tab - you should see the fetch request / response - can you show us what the response from the server looks like (this should indicate whether the 'totalRows' are being correctly passed to the client from the server or whether the problem is occurring before the response reaches the client).

Thanks
Isomorphic Software

EDIT: Also - can you confirm whether this occurs for you in compiled mode or whether it's a problem in hosted (development) mode only? And whether this is specific to a particular browser. Thanks.

[Isomorphic]

Actually, we just realized what this is - see operationBinding/dataSource.progressiveLoading.

You can either turn this feature entirely off, or, if you still want to use progressiveLoading by default, but need an accurate totalRows value for some kind of separate display from the grid, create a second operationBinding where progressiveLoading is off and use that when fetching the totalRows value.

[jarekevi]

Dosn't seem to be it.The problem still occurs.
request (expected over 200k):

{
    dataSource:"TestProd_messages", 
    operationType:"fetch", 
    data:{
        operator:"and", 
        criteria:[
            {
                fieldName:"review_status", 
                operator:"notEqual", 
                value:"6"
            }, 
            {
                fieldName:"query_info_id", 
                operator:"equals", 
                value:12083
            }
        ]
    }, 
    startRow:0, 
    endRow:1, 
    textMatchStyle:"exact", 
    showPrompt:true, 
    oldValues:{
        operator:"and", 
        criteria:[
            {
                fieldName:"review_status", 
                operator:"notEqual", 
                value:"6"
            }, 
            {
                fieldName:"query_info_id", 
                operator:"equals", 
                value:12083
            }
        ]
    }, 
    requestId:"TestProd_messages$62772", 
    fallbackToEval:false, 
    progressiveLoading:false, 
    lastClientEventThreadCode:"MUP8", 
    bypassCache:true
}

Response:

[
    {
        affectedRows:0, 
        data:[
            {
            [ data ]
            }, 
            {
           [ data ]
            }
        ], 
        endRow:2, 
        invalidateCache:false, 
        isDSResponse:true, 
        operationType:"fetch", 
        queueStatus:0, 
        startRow:0, 
        status:0, 
        totalRows:21 <--------- Black Jack
    }
]

for small data :

{
    dataSource:"TestProd_messages", 
    operationType:"fetch", 
    data:{
        operator:"and", 
        criteria:[
            {
                fieldName:"review_status", 
                operator:"equals", 
                value:"6"
            }, 
            {
                fieldName:"query_info_id", 
                operator:"equals", 
                value:12083
            }
        ]
    }, 
    startRow:0, 
    endRow:1, 
    textMatchStyle:"exact", 
    showPrompt:true, 
    oldValues:{
        operator:"and", 
        criteria:[
            {
                fieldName:"review_status", 
                operator:"equals", 
                value:"6"
            }, 
            {
                fieldName:"query_info_id", 
                operator:"equals", 
                value:12083
            }
        ]
    }, 
    requestId:"TestProd_messages$62775", 
    fallbackToEval:false, 
    progressiveLoading:false, 
    lastClientEventThreadCode:"MUP8", 
    bypassCache:true
}

[
    {
        affectedRows:0, 
        data:[
            {
               [ data ]
            }
        ], 
        endRow:1, 
        invalidateCache:false, 
        isDSResponse:true, 
        operationType:"fetch", 
        queueStatus:0, 
        startRow:0, 
        status:0, 
        totalRows:5467 <--- Over
    }
]

[Isomorphic]

Then we will need all the information previously requested.

[jarekevi]

Standart sql, no additional server logic, simple fetch

<DataSource 
	ID="messages"
	serverType="sql"
	schema="dev"
	tableName="message_results"
	titleField="messages"
	
>
	<fields>
		<field name="id" type="integer" title="id" primaryKey="true" hidden="true"/> 
		<field name="bcc" type="text" title="bcc" />
		<field name="cc" type="text" tite="cc" />
		<field name="extern_type" type="integer" title="type"/>
		<field name="import_date"  type="date"       title="import_date" /> 
		<field name="recipient" type="text" title="to" />
		<field name="sender" type="text" title="from" />
		<field name="sent_date"  type="datetime" title="datefield"/> 
		<field name="subject" type="text" tite="subject" />
		<field name="sys_creation_date"  type="date" title="sys_creation_date"/> 
		<field name="technology" type="integer" title="type"/>
		<field name="query_info_id" type="integer" primaryKey="true" title="query_info_id" hidden="true"/>
		<field name="attachments_hadoop_files_ids" type="any" title="attachments_hadoop_files_ids" />
		<field name="source" type="integer" title="source" />
		<field name="risk_level" type="integer" title="risk_level" />		
		<field name="with_attachment" type="boolean" title="with_attachment" />
				
	</fields>
</DataSource>

[Isomorphic]

Sorry, we forgot to reiterate it, but with any server issue, we *always* need the complete server logs for the request.

[Isomorphic]

Also, please let us know what database you are using (including exact version and OS/platform).

[jarekevi]

psql (PostgreSQL) 9.1.7

I found the culprit, trusted the RCP track, but server logs revealed:

(default task-1) === 2014-04-18 12:30:11,809 [sk-1] INFO  SQLDataSource - [builtinApplication.TestProd_messages_fetch] Row count query found 216541 rows, which exceeds the DataSource's progressiveLo
adingThreshold of 200000.  Switching to progressive loading mode.

what is wierd, because I'm setting

setProgressiveLoading(false);
progressiveLoading="false"

on the DSRequest

getView().getMessageGrid().getDataSource().fetchData(criteria.asAdvancedCriteria(), new DSCallback() {
			@Override
			public void execute(DSResponse response, Object data, DSRequest dsRequest) {
				resolveMessageFetchResponse(response, criteria);				
			}
		}, request);

[Isomorphic]

We'll look into whether we can reproduce a problem with the progressiveLoading flag not being honored when passed from the client.

In the meantime, we suggest the remedy we previously posted:

You can either turn this feature entirely off, or, if you still want to use progressiveLoading by default, but need an accurate totalRows value for some kind of separate display from the grid, create a second operationBinding where progressiveLoading is off and use that when fetching the totalRows value.

[Isomorphic]

We've had a look at this issue and it appears that DSRequest.progressiveLoading was only being partially honoured for SQL data sources. This has now been fixed and will be available in a nightly build in a day or two.

You should with this fix be able to create a DSRequest, set both startRow and endRow to zero and progressiveLoading to false to just get a row count and no data items back.

Regards,
Isomorphic Software

[jarekevi]

Thank you, works fine

[jarekevi]

v9.1p_2014-05-18/PowerEdition Deployment (built 2014-05-18)

It seems, that it still not working fine. Only the values changed to higher ones.

{
    dataSource:"messages", 
    operationType:"fetch", 
    data:{
        operator:"and", 
        criteria:[
            {
                fieldName:"query_info_id", 
                operator:"equals", 
                value:50775
            }
        ]
    }, 
    startRow:0, 
    endRow:528, 
    sortBy:[
        "-sent_date"
    ], 
    textMatchStyle:"exact", 
    showPrompt:true, 
    oldValues:{
        operator:"and", 
        criteria:[
            {
                fieldName:"query_info_id", 
                operator:"equals", 
                value:50775
            }
        ]
    }, 
    requestId:"messages$627247", 
    fallbackToEval:false, 
    progressiveLoading:false, 
    lastClientEventThreadCode:"KPR4", 
    bypassCache:true
}

        endRow:529, 
        invalidateCache:false, 
        isDSResponse:true, 
        operationType:"fetch", 
        queueStatus:0, 
        startRow:0, 
        status:0, 
        totalRows:548
    }

The total rows value is 548 but expected is over 3 000k. It also behaves the same way for fetches with total row number over 300k

[jarekevi]

Is anyone looking into it ?