Problems with autoFetchData

[svonmiller]

Greetings,

I'm using 7.0rc2. I've got a listgrid that I'd like to autofetch as the user scrolls through large numbers of records. For some reason, it's not working. The only thing I'm doing differently that might be causing a problem is that I do a .setFields() to the grid to dynamically set the fields, then I .fetchData(criteria). Would this cause it to not to autofetch while the user scrolls? The result set I'm testing with is 4,000 records.

	isc.ListGrid.create({
	    ID: "searchResultsList",
	    loadingDataMessage:"Searching...",
	    dataSource:"SearchResults",
		width:panelWidth-30,
		bodyBackgroundColor:"#FFFFFF",
		alternateRecordStyles:true,
		selectionAppearance:"checkbox",
		canPickFields:true,
	    autoFetchData:true,
	    headerHeight:30,
	    autoFitData:"vertical",
	    autoFitMaxHeight:550,
	    showAllRecords:false,
	    wrapCells:true,
	    cellHeight:30,
	    left:0,
		autoDraw:true
	});

[Isomorphic]

No, setFields() should not cause this. Just troubleshoot it normally; check the RPC tab to see if correct totalRows values are being sent, and enable the "ResultSet" log category.

[svonmiller]

I'm setting totalRows to the entire result set size (4026). I'm under the impression setTotalRows should be for the entire result size, not the desired paging size.

I used fiddler as well as the RPC trace to make sure it wasn't actually making calls. And as well, totalRows are being set (but to the total value as I mentioned). Is there anything else that might keep it from fetching everything at once?

Below are the log entries.
Server log:

=== 2010-11-11 13:48:48,969 [sor3] INFO  RequestContext - URL: '/iap/include/isomorphic/IDACall', User-Agent: 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; GTB6.6; .NET CLR 2.0.50727; .NET CLR 1.1.4322; .NET CLR 3.0.04506.30; .NET CLR 3.0.04506.648; MS-RTC LM 8; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; OfficeLiveConnector.1.4; OfficeLivePatch.1.3)': MSIE with Accept-Encoding header, ready for compressed JS
=== 2010-11-11 13:48:48,969 [sor3] INFO  IDACall - Performing 1 operation(s)
=== 2010-11-11 13:48:49,172 [sor3] INFO  DSResponse - DSResponse: List with 4261 items

And the RPC trace:

[
    {
        invalidateCache:false,
        data:[
....


],
        status:0,
        totalRows:4261,
        isDSResponse:true
    }

[svonmiller]

There's another oddity as well. In one instance, when setting the fields using setFields(), I set a couple of columns frozen. This works fine, unless on this same listGrid, I set autoFitData:"vertical". If this is true, then when using those frozen columns I get the error below. I don't know if this is a related problem or not.

14:05:55.499:RDQ8:WARN:Log:Error:
	'Object doesn't support this property or method'
	in http://svonmiller.highlinedata.com:81/iap/include/isomorphic/system/modules/ISC_Grids.js
	at line 722
    GridBody.getAutoFitMaxHeight()
        "return this.parentElement?this.parentElement.getAutoFitMaxBodyHeight():null"
    GridBody.adjustOverflow(_1=>"redraw", _2=>null, _3=>true, _4=>undef, _5=>undef)
    Canvas.$ra()
    Canvas.redraw(_1=>false, undef, undef, undef, undef, undef, undef, undef)
    Class.invokeSuper(_1=>[Class GridRenderer], _2=>"redraw", _3=>false, _4=>undef, _5=>undef, _6=>undef, _7=>undef, _8=>undef, _9=>undef, _10=>undef)
    GridRenderer.redraw(_1=>false, _2=>undef, _3=>undef, _4=>undef, undef, undef, undef, undef)
    ** recursed on Class.invokeSuper

[Isomorphic]

Yes, that's what totalRows means.

Look closely at the data array - are you maybe sending an Array of length 4026, with a bunch of blank spots?

What specific behavior do you see when you scroll down? Blank records? Scrollable area stops after the first batch of loaded records?

[Isomorphic]

If you're getting a JS error, you can ignore all behavior after that error as not being important, because things are potentially already in a bad state. Do you ever get a problem without this JS error?

[svonmiller]

As long as I'm not using autoFitData, autoFitMaxHeight or autoFixMaxRecords, it works without throwing errors. And it works just fine, scrolling is normal, and there are no blank records or other odd behaviors, except that it sends all 4026 rows to the client and never fetches again.

I've noticed in the feature explorer fetchdata example that startrow and endrow are being set. I'm not doing this in my fetchdata code. I'm betting this is the problem. My fetchData method simply does

DSResponse dsResponse = new DSResponse();

And so I'm not passing through a startrow or endrow, so I would imagine this is the problem. Sound right?

[Isomorphic]

Correct. If you manually create a DSRequest and call DataSource.fetchData(), startRow and endRow are not set. It's the ResultSet created by calling ListGrid.fetchData() that automatically adds startRow and endRow based on scroll position.

[svonmiller]

Setting startRow and endRow didn't solve any problems.

After I setFields(), and then do a fetchData(criteria, callBack); the fetchData() won't make a call to the server (this was always the case). The only way I could get it to was to fetchData, then invalidateCache(). This forces the call. I don't know why I have to force it, but I do. Would setting invalidateCache() cause this pagination problem? I don't fully understand what invalidateCache is doing internally, and again, I don't know why fetchData() doesn't fetch without using it. But now that I'm thinking about it, invalidateCache just sounds like it would cause this problem. If this is the problem, why won't fetchData(criteria) fetch? I'm pretty sure the criteria is changed.

[Isomorphic]

At this point we're really not sure what methods you're calling. Please take a look at the previous responses - setting startRow and endRow manually apply *only* to calling DataSource.fetchData(). Are you or are you not calling this API?

Your description of the behavior - loading all records - sounds like either setting clientOnly:true or cacheAllData:true on the DataSource, or setting dataFetchMode:"basic" or "local" on the ListGrid. All of these settings explicitly disable paging.

[svonmiller]

I'm not setting startRow and endRow manually. I'm passing it through:

public DSResponse fetch(HttpServletRequest request, DSRequest dsRequest) throws Exception {

    	DSResponse dsResponse = new DSResponse(dsRequest == null ? (DataSource)null : dsRequest.getDataSource());
    	dsResponse.setStartRow(dsRequest.getStartRow());
    	dsResponse.setEndRow(dsRequest.getEndRow());

    	try {
	    	List criteriaList = dsRequest.getCriteriaSets();

The datasource is a totally standard setup. I'm not setting any of the methods on the listgrid or datasource that you mention.

	isc.ListGrid.create({
	    ID: "searchResultsList",
	    loadingDataMessage:"Loading...",
	    dataSource:"SearchResults",
		width:panelWidth-30,
		bodyBackgroundColor:"#FFFFFF",
		alternateRecordStyles:true,
		selectionAppearance:"checkbox",
	    autoFetchData:true,
	    headerHeight:30,
	    height:200,
	    wrapCells:true,
	    cellHeight:30,
	    left:0,
		autoDraw:true
	});

The only thing I'm doing out of the ordinary is, when the user clicks a "search" button, I do an RPC call that sets the fields, then fetchData, which does nothing until I invalidateCache. That's all i'm doing:

	    switch(rpcResponse.status)
	    {
	    	case 0: //Search successful and results are there
	    		<%-- Configure search list results grid --%>
				searchResultsList.setFields(rpcResponse.data);
				<%-- Now that new fields are set, load the grid --%>
				searchResultsList.fetchData(criteria, setupResultsPanel());
				searchResultsList.invalidateCache();
	    		break;

The third the last line, fetchData, does nothing unless I run the next line, invalidateCache(), then it does the fetch. The only thing I can come up with is that the invalidateCache prevents the pagination.

[svonmiller]

This is the datasource. I contains all possible fields I might need, and I use setFields() to setup a subset of these fields:

<DataSource ID="SearchResults" serverType="generic" dataFormat="iscServer">
	<fields>
		<field name="company_code" type="integer"	/>
		<field name="ins_type" 	type="integer"	/>
		<field name="company_code_dis" type="text" />
		<field name="company_name" type="text" />
		<field name="ticker" type="text"  />
		<field name="city" type="text"  />
		<field name="state" type="text"  />
		<field name="total_assets" type="integer"  />
		<field name="financial_date"  type="date" />
		<field name="data_status"  type="text" />
		<field name="group_name"  type="text" />
		<field name="group_code"  type="text" />
		<field name="net_income" type="integer" />
		<field name="net_prem_written" type="integer" />
		<field name="dpw" type="integer" />
		<field name="net_assets" type="integer" />
		<field name="member_months" type="integer" />
		<field name="fein" 		type="text" />
		<field name="domicile" type="text" />
		<field name="filing_type" type="integer" />
		<field name="assumed_prem_filed" type="integer" />
		<field name="assumed_prem" type="integer" />
		<field name="ceded_prem_filed" type="integer" />
		<field name="ceded_prem" type="integer" />
		<field name="filer" type="integer" />
		<field name="assumed_life" type="integer" />
		<field name="ceded_life" type="integer" />
		<field name="assumed_ah" type="integer" />
		<field name="ceded_ah" type="integer" />
	</fields>
    <serverObject lookupStyle="new" className="com.insx.isomorphic.ds.AdvancedSearchController" />
    <operationBindings>
        <binding operationType="fetch" serverMethod="fetch" customCriteriaFields="searchType" methodArguments="$request, $dsRequest"/>
    </operationBindings>
</DataSource>

[Isomorphic]

startRow and endRow can be set either client-side (which means requesting a specific range of data from the server, and which the ResultSet does automatically) or server-side (which is required if you are implementing your own server-side DataSource / DMIs and you want to support paging - you have to tell the client what rows you are returning - see the samples).

However, having seen more of the code, what appears to be happening is that you are setting autoFetchData:true, causing the grid to make a request and receive a presumably empty result before setFields() is ever called. Then, when setFields() and fetchData() is called, it already has an empty result for the current criteria, so no new fetch is issued (which is correct).

If you don't want data fetched until you call fetchData(), do not set autoFetchData.

[svonmiller]

Yes I suspected that was the problem early on in the troubleshooting loop I've been in. So I set autoFetchData:false, and I don't need the invalidateCache() method as suspected. But it has no effect on the pagination problem. All records still come down to the client. So when it had no effect, I started thinking that if autoFetchData was false, then pagination wouldn't happen.

So are you saying that even if autoFetchData is false, pagination should still happen? Does that attribute only apply to when the grid is drawn?

If invalidateCache prevents pagination, and autoFetchData:false prevents it, that would explain the problem. I don't know how to solve it yet, but that would explain what is going on. But in reading your responses, I've not seen anything that states either one of those settings will actually prevent pagination.

[Isomorphic]

Right, none of those settings have anything to do with pagination.

We listed off the ways to disable pagination on the client-side in a previous post. These would result in no startRow/endRow parameters being sent to the server. This is easy to detect by looking at the RPC tab.

For server-side, you need to:
1. return only the data you mean to return in your call to setData(). This will not be automatically trimmed by the server
2. set startRow, endRow and totalRows indicating the range of records returned and the total available.

This is shown in all samples. Whether you are doing this correctly is easily determined by looking in the RPC tab.