RestDataSource: can totalRows be omitted?

[ljnelson]

I was wondering what the behavior is defined to be when a backend web service returns an XML document to a RestDataSource, where the following things are true:
* paging is in effect {handwave handwave}
* <startRows> is present in the output in the appropriate place
* <endRows> is present in the output in the appropriate place
* <totalRows> is OMITTED from the output

I understand what <totalRows> does. I understand that it drives paging and caching. I am effectively asking what its default value is (some big number, or something approaching zero?), or whether any errors I might encounter by virtue of omitting <totalRows> are to be interpreted as part of the defined behavior of SmartGWT and not just a happy or unhappy accident.

Thanks very much for any input.
Best,
Laird

[Isomorphic]

The behavior in the absence of totalRows in not defined. You may be thinking about progressive loading - see "Paging and total dataset length" under the ResultSet docs.

[ljnelson]

Thanks. Progressive loading isn't what I was thinking of directly, but yes, perhaps I was indirectly. Thanks for the tip. Is it reliable and guaranteed to infer the DataSource's page size from the _startRow and _endRow parameter? And is the _endRow inclusive?

To belabor: if I have rows 0, 1 and 2, I would say that the page size is three, and the end row is 2. Is that correct?

If so, then progressive loading would be enabled by reporting totalRows of 6. Is that correct?

Thanks,
Laird

[Isomorphic]

That's not completely reliable, since there are edge cases of eg filling in the top few rows of a dataset where you start with cache in the middle. Better to just use a fixed page size that you configured server-side, the client side system will basically handle whatever you return.

endRow is exclusive. The row parameters are inclusive at the beginning, exclusive at the end, like String.substring()

[ljnelson]

Thanks again. To be very explicit, then:

My server side component might get _startRow=0 and _endRow=6.
That means rows 0 through 5 at a minimum SHOULD be returned.
My server side component decides for whatever reason to return rows 0 through 240, because this is, say, its configured page size.
The client, although it asked for five rows, will swallow hard and deal with the 240 rows I send back.

Yes?

Best,
Laird

[Isomorphic]

Yup. The client is OK with any number of rows being returned so long as either all requested row are return or you set a totalRows amount indicating that there are no more rows to be had. Because of this, you can drive caching behaviors from the server (such as deciding, based on physical office location of the user, how aggressively to cache - yes this scenario came up).

[ljnelson]

Yup. The client is OK with any number of rows being returned so long as either all requested row are return or you set a totalRows amount indicating that there are no more rows to be had.

To help my slow brain, then:

Assume a start of 0 and an end row of 10.
Assume a server-side fixed page size of 5, let's say.
Assume a real, physical count of rows on disk of 100.

Is <totalRows> supposed to indicate the physical count of rows on disk, understanding that in some cases the server will fake this number? http://www.smartclient.com/smartgwt/javadoc/com/smartgwt/client/data/DSResponse.html#getTotalRows%28%29 indicates that it does.

(Or is it supposed to indicate the actual number of rows returned in response to the original startRow/endRow query? I suspect no.)

Given the example above, would you kindly assess the truthfulness of these statements:

The server SHOULD in a perfect world return rows 0-9, inclusive.
The server in such a case MUST return a <startRow> of 0 and an <endRow> of 10.
The server MUST in all cases supply a <totalRows>. This doesn't HAVE to match the actual physical count.
The server MAY return fewer or more than the number of rows requested.
That is, the server MAY return rows 0-2 inclusive, or 0-49 inclusive.

Because of this, you can drive caching behaviors from the server (such as deciding, based on physical office location of the user, how aggressively to cache - yes this scenario came up).

:-D Wow.

Best,
Laird

[ljnelson]

The server MAY return fewer or more than the number of rows requested.
That is, the server MAY return rows 0-2 inclusive, or 0-49 inclusive.

Quick follow up: MUST the server honor the startRow and endRow requests? That is, if the request asks for rows 7 through 15, exclusive, could the server (in a fit of stupidity or something else) return 0 through 4, exclusive? Or is it assumed (and therefore required of the server) that the <startRow> it returns is equal to the <startRow> supplied?

Just looking for the boundaries and edges of this contract.

Best,
Laird

[Isomorphic]

Yes, all correct, except just to clarify the only case in which the server is allowed to return *less* than the requested rows is if there are no such rows, for example, the client requested past the end of the dataset because it does not know the total dataset length on initial fetch.

[ljnelson]

Yes, all correct, except just to clarify the only case in which the server is allowed to return *less* than the requested rows is if there are no such rows, for example, the client requested past the end of the dataset because it does not know the total dataset length on initial fetch.

OK, great.

So then, to finish up:

* A RestDataSource request, when paging is enabled, will send startRow and endRow parameters.
* The request endRow is exclusive.
* It is expected, BUT NOT REQUIRED, that the server will honor these parameters.
* The server MUST return <startRow>, <endRow> and <totalRows> elements in its response.
* If the server has chosen to honor the paging request parameters, it SHOULD return the same <startRow> and <endRow> values.
* The <totalRows> returned by the server should be its best guess at representing the total number of possible rows that could be returned (total rows on disk).
* The server can indicate that it has returned fewer rows than requested by making its <endRow> element LESS THAN the requested endRow parameter value. It may do this ONLY when it is actually impossible for it to "fill up" the full amount of the rows requested by the client (endRow - startRow).
* The server can indicate that it has returned more rows than requested by making its <endRow> element GREATER THAN the requested endRow parameter value. It may do this whenever it wishes, although obviously this isn't what the client asked for.
* The server can, conceivably, "reanchor" its <startRow> element, though it is hard to see what benefits this would confer. That is, the client could ask for a startRow of 7, and the server could respond with a startRow of 3.
* The server MUST NOT have an endRow element in its response that is less than its corresponding startRow.
* The server's <totalRows> element serves another purpose, which is to control client caching. If the server wishes to "lie" about the rows on disk, it could return a value of 0 here, and then the client would never check again for new data.

Please feel free to correct everything I've gotten wrong. Thanks again for your time and attention on this long thread.

Best,
Laird

[Isomorphic]

All correct.

[ljnelson]

All correct.

And with that, we return you to your regularly scheduled programming. Thanks again for the time, and for a great toolkit!

Best,
Laird

[ljnelson]

All correct.

Eek; sorry to surface again.

The Javadoc you pointed me at in your comment to my blog article (http://weblogs.java.net/blog/ljnelson/archive/2010/04/28/pushing-jersey-limit#comment-14692) which is here (http://www.smartclient.com/smartgwtee/javadoc/com/smartgwt/client/data/DSRequest.html#getStartRow%28%29) says:

"Note that startRow and endRow are zero-based, so startRow: 0, endRow: 1 is a request for the first two records."

That implies that endRow is inclusive, not exclusive. Is this conversation authoritative, or is that Javadoc authoritative?

Thanks,
Laird

[Isomorphic]

Oops - yes in reality it's exclusive at the end, it looks like someone added that line to clarify that zero-based numbering was in use and erroneously put in "two records". Fortunately, since the client handles extra data, getting an extra record is fine :)