[wallytax]

Scenario A: Show 800.000 records, paginated but without progressive loading:

The server sends back a response like this:

{
  "response": {
    "status": 0,
    "data": [....],
    "endRow": 79,
    "startRow": 0,
    "totalRows": 800000
  }
}

I'm using the getVisibleRows() and getTotalRows() methods inside a dataChanged() event to show the visible and total number of records. It all works fine, until I noticed the warning in the browser's Developer Tools. So I'm trying to build...

Scenario B: Show 800.000 records, paginated but with progressive loading:

The server first counts the number of rows. If it is less than 200.000 (which isn't the case in this example), it returns a response like the one from scenario A. If it is more than 200.000, it sends back this response:

{
  "response": {
    "status": 0,
    "data": [....],
    "endRow": 79,
    "startRow": 0,
    "totalRows": 99,
    "myCustomRowCount": 800000
  }
}

I don't know how to access that "myCustomRowCount" property inside the grid's dataChanged() event.

Beware: this is what I had before I started looking into that SmartClient 14 API, which was after your post hinted me into that direction. It looks promising, so what I did was replace the SmartClient code to version 14 and changed the response only at two places:

• renamed "myCustomRowCount" to "estimatedTotalRows"
• added "progressiveLoading" with value true

That didn't do anything. Then I started debugging a bit, to find out that internally the "progressiveLoading" and "estimatedTotalRows" properties (from the original JSON response) were not taken over in a DSResponse instance.

That's when I stopped my endeavor and write you this post. To conclude it positively, I would like to know if it is possible to use the "myCustomRowCount" property inside the grid's dataChanged() event. The only way I can think of to be able to access it, is by inheriting the transformResponse() method of the data source, check the component ID that invoked the request and if that's a grid, set a property on the grid with that value. This seems to be doable, but is not favorable because my data sources are generated taking user grants into account and I don't need that code everywhere, only in this particular grid. Therefore, I would prefer doing something special in this particular grid.

Hope this all makes sense...

[Isomorphic]

Wallytax, we are kind of lost as to how to respond here. You talk about being "curious and desperate" when features from 14.0 could not be found in the 13.0 documentation.

But this is just normal software stuff: new features are introduced, and they are only available in the new version - you can't use them in the old version.

Everything from Windows to MacOS to Java works this way - it's not just us!

And we can't do anything further in terms of moving new features back - if we just moved everything from 14.0 back to 13.0 or 12.0 - then all versions would be the same, right?

We can't do any better in this area, so far as we can see. And surely in your application, you have the same problem of managing customer expectations around versions.

---

The only other thing we can find in your messages is that you are struggling to deliver a custom DSResponse property to the client. Here - yes sorry, you've said you've read it a 100 times - it's in the QuickStart Guide! It's the Data Integration chapter.

As far as how to apply what's in the QuickStart Guide: you've never mentioned what kind of DataSource you are using, which is critical basic information that anyone trying to help you needs to have, and which you must always provide when posting here.

If we ignore that requirement and just try to take a guess from your JSON responses - it could be isc.RestDataSource, or could be just isc.DataSource with various settings. Either way, yes, as the QuickStart tells you, transformResponse() is the right place to pull custom properties from your raw JSON response and put them in the DSResponse if you want to respond to such properties in a component-level handler.

It's extremely straightforward to just inspect your data object from the server, as passed to transformResponse(), and then just modify the dsResponse object, also passed to transformResponse().

This would typically be literally one line of code aside from the call to Super() / method declaration.

So we don't know what else to tell you in this area to help you succeed. Specifically, we don't have even an outline of your DataSource and we cannot test your code. But it looks like you need a line like dsResponse.wallyRealTotalRows = data.estimatedTotalRows... something like that. Then you have a dataArrived handler that looks at DSResponse.wallyRealTotalRows and does whatever you want.

Note also that none of this even seems necessary to complete your use case. The DataSource API is bristling with APIs that give you access to the raw data returned by your server. Use any of them. If you can't succeed from the docs, well, just set a breakpoint on any one of them, and there you go - parameters right there and you can drill down into them with the browser's native tools. Can't miss!

---

Finally, as far the developer-only message about browser limitations: in order to render the first 80 records of a dataset, but render space as if 1M records exist (so that the scrollbar works), then we need to create a space of 20,000,000 pixels (20 x 1M).

Some browsers choke when you try to render this space - they start returning nonsense values from various browser APIs, for example, they might report the space as only 65,536 pixels instead of the full amount.

That's what the message means: some browsers, on some platforms, cannot handle this - and even if they could, it's a bad UE. As you said, moving the thumb 1px might move past 1,000 records. We put a warning in there both to inform developers that some versions and some browsers might fail, and also to say: if you are doing this, you may have a bad UE, please reconsider.

[Isomorphic]

You added more after we replied..

Yes, as the QuickStart says, and as covered above, transformResponse() is your opportunity to turn your custom response into a standard DSResponse object, and put whatever properties you want on the DSResponse object that all component-level handers receive.

As discussed in the QuickStart, this is a key part of the SmartClient Architecture. UI components do not need to know anything about the protocol used by a DataSource to contact the server. They just work with DSRequest / DSResponse. This is why exactly the same UI, with code completely unchanged, can work with a SQL-based server, REST DataSource, .NET backend, Node backend, whatever.

This is all covered in the QuickStart Guide.

This is why you need basically one line of code to have your custom property appear. It doesn't matter if the DataSources are generated or not. You can create a simple subclass that has your transformResponse() implementation with its simple override to set up your custom property. Then all of your DataSources can be instances of that subclass.

Dead simple. This is barely any code, and by the same pattern, your UI could connect to LDAP, a custom router backend written in C++, whatever.

[claudiobosticco]

Hello Isomorphic, just chiming in for a quick question about transformResponse (which indeed is a powerful feature), the quickstart guide also mention a DataSource.transformRequest method, but I can't find it in the javadoc, could you please add it?

[wallytax]

Please read my messages better. I'm pretty sure I've explained extensively what happened. I was not starting the version 14 stuff...

I will respond as short as possible in different posts. First, about SmartClient 14.

I was not using anything above stable version 13 at all, none of my recent posts will mention that. However, 6 posts above, you show a link and when clicking that, it shows the documentation of version 14 Enterprise. I didn't notice that at first and was surprised to find that whole row range API. I've dismissed that reference and later opened my own bookmark (to version 13) and found out the documentation was gone. Then I reopened the link from your post and found the documentation again and it was then that I discovered it was version 14 documentation.

Curious and desperate, I've checked for 15 minutes if it would all work with this version 14, but it didn't either.

BTW, claudiobosticco mentions two posts later that the documentation is also in version 13.1, which I don't use either.

I did however searched the code in version 13 for the properties and found them. Gave this one more try, but that approach led to nothing. So, I'm back at my original code.

[wallytax]

You added more after we replied..

Nope, my messages were all before your reaction... Before you mention it, after writing them, I read them once again and fixed some grammatical errors. I do my best to write as good English as I can, since it's not my native language... but even the edited messages were finished before your reply.

Yes, as the QuickStart says, and as covered above, transformResponse() is your opportunity to turn your custom response into a standard DSResponse object, and put whatever properties you want on the DSResponse object that all component-level handers receive.

I will look into that, thanks!

As discussed in the QuickStart, this is a key part of the SmartClient Architecture. UI components do not need to know anything about the protocol used by a DataSource to contact the server. They just work with DSRequest / DSResponse. This is why exactly the same UI, with code completely unchanged, can work with a SQL-based server, REST DataSource, .NET backend, Node backend, whatever.

This is all covered in the QuickStart Guide.

This is why you need basically one line of code to have your custom property appear. It doesn't matter if the DataSources are generated or not. You can create a simple subclass that has your transformResponse() implementation with its simple override to set up your custom property. Then all of your DataSources can be instances of that subclass.

Dead simple. This is barely any code, and by the same pattern, your UI could connect to LDAP, a custom router backend written in C++, whatever.

Well, I knew about transformResponse() and I use it. What was more unclear - at least for me - that if I do this in the data source, my custom property will show up later in the call stack of methods.

[wallytax]

Wallytax, we are kind of lost as to how to respond here. You talk about being "curious and desperate" when features from 14.0 could not be found in the 13.0 documentation.

But this is just normal software stuff: new features are introduced, and they are only available in the new version - you can't use them in the old version.

Everything from Windows to MacOS to Java works this way - it's not just us!

And we can't do anything further in terms of moving new features back - if we just moved everything from 14.0 back to 13.0 or 12.0 - then all versions would be the same, right?

We can't do any better in this area, so far as we can see. And surely in your application, you have the same problem of managing customer expectations around versions.

Please read back, I did only look into version 14 after your post. I think I did something quite "humanly reasonable": attempted several things, then got a reaction suggesting there's built-in support, it turned out from version 14, attempted that (which failed), saw (parts of) the documented code in version 13 and attempted once more. After that, quit those attempts because it looked unsupported.

The only other thing we can find in your messages is that you are struggling to deliver a custom DSResponse property to the client. Here - yes sorry, you've said you've read it a 100 times - it's in the QuickStart Guide! It's the Data Integration chapter.

I've posted the response my server sends (multiple times).

As far as how to apply what's in the QuickStart Guide: you've never mentioned what kind of DataSource you are using, which is critical basic information that anyone trying to help you needs to have, and which you must always provide when posting here.

If we ignore that requirement and just try to take a guess from your JSON responses - it could be isc.RestDataSource, or could be just isc.DataSource with various settings. Either way, yes, as the QuickStart tells you, transformResponse() is the right place to pull custom properties from your raw JSON response and put them in the DSResponse if you want to respond to such properties in a component-level handler.

It's extremely straightforward to just inspect your data object from the server, as passed to transformResponse(), and then just modify the dsResponse object, also passed to transformResponse().

This would typically be literally one line of code aside from the call to Super() / method declaration.

So we don't know what else to tell you in this area to help you succeed. Specifically, we don't have even an outline of your DataSource and we cannot test your code. But it looks like you need a line like dsResponse.wallyRealTotalRows = data.estimatedTotalRows... something like that. Then you have a dataArrived handler that looks at DSResponse.wallyRealTotalRows and does whatever you want.

Note also that none of this even seems necessary to complete your use case. The DataSource API is bristling with APIs that give you access to the raw data returned by your server. Use any of them. If you can't succeed from the docs, well, just set a breakpoint on any one of them, and there you go - parameters right there and you can drill down into them with the browser's native tools. Can't miss!

Again, I did want to avoid this kind of discussion. I've also mentioned that I don't use a SmartClient server, so big chance I'm using the RestDataSource, but I don't think that really matters that much. The concept of transforming the response inside the data source (which is not exclusively meant for grids) wasn't the first that came into my mind: I was focusing on implementing this in the grid itself.

Finally, as far the developer-only message about browser limitations: in order to render the first 80 records of a dataset, but render space as if 1M records exist (so that the scrollbar works), then we need to create a space of 20,000,000 pixels (20 x 1M).

Some browsers choke when you try to render this space - they start returning nonsense values from various browser APIs, for example, they might report the space as only 65,536 pixels instead of the full amount.

That's what the message means: some browsers, on some platforms, cannot handle this - and even if they could, it's a bad UE. As you said, moving the thumb 1px might move past 1,000 records. We put a warning in there both to inform developers that some versions and some browsers might fail, and also to say: if you are doing this, you may have a bad UE, please reconsider.

I think there's other options when you're not using the native scrollbar...

[wallytax]

So we don't know what else to tell you in this area to help you succeed. Specifically, we don't have even an outline of your DataSource and we cannot test your code. But it looks like you need a line like dsResponse.wallyRealTotalRows = data.estimatedTotalRows... something like that. Then you have a dataArrived handler that looks at DSResponse.wallyRealTotalRows and does whatever you want.

How? dataArrived gets two arguments (startRow and endRow), no dsResponse.

I've created this transformResponse in my custom DataSource class (which extends from RestDataSource):

transformResponse(response, request, data) {
  const memo = this.Super('transformResponse', [response, request, data]);
  memo.myCustomRowCount = data.response.myCustomRowCount;
  return memo;
}

And the documentation of dataArrived mentions this:

Note that dataArrived(), unlike dataChanged(), only fires in limited circumstances - when data for a ResultSet arrives from the server due to a fetch or cache invalidation, or as a result of filtering. If you want to catch all data changes, you should instead react to dataChanged().

So I've used dataChanged() so far, which also has no access to the dsResponse object.

[wallytax]

I could do this, but that seems to be ugly:

transformResponse(response, request, data) {
  const component = window[request.componentId] || {};
  component.myCustomRowCount = data.response.myCustomRowCount;
  return this.Super('transformResponse', [response, request, data]);
}

And then I can access the this.myCustomRowCount inside the dataChanged() handler.

[Isomorphic]

So again, the type of DataSource you are using is critical information that you should always include if you are asking about server contact.

Note that, as mentioned above, posting the response from your server does not tell us what kind of DataSource you are using.

Anyway, we now know that you are using RestDataSource., and you are finally using transformResponse, so that's good.

You can choose to put the custom property on the DSResponse or on the component or somewhere else- all are fine. Obviously if you place the property only on the DSResponse, you will need to use a callback where the DSResponse is available, such as the callback for fetchData().

Which way to do it depends on how frequently you want to update the row count. Note that your current code will overwrite component.myCustomRowCount on every DSRequest, including non-fetch requests, and including fetch requests in which your server doesn't include myCustomRowCount, so you probably don't want to do that.

[Isomorphic]

claudiobosticco we're not sure what you mean about transformRequest - this is a client-side API, so for SmartClient it appears in the SmartClient Reference, not in JavaDoc:

https://smartclient.com/reifyOnSite/...ansformRequest

If you meant to ask about SmartGWT, it's in the JavaDoc here:

https://smartclient.com/smartgwt/jav...ata.DSRequest-

[wallytax]

So again, the type of DataSource you are using is critical information that you should always include if you are asking about server contact.
You can choose to put the custom property on the DSResponse or on the component or somewhere else- all are fine. Obviously if you place the property only on the DSResponse, you will need to use a callback where the DSResponse is available, such as the callback for fetchData().

Which way to do it depends on how frequently you want to update the row count. Note that your current code will overwrite component.myCustomRowCount on every DSRequest, including non-fetch requests, and including fetch requests in which your server doesn't include myCustomRowCount, so you probably don't want to do that.

Well, this is crucial: I use autoFetchData: true and I don't believe there's a callback available then (at least one with response as argument).
My preferred approach would be to have the response available. This means I need to disable autoFetchData and call fetchData myself and then the question is, where?

[claudiobosticco]

claudiobosticco we're not sure what you mean about transformRequest - this is a client-side API, so for SmartClient it appears in the SmartClient Reference, not in JavaDoc:

https://smartclient.com/reifyOnSite/...ansformRequest

If you meant to ask about SmartGWT, it's in the JavaDoc here:

https://smartclient.com/smartgwt/jav...ata.DSRequest-

sorry, I misunderstood the sentence in the QuickStart Guide "you can override DataSource.transformRequest() and DataSource.transformResponse() and add Java code to handle those cases", and I thought that "and add Java code" was also referred to the transformRequest.

[wallytax]

Having read back this whole thread I come to the conclusion that there has been quite some noise and the bottom line is, that I have failed so far to use data from this response:

{
  "response": {
    "status": 0,
    "data": [....],
    "endRow": 79,
    "startRow": 0,
    "totalRows": 99,
    "myCustomRowCount": 800000
  }
}

Yes, it is true that I can add that "myCustomRowCount" into a response object using transformResponse() in the data source. But I cannot access it inside the grid after that has done an auto fetch. And no, I don't think I should get rid of that auto fetching just because of this, unless that is super simple. The point is that to get this particular thing to work, it is very hard to figure out what a solution is and mostly that solution means changing something that I was using for other good reasons.

I hope somebody has a good solution, which might be to add an extra argument to dataArrived() and/or dataChanged() with the response object, if that is possible.

[Blama]

Hi wallytax,

to solve your no-CallBack-with-DSResponse problem, how about this:

• Add a number field myTotalRows to your DataSource
• Clientside, add notEqual -1 criteria to your myTotalRows-field (does have no effect on the result, but allows you to "mark" requests whenever you want)
• Serverside, when you see criteria for myTotalRows do your rowCount the way you do it now (so you don't do it with every call, as it might be an expensive operation)
• Serverside, add the result of the rowCount to myTotalRows, either in one row or all rows of your return data
• Clientside, use addDataArrivedHandler and check for myTotalRows
• Clientside, do your UI changes based on this number, if it is present

I assume that you could even update the UI for e.g. deleted records in background when you ask you ListGrid's ResultSet.getRange() with the startRow and endRow from the DataArrivedHandler.

In order to "smuggle" your rowCount in data rather in the DSResponse metadata, you need a DataSourceField (step 1), it does not work without.
I'm using this technique (or hack, Isomorphic?) to transfer information to the server that I can't transmit otherwise.

Best regards
Blama