Why warning over lots of rows

[wallytax]

I have a paginated ListGrid that warns "Due to native rendering limitations, grids with this many rows may not appear correctly on all browsers. Consider filtering the data displayed to the user to reduce the total number of rows displayed at a time. This will improve usability as well as avoiding unpredictable behavior."

In this particular case there are about 800k records, but less than 100 are retrieved at a time. I'm wondering why this warning occurs and if there's a way of getting rid of it. I don't want the data to be filtered, we use pagination to avoid large data transferrals.

[Isomorphic]

As the message says, the problem is due to native rendering limitations, that is, some browsers have problems when we try to create very large scrollable areas like this.

As the message says, a fully drag-scrollable area like this generally considered a poor UE, so you should consider either requiring the user to filter further, or switching to a progressiveLoading interaction, for both UE and database performance reasons.

[wallytax]

Using a non-SmartClient-server, I've implemented progressive loading by returning a "response" object containing (amongst others) a "totalRows" property with a value that is 20 higher than the "endRow" property (the value 20 is chosen because it equals the "endGap" property in the DataSource class). I only do this if the number of records (that I retrieve via a SQL COUNT) is higher than 200.000 (that value equals the "progressiveLoadingThreshold" property as in the DataSource class).

In a grid I would like to show the total number of records, which is no longer equal to the "totalRows" property. I return that value in a custom property. Is there a way to use that value whenever the data arrives?

[wallytax]

So, yes I would like to count the number of records so that the end user knows how many records there are, but setting that value in "totalRows" results in the warning "Due to native rendering limitations...". Therefore I use a different, custom, property but I don't know how to access it from the grid.

[wallytax]

The way the grid displays the total number of records, is similar to the Custom Toolbar example of grids,

[wallytax]

We have some internal discussions about whether it is reasonable to require an user to always filter the data to get less records. Does that suggest the user starts with an empty grid and then should filter? Do you have any advice on that?

Apart from that, how would an user be able to know how many records there are in total for a certain set of filters (or none)? Thus, without the need to see all those records, it would be nice to be able to know the total number of records in the database.

[Isomorphic]

See the Grid Range and row count display feature:

https://smartclient.com/smartclient-...owRangeDisplay

There's a built-in way to display row ranges, and a feature to delay fetching the rowCount but allow the user to do it.

As far as the design question, when there are so many rows that fetching the full count and/or fetching rows near the end of the dataset is expensive, you can either:

1. require the user to enter certain criteria before fetching data, to reduce data set size

.. or ..

2. turn on progressiveLoading

It would be rare to do both.

[wallytax]

When I click on that link, it shows information from version 14.0 development. Looking in the version 13.0 reference, nothing is described, but it seems like most methods and properties exist in the ListGrid. Trying them however, seems not to be working.

I have filled the “estimatedTotalRows” property when sending back the response but it doesn’t seem to make a difference.

For now, I will use the getRowRangeDisplay() method to show the number of records. It is not documented in version 13, but it works.

[claudiobosticco]

Hello wallytax, here is the 13.1 link https://www-demos.smartclient.com/sm...owRangeDisplay

[wallytax]

Thanks claudiobosticco ! I’m using the latest stable version, which is 13.0 and that documentation is lacking the information. It looks like the code exists. I’ve tried to get things working with the 14.0 development but no luck either.

I can’t get this to work at all… I’m not sure what to set in the request and/or response. I’ve been debugging for quite some hours now and find that the response object that is used internally does not have the progressiveLoading and/or estimatedTotalRows property set, while the actual response (looking in my Developer Tools) has those properties:

{
    "response": {
        "status": 0,
        "data": [....],
        "endRow": 79,
        "estimatedTotalRows": 809771,
        "progressiveLoading": true,
        "startRow": 0,
        "totalRows": 99
    }
}

Do I need to set something on the request?

[wallytax]

Without the whole technical stuff, I’m just wondering:

Suppose I show a list of orders (and the database contains 100.000’s of them), why is it so strange to be able to inform the user that there are like 809.771 records (in my case)? And by using pagination, I’m only retrieving 80 of them. There’s no rendering of those 100.000’s of records. And yes, if the user wants to filter, fine, but I’m confused that I get a warning in case (s)he doesn’t.

And yes, I understand that the scrollbar then contains a very small handle and if you move that 1 or 2 pixels, immediately the next 1000 or 10.000 rows will be returned, but it also gives the user a chance to move to the end of the results. But for me, the most important thing is that I want to show the user the total number of records, unfiltered.

[wallytax]

For me, as developer, it’s still unclear what your philosophy about things is. I’m reading the QuickStart guide for the “100th” time right now, just to see if I miss things somewhere. In this particular case, that started with a browser warning, that end users don’t see unless they open the development console, it is not so bad, but trying to get rid of those warnings makes me need to do things that I don’t think I should be doing. I simply want to return the records that the grid requested and return the total number of records that apply to the given filter. If that number is too high, a warning appears indicating the user should filter or I should use progressive loading. But that gives all kinds of side effects that I don’t want.

[Isomorphic]

Wallytax, it looks like you saw the 14.0 docs and then tried to use them with 13.0. That's a recipe for wasting a ton of time - so no surprises there!

You should either upgrade to 14.0, or don't try to use features from 14.0. Specifically, you should not continue to try to use 14.0 features in 13.0 - there are no special properties to set in 13.0 to make 14.0 features work - it simply cannot be done. You need to use version 14.0 to use features documented as part of version 14.0.


On the other points:

1) we do not have a company-specific "philosophy" regarding the best UE for dealing with huge numbers of records

We are just telling you off-the-shelf standard advice, well known amongst UE experts.


2) the warning that we log for developers in the SmartClient Developer Console is not intended for end users, and no normal end user would ever be aware of it.

As far as the possibility of end users seeing this - which would require them to become aware that the application was implemented in SmartClient, and then, for example, to go to the URL bar and open the SmartClient Developer Console by typing in "javascript:isc.showConsole()" in order to see a warning... we wouldn't regard this as "not so bad", but as completely a non-issue.


3) it is not "strange" to inform the user of the row count.

Indeed, the 14.0 features regarding row range display are explicitly designed to tell the user the row count, and to allow the user to explicitly request the row count in situations where it might take a while (such as in some pharma apps we have recently implemented - over 20 million records may match).

So, when you say "why is it so strange to be able to inform the user that there are like 809.771 records", we're not sure whose ideas you are addressing, but certainly they are not ours.

Showing the row count is something we routinely do, and support directly in the framework, in 14.0.

--

We would recommend again that you either require the user to enter certain criteria, or use the progressiveLoading mode.

This is the best approach both for universally recognized UE principles, and also because of limitations in certain browsers.

If you decide on the progressiveLoading mode, there is the rowRangeDisplay feature in 14.0 - but this is not present in 13.0.

You should either:

1) upgrade to 14.0

.. or ..

2) stick with 13.0 and come up with your own way of displaying the total rows and/or currently visible row range

As far as #2, from your previous posts, it sounded like you have already found the correct approach for sending additional properties from the server, that you can then access from the client. So there you are: you just take the real totalRows value and put it in an isc.Label - easy!

Just definitely do not continue to try to use 14.0 features on 13.0. That is certain to be a waste of time.

[wallytax]

I will respond as short as possible in different posts. First, about SmartClient 14.

I was not using anything above stable version 13 at all, none of my recent posts will mention that. However, 6 posts above, you show a link and when clicking that, it shows the documentation of version 14 Enterprise. I didn't notice that at first and was surprised to find that whole row range API. I've dismissed that reference and later opened my own bookmark (to version 13) and found out the documentation was gone. Then I reopened the link from your post and found the documentation again and it was then that I discovered it was version 14 documentation.

Curious and desperate, I've checked for 15 minutes if it would all work with this version 14, but it didn't either.

BTW, claudiobosticco mentions two posts later that the documentation is also in version 13.1, which I don't use either.

I did however searched the code in version 13 for the properties and found them. Gave this one more try, but that approach led to nothing. So, I'm back at my original code.

[wallytax]

Then about the pagination stuff.

For more than 15 years, I'm using SmartClient and one of the nicer things is this infinite scrolling grid as opposed to page numbers. Never had I seen the Developer Tools warning (the one of the browser, not SmartClient's Developer Console). So normal end users that know a bit more about a browser will be able to find it. I'm still wondering why I have never seen it, but most likely because most of my tables only contain 100.000 records at most. But there's one containing almost a million.

So driven to fix all kind of warnings, I've looked into that issue and I wish I hadn't, because it's so time consuming and we have so much communication difficulties. My reasoning is that if a grid row is like 20 pixels high and SmartClient requests 80 records, than only 1.600 vertical pixels are needed to render that data. The scrollbar could be a component rendered by SmartClient. It doesn't need to be higher than the grid's height. Everytime new data arrives if overwrites the rows one-by-one and if the last data set is less than 80 records, you empty the existing rows. In my opinion no need for a warning, but I guess (I could not find that out so easily) that the grid renders the total number of rows times the row height, so like a DIV of 800.000 x 20 = 16.000.000 pixels and that might "upset" the browser. Again, I don't know the inner workings of SmartClient and/or browsers that well.

The 3 points in your last post make no sense to me (in the context of my earlier posts). But please let's not end in a huge discussion. I will tell once more what I would like to achieve in my next post.