DataSource update duration for large number of records

[slick07]

Hello,

We are encountering significant time delays for bulk update operations on a DataSource for a large number of records. For a simple test case with a simple ListGrid with 1000 records the update of all records takes 5 seconds. In our production application we might even have 10k records which takes about 1 minute which is unacceptable.

Why does it take that much to update the cache? Even without any component connected to the datasource the update for 1000 records still takes 1-2 seconds... And how can this be improved?

Using SmartClient Version: SNAPSHOT_v13.0d_2021-08-21/LGPL Development Only (built 2021-08-21)

Thank you!

[Blama]

slick07,

I think the answer to this question depends a lot on if you are only talking about clientside DS, like in your sample, or also serverside updates.

Best regards
Blama

[slick07]

Blama The datasource in our application is not client side. Updates are coming from the server and based on these updates we do some additional computations that may affect thousands of data elements and then update the datasource for the corresponding records. The provided test case is just a simple way to isolate the problem.

Initially the computations were made server side which in turn generated thousands of updates over the network which was obviously a no go. Hence we adjust the code to make the computations on the client side for increased speed and lower bandwidth loading. But we are still hitting this datasource bulk update duration...

First we would like to know why it takes that long since 1k records is not such a big number... And if this can not be improved then what other solutions are there?

[Blama]

Hi slick07,

there is always invalidateCache with a response that will run a new fetch from the clientside after it arrives on the clientside.
Then you don't have to send many relatedUpdates, if that's what you are doing now.

Apart from that: Did you try running your many updates on the DB itself (in one transaction or one transaction per statement, if that's what you are doing on the client? What you get there is the bare minimum of what will be possible for any program using that DB.
Depending on the data amount in the table, indexes to update and row access I think that 10k records in 1 minute is not necessarily bad.

I think that a lot more detail here will help.

• Table size
• Example Update statement(s)
• Who sends these statement(s) when?
• What is expected on the client when?

Best regards
Blama

[slick07]

Dear Blama,

Maybe I am wrong but invalidateCache will determine the lost of selection, the position of the scroll etc. And we do not want that. We are not sending relatedUpdates.

What we have is a hierarchical structure of projects and tasks. Each task has an ID. These are in consecutive order and the end user uses this ID to visually identify a task. When a task is inserted at a given position, all the next IDs are recomputed to stay in consecutive order. This recomputation process is done instantly (a few miliseconds) even for 10k tasks. But updating the corresponding records in the datasource with several components attached takes around 1 minute! Without any component attached it takes 1 second. A lot more faster but still 1 second instead of a few miliseconds...

- The TreeGrid size differs based on the filter that the user applies and even if it has 100 records it takes almost the same time. Important is the number of records to be updated.

- This is the update code if this is what you wanted:

List<TaskRecord> dirtyTaskList = computeTasksId(); // recomputes the ID of tasks

DSResponse dsResponse = new DSResponse();
dsResponse.setOperationType(DSOperationType.UPDATE);
dsResponse.setData(toTreeNodes(dirtyTaskList).toArray(new TreeNode[0])); //convert the list of dirty tasks into TreeNodes
updateCaches(dsResponse);

- The above code is called for example when a task is inserted at a given position and the structure is changed
- The expected result should be to see the new IDs in the ID column of the TreeGrid

Thanks!

[Isomorphic]

You are using an API designed for incremental cache updates to update every single record in the cache, and the reason its slow is that, because it's intended to refresh incrementally, components such as the TreeGrid or ListGrid will immediately refresh the display of just one row.

So you have a few options:

1. flush the complete cache, and preserve selection and/or scroll position yourself

2. sponsor some kind of feature to allow you to continue using updateCaches(), but in some kind of batch mode

3. switch your data model so that adding a new record to the dataset does not require every other record in the dataset to be updated

[slick07]

2. sponsor some kind of feature to allow you to continue using updateCaches(), but in some kind of batch mode

Please send us a quote for this. If it is not too pricey we might go ahead with it.