[Isomorphic]

Thanks for the test case. This is under investigation and we'll get back to you as soon as we have anything to share.

Regards
Isomorphic Software

[Isomorphic]

We've made a change which improves performance significantly for the "Add 50 records (block)" case. Please try the next nightly 5.0p build (dated Nov 13 or above)

We'll follow up on the other issue (RE GroupBy) separately

Regards
Isomorphic Software

[Isomorphic]

We've now resolved the second problem in post #30 whereby if you called "updateCaches()" on a grouped listGrid with multiple records, the grid would fail to regroup and display the changes.

[bbaker]

Thanks for all the updates, we've seen a significant improvement in performance in our sample, and the group by issues we reported were resolved.

For consistency, we want to update records in the same manner as the add and remove operations (via updateCaches). We've implemented this approach (sample attached) and
we've discovered a likely defect in the framework.

1. Launch RecordUpdatesViaUpdateCaches.java sample
2. Group by Status
3. Expand any group
4. Select a single record
5. Click 'Update status field' button
6. Notice the Status changes, but regrouping does not occur
7. Select multiple records
8. Click 'Update status field' button
9. Notice the status changes, and the grid is regrouped as expected

Another oddity in Group By mode that we see is the groups will occasionally change their order:

1. Launch RecordUpdatesViaUpdateCaches.java sample
2. Group by Status
3. Note the order of the groups (i.e. Warning, Unknown, OK, Error)
4. Expand any group
5. Select a record
6. Continuously click the 'Update sort field' button
7. Note (eventually) the order of the displayed groups changes (i.e. Warning, OK, Unknown, Error)

Is this normal behavior? We think this could be highly disruptive to our users

[Isomorphic]

On this issue:

1. Launch RecordUpdatesViaUpdateCaches.java sample
2. Group by Status
3. Expand any group
4. Select a single record
5. Click 'Update status field' button
6. Notice the Status changes, but regrouping does not occur
7. Select multiple records
8. Click 'Update status field' button
9. Notice the status changes, and the grid is regrouped as expected

The problem is actually a subtle one in the application code. Your logic to update the record(s) does this:

        button.addClickHandler(new ClickHandler() {
            @Override
            public void onClick(ClickEvent event) {
                Record [] records = grid.getSelectedRecords();
                
                for (Record record : records) {
                    record.setAttribute("statusField",  GridData.getStatusValue());
                }
                
                ds.updateData(records);
            }
        });

getSelectedRecords() is returning references to records actually in the grids data object. By calling 'setAttribute' on these, you're updating them directly, before notifying the DataSource of the changes via the updateData call.
When the grid subsequently gets a notification from the dataSource that its data has changed, it looks at the changed records on the response object, and compares to the values currently stored in the grid's data object.
In this case it doesn't detect a difference and so assumes the value is unchanged [though actually the change has already been stealthily applied by the setAttribute call]. As such it fails to recognize that it needs to update its group tree.

Here's a reworked version which avoids manipulating the records in the data set directly. Changing to this approach fixes the problem

        button.addClickHandler(new ClickHandler() {
            @Override
            public void onClick(ClickEvent event) {
                Record [] selection = grid.getSelectedRecords();
                Record [] records = new Record[selection.length];
                for (int i =0; i < selection.length; i++) {
                    Record record = new Record();
                    // primary key (for identifying the record)
                    record.setAttribute("idField", selection[i].getAttribute("idField"));
                    // Modification:
                    record.setAttribute("statusField", GridData.getStatusValue());
                    records[i] = record;
                }
                
                ds.updateData(records);
            }
        });

On the other issue - we haven't seen the problem you describe with the groups getting arbitrarily reordered. There's a possibility it is related to this same issue. If you continue to see it, let us know and we'll take a look at what might be responsible.

Regards
Isomorphic Software

[bbaker]

Your suggestion fixed the first issue, thanks for your help there.

Following the same steps I described above, I'm still seeing the groups swapping order. Sometimes it doesn't take much,
other times it takes a minute or two of changing the sort field. I've also seen the reorder with selecting multiple records, and leaving
some groups collapsed. It eventually swaps.

I've attached the latest sample code. Let me know if you need more information.

[Isomorphic]

What you're seeing here is actually an effect of sorting. The effect is that each group's first record appears in sorted order. So if you change the sort-field value of some record such that it falls before some other group's first-record-value, the groups get shifted around to accommodate this.
Probably the easiest way to avoid this would be to have the groups be explicitly sorted themselves (and then have data sorted within the groups by another field).
This can be achieved via the ListGrid.sortByGroupFirst attribute.

[bbaker]

We've been working hard to achieve the same performance in our app that we're seeing in the sample, yet unfortunately we aren't quite there.

In our sample, the updateCaches() function is taking around 100-200ms with 12000 records, and in our app with only 5000 records, updateCaches() is
taking around 1,000ms.

We're having trouble identifying what the cultprit is. Any insight from you guys, even a methodical approach to finding a resolution would be very helpful.

Thanks

[bbaker]

We debugged down into the javascript of the updateCaches() routine, and discovered this.applyHilites() was taking a considerable amount of time, in function isc_ListGrid_dataChanged() in ISC_Grids.js. We disabled our hiliting rules and saw a 60% performance improvement in our app, which is big, yet still takes 5 times longer than our sample. Is it possible the hilites are getting applied to all of the records instead of just the visible ones? It seems to slow down in relation to the number of records in the grid.

[Isomorphic]

We've done some analysis of that code flow and eliminated some inefficient logic which may have an impact for you (particularly if you're using formula or summary fields).
Please try the next nightly 10.0 build (Dec 10 or above) and see if there's any difference in either of your test cases.

If not (or if performance is still bad), it would really help if we could get enough information to see the performance hit you're seeing in your app in a test case. I'd recommend you try modifying your test case to have as similar as possible hilite definitions to your live app, and also duplicate any summary / formula rows into your test case, and look for any methods which run once-per-cell such as custom formatters, etc. If you can get us a test case where there's a clear performance problem that's representative of what you're seeing we should be able to debug it from our end.

Oh also - be sure to mention which browser(s) you're seeing what performance in just to make sure it's not a native browser-specific issues. Different browsers do have different performance bottlenecks.

Regards
Isomorphic Software

[EDIT]:
Just to expand on this slightly so you know what we tested. We took the test case you posted, and expanded the data set to contain 20k records, and we added a couple of hilites to the grid which impacted a couple of fields (with criteria).

The results we saw (testing on Firefox) were a under 1000ms to run through the applyHilites() flow - and the changes we made addressed a little under half that time in our test case. This may have a different level of impact on your actual usage of course.

And - yes - the size of the data set would have an impact on the performance. Ultimately some more targetted 'applyHilites' logic may be possible, but it's not trivial to achieve this so would make more sense to ensure we're really seeing what you're seeing before we try to go down this route

Regards
Isomorphic Software

Thank you for the changes. This did noticeably improve the times.

We are seeing something unexpected. In our application we have hilites associated with the grid that are based upon comparing a date field to some offset from "now". Since "now" is always changing, the application of the highlight can change over time with no change occurring in the data. So, we needed a way to "refresh" the highlight evaluation over time. We did this by using a timer that fires every 10 seconds and resets the grid hilites by calling grid.setHilites(grid.getHilites()). The unexpected thing is that this is VERY fast and even with 30,000 records in the grid is barely even noticeable. It appears that this "reset" of the hilites on a grid is taking far less time than the update takes when calling applyHilites(). This seems curious to us because we would expect them to take roughly the same amount of time. If the update handled re-applying the hilites as fast as resetting the hilites on the grid, we would not have even brought this up as an issue. Any thoughts about why the difference and if this is something that could be used to easily improve the update performance?

To answer some of your questions from the previous post:

1. We are not using formula or summary fields.

2. We are attaching our latest sample code (main code is in RecordUpdatesViaUpdateCaches). It can apply and clear both filters and hilites using buttons on the bottom of the screen. We typically select all the visible records and then use the "update" button to get some timings. Then we apply the hilites and do the same thing. We've included buttons for two different highlighting methods. The first uses the built-in grid hilites feature. The second method does it manually using the getCellCssText method override (with large numbers of records, this second method performs about as well as turning highlighting off). We can do this because we are only using the foreground/background features of highlighting and not doing things like replacing text or adding icons. We also have a timer running in both scenarios that refreshes things every 10 seconds. We typically judge performance of this by moving the mouse up and down over the grid (so the hover highlighting follows the mouse). When the system is busy doing updates the mouse hover highlighting temporarily stops (we refer to it as a blip or hanging up). Performance is judged based upon how long the browser goes out-to-lunch doing the update and watching the mouse hover highlighting (or lack thereof).

3. We do have some custom cell formatting, but the timing does not change when we comment this out, so we don't think there is any significant overhead coming from this.

4. We have been trying these things in IE 11, Chrome (latest), and Firefox (latest). The times are very different. Chrome is the fastest, IE is noticeably slower, and Firefox has been all over the map, but is typically slower.

Using Firefox, with our current sample app, trying to add 1 record seems to hang every time. Trying to update a set of 10 selected records is inconsistent. One update will be very fast, then try it again and it's slow, then fast, then it might just seem to hang completely like the "Add 1". We're testing with: SmartGWT Version: v10.0p_2015-12-10/LGPL Development Only.

[Isomorphic]

We are seeing something unexpected. In our application we have hilites associated with the grid that are based upon comparing a date field to some offset from "now". Since "now" is always changing, the application of the highlight can change over time with no change occurring in the data. So, we needed a way to "refresh" the highlight evaluation over time. We did this by using a timer that fires every 10 seconds and resets the grid hilites by calling grid.setHilites(grid.getHilites()). The unexpected thing is that this is VERY fast and even with 30,000 records in the grid is barely even noticeable. It appears that this "reset" of the hilites on a grid is taking far less time than the update takes when calling applyHilites(). This seems curious to us because we would expect them to take roughly the same amount of time. If the update handled re-applying the hilites as fast as resetting the hilites on the grid, we would not have even brought this up as an issue. Any thoughts about why the difference and if this is something that could be used to easily improve the update performance?

It seems possible that things are getting short-circuited with this usage - since the hilites are unchanged (literally the same objects) setHilites() may be avoiding doing necessary work.
One obvious question that comes to mind - is this re-applying actually working? Are you seeing cases where the current time has changed enough that a record which was not hilited becomes hilited or vice versa without the data changing using this mechanism?
Also - what does your criteria object look like? It seems like if you're reapplying the same hilite object, you're going to be ending up with the same criteria rather than updating them to reflect a new (current) date-time, unless we misunderstand how you're doing this.

2. We are attaching our latest sample code (main code is in RecordUpdatesViaUpdateCaches).

One note here: we see some references to "DateUtils" - seemingly an external class of yours? within the SmartGWTFilterBuilder.java class. Not sure if this actually matters - we seem to be able to run the sample regardless

We have taken a preliminary look and will experiment some more, but I think we could still use a few clarifications with respect to this test case:

Using Firefox, with our current sample app, trying to add 1 record seems to hang every time.

Interestingly we saw this too, running in super-dev mode only. At first glance it seems like this may be a GWT SuperDev bug, and we saw some errors logged in the developer console when it occurred.
Are you seeing any errors logged in the SmartGWT developer console, or elsewhere indicating an actual "crash" or similar here?
Are you testing in SuperDev mode or in compiled mode? For performance testing you should always test the compiled app to get a proper picture of what the end-user will see.

A couple of other questions
- would it be accurate to say that your exact steps for testing would be
1) Load the app
2) Hit "Add Hilites"[or skip this step]
3) Select a viewport of data (that's about 32 records)
4) Hit "Update Contents"
5) Observe the UI becoming temporarily unresponsive (for approx how long?)

- on the performance you are seeing in the various browsers - how bad is the lag for you on each browser in your app?
Are we talking a measurable number of seconds, a brief (hard to measure) blip in interactivity or somewhere in between?
How well does the test case you've shared with us here reflect the performance you're actually seeing in the app for you?
It sounds like the lag is more significant when hilites are applied - is it acceptable for you when they are not?

- from your description it sounds like using custom cssText is giving you better performance than using Hilites. Is the difference great enough that you have an acceptable performance in your app in this case? If so, will this approach actually work for you in your real application? While we would obviously like to improve the hiliting performance, if you already have an approach that gives you the behavior that you need with workable performance it probably makes sense to take that approach.

Thanks and Regards

Isomorphic Software

Let's see if I can do a good job of answering all of your questions :)..

1. Yes, the re-apply of the same highlights is working, the cells change color over time. The criteria was created with the highlights editor with relative time references to things like 0 minutes from now. So for example if the due date of a job has passed we turn it red so the expression is something like "due_date <= $now" (I think the rule looks something more like "+0s[-0ms]" if you call getHiliteState() - I know this is not right, but I am not at work and cannot check right now for the exact expression and this should give you the general idea).

2. Sorry about the missing DateUtils class. The SmartGWTFilterBuilder is a very stripped down version of the class used by our product. We were using it for some of our grid test code variations, but apparently not this particular sample - so you can ignore this file.

3. We have been doing a lot of the testing in super-dev mode because its timing was much closer to compiled mode than dev mode. We'll try to make sure we run the tests in compiled mode from now on. Although we have not run the sample in compiled mode, we have been testing our product in compiled mode and it is actually running 5x slower than the sample in super-dev mode. We have not been able to narrowed down why the app times are so different. If the FF issue is only in super-dev mode then we can live with that. Our primary test and customer environments are Chrome and IE (most customers are using IE). I am not certain if there were errors logged in the SmartGWT developer console when running FF and I'm out of the office until 1/4 so I can't easily check right now - if this is a super-dev mode issue then maybe that doesn't matter?

4. Test case looks like this:
1) Load the app
2) Select view port (our displays are large I think it's about 50 records).
3) Hit "Update Status" (alert shows the time). We typically do this a few times to get idea of the range since the timing varies.
4) Hit "Add Hilites".
5) Hit "Update Status" (alert shows the time). Again we do it multiple times.
6) Hit "Custom Hilites".
7) Hit "Update Status" (alert shows the time). Again we do it multiple times.

5. With regard to the "lag". The alerts in our sample only seem to measure part of the time. A noticeable amount of additional time seems to be after this point (I assume it has something to do with updating the DOM and the browser rending it). Whatever it is, we don't seem to have a way of measuring the total time things take other than a stop watch (and when your in the 1 second range, this is hard for a person to get accurate measurements). Our app is between 5 and 8 times slower than the sample for some reason. With about 4500 records in the grid and showing about 8 columns, Chrome takes 700-800ms to update 10 records and IE takes 1100-1400ms (note these times are just the part we can measure and do not include the time beyond this so what the user actually experiences is longer than this). Also note that the technology we are using to gather the metrics (i.e., the times) is different in our application than in the sample so these differences could be caused by the measurements themselves. We will try to normalize our metric collection to eliminate this as a potential factor.

6. The custom cssText solution seems to be faster, we are in the process of converting our app to use it in a way similar to the sample. We do not know yet if what we are seeing in the sample will translate to something meaningful in our application, but preliminary tests indicate we could shave of about 25% of the updateCaches time. We'll let you know. If this does work, then I don't see any reason for you to continue to improve the built-in highlighting performance.

7. With regard to questions around whether this is "acceptable" performance... When we started this, some browsers were taking 2 1/2 minutes to update. We are now down in the fraction of a second to the 3 second range in our app depending upon the browser and how many events our client is trying to field from our server. This is with 5000 records in the grid. We have customers who want to be more in the 12,000 record range. So, the question of if this "good enough" for now is a good one. We are currently testing a lot of scenarios in our app with the latest changes in compiled mode and trying to use a stop watch to measure the overall times and just plain old end-user impressions to answer the "good enough" question. So, at this point the answer is "we don't know yet, but answering this question is in progress". Due to vacation and holidays we have a lot of staff out and will probably not have a more definitive answer to this until the first week in January.

Note, the main reasons for my previous communication were:
1) Give you feedback about the highlights changes you made for us after we had tested.
2) Have a discussion about the curious difference in the highlight management between the reset and the update incase there is an issue there.
3) Let you know where we are at in the process of trying to get "good enough".
4) Give you some feedback before the holidays so weeks don't go by without responding to you.
5) Let you know we were having a strange failure for the first time in FF incase this was something significant.

This was not intended to ask for anything new from you beside info/communication unless our questions and additional info generated some thoughts on your end about something you would want to look at more closely. There are no specific issues or potential defects here that we are asking you to address. We are just trying to gain a better understanding of what we are seeing in our app to make certain we are testing and considering the right things to get the best performance we can and determine if that is good enough.

If you do have a good way to measure the exact time it take from when we begin an update to when the user sees the changes in the browser and can interact with it again, we'd love to know how to do this to help with this process.

Thanks for all of your help with this! The improvements so far have been dramatic and we and our customers are very excited about the improvement we are seeing.

[Isomorphic]

Ok - thanks very much for the clarifications - very helpful and gives us a pretty clear picture of where you are.

It's good to hear that the relative date criteria hilites work - your description makes perfect sense.
We'll do a sanity check and see if the quickness of reapplying the same hilites does suggest any way to improve the initial time to apply, but it seems likely that our guess about being able to skip most of the work as the hilites have already been associated with the data is correct in which case this won't help anything.

Aside from that, let us know when you have more information on whether this does indeed need further optimization (probably next year) and we'll revisit this area as necessary

Regards
Isomorphic Software

[Isomorphic]

A couple of additional points to throw out that might help you diagnose the odd things you're seeing:

1. one technique we have previously used to measure the time before the browser actually responds is to set a zero-ms delay timer (natively, JavaScript setTimeout()) and considering the test done only when this timer fires. At least on some combinations of browser and platform, you can see that this timer only fires at the moment that native browser rendering is complete and interactivity is restored

2. about the extraordinarily long times you are seeing for DOM updates, consider possible causes such as:
a. a blizzard of complex CSS selectors applied to the HTML in your actual app
b. any kind of global "shim" technologies you might be using to replicate CSS behaviors the browser lacks natively. An example would be use of "DHTML behaviors" to "upgrade" IE to support certain CSS features correctly. That particular example no longer applies for IE10+, but other shim approaches might have weird effects on performance
c. really really deep nesting, which can cause the browser to take into account whether a bunch of other elements on the page might need to change, even if you can see this is impossible. You might try popping the grid out into a modal dialog just to see if this improves performance

3. just as a sanity check: your weird timings, including both getting large times for areas where we see negligible times, and getting erratic behaviors from Firefox, suggest problems like:
a. leaving developer tools open during tests: both the native browser Developer Tools and SmartGWT's Developer Console
b. diagnostics such as verbose log categories being left enabled, or expensive logging being performed by your code. Both of these would have an impact whether or not the Developer Console is actually open
c. leaving "Track RPCs" enabled (in the RPC tab of the Developer Console). This again would have an impact whether the Developer Console is open or not
d. testing in a memory-constrained VM. This might make Firefox in particular swap to disk like crazy, causing erratic timings