issues with grid editing performance with editByCell

[senordhuff]

Hi there, we are trying to enable editing on a grid that has approximately 800 fields defined on the Datasource. This is a grid with lots and lots of read-only data. The visible, editable fields on the grid typically total only about 10-15 and we have editByCell turned to true. However, we are seeing major performance problems when clicking from one editable cell to another. I have a few questions to help me sort out how to resolve these issues

1. It looks like the hideInlineEditor() call when you click from one cell to another calls recordHasChanges() which is very expensive with so many datasource fields. If we have editByCell turned on and thus you can only edit one cell at a time, does the framework need to call recordHasChanges() in that case?

2. With validation turned off, we see storeUpdatedEditorValue() being called which in turn calls getEditDisplayValues() which returns all 800+ properties on the record. Does the framework need to return all 800 properties when editByCell is enabled and only one cell has been edited? We are considering a patch to only evaluate fields that are visible and/or fields that are editable to avoid processing so many loops that are doing nothing.

3. When validation is turned on, even though we have editByCell turned on, it looks like the changeEditCell() method is firing and calling validateRow() which also makes expensive calls to validateRecord() which calls storeUpdatedEditorValue() . Shouldn't it call validateCell() if editByCell is turned on?


Sorry for the dense questions but this is extremely time consuming on our part to try to sort through framework code to find performance bottlenecks. Let me know what else I can do to help sort through how to speed this up for our use case.

[senordhuff]

Hi, for the time being, we are setting allowEditFormValueManipulation=false for our grid and that seems to bypass most of the performance issues that are originating from storeUpdatedEditorValue(). I see that is an undocumented flag but it resolves our issue for now. Is that a safe modification for us?

[Isomorphic]

We're considering optimizations to the routines you mentioned in your original post but it won't be an instant fix as in the process we'll need to verify that existing use cases still work and make any needed adjustments. We'll update this thread when something's been committed.

You've changed directions a bit in your second post, so we'll have to evaluate that separately.

[senordhuff]

Hi, thank you for the update. Just to clarify, my second post was meant to highlight that most of the issues I pointed out in my first post seem related to performance bottlenecks in storeUpdatedEditorValue() and setting that allowEditFormValueManipulation=false is an effective workaround for the time being. Can you comment on whether setting that flag (which appears to be undocumented) will cause any issues for us in the meantime?

[Isomorphic]

We're still deciding whether to expose that flag, but the impact of turning it off is that changes made to the edit values of a record other than through the UI and ListGrid APIs such as setEditValue() would not be saved. So, for example, if you had a change handler on one edit item that modified the value of another through the form APIs, that wouldn't be saved with the flag false.

[senordhuff]

Hi there, just curious if you have any ETA on when these proposed optimizations may make into the framework? I also wanted to clarify that we also see the same performance issues with editByCell:false so hopefully any enhancements you make will improve performance then. Typically only 10-15 fields will be opened for editing so hopefully there is some way to avoid looping over all 800 properties on the datasource excessively.

[Isomorphic]

Earlier you indicated that configuring the undocumented property you found was an "effective workaround." Is that not proving to be true?

[senordhuff]

That workaround helped some but it is still really slow so I was hoping that you were still investigating potential optimizations you mentioned. If I need to provide more details, I can. Please let me know. I also see significant slowness when executing cancelEditing() and discardAllEdits() to shut down editing on this grid as well.

[Isomorphic]

We can try to replicate your situation by building a sample from the first paragraph of your first post, but you may want to provide us with some standalone sample code to make sure we see the correct performance impact of various internal improvements.

[Isomorphic]

Some performance improvements here are in progress. Bear in mind that changes are always made in a way that preserves the correctness of existing customer usage. So even if your restricted use case would allow us to make a quick modification, a more involved change may be needed that handles all known cases without breaking.

We'll update this thread when something's been committed. You may also want to provide some sample code if you'd like it to be verified against the commit(s).

[senordhuff]

Hi, thank you for the update. I saw your request for a sample and I've been deep diving into the framework code myself before I produce the sample to better pinpoint the slowness we are seeing. Just to summarize, our two slow use cases are a button where the user can cancel all edits on the grid. And, where the user clicks from one cell to another and across rows to open editors on a grid with 800+ fields available but only perhaps 30 visible and 10 editable it is very slow with each cell click taking 1 to 2 seconds.

Here are some initial findings/customizations from my deep dive...

1. discardAllEdits() is slow because of an excessive number of calls to recalculateRecordSummaries. I've applied overrides that turn off grid and group summaries before calling cancelEditing() and discardAllEdits() and then will turn them back on after those methods have completed. I noticed that summaries were being recalculated in cases where they didn't need to be. The values didn't change and they weren't being refreshed by the edits anyway. Or, the column had no summary function defined.

2. Turning canChangeNonFieldValues to false seems to help a bit.

3. As previously mentioned, allowEditFormValueManipulation set to false helps

4. in isc_ListGrid__showEditForm, there is a call to redraw() the entire grid. I am experimenting with just calling refreshCell or refreshCellValue instead here and seems to work ok so far. Experimenting with some of the other params on refreshCell still.

5. In makeEditForm, there is a loop over all drawn fields, I am experimenting with having that only process fields that are editable instead of all fields. Also, the call to getRecord() and getEditedRecord() should only happen once and not during each loop iteration.

6. I noticed that recordHasChanges() loops over all Datasource fields which is very expensive for us. Can it be updated to allow for only looping over editable fields for this use case?

7. I am still experimenting but it seems that other slowness revolves around a few specific features such as grid and group summaries, frozen fields, and record components which are all common features being used on our grid. I am wondering if there are ways to eliminate or reduce slowness related to those features and I'm still experimenting.

So, I am going to keep pulling on some of these threads to find some suitable workarounds while you guys are working on framework optimizations. But, I do hope to get you a sample soon. Thank you for the help!

[senordhuff]

Hi there, I'm attaching a sample. Sorry it is pretty messy and longer than you would probably like but it illustrates the general way our grid works. The slowness you see with the sample is noticeable as you add more datasource and grid fields. But, it still doesn't equate to the level of slowness we are seeing in our environment. So, there is something still missing. But, I hope this is helpful. See attached you can drop into your SDK. In my debugging, I noticed a few other areas of potential slowness I wanted to point out when I was debugging what is happening during the call to redraw() made from _showEditForm()

1. _editItemsDrawingNotification is looping over all drawn items. It would be nice if that only processed editable items

2. DynamicForm.setItemValues() is called twice in response to one double-click in an editable cell. One call comes from updateEditRow() and not sure where the other one comes from yet. setItemValues() is processing every visible field and not just those that are editable. Also the values passed in encompass all 800+ values associated with a record. It might help if it only included the values that can be edited for the form.

[Isomorphic]

We're comparing a few different possible improvements now for situations with a large number of DSF. Thanks for providing the sample. We'll make sure to check how it's impacted.

[senordhuff]

Thank you for the update. I ended up creating a series of patches to address many of our performance issues in the methods I have highlighted in this thread. I am happy to share that with you if you want to see it. I'm sure that some of my patches would not be acceptable for all use cases. One of the biggest improvements I saw was eliminating some instances where the native logic iterates over all properties on a record because we do add a variety of objects and properties to our records that would never be edited by a user. That is a patch we made to getEditDisplayValues() for example. I don't think my sample adequately captures the variety of properties and objects we hang onto the record and that is why the performance impact wasn't quite as dramatic in my sample. In our application, it was slowing down things dramatically to iterate over all the properties and objects. Performance is still slow with my patches but it is in the range of acceptable. Hopefully, your planned improvements will help further.

Also, just to clarify we have 800+ columns on the grid AND on the datasource. So, a typical use case is to expose a view with 10 editable visible fields, 30 total visible fields, 800+ total fields including those that are hidden from the current view, and 800+ datasource fields. This occurs because we dynamically add many fields at run-time based on some child objects. The user will edit these child objects in a "detail" view and then we display a series of "flattened" columns representing those child objects in this main grid.

Thanks again for your help with these issues and let me know if you need any more feedback.

[Isomorphic]

While looking at your sample in the context of our Framework optimizations, we noticed that it's not configured with editByCell:true. Obviously, that means the logic path during field navigation is different, rendering some optimizations moot. Did you settle on edit-by-row instead?

With your current sample, we noticed that during field navigation, the editor may exit when crossting the "Status_2_2" field. Is this something intentional, or should we look into it?

When running comparisons, we're not able to only look at canEdit:true fields (which have formItems) in our optimizations as we want to capture programmatic changes as well to any field. So our changes may not duplicate what you've done, but hopefully they'll minimize any required patching.