Client/Server data communications...

[phaven]

I'm looking for some help/guidance with how to best implement my data communications model.

I've spent quite a bit of time with the documentation (which is great, btw) and experimented with different approaches, but continue to get stuck on various performance issues - which leads me to believe I'm just not doing it right, or am missing a couple key points.

To load data, I'm using datasources with dataURLs that point to .net handlers which return JSON formatted data. This is working great for loading data into the listgrids.

The problem I'm running into, is that once I load the data into the grid, I don't need to keep it in sync with the server. The data I'm loading is just meant to be a template for the user... from which they can add, remove update, etc. I only need to send the data in the grid back to the server once the user indicates they are finished. And when I do that, I won't be updating the same source that originally sent the template data. So from the server's standpoint, the update is actually new data, not an updated dataset.

At the largest, my listgrids will contain around 10,000 rows. I am having trouble figuring out the best way to post the data to the server. My server side handlers are currently expecting XML data in the request body. I've tried using xmlSerialize on the listgrid.data object, but on the larger datasets, I get a slow script warning.

Any advice on methods that I'm overlooking or other approaches would be great.

Thanks,

Philip

[abhi_shrut]

We too are facing a similar issue, we have a list grid embedded in canvas item. This helps us to include a list grid in a form and data can be submitted with other form fields.

Now, when we populate say a thousand record in the list grid and submit the form. We observed a javascript warning message saying 'Javascript taking too long to execute.' When we analyized the issue further our observation was the same as above it was XMLSerialize method which was taking too long. Now we are stuck.. :(

Isomorphic, do you have any suggestions, how do we improve the performance of script in this scenario?

Thanks

[Isomorphic]

First question would be to reconsider whether what you want to send 10,000 (or however many) XML serialized records. Obviously, the user isn't typing in that much data. If the user is selecting among many records and you're trying to save the selection, it would make more sense to send just the ids of the selected records, possibly as a single comma-separate String. Similarly, if you're trying to save edits, send just the changes.

However if you really need to XML serialize that many records, as with any long-running code, you can split up the execution using Timer.setTimeout(). For example, serialize in batches of 100, and combine all the Strings at the end.

[abhi_shrut]

Thanks Isomorphic,

I'll try this approach and update the thread, FYI we are using an excel sheet to upload data in the grid. So we have 10000 new items with their respective attributes. So this way user is adding data and which needs to be shown on the form.

I have analyzed the javascript method call trace and found XMLSerialize method is called by datasource's saveData method when the form is submitted. I don't know to intervene and override the behavior to split serialization in batch. It will be great help if you can provide us some pointer how can this be achieved.

BTW, this may sound odd but if this feature can be included inside the XMLSerialize, i.e. script auto timeout while processing large data no one will face this kinda issue and can be a great help to community.

[Isomorphic]

It's a useful general enhancement, so consider sponsorship if you prefer not to have to split up the work into timers yourself.

However for your particular use case, shuttling 10k records back and forth between the server and client might be less efficient than storing the data temporarily on the server, and showing the user just records that have errors to correct.

[abhi_shrut]

Isomorphic,

I have been trying hard to get rid of this problem, but I am still blocked. I tried out splitting data in chunks by timing out. I did this in RestDataSource implementation below is the code snippet:

function getItemArray(array, boundary){
                var result = new Array();
                var i =0;
                while(boundary > 0){
                                result[i++] = array.shift();
                                boundary--;
                }
                return result;
}

function chunk(array, process, context){
        Timer.setTimeout(function(){
        var itemArray = getItemArray(array, 50);
        serializeInParts(context, itemArray);

        if (array.length > 0){
            Timer.setTimeout(arguments.callee, 100);
        }
    }, 100);
}

function serializeInParts(context, itemArray){
                var ds = isc.DataSource.create({
                fields:[
                    {name:"data", multiple:true, type:context.getID()}
                ]
            });
                ds.xmlSerialize(itemArray, null, null, "data");
                
}

In RestDataSource transformRequest method we are serializing the data now in parts instead of calling XMLSerialize method.

But we observed that the script doesn't get blocked while serialization is done. Instead the request is submitted before the completion of serialization.

Please let us know how can this be achieved.

Thanks.

[Isomorphic]

Yes, transformRequest is a synchronous API. You should do the serialization before calling DataSource APIs instead.