OK, I am still trying to get my head around your original definitions:

There are 3 relevant modes to consider:

1. ListGrid with DataSource and using a ResultSet (as a result of a call to fetchData())

2. ListGrid with DataSource and using a RecordList passed to setData()

3. ListGrid with no DataSource, using a RecordList.

Starting with #3 which is similar to what we were originally doing and having trouble with, I want to clarify that we were calling set data with a parameter of ListGridRecord..., not RecordList. I have know idea if this is an important distinction.

With regard to #2, this makes perfect sense to me (I think).

#1 is where all of my confusion is. First of all, it is not clear whether you are talking about fetchData() on the DataSource or the ListGrid. Also it is unclear how fetchData is getting called. We don't have any calls to it at all in our code at the moment. Should we be adding one, and if so where/when should that call be made. Next, I would want the fetch data to create the ResultSet from an in-memory List. Where would the code exist to build the ResultSet object from the in-memory data and how what that end up becoming the result from calling fetchData? Also there are several fetchData overloads, are you referring to fetchData generically or the specific overload that takes no parameters?

[Isomorphic]

Again, dataProtocol "clientCustom" gives you a completely arbitrary, programmatic entry point to do whatever you want.

So to spell it out: you could go to the server, or not go to the server, and instead work with data you've already loaded, as you please. You can do different things on a per-operation basis. If you have a transformation between some data structure you've already loaded and the Records you need to populate grids and forms (which you've indicated you do), you can execute it here, and place the resulting Records in the DSResponse you create.

I think I am finally starting to grasp what you are trying to tell me, but I have been doing web searches and searches in the showcase, but I can't seem to find any examples of something using a custom protocol. So I'm kind of flying blind here. I've attached some modified sample code that I think is headed in the direction you are recommending, but I still seem to be missing a piece.

In the sample I've created a derivation of DataSource and overrode the trasnformRequest method. However, I'm having trouble handling an UPDATE request because I can't seem to figure out how to get at the instance of the record with the updates in it from within the DSRequest object. How do I get at that data?

[Isomorphic]

The problem with providing samples of dataProtocol "clientCustom" is that it exists largely for legacy integration and/or purposes for which the framework already has support (such as Offline storage). Experience has shown that if you provide a sample showing how to solve a legacy problem or that shows a simplified version of a framework feature, someone will follow the approach shown in the sample, no matter how carefully you label it as code that should not be copied..

You seem to be far enough along that you probably can't make use of sample code at this point, but note there is a short bit of sample code linked right from the clientCustom docs.

You can use either getCriteria() or getData() to access the dsRequest.data. Note getData() is from the superclass RPCRequest. It returns a JavaScriptObject, which is readily converted to Record via new ListGridRecord(jso).

That makes sense. The part I was missing was the knowledge that the JavaScriptObject represented something that could be passed to a Record. Just to confirm, if the objects I put into the data source were of type Record, should I still use new ListGridRecord on the JS object, or should I just new Record?

[Isomorphic]

Yes, use ListGridRecord, just as you would with setCacheData().

I'm getting close, but still not quite there. If you need the code to answer the following question, just let me know and I'll upload the latest.

The only thing that isn't working they way I want is that I am not getting filter criteria applied on the initial load. I assume the grid is passing sort and filter criteria to the transformRequest method. However, my implementation is ignoring this. With the sort, the grid is still presenting the records in sorted order correctly (even though my data source does not return the records in sorted order). Is there some combination of settings I can put on the DataSource/ListGrid that will allow the transformRequest fetch to return all the records and let the grid automatically filter out the ones that don't match the criteria on the grid instead of assuming that the fetch will do the filtering?

[Isomorphic]

Yes, set dataFetchMode:"local" on the grid.

Thank you for all of your help here! I appreciate all of your patience with me while I tried to get my head around how to effectively/correctly use the grid/datasource in the different ways. I don't know why I was hitting such a mental block here because it really is very simple. I think I must just have had a different model in mind based upon other GUI libraries I've used in the past.

Please keep me posted on the groupBy issues in mode #2 since we still have the use case that started all of this discussion of trying to programmatically rearrange the records. I assume the recommendation you made early on in this thread to use mode #2 for this case still applies.

I've updated our sample code, and begun converting some of our application grids based upon this discussion to use mode #1 with a custom protocol. Although I've only finished the conversion on a couple of our grids and I have had limited time to test it, everything is looking very good.

I do have a couple of question I could use clarification on.

1. When using this technique, if I call addData/updateData/removeData via my ListGrid object, it appears that the operations are synchronous. Is that correct? So for example if I call addData, then I would be able to select that record in the grid immediately upon return from addData.

2. If appears that calling addData via the ListGrid object does not do anything initially. It appears to be based upon some other bit of initialization. I suspect that it cannot be used until something like the first fetch is done, or maybe until after the first dataArrived event is received by the ListGrid. Is this correct? If so, can you clarify what has to occur before calling addData? I assume any behavior here is the same for updateData/removeData as well. Correct?

[Isomorphic]

Glad to hear of the good progress.

We'll keep you posted on fixes to mode #2, but no, as far as we can tell, mode #2 is not appropriate for your print queue reorder use case, because the user is actually modifying server-side state reflecting the persistent order of the queue. So again we'd go with our original recommendation:

When working with a DataSource, the only meaningful way to change order of records is to have a field that tracks the order, sort by that field, and modify that field when the user reorders records.

1. When using this technique, if I call addData/updateData/removeData via my ListGrid object, it appears that the operations are synchronous. Is that correct? So for example if I call addData, then I would be able to select that record in the grid immediately upon return from addData.

No, it's asynchronous unless saveLocally is set (which would no longer really be mode #1).

2. If appears that calling addData via the ListGrid object does not do anything initially. It appears to be based upon some other bit of initialization. I suspect that it cannot be used until something like the first fetch is done, or maybe until after the first dataArrived event is received by the ListGrid. Is this correct? If so, can you clarify what has to occur before calling addData? I assume any behavior here is the same for updateData/removeData as well. Correct?

Having a ResultSet is how you participate in cache synch. If you haven't either called fetchData() and gotten a reply, or provided a ResultSet directly to the ListGrid via setData(), added records will not automatically appear in the grid (and this is true whether you call DataSource.addData() or ListGrid.addData()).

In my reorder case, there is no field to sort by. The order is the physical order the records come from the server in. Also, they are not written back to the server until the user presses the OK button on the dialog so I need a way to reorder locally.

Seems a little strange that mode #1 operations would be asynchronous even if my custom protocol doesn't do any async operations in response to an add/update/remove. From looking at the call stack in the debugger it seems to all be happening on the same thread. Are you certain that this will be asynchronous if the protocol just updates an in-memory java object?

I don't quite understand your last paragraph. Will you please look at that attached implementation of a custom data source to see if I am doing everything you expect correctly. In my code I never call fetchData (I have the grid set to autoFetch). I never call setData on the grid, I build a RecordList externally and call my own version of setData on my custom data source class to save this data locally in the custom data source object for later use. I never use a ResultSet and don't do any cache management. Maybe all of this is incorrect, but with no examples to go off of, it's difficult to know what your expectations are on how these operations would be implemented.

[Isomorphic]

In my reorder case, there is no field to sort by. The order is the physical order the records come from the server in.

This still means that a persistent sort order is being saved, and so still fits the recommended approach if you just synthesize a sort field from the record order, however..

Also, they are not written back to the server until the user presses the OK button on the dialog so I need a way to reorder locally.

This - not saving order changes until the user arrives at a final order, when the dataset is known to be relatively small - is the kind of thing that mode #2 is validly used for. However, if there is any possibility of two users rearranging the print queue at the same time, you might want to consider immediate saves for conflict resolution reasons.

Seems a little strange that mode #1 operations would be asynchronous even if my custom protocol doesn't do any async operations in response to an add/update/remove.

To clarify, the contract of the DataSource in this case is that it is *allowed* to respond asynchronously and the framework logic avoids any assumptions that it's going to get a synchronous response. If in fact you're able to synchronously update data, great.

my code I never call fetchData (I have the grid set to autoFetch).

Have you looked at the docs for autoFetchData? It calls listGrid.fetchData() when the component is drawn.

If you are subsequently also providing data via listGrid.setData(), you're wiping out the ResultSet that fetchData() automatically created for you.

But at this point your description of what you're doing doesn't match your last sample code (which has no calls to ListGrid.setData()), so we're not really sure what you're up to.

I don't want to have to synthesize a sort field. As I mentioned earlier, we are also supporting drag & drop reordering of the records and this is working fine without any extra fields/code. I just want to programmatically do what the drag & drop is doing.

Our data is versioned. Two users cannot rearrange at the same time. We definitely do not want to do immediate saves. The device will start reacting immediate on the save and the user may cancel out of the dialog and never approve the changes.

My mistake about the setData. I am calling the one on the custom data source class that I send you in my last post. This is not wiping out the fetchData work, rather it is establishing the data that will be used to response to the fetch. The setData call is made before the grid is drawn.

You did not answer my question about caching. I am not explicitly managing any cache, so I'm not certain what cache you are referring to. Should I be calling setCacheData on the data source with the content I am passing into my setData function? If so, should I skip storing my data as a member variable in the class? If I get rid of my local copy it makes things like find more difficult because RecordList has a find, but getCacheData just returns an array of records so I'll have to do a find some other way. Should I be explicitly calling updateCaches somewhere in my transformRequest instead of (or in addition to) processResponse?

How do I choose between calling addData on the grid versus the data source? If I call it on the data source with a callback, is the grid guaranteed to be up-to-date by the time the callback is executed?

[Isomorphic]

You did not answer my question about caching. I am not explicitly managing any cache, so I'm not certain what cache you are referring to. Should I be calling setCacheData on the data source with the content I am passing into my setData function? If so, should I skip storing my data as a member variable in the class? If I get rid of my local copy it makes things like find more difficult because RecordList has a find, but getCacheData just returns an array of records so I'll have to do a find some other way. Should I be explicitly calling updateCaches somewhere in my transformRequest instead of (or in addition to) processResponse?

We're not following this discussion at all. You seem be mixing together concepts that apply to a clientOnly DataSource (setCacheData()) with concepts that apply to a dataProtocol:"clientCustom" DataSource (transformRequest & processResponse), and these are two different use cases - you can't combine these APIs.

How do I choose between calling addData on the grid versus the data source? If I call it on the data source with a callback, is the grid guaranteed to be up-to-date by the time the callback is executed?

With saveLocally:true, addData() on a ListGrid will never call the DataSource. If you call addData() on ListGrid, properties such as listGrid.addOperation apply, whereas they obviously would not for a direct DataSource call. Otherwise, they are equivalent, and yes the data will have been updated before the callback is fired.

My discussion on caching was triggered by your statement:

Having a ResultSet is how you participate in cache synch.

I was trying to understand what cache you were talking about since I am not doing anything explicitly related to caching. I was speculating that maybe I was supposed to be doing something explicitly. I mentioned setCacheData because I was speculating what that might be. So I ask again, what cache are you referring to in this statement, what kind of synchronizing is being managed, and is there anything I need to do to support it correctly besides calling processResponse in transformRequest?

You have now brought up ListGrid.addOperation for the first time. I am not certain whether I should care about this or not. The documentation just says it is an Operation ID, it is a String, and it defaults to null. That doesn't really tell me much. What value might be in the String? What is it used for? Why would I want to set it to something other than the default? You also said "properties such as" addOperation. This implies that there are others. So, even if it turns out I don't care about the addOperation being applied, what is the best way to find out the full list of properties that would not be applied if I called addData directly on the DataSource? Would it be all of the settable properties on DataBoundComponent, or some other set?