[Isomorphic]

Yes, DataSources can handle any kind of client-to-server and server-to-client data massaging that might be required - see dataProtocol:"clientCustom" for the most flexible approach, which gives completely arbitrary control over both format and timing of when requests and sent.

As we noted yesterday, we're already looking into the issues with mode #2, however, just for completeness, it still appears, even after your description, that mode #1 is the most appropriate. Again, it's inherent to your application that updates to the queue are being asynchronously saved to the server while additions from other users may arrive asynchronously at any time. It seems like in your current implementation you've somehow ended up at a point where synchronously updating the dataset simplifies things - we think it's likely that this is actually an artifact of your approach of using clientOnly DataSources and a globally simpler solution is possible.

Note also that you can actually make a synchronous update to the dataset even in mode #1 - you could take the current ResultSet and replace it with a new ResultSet using the same data plus whatever additions or changes you've made.

[Isomorphic]

A quick follow up - we are investigating a couple of potential bugs in the process described in #2 - we'll update here shortly

[Isomorphic]

As previously noted, method #2 is probably not the most appropriate route for you to take, but we've made some changes which should improve things in that case.

You can test these changes in today's build, dated December 11, when it arrives at smartclient.com/builds - you can let us know if you still see any issues.

SmartClient Version: v9.1p_2014-12-11/LGPL Development Only (built 2014-12-11)

The use case we have been discussing I have converted to use mode #1. However, as I have stated many times in this discussion, we have many more use cases and our architects still believe that mode #2 is more appropriate for many of them. I don't want to waste any more of your time or mine arguing over what is the best mode. What I want is for all of the modes to actually function as expected.

I tried your latest changes and there does not appear to be any change in behavior. I've attached my latest sample code. The only difference from last time is that I removed a line to setClientOnly in the data source. I did this because this is now causing an error and the grid data does not load. Also it seems to be consistent with our previous conversations about how to manage a data source locally when I am not trying to simulate a remote data source.

I am still seeing the same problems. On updates the records do not move to the correct sort location nor do they move to the correct group if the groupBy field changes. The filter does not appear to work at all. I've even tried changing the filter criteria in the sample such that it should filter out all the initial records (by reversing the test). The initial load of the data still shows all records (even though they do not match the filter criteria).

Did you actually try running my sample code against your changes?

Sorry, ignore all that - I pulled down the wrong version build. I'll get the correct version and try again...

[Isomorphic]

Just to spell out something we thought you already realized: the reason that discussing mode #1 vs mode #2 is not a waste of time is that mode #2 is a much more rarely used, specialized mode intended for just a handful of unusual use cases.

Rarely used modes in general have more bugs (in any kind of software).

You may also discover limitations of mode #2 where we would decline to add features to address your usage, in a case where we think mode #1 is a better fit anyway.

So these are all reasons to very carefully consider the choice to use mode #2, for this specific scenario or future scenarios.

I tried again with this version:

SmartClient Version: v10.0p_2014-12-11/LGPL Development Only (built 2014-12-11)

and as in my previous message that I said to ignore (because I had pulled down the wrong version), I don't see any change in behavior. Did you actually tried my sample code after you thought the issues were fixed. See code attached to that message if you need the source.

Also, it would be helpful if you gave more detail then simply saying you made changes that should help and instead actually enumerated exactly which issues you think should have been solved by updating.

I think I must be missing something about the capabilities of mode #1. We have hundreds of use cases in our application that already manage server interactions getting large blocks of data where only few small parts of that data are lists of things that show up in different grids. We really strongly object to any data source associated with these grids doing any interaction with the server. We just want to place the data we already have in the grid, do updates on the grid when events tell us something changed, and allow the user to make changes. User initiated changes are NOT done directly on the grid, they are done through dialogs that allow the user to modify key pieces of information which result in one or more server calls that indirectly update the server data and fire targeted events telling the UI exactly which piece of data changed. During modifications we do not deal with whole records. The sets and events are targeted and just contain something akin to a record id, the field that changed and it's new value.

Are you suggesting that we create a data source that just stores a reference back to the actual data in the object we retrieved via rpc and then override the request/response pairs to manipulate the underlying data? I assume if this is it, I would also somehow tell the data source that "all" of the data has arrived and that I want to do all filtering and sorting locally. Is that what you are suggesting?

It seemed like your whole rationale for using mode #1 was based upon there being some tie between the grid, its data source, and needing to do remoting calls to get and update the data. If that is correct, that seems to break down in situation where we don't want the data source associated with any remoting. That is why I keep trying to use mode #2.

[Isomorphic]

Have you setFilterLocalData(true)? That's necessary, see the doc, and your sample works fine once that attribute is set.

This helped some, not certain why the need to set this for mode #2 didn't come up before now. It did fix the filtering and the sorting (doc doesn't mention anything about it affecting sorting), but it made groupBy worse. Now the sample won't groupBy at all. It does nothing. Before it would at least groupBy but an update would not move the record to the correct group.

[Isomorphic]

Can you try reducing your row-count from 20,000? Just a small test value, like 50.

Do you see any improvement? If not, do you see any logs in the Smartclient Developer Console? (SC.showConsole())

Well, it's interesting that you chose the value 50 because that happens to work, but 51 doesn't.

I do have the call setGroubByMaxRecords(20000) in my code and this does work when using mode #1.

51 is too small a limit for some of our use cases.

[Isomorphic]

To address the mode #1/#2 discussion while we're looking at the grouping issue, again, DataSources are designed to allow you to do the remoting in any way you want (refer back to our mention of dataProtocol:"clientCustom"). Your description of sending updates to the server and receiving responses exactly fits the concept of a DataSource. A DataSource is a client-side interface to a remote data model and that is your situation exactly.

If you were doing your data loading via a normal DataSource "fetch" operation, there would be no need to separately or specially tell the grid that the data had arrived. It would be just the response to a fetch.

Similarly, if you represented your updates to the dataset as DataSource "update", "add" and "remove" requests, you would never run into a problem of having to manually, synchronously update a client-side cache to reflect changes you've made to server-side state, because the update of the cache would just be the automatic consequence of the successful update/add/remove operation.

It's your choice of ignoring the DataSource operations architecture and instead manually managing a client-side cache that makes mode #2 look appealing, when in reality, your use case is addressed by mode #1.

So again, for the reasons previously explained, we would highly recommend moving to mode #1 unless you actually discover a compelling reason to use the more specialized, much less common mode #2.

Sorry I'm being dense here, but I still do not understand what the expectation is in doing a custom data source in some of my scenarios. In all the cases I've seen the following doesn't seem to fit:

1) The data source seems to want to think in terms of records. The server queries we make pass large objects that contain multiple lists in them as well as other data that is not record based at all. I don't see how I would manage the content in the data source that is not tied to any notion of a record or how I would differentiate which list I was referring to when there are multiple lists in the data.

2) In many contexts in our application the client gets the data during startup before the user chooses which part of the UI they are going to work in. So by the time they make that selection, any fetch has already been done. Also, the changes may not be written back to the server until they leave that section of our UI (even though a dialog that displayed a list from the data as a grid has long since been closed). No "fetch" operation needs to be done (at least not as far as contacting the server - I can see how the grid might view it as a fetch from the data source, but the data source does not need to do any fetch) when the grid is initialized. As we exit the dialogs we save the updated list back into the original java object (the one that contains multiple lists and other non-record oriented data). Eventually the user does something that triggers writing the object back to the server. There is not a separate server update for each list in the object. It was one rpc call that sends the updated super object.

I understand how your DataSource operations architecture would account for these issues.

Also your assertion that doing updates to a dataset would avoid the problem of needing to manually synchronize updates to a client-side cache to reflect changes seems incorrect. We are not talking about a single client here. The environment is multi-client.

You are correct that an "add" could be done on the client that initiated the request, but you are ignoring the fact that when the server sees the addition it is going to fire an event so other clients can update themselves without needing to re-fetch a larger set of data from the server. That event will not only be received by all of the other clients, but the client that originally did the add will also receive the event. There is no clean way for the client to keep track around all of the asynchronous operations that this event is for the same entry it is trying to add or an add done by another client. It could receive and start processing the event before it even sees a successful return from the server call that does the add because of the asynchronous nature. Or the event processing could trigger after it knows the server add was successful, but during the asynchronous add to the grid. Or it could occur after the all the local adding finished. It's all basically random which scenario is seen each time because of the asynchronous environment. Because of this, the local adding of the record must be synchronized with the event handling so that the first one in does and add and all the rest call update.

Therefore, both the code after the local add and the event handling code must check to see if the item in question has already been added by the other path. If it has not then add it, otherwise update it. Since the addData call on the grid is asynchronous, we are hitting the case where both paths check if the record already exists; both paths see that is doesn't; both paths try to add the item. Only one of the two adds succeeds, the other exceptions with a duplicate key. Hence in mode #1 some type of additional external synchronization is necessary.

So again, from our perspective mode #2 simplifies these interactions and fits the server interaction model better. Also, the application already has hundreds of grids structured to work in mode #2 and we are unlikely to invest the weeks of engineering effort it would take at this point to refactor them to use mode #1 instead (unless we cannot get mode #2 to work correctly and have no other choice). It would be a different discussion if we saw a reasonable way to address the two items I outlined above and we creating these grids from scratch.

[Isomorphic]

The grid works in terms of Records, so you are already doing this transform somehow.

The clientCustom dataProtocol we keep referring to does not assume that every DataSource operations implies a network RPC. If you would read the docs on this, it would probably help immensely to clarify things.

Finally, you would still need to use updatedCaches() to communicate changes made by other users, and you would now be eligible to use this API since you would no longer be using a clientOnly DataSource.

As I said before the transformation is done by looping through a java list object creating Record objects based upon my list and then using grid.setData/addData/updateData to update the local data source. At no time do I want the data source to do any networking. Other code in the client already does that because the network calls to get/set the data are NOT record oriented. The only difference I have seen in these discussions between mode #1 and mode #2 is the remoting. I see no way in any the documentation you are referring to of using a DataSource object to interface to a pre-existing java object that looks something like:

public class PropertyMap extends HashMap<String, String> {
}

public class Job extends PropertyMap {
}

public class Capability extends PropertyMap {
}

public class Fubar {
  Date lastModification;
  String mode;
  List<Job> jobs;
  List<Capability> capabilities;
}

With appropriate getters and setters. Code to fill in a grid looks something like:

public void someMethod() {
  loadGridData(grid, fubarObject.capabilities);
}

public void loadGridData(ListGrid grid, List<Map<String, String>> data) {
  List<Record> records = new ArrayList<Records>();
  for (Map<String, String> item: data) {
      Record record = new Record();
      for (Map.Entry<String, String> entry: item.entrySet()) {
        record.setAttribute(entry.key, entry.value);
      }
      records.add(record);
  }
  grid.setData(records.toArray(new Record[records.size()]));
}

I have read all of the documentation you have referred to and I still don't see any mention or examples of how you would define a structure like this as a DataSource because it is not record oriented. If it does support this please point me at an example. If there's no example, but it's documented then please give me a more precise pointer to this documentation instead of just saying it's in the docs.

My startup code makes an RPC call that returns this object. At some arbitry time in the future, the user navigates to some part of the UI that wants to display the capabilities in a grid. Exactly where is the documentation does it show me how define a datasource that reflects this structure and does NOT do any server interaction (because it was already done at startup and I don't need to do it again)? I understand that clientCustom dataProtocol does not assume that every operation implies networking, but I don't want any operations to do any networking ever. From your descriptions, this sounds more like mode #2 to me.