Custom DataSource implementation questions

[csillag]

I am considering to create a custom DataSource above Google AppEngine's DataStore, and I have a couple of generic questions about implementing DataSources.

Q1: Ideally, all the requested criteria (and sort instructions) can be fully translated to the query language of the underlying data storage. However, this is not necessarily true; there can be situations when the wanted query is not supported. (For example, DataStore queries to not support more than one inequality conditions at a time.) Is it required for all DataSource implementations to support all possible combinations of criteria and advanced criteria? (Since I have never ready anything to suggest that the notion of "unsupported criteria" would exist, I guess it is required, but want to be sure.)

Possible answers:

A) No, in these cases, I can throw some kind of "CriteriaNotSupportedException". Q2: What happens in these cases?

B), Yes, it's required to implement support for all possible criteria, independently of whatever the underlying layer supports. If something is not natively supported, then I just have to do the required filtering/merging/sorting in the DataSource implementation layer. Is there some built-in support for this? (I am thinking of a method to be called when a (probably partially filtered) set of objects was already received from the underlying data storage, and translated to a list of maps. Then this method could do one more round of filtering, executing any remaining (not natively supported) criteria. Such common method could be used by all DataSource implementations.) Q3: Does such thing exists? If no, then is it planned? (If no and no, then why not?)

C) I am misunderstanding something, so the above questions are wrong, or the answers are completely different. (In this case, please elaborate.)

Thank you for your help!

[Isomorphic]

Generally, just use the various features related to configuring filtering, such as ComboBox.setTextMatchStyle() or DataSource.setTypeOperators() to ensure that you never issue criteria that your DataSource can't handle.

If you like, add some additional metadata to the DataSource that advertises what it can and can't support, and subclass the core components to check this automatically.

[csillag]

Generally, just use the various features related to configuring filtering, such as ComboBox.setTextMatchStyle() or DataSource.setTypeOperators() to ensure that you never issue criteria that your DataSource can't handle.

Thank you for the explanation. However,

1. DataSource.setTypeOperators seems to be missing from the Java wrapper

2. There are situations when all the used operators are supported, only a given combination of them is forbidden (like, as I have mentioned, GAE's DataStore can not handle more than one inequality condition in filters); how can I handle situations like these?

If you like, add some additional metadata to the DataSource that advertises what it can and can't support, and subclass the core components to check this automatically.

This would make sense, but it would be best if this would be done in a standardised manner, probably by Isomorphic. (At least the definition of the metadata language, and the modification of the filter-emitting components to honor it.)

[Isomorphic]

That's a very unfortunate and strange limitation, and for something like that, we really can't suggest anything better than flagging it as a general error and sending back an error message as data (default behavior will be to show this to the user).

[csillag]

That's a very unfortunate and strange limitation,

Indeed, but there is nothing I can do about this, since it was Google who came up with this, see here: http://code.google.com/intl/en/appen...ons_on_Queries

[...] for something like that, we really can't suggest anything better than flagging it as a general error and sending back an error message as data (default behavior will be to show this to the user).

Well, bad luck. How about adding a specific exception (or error code) for this kind of situation, so that the client side can react in a decent way?

[Isomorphic]

How would you like the client to react (and what prevents you from implementing it?)

[csillag]

Without giving the problem too much thought, I would say that the ideal behavior on the client-side would be the something like this:

- by default, display an error dialog (but not necessarily the same one that is shown for general errors)
- at com.smartgwt.client.rpc.RPCManager, there should be a method called setHandleUnsupportedFilterCallback (just like setHandleTransportErrorCallback) which could be used to override the default behavior.

To implement this, a new constant (called something like STATUS_UNSUPPORTED_FILTER) should be added to the com.smartgwt.client.rpc.RPCResponse and com.isomorphic.datasource.DSResponse classes, so that we can mark this kind of error properly in the DataSource implementations.

As an added bonus, I guess the FilterBuilder could give some more meaningful feedback to the user, but that's not so important.

Also, to help avoiding unsupported calls, something like isCriteriaSupported(Criteria, BooleanCallback) could be added to the client-side DataSource class, which should be channeled to the server-side DataSource implementations. There should be a default implementation for this in one of the common superclasses of the DataSources, always returning true, but DataSources not supporting all criteria should override this method to decide that is supported, and what is not.

At least FilterBuilder should check this call before any operation that modifies the filter.

* * *

Well, this might be too much just for one quirky DataSource (although I suspect there will come more than one, in time); for a starter, just a new error code and override function (see the first two points) would suffice.

[Isomorphic]

Just send back an error code of your choosing from the server and implement what you want with an RPCManager HandleErrorCallback.

[csillag]

Implementing AutoDeriveSchema

How can I implement autoDeriveSchema behavior for my DataSource? (I can not find anything about this in the docs at http://www.smartclient.com/smartgwte...ataSource.html.)

I understand that getting the meta-data is my task; the question is, how do I feed it into SmartGWT?

As far as I can tell from watching the SQL backend, I should automatically create a DS definition, so that the actual DS can inherit from it... but how do I get this into the system?

[Isomorphic]

DataSource.fromXML()

[csillag]

To rephrase my question:

I understand that DataSources are loaded by the system on demand, from the .ds.xml DataSource descriptor files. Let's suppose there is a descriptor that specifies my custom DataSource implementation in the serverConstructor, and it also specifies autoDeriveSchema=true.

If I understand it right, this means that when this DataSource is initiated, it will automatically get an inheritsFrom property, and this will reference a new, autogenerated DataSource definition, which should by created using meta-data gathered for the given tableName.

So, I assume that my DataSource implementation is supposed to provide this autogenerated DS definition, when required.

My question is this: where is the API for this? (How does the DataSource know that an auto-generated DS definition is needed? How do I get the TableName to collect mete-data for? When I have the name, collected the info and created the DS, where do I pass it?)

* * *

Or am I misunderstanting how autoDeriveSchema works?

Thank you for explaining!

[Isomorphic]

If you want to build DataSources on the fly from your own metadata, use DataSource.fromXML(). This gives you the ability the control the entire DataSource definition, so it gives you the ability to build something similar to the builtin autoDeriveSchema behavior if you want you. But you would not use the autoDeriveSchema property as such.

[csillag]

Sorry, I don't follow. (I guess my understanding of these parts is lacking.)

Use it how? Where?

I would assume that in order to intercept the creation process of a DataSource, I need to override some methods in my DataSource implementation, or something like that.

DataSource.fromXML() is a static method, so I assume you are not suggesting to override that...

[Isomorphic]

Use it in a servlet before requests are processed in the default way, hence, before the server tries to look up DataSources at all. This thread may help.

[csillag]

Use it in a servlet before requests are processed in the default way, hence, before the server tries to look up DataSources at all. This thread may help.

Yes, the referenced thread is most helpful.

After reading through it, I came to the same conclusion that I reached last time I had to do something similar: it seems to me that simply saving to modified DS as a new DS XML file in the shared/ds directory (before anybody asks for it) is way easier that installing all the required hooks and overrides.

So this is what I will continue to do.