dynamic DataSources

[paulf]

We have a need to be able to dynamically create DataSource fields for user-defined searches. The scenario is as follows:

The user can create new searches for any type of object using a search builder.

The search builder will have a pane for defining the search target, the filter criteria, and the select columns. Imagine this pane corresponding to an arbitrary 'select' statement over the database.

The search will be represented by an xml structure.

The result grid will be populated by parsring the xml and executing the search.

The primary question we have is around the definition of the datasource for the result grid. The fields in the result grid need to be dynamically created by parsing the xml search object (which we can do now) but we would also like to have the DataSource ultimately known to the server so that the result grid can take advantage of data paging.

Could you recommend the best strategy for doing this?

A related question is, once a datasource has been created can we reuse it by modifying its fields on both the client and the server?

thanks,
-pf

[Isomorphic]

In 8.0 there's a new, dedicated API for this DataSource.addDynamicDSGenerator(). In 7.0 the same thing was achievable with more work (custom IDACall servlet that would use DataSource.fromXML() to dynamically provide DataSources).

If the DataSource is changing over time, just use a different ID to load the changed DataSource as a fresh DataSource. There's not currently support for any kind of "upgrade in place".

[paulf]

We are using the new dataSource API (DataSource.addDynamicDSGenerator(dds, String prefix)) and have two questions.

Is it necessary to manually load the DataSource on the client via DataSource.load (dsID, callback, forceReload)?

Also, once a DataSource has been generated for a given id (with mathcing prefix), on successive fetches, does the server look up a DataSource by first checking its own internal cache or will it always turn to the DynamicDSGenerator to find a DataSource with the matching prefix?

Thanks,
-p

[Isomorphic]

How do you mean "manually", what approach are you contrasting that with? The loadDS JSP tag and DataSourceLoader servlet both work with dynamic DataSources if that's what you mean.

Dynamic DataSource instances are pooled and reused like other DataSources. Note that DataSource pooling is typically disabled in the SDK for ease of trying out changes (no server restart required).

[paulf]

By manually I just meant using the client side API (DataSource.load()).

The question we have is that when we call DataSource.load() the DataSourceGenerator we created does in fact get called with getDataSource(dsid, dsRequest). The problem is that the dsRequest is null. For our application it is necessary to have the dsRequest (or more specifically the HttpSession object which we can get from the dsRequest) in order to create the data source. The main question for us is under what conditions will the dsRequest parameter not be null?

-p

[Isomorphic]

A DSRequest will be available if what triggers the loading of the DataSource is a DSRequest. This typically only applies if you're not loading the DataSource definition as such to the client at all (eg, the client has a stub).

What are you planning to use the session for? If you plan on creating user-specific DataSources this may not be the right API. That's going to result in pools of various DataSources per user.

[paulf]

In our product, the user has the ability to create queries dynamically. Neither the result set nor the select columns can be predicted in advanced. So we were looking for a mechanism by which we could generate a datasource based on an arbitrary query.

Our current scheme using the DynamicDSGenerator is as follows:

1) the user defines the query filters and select columns on the client (using a modified filterBuilder)
2) we parse the query and generate the datasource dynamically on the server using a DMI call.
3) in the DMI callback we load the datasource, and set myListGridResults.dataSource = "the new loaded datasource id"
4) we execute a myListGridResults.fetch({searchId: "mySearchId"}) which allows us to leverage the existing mechanism for data set pagination.

In other words, in our world dynamic dataSources are transitory based on the user session. At this point we figured out a way to manage the generation of new DataSources fine. The only issue remains is reclaiming them when a user session is ended. Is there something like a DataSource.removeDataSource(dsid) method on the server?

Alternatively, the main reason we are using dataSources on the server for this is to be able to use pagination (startRow, endRow) with a DSRequest. Is the pagination mechamism still available in DSRequest when the DataSource is purely client-side.

Thanks
-p

[Isomorphic]

Since they are per-user and transitory, you don't want to be using the DynamicDSGenerator API, which will just introduce pools of what you intend as transitory DataSources. Instead, use only DataSource.fromXML(). The resulting DataSource can be turned into a JSON string via the JSTranslater which you can eval() client-side to bring the DataSource into existence. To execute requests against the DataSource, create a DSRequest and call DataSource.execute().

Note, you don't need to have a server-side DataSource to enable paging - in fact it's questionable whether you want to go down this path in general. For one, you could have a single .ds.xml for all your dynamic DataSources, which does not declare any fields and therefore cannot do server-side validation or execute SQL queries, but if you don't need either of these features, this doesn't matter.

Second, if paging is the only thing you're worried about, you can also use a RestDataSource, but the former approach (single .ds.xml file) is easier.

[jooky68]

I am currently in the process of evaluating the SmartGWT Analytics module for the CubeGrid component. My project has similar requirements to those specified by paulf (transitory datasources based on user session) and I was hoping you could elaborate on your response.

Use case: A user requests a report via a report name. The report information is retrieved from code which runs on the server and is completely dynamic, meaning we will never know ahead of time the structure of the report. I have an existing RESTful web-service which serves the requested report as a JSON response. I am not dead-set on this approach, and would like to use the approach which best fits with the SmartGWT framework.

Given my use case:
1.) how does the framework handle a single .ds.xml file, with no field definitions, that is used across all of my dynamic datasources? Does this mean that I would then need to make use of DataSource.transformRequest/transformResponse ?
2.)Does a DS file need to be specified at all when dealing with dynamic datasources?
3.) I also must deal with CubeGrid updates which can add new facets. I was originally under the impression that when dealing with an update like this, I would have to create a new DataSource on the server via DataSource.toXML(). Then serialize the DataSource to the client, call DataSource.load(), and then set the new DataSource into each databound via <databoundcomponent>.setDataSource(). How does this work if the same ds.xml file is used for all dynamic DataSource bound components?


What is the SmartGWT recommended way to approach this problem? I feel there are many paths to achieve a solution, but I am having trouble gathering enough information and understanding the nuances of each path to make a sound engineering decision. I have read the Quick Start Guide from front to back. While it was a great introduction to the framework, it seems my use case is not the norm and it wasn't covered in any great detail. I understand, at a macro-level, the different server-side and client-side approaches, but I still am lacking the knowledge to understand which path is right for this use case. Any help would be greatly appreciated.

[Isomorphic]

1) this is fine, but as you might expect, this drops all server-side features related to fields (such as validation, trimming responses via dataSource.dropExtraFields, etc)

2) no, not required

3) a CubeGrid can display a subset of the available facets (see CubeGrid.setFixedFacetValues()). Otherwise, if the list of facets is truly dynamic, provide them dynamically - this articule shows a sample of doing this when integrating with Pentaho Mondrian.

[mcardinale]

Since they are per-user and transitory, you don't want to be using the DynamicDSGenerator API, which will just introduce pools of what you intend as transitory DataSources. Instead, use only DataSource.fromXML(). The resulting DataSource can be turned into a JSON string via the JSTranslater which you can eval() client-side to bring the DataSource into existence. To execute requests against the DataSource, create a DSRequest and call DataSource.execute().

I'm trying to do something similar to this and I'm not following what needs to be done to make this work. Can you show and example or point me to a place I can get more information. I do not see the DataSource.execute method define and I'm not sure how to use the DSRequest object clientside. Will I still be able to bind this datasource to a list grid to display the data? Thanks!

[Isomorphic]

This entire thread is discussing a server-side approach for dynamic DataSources. You'll find DataSource.execute() in the server-side JavaDoc - it's also discussed in the QuickStart Guide under Custom DataSources.

[mcardinale]

I'm sorry - I'm not seeing any mention of how to create dynamic columns in the QuickStart or any of the related references. I have done coding like described, but I'm not seeing how to generate the columns dynamically. You indicate that you use DataSource.fromXML (server side), encode to JSON String and I guess some how get it to client and then eval it. This will then make it available clientside and can you then use the datasource like normal? This is the part I'm missing - Thanks!

[Isomorphic]

Again, if you aren't looking at the server-side JavaDoc, you're going to be lost. The QuickStart Guide shows you where this is (both online and in your SDK package).

Now, are you working with SmartClient or SmartGWT? You've posted in each forum once. eval() is a reference to JavaScript so if you're using Java maybe that's why this thread is hard to follow. The equivalent for SmartGWT is just to have a <script src=> tag in your host .html file that points at a .jsp that returns JavaScript, again via using JSTranslater with the DataSource instance obtained from DataSource.fromXML().

[mcardinale]

Sorry I am using SmartGWT, did not realize this was the SmartClient part of the forum. Just did a search and found this posting that was dealing with the same problem. I belive I have determined a way to do this, not sure it correct, but it seems to work. Cannot do the jsp route as I need to be able to changes the definition of the columns bases on the inputs of the query. Again sorry did not mean to cause problems.