DataSource.load usage

[bbruyn]

The application I'm working on has lots and lots and lots of datasources. I'd rather not load them all at once from the host page because it just takes forever.

The app is broken up into logical 'modules' with code splitting. What I thought I would do is load the datasources needed by each module when the module starts. So if I have 500 datasources in total, but the current module only needs 15 of them, I load the 15 and move on.

So the way I thought that could work is by using the DataSource.load client API.

The first thing I found is that you can't seem to localize datasource elements that way out of the box, because the standard DataSourceLoaderServlet doesn't know how.

So I wrote this goofy servlet that basically just includes a JSP which uses the isc i18n taglib. This works, but already I see that I have at least one problem with this approach:

I have 2 datasources, parentDS and childDS, where childDS inheritsFrom parentDS. parentDS has autoDeriveSchema="true", childDS does not. Which I think is right, otherwise childDS won't inherit fields from parentDS.

When I use childDS in some DynamicForm, the fields that are defined explicitly in either datasource are fetched & rendered correctly. The fields that are derived from the schema are not rendered. Setting form.setUseAllDataSourceFields(true) has no effect.

However, I _can_ actually refer to the 'missing' fields explicitly and have them appear. So if parentDS overrides the field1 definition as derived from schema, and childDS overrides field2 (and inherits field1) then form.setDataSource(childDS) gets me a form with only field1 and field2 on it. I have to create a TextItem(field3) and setFields (field1, field2, field3) to get it on my form.

In practice, this is what I'll almost always do anyway. So no big deal on its own. But if I use the provided DataSourceLoader servlet, this all just works out of the box.

So the point of all this is just that I'm beginning to doubt my usage of the DataSource.load API. Am I barking up the wrong tree here?

[Isomorphic]

See the docs for DataSource.inheritsFrom. There are flags you can set for whether all fields are inherited, only fields you specify are inherited, etc.

[bbruyn]

Well you didn't come out and tell me to stop, so that's good. :-)

I have read that doc though. I think you're talking about setShowLocalFieldsOnly? That wouldn't seem to make much sense, as I am getting all of the fields that are defined explicitly - just not the ones that are 'derived' from the schema.

And as I say, setting that thing is unecessary when using the standard ds loader.

I tried it anyway by adding showLocalFieldsOnly="false" to my ds xml. No change.

Again, if this is the only problem I can expect, then I'm good. Just looking for confirmation that I'm not abusing the API in a way that's going to give me a headache later. I wonder, because there are a whole mess of request parameters that I do nothing with in my custom servlet. isc_tnum, isc_xhr, isc_rpc, and so on...

[Isomorphic]

About those extra parameters - you don't necessarily need to be using DataSource.load() here, the only thing you need to do to load a DataSource is evaluate the JavaScript that the server produces when a DataSource is transformed to JSON via a JSTranslater. So you can just use an rpcRequest with evalResult:true, for example. But it's also OK to just use DataSource.load and ignore those parameters.

As far as this inheritance issue - we can't really tell definitively without seeing the DataSources and code. But what this sounds like is either a load order issue, or perhaps something to do with your "goofy servlet" - there's an explanation of how to do i18n with .jsp parsing of .ds.xml files here - if your servlet differs from this it could, for example, prevent the information obtained from autoDeriveSchema from ever being delivered to the client.

If you look, for example, at what is delivered by the DataSourceLoader servlet when a DataSource has autoDeriveSchema="true", you'll see there's an implicit parent DataSource that represents the information inherited from the SQL table / Hibernate bean. If that's missing in what your servlet spits out, that's the problem.

[bbruyn]

if your servlet differs from this it could, for example, prevent the information obtained from autoDeriveSchema from ever being delivered to the client.

If you look, for example, at what is delivered by the DataSourceLoader servlet when a DataSource has autoDeriveSchema="true", you'll see there's an implicit parent DataSource that represents the information inherited from the SQL table / Hibernate bean. If that's missing in what your servlet spits out, that's the problem.

Yes, that seems to be what's happening. I can see the implicit parent in the server log, but not in the js result.

All the servlet actually does is try to hand off to a JSP that uses the taglib as described in the docs you've linked to:

public class DataSourceLoaderServlet extends HttpServlet {

	private static final long serialVersionUID = 1L;

	public static final String PARAM_DATASOURCE_ID = "dataSource";
	public static final String ATTR_DATASOURCE_FILENAME="dataSourceFileName";
	
	protected void doGet(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException {

		String path = (String) Config.getProperty("project.datasources");
		String param = request.getParameter(PARAM_DATASOURCE_ID); 

		String[] datasources = param.split(",");
		for (int i = 0; i < datasources.length; i++) {
			String ds = datasources[i];
			String file = request.getContextPath().concat("/") + path.concat("/").concat(ds).concat(".ds.xml");
			request.setAttribute(ATTR_DATASOURCE_FILENAME, file);
			request.getRequestDispatcher("/WEB-INF/datasources.jsp").include(request, response);
		}
	}

}

And all the JSP does is

<%@ taglib uri="/WEB-INF/iscTaglib.xml" prefix="isc" %>
<%@ page import="DataSourceLoaderServlet;" %>
<%
    String fileName = (String) request.getAttribute(DataSourceLoaderServlet.ATTR_DATASOURCE_FILENAME);
 %>
<isc:XML> 
  <jsp:include page="<%=fileName%>"></jsp:include>  
</isc:XML>

I don't think there's anything special about the datasources themselves, but I'll dumb them down and include them here if there's no obvious problem here...

[Isomorphic]

If you use the steps from the DataSource localization topic and you just hit the .ds.xml files directly (just type the URL into your browser) do you get the complete DataSource JSON?

Does it fix your problem to load the DataSources in strict parents-before-children order?

[bbruyn]

No, I get the localized XML definition, not JSON. I get JSON if I go through the servlet. e.g.,

http://127.0.0.1:8888/MyApp/ds/Contact.ds.xml

yields XML.

http://127.0.0.1:8888/MyApp/sc/DataSourceLoader?dataSource=Contact&isc_rpc=1&isc_v=SC_SNAPSHOT-2011-12-05&isc_xhr=1&isc_tnum=0&_transaction=%3Ctransaction%20xmlns%3Axsi%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2F10%2FXMLSchema-instance%22%20xsi%3Atype%3D%22xsd%3AObject%22%3E%3CtransactionNum%20xsi%3Atype%3D%22xsd%3Along%22%3E0%3C%2FtransactionNum%3E%3Coperations%20xsi%3Atype%3D%22xsd%3AList%22%3E%3Celem%3E__ISC_NULL__%3C%2Felem%3E%3C%2Foperations%3E%3C%2Ftransaction%3E&protocolVersion=1.0

Yields JSON.

I am loading them parent first now.

[Isomorphic]

OH.

Well the fact that this even appears to work in a limited way is a surprise. You should be returning JSON, and that's what the DataSource Localization instructions will lead to. If you look, the key step that results in JSON being returned is the addition of the <isomorphic:XML> JSP tag around the included XML generated by the .ds.xml file when processed as a JSP.

[bbruyn]

the key step that results in JSON being returned is the addition of the <isomorphic:XML> JSP tag around the included XML generated by the .ds.xml file when processed as a JSP.

Well, right. Per the docs, "you should create a JSP that uses SmartGWT Server's XML conversion tag to return Javascript DataSource definitions to the browser... You then refer to this JSP in a <script src=...> tag, in place of the DataSourceLoader reference"

But when you make a request for the .ds.xml directly, as you asked me to, you're not going through that JSP, or any other kind of datasourceLoader. That's what my servlet does, and that's why it returns JSON.

Unless I'm missing some very important sentence from the same doc, I don't at all understand how a request for the ds file is supposed to result in JSON.

[bbruyn]

Maybe I just misunderstood what you were asking for. If I bypass the servlet entirely, and go straight to a JSP that includes the datasources explicitly (as in the doc), then yes, I get a JSON result. FWIW, that JSON result matches what I get back from the servlet, but you can see the same behavior here.

The datasources:

<!-- <%@ taglib prefix="fmt" uri="http://java.sun.com/jsp/jstl/fmt" %> -->

<DataSource xmlns:fmt="urn:jsptld:http://java.sun.com/jsp/jstl/fmt"
    ID="Individual"
    dbName="PolarisData"
	tableName="IND"
	serverType="sql"
	autoDeriveSchema="true">
	 
	<fields>
		<field name="IND_UK" type="sequence" sequenceName="S_5082_1_IND" primaryKey="true" />
		<field name="FULL_NAME" type="text" customSelectExpression="CONCAT(CONCAT(first_name, ' '), last_name)" />
	</fields>
	
</DataSource>

<DataSource xmlns:fmt="urn:jsptld:http://java.sun.com/jsp/jstl/fmt"
	ID="Contact" 
	serverType="sql" 
	tableName="IND" 
	dbName="PolarisData"
	inheritsFrom="Individual"
	showLocalFieldsOnly="false"> 

	<fields>
		
		<fmt:setBundle basename="com.experis.polaris.datasource.shared.ds.Company"/>
		<field name="COMPANY_UK" type="integer" hidden="true" foreignKey="Company.COMPANY_UK">
			<title><fmt:message key="companyIdTitle"/></title>
		</field>
		<field name="COMPANY_NAME" includeFrom="Company.NAME" canEdit="false">
			<title><fmt:message key="companyNameTitle"/></title>
			<prompt><fmt:message key="companyNamePrompt"/></prompt>
		</field>

	</fields>
</DataSource>

The JSP:

<%@ taglib uri="isomorphic" prefix="isc" %>
<isc:XML> 
  <jsp:include page="/MyApp/ds/Individual.ds.xml"></jsp:include>  
  <jsp:include page="/MyApp/ds/Contact.ds.xml"></jsp:include> 
</isc:XML>

The response:

isc.DataSource.create({
    ID:"Individual",
    autoDeriveSchema:true,
    dbName:"PolarisData",
    serverType:"sql",
    tableName:"IND",
    fmt:"urn:jsptld:http://java.sun.com/jsp/jstl/fmt",
    fields:[
        {
            name:"IND_UK",
            primaryKey:true,
            sequenceName:"S_5082_1_IND",
            type:"sequence"
        },
        {
            customSelectExpression:"CONCAT(CONCAT(first_name, ' '), last_name)",
            name:"FULL_NAME",
            type:"text"
        }
    ]
})


isc.DataSource.create({
    ID:"Contact",
    dbName:"PolarisData",
    inheritsFrom:"Individual",
    serverType:"sql",
    showLocalFieldsOnly:false,
    tableName:"IND",
    fmt:"urn:jsptld:http://java.sun.com/jsp/jstl/fmt",
    fields:[
        {
            foreignKey:"Company.COMPANY_UK",
            hidden:true,
            name:"COMPANY_UK",
            type:"integer",
            title:"Company ID"
        },
        {
            canEdit:false,
            includeFrom:"Company.NAME",
            name:"COMPANY_NAME",
            title:"Company Name",
            prompt:"The legal or DBA name of the Client"
        }
    ]
})

[Isomorphic]

That's what we meant (and we could have been more clear).

Just to confirm - this .jsp that is showing JSON results is not the same as what you see from the DataSourceLoader, because the DataSourceLoader result includes an implicit inherited DataSource on the DataSource that has autoDeriveSchema:true?

[bbruyn]

No, they're the same. From the DataSourceLoader (my custom loader, not yours)

isc.DataSource.create({
    ID:"Individual",
    autoDeriveSchema:true,
    dbName:"PolarisData",
    serverType:"sql",
    tableName:"IND",
    fmt:"urn:jsptld:http://java.sun.com/jsp/jstl/fmt",
    fields:[
        {
            name:"IND_UK",
            primaryKey:true,
            sequenceName:"S_5082_1_IND",
            type:"sequence"
        },
        {
            customSelectExpression:"CONCAT(CONCAT(first_name, ' '), last_name)",
            name:"FULL_NAME",
            type:"text"
        }
    ]
})

/**** a bunch of whitespace ***/
  
isc.DataSource.create({
    ID:"Contact",
    dbName:"PolarisData",
    inheritsFrom:"Individual",
    serverType:"sql",
    showLocalFieldsOnly:false,
    tableName:"IND",
    fmt:"urn:jsptld:http://java.sun.com/jsp/jstl/fmt",
    fields:[
        {
            foreignKey:"Company.COMPANY_UK",
            hidden:true,
            name:"COMPANY_UK",
            type:"integer",
            title:"Company ID"
        },
        {
            canEdit:false,
            includeFrom:"Company.NAME",
            name:"COMPANY_NAME",
            title:"Company Name",
            prompt:"The legal or DBA name of the Client"
        }
    ]
})

It's interesting though. I (think I) do see the implicit parent get created in the server log - but that doesn't seem to happen until IDACall is handled.

http://127.0.0.1:8888/Polaris/sc/DataSourceLoader?dataSource=Individual,Contact&isc_rpc=1&isc_v=SC_SNAPSHOT-2011-12-05&isc_xhr=1&isc_tnum=0&_transaction=%3Ctransaction%20xmlns%3Axsi%3D%22http%3A%2F%2Fwww.w3.org%2F2000%2F10%2FXMLSchema-instance%22%20xsi%3Atype%3D%22xsd%3AObject%22%3E%3CtransactionNum%20xsi%3Atype%3D%22xsd%3Along%22%3E0%3C%2FtransactionNum%3E%3Coperations%20xsi%3Atype%3D%22xsd%3AList%22%3E%3Celem%3E__ISC_NULL__%3C%2Felem%3E%3C%2Foperations%3E%3C%2Ftransaction%3E&protocolVersion=1.0

=== 2011-12-13 18:29:19,754 [l0-0] INFO  RequestContext - URL: '/Polaris/sc/DataSourceLoader', User-Agent: 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:7.0.1) Gecko/20100101 Firefox/7.0.1': Moz (Gecko) with Accept-Encoding header
=== 2011-12-13 18:29:22,671 [l0-0] DEBUG XML - Parsed XML from (in memory stream): 16ms
=== 2011-12-13 18:29:22,703 [l0-0] DEBUG XML - Parsed XML from __USE_CONTAINER__/Polaris/sc/system/schema/builtinTypes.xml: 31ms
=== 2011-12-13 18:29:22,719 [l0-0] INFO  PoolManager - SmartClient pooling disabled for 'simpleTypes' objects
=== 2011-12-13 18:29:22,766 [l0-0] DEBUG XML - Parsed XML from __USE_CONTAINER__/Polaris/sc/system/schema/action.ds.xml: 0ms
=== 2011-12-13 18:29:22,781 [l0-0] DEBUG XML - Parsed XML from __USE_CONTAINER__/Polaris/sc/system/schema/DataSource.ds.xml: 0ms
=== 2011-12-13 18:29:22,797 [l0-0] DEBUG XML - Parsed XML from __USE_CONTAINER__/Polaris/sc/system/schema/valueMap.ds.xml: 16ms
=== 2011-12-13 18:29:22,797 [l0-0] DEBUG FileSystemDSRepo - dsName case sensitivity mismatch - looking for: valueMap, but got: ValueMap
=== 2011-12-13 18:29:22,812 [l0-0] DEBUG XML - Parsed XML from __USE_CONTAINER__/Polaris/sc/system/schema/serverObject.ds.xml: 0ms
=== 2011-12-13 18:29:22,812 [l0-0] DEBUG FileSystemDSRepo - dsName case sensitivity mismatch - looking for: valueMap, but got: ValueMap
=== 2011-12-13 18:29:22,812 [l0-0] DEBUG FileSystemDSRepo - dsName case sensitivity mismatch - looking for: serverObject, but got: ServerObject
=== 2011-12-13 18:29:22,875 [l0-0] WARN  Validation - No such type 'string', not processing field value at /DataSource[@ID=Action]/ID
=== 2011-12-13 18:29:22,875 [l0-0] DEBUG XML - Parsed XML from __USE_CONTAINER__/Polaris/sc/system/schema/DataSourceField.ds.xml: 0ms
=== 2011-12-13 18:29:22,890 [l0-0] DEBUG FileSystemDSRepo - dsName case sensitivity mismatch - looking for: valueMap, but got: ValueMap
=== 2011-12-13 18:29:22,922 [l0-0] WARN  Validation - No such type 'string', not processing field value at /DataSource[@ID=Action]/fields/0/field[@name=target]/name
=== 2011-12-13 18:29:22,922 [l0-0] WARN  Validation - No such type 'string', not processing field value at /DataSource[@ID=Action]/fields/0/field[@name=target]/type
=== 2011-12-13 18:29:22,922 [l0-0] WARN  Validation - No such type 'string', not processing field value at /DataSource[@ID=Action]/fields/1/field[@name=name]/name
=== 2011-12-13 18:29:22,922 [l0-0] WARN  Validation - No such type 'string', not processing field value at /DataSource[@ID=Action]/fields/1/field[@name=name]/type
=== 2011-12-13 18:29:22,922 [l0-0] WARN  Validation - No such type 'string', not processing field value at /DataSource[@ID=Action]/fields/2/field[@name=title]/name
=== 2011-12-13 18:29:22,922 [l0-0] WARN  Validation - No such type 'string', not processing field value at /DataSource[@ID=Action]/fields/2/field[@name=title]/type
=== 2011-12-13 18:29:22,922 [l0-0] WARN  Validation - No such type 'boolean', not processing field value at /DataSource[@ID=Action]/fields/3/field[@name=mapping]/multiple
=== 2011-12-13 18:29:22,922 [l0-0] WARN  Validation - No such type 'string', not processing field value at /DataSource[@ID=Action]/fields/3/field[@name=mapping]/name
=== 2011-12-13 18:29:22,922 [l0-0] WARN  Validation - No such type 'string', not processing field value at /DataSource[@ID=Action]/fields/3/field[@name=mapping]/type
=== 2011-12-13 18:29:22,922 [l0-0] DEBUG FileSystemDSRepo - dsName case sensitivity mismatch - looking for: action, but got: Action
=== 2011-12-13 18:29:22,937 [l0-0] DEBUG XML - Parsed XML from __USE_CONTAINER__/Polaris/sc/system/schema/method.ds.xml: 0ms
=== 2011-12-13 18:29:22,937 [l0-0] DEBUG FileSystemDSRepo - dsName case sensitivity mismatch - looking for: method, but got: Method
=== 2011-12-13 18:29:22,937 [l0-0] DEBUG FileSystemDSRepo - dsName case sensitivity mismatch - looking for: valueMap, but got: ValueMap
=== 2011-12-13 18:29:22,953 [l0-0] DEBUG XML - Parsed XML from __USE_CONTAINER__/Polaris/sc/system/schema/isomorphicXML.ds.xml: 0ms
=== 2011-12-13 18:29:23,639 [l0-0] DEBUG XML - Parsed XML from (in memory stream): 0ms
=== 2011-12-13 18:29:23,670 [l0-0] INFO  Compression - /Polaris/sc/DataSourceLoader: 1195 -> 482 bytes

And then what appears to be a result of getting some data

=== 2011-12-13 18:32:10,994 [l0-4] INFO  RequestContext - URL: '/Polaris/sc/IDACall', User-Agent: 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:7.0.1) Gecko/20100101 Firefox/7.0.1': Moz (Gecko) with Accept-Encoding header
=== 2011-12-13 18:32:11,025 [l0-4] DEBUG XML - Parsed XML from (in memory stream): 15ms
=== 2011-12-13 18:32:11,041 [l0-4] DEBUG XML - Parsed XML from __USE_CONTAINER__/Polaris/sc/system/schema/List.ds.xml: 16ms
=== 2011-12-13 18:32:11,057 [l0-4] DEBUG RPCManager - Processing 1 requests.
=== 2011-12-13 18:32:11,072 [l0-4] DEBUG XML - Parsed XML from C:\Development\workspace\polaris\polaris-webapp\target\polaris-webapp-0.0.1-SNAPSHOT\Polaris\ds\Contact.ds.xml: 0ms
[WARN] Server class 'org.apache.xalan.processor.TransformerFactoryImpl' could not be found in the web app, but was found on the system classpath
   [WARN] Adding classpath entry 'file:/C:/Development/.m2/repository/com/google/gwt/gwt-dev/2.4.0/gwt-dev-2.4.0.jar' to the web app classpath for this session
=== 2011-12-13 18:32:11,229 [l0-4] DEBUG XML - Parsed XML from C:\Development\workspace\polaris\polaris-webapp\target\polaris-webapp-0.0.1-SNAPSHOT\Polaris\ds\Individual.ds.xml: 15ms
=== 2011-12-13 18:32:11,230 [l0-4] INFO  SQLDataSource - Deriving dataSource Individual from table: IND
=== 2011-12-13 18:32:11,277 [l0-4] DEBUG PoolableSQLConnectionFactory - Initializing SQL config for 'PolarisData' via JNDI
=== 2011-12-13 18:32:11,277 [l0-4] DEBUG JNDI - No configuration for JNDI context '' - assuming default initialContext
[WARN] Server class 'oracle.jdbc.pool.OracleDataSource' could not be found in the web app, but was found on the system classpath
   [WARN] Adding classpath entry 'file:/C:/Development/.m2/repository/com/oracle/ojdbc6/11.1.0.7.0/ojdbc6-11.1.0.7.0.jar' to the web app classpath for this session
=== 2011-12-13 18:32:12,478 [l0-4] DEBUG PoolableSQLConnectionFactory - Returning pooled Connection
=== 2011-12-13 18:32:12,588 [l0-4] INFO  SQLDSGenerator - Fetching column metadata for table: IND
=== 2011-12-13 18:32:12,588 [l0-4] INFO  SQLDSGenerator - =============Using catalog: null
=== 2011-12-13 18:32:14,553 [l0-4] INFO  SQLDSGenerator - Fetching column metadata for IND complete
=== 2011-12-13 18:32:14,648 [l0-4] WARN  SQLDataSource - ds:isc.DataSource.create({
    schema:"FOXPRODDBO",
    dbName:"PolarisData",
    tableName:"IND",
    ID:"Individual_inheritsFrom",
    dataSourceVersion:"1",
    serverType:"sql",
    generatedBy:"SC_SNAPSHOT-2011-12-05/PowerEdition Deployment 2011-12-05",
    fields:[
        /* loads and loads of fields all derived correctly from the table */
    ]
})

=== 2011-12-13 18:32:14,663 [l0-4] DEBUG XML - Parsed XML from C:\Development\workspace\polaris\polaris-webapp\target\polaris-webapp-0.0.1-SNAPSHOT\Polaris\ds\Company.ds.xml: 0ms
=== 2011-12-13 18:32:14,695 [l0-4] DEBUG RPCManager - Request #1 (DSRequest) payload: {
    criteria:{
        IND_UK:"3001"
    },
    operationConfig:{
        dataSource:"Contact",
        operationType:"fetch"
    },
    componentId:"isc_DynamicForm_0",
    appID:"builtinApplication",
    operation:"Contact_fetch",
    oldValues:{
        IND_UK:"3001"
    }
}
=== 2011-12-13 18:32:14,695 [l0-4] INFO  IDACall - Performing 1 operation(s)

But both datasources are already loaded by that time? Does that make any sense?

[Isomorphic]

OK, we think we know what this is. This particular way of loading DataSources is skipping some processing steps. We think we'll have a fix for you that won't require changing your code, in the meantime, just use the built-in DataSourceLoader servlet to test and ignore the unprocessed <fmt> tags if you can.