reading from DataSource on Server

[atomatom]

Is it possible to use DataSource definitions on the server side? For example, I want to create a DMI DataSource that reads from some other DataSources.

I've looked through the manual and examples, javadoc, but I'm not finding how to do it. I know there is a DataSource.fromXML() method, but it threw this exception when I tried to fetch()

=== 2011-01-14 11:47:14,632 [10-2] DEBUG DataSourceDMI - Invocation threw exception
com.isomorphic.xml.XMLParsingException: [
    "XML parser fatal error: file '(in memory stream)' line 1: org.xml.sax.SAXParseException: Content is not allowed in prolog."
]
	at com.isomorphic.xml.XML.parseXML(XML.java:205)
	at com.isomorphic.xml.XML.parseXML(XML.java:124)
	at com.isomorphic.xml.XML.toDSRecords(XML.java:289)
	at com.isomorphic.xml.XML.toDSRecords(XML.java:285)
	at com.isomorphic.xml.XML.toDSRecords(XML.java:277)
	at com.isomorphic.datasource.DataSource.fromXML(DataSource.java:467)
	at com.isomorphic.datasource.DataSource.fromXML(DataSource.java:436)
	at com.project.MyDMI.fetch(MyDMI.java:43)

[atomatom]

Ok, so, I was passing the filename instead of an XML string. My bad.

I'm interested to have my DMI datasources read from other DataSources because it decouples my application from the actual data storage. Is this good practice?

[atomatom]

All good now.

In case anyone else was trying to do this:

In my server DMI code I get a DataSource like this (note the use of the com.isomorphic packages - the server version of DataSource)

import com.isomorphic.datasource.DSRequest;
import com.isomorphic.datasource.DSResponse;
import com.isomorphic.datasource.DataSource;

....


		String dSFile = "ds/myRegularDS_DataSource.ds.xml";
		FileReader fileReader = new FileReader(dSFile);
		DataSource regularDS = DataSource.fromXML(fileReader);

		List<MyDataBoundClass> objects = regularDS.fetch(null);

Can I suggest you add an example to the docs that illustrates this. Likewise, if it's there and I missed it, apologies.

[Isomorphic]

This is overkill - you just need to create a DSRequest and execute() it - see the DMI sections in the Server Framework chapter of the QuickStart.

[atomatom]

I just re-read those sections and I don't see what you mean.

I tried this:

			DSRequest dsRequest = new DSRequest();
			dsRequest.setDataSourceName(dsName);
			DSResponse dsResponse = dsRequest.execute();
			List<T> results = dsResponse.getDataList();

Which blows up. Presumably because simply setting the data source name is not enough. I could load the datasource as I did earlier, but I'm not sure I see how this is any less overkill than my example?

[Isomorphic]

Can you elaborate on "blows up"?

[atomatom]

ok so, the answer was in the stack trace.

java.lang.Exception: Auto-operation name (null) must either be of the format dataSourceId_operationType or a public zero-argument method
	at com.isomorphic.application.AppBase.execute(AppBase.java:457)
	at com.isomorphic.datasource.DSRequest.execute(DSRequest.java:1443)
	at com.MyCode.getDataSource(MyCode.java:32)

so I added

 
			DSRequest dsRequest = new DSRequest();
			dsRequest.setDataSourceName(dsName);
			[b]dsRequest.setOperationType("fetch");[/b]
			DSResponse dsResponse = dsRequest.execute();
			List<T> results = dsResponse.getDataList();

and now it works.

Thank you.

[Isomorphic]

Great. Note the convenience constructors by the way - you could also be doing:

List<T> results = new DSRequest(dsName, opType).execute().getDataList();

[atomatom]

Version tested: SC_SNAPSHOT-2011-01-18/EVAL Deployment 2011-01-18

I'm getting exceptions when I use DSRequest from within a server method. I wonder if there is any other cleanup/etc I need to do.

Here is my server method that makes an additional DSRequest.

First I get a warning, (highlighted in bold), then I get exceptions. Is there a bug in the isomorphic JPA code?

The test case below is just a modified version of the GAE-DS sample, and bound the "countryGrid" to my DMI based datasource. I added a button to the code to force a re-fetch, clicking this button a few times will get the behavior below.

public DSResponse fetchDMI(DSRequest request) throws Exception {
		String dataSourceName = "country_DataSource";

		DSRequest newRequest = new DSRequest(dataSourceName, "fetch");
		DSResponse newResponse = newRequest.execute();

		DSResponse response = new DSResponse();
		response.setStartRow(newResponse.getStartRow());
		response.setEndRow(newResponse.getEndRow());
		response.setTotalRows(newResponse.getTotalRows());
		response.setData(newResponse.getData());
		System.out.println("OINK " + newResponse.getData());
		return response;
	}

=== 2011-01-19 13:32:34,340 [75-5] DEBUG GAEJPADataSource - [builtinApplication.null] Query string: select _Country from Country _Country
=== 2011-01-19 13:32:34,857 [75-5] INFO  DSResponse - [builtinApplication.null] DSResponse: List with 5 items
=== 2011-01-19 13:32:34,857 [75-5] INFO  DSResponse - DSResponse: List with 5 items
OINK [com.smartgwt.sample.server.Country[countryId=agZkcy1nYWVyDQsSB0NvdW50cnkYAQw, countryCode=LT, countryName=Lithuania], com.smartgwt.sample.server.Country[countryId=agZkcy1nYWVyDQsSB0NvdW50cnkYBQw, countryCode=LV, countryName=Latvia], com.smartgwt.sample.server.Country[countryId=agZkcy1nYWVyDQsSB0NvdW50cnkYBww, countryCode=EE, countryName=Estonia], com.smartgwt.sample.server.Country[countryId=agZkcy1nYWVyDQsSB0NvdW50cnkYCgw, countryCode=bO, countryName=nAH], com.smartgwt.sample.server.Country[countryId=agZkcy1nYWVyDQsSB0NvdW50cnkYDAw, countryCode=asdas, countryName=dasd]]
=== 2011-01-19 13:32:34,858 [75-5] DEBUG RPCManager - Content type for RPC transaction: text/plain; charset=UTF-8
=== 2011-01-19 13:32:34,860 [75-5] DEBUG RPCManager - DMI response, dropExtraFields: true
[b][WARN] Request completed without committing or rolling back transaction with id 0.  Transaction will be rolled back.[/b]
=== 2011-01-19 13:33:06,374 [75-7] INFO  RequestContext - URL: '/GAEDS_js/sc/IDACall', User-Agent: 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.2.13) Gecko/20101206 Ubuntu/10.04 (lucid) Firefox/3.6.13': Moz (Gecko) with Accept-Encoding header
=== 2011-01-19 13:33:06,376 [75-7] DEBUG XML - Parsed XML from (in memory stream): 2ms
=== 2011-01-19 13:33:06,380 [75-7] DEBUG RPCManager - Processing 1 requests.
=== 2011-01-19 13:33:06,381 [75-7] DEBUG RPCManager - Request #1 (DSRequest) payload: {
    criteria:{
    },
    operationConfig:{
        dataSource:"country_DataSource2",
        operationType:"fetch",
        textMatchStyle:"exact"
    },
    startRow:0,
    endRow:75,
    componentId:"isc_ListGrid_0",
    appID:"builtinApplication",
    operation:"country_DataSource2_fetch",
    oldValues:{
    }
}
=== 2011-01-19 13:33:06,381 [75-7] INFO  IDACall - Performing 1 operation(s)
=== 2011-01-19 13:33:06,382 [75-7] DEBUG AppBase - [builtinApplication.null] No userTypes defined, allowing anyone access to all operations for this application
=== 2011-01-19 13:33:06,383 [75-7] DEBUG AppBase - [builtinApplication.null] No public zero-argument method named '_null' found, performing generic datasource operation
=== 2011-01-19 13:33:06,383 [75-7] DEBUG JPADataSource - [builtinApplication.null] Creating EntityManager and starting transaction.
=== 2011-01-19 13:33:06,384 [75-7] DEBUG GAEJPADataSource - [builtinApplication.null] Query string: select _Country from Country _Country
=== 2011-01-19 13:33:06,405 [75-7] INFO  DSResponse - [builtinApplication.null] DSResponse: List with 5 items
=== 2011-01-19 13:33:06,405 [75-7] INFO  DSResponse - DSResponse: List with 5 items
OINK [com.smartgwt.sample.server.Country[countryId=agZkcy1nYWVyDQsSB0NvdW50cnkYAQw, countryCode=LT, countryName=Lithuania], com.smartgwt.sample.server.Country[countryId=agZkcy1nYWVyDQsSB0NvdW50cnkYBQw, countryCode=LV, countryName=Latvia], com.smartgwt.sample.server.Country[countryId=agZkcy1nYWVyDQsSB0NvdW50cnkYBww, countryCode=EE, countryName=Estonia], com.smartgwt.sample.server.Country[countryId=agZkcy1nYWVyDQsSB0NvdW50cnkYCgw, countryCode=bO, countryName=nAH], com.smartgwt.sample.server.Country[countryId=agZkcy1nYWVyDQsSB0NvdW50cnkYDAw, countryCode=asdas, countryName=dasd]]
=== 2011-01-19 13:33:06,406 [75-7] DEBUG RPCManager - Content type for RPC transaction: text/plain; charset=UTF-8
=== 2011-01-19 13:33:06,406 [75-7] DEBUG RPCManager - DMI response, dropExtraFields: true
[WARN] Request completed without committing or rolling back transaction with id 1.  Transaction will be rolled back.
=== 2011-01-19 13:33:07,106 [izer] DEBUG JPADataSource - Committing transaction.
=== 2011-01-19 13:33:07,120 [izer] ERROR JPADataSource - Failed to commit transaction. Rolling back.
javax.persistence.RollbackException: Transaction failed to commit
	at org.datanucleus.jpa.EntityTransactionImpl.commit(EntityTransactionImpl.java:118)
	at org.datanucleus.store.appengine.jpa.DatastoreEntityTransactionImpl.commit(DatastoreEntityTransactionImpl.java:55)
	at com.isomorphic.jpa.EMFProviderLMT.commitTransaction(EMFProviderLMT.java:153)
	at com.isomorphic.jpa.EMF.commitTransaction(EMF.java:131)
	at com.isomorphic.jpa.JPADataSource.freeResources(JPADataSource.java:578)
	at com.isomorphic.datasource.DSRequest.freeResources(DSRequest.java:2771)
	at com.isomorphic.datasource.DSRequest.finalize(DSRequest.java:1758)
	at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
	at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:83)
	at java.lang.ref.Finalizer.access$100(Finalizer.java:14)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160)
Caused by: javax.persistence.PersistenceException: Transaction rolled back due to failure during commit
	at org.datanucleus.jpa.NucleusJPAHelper.getJPAExceptionForNucleusException(NucleusJPAHelper.java:283)
	at org.datanucleus.jpa.EntityTransactionImpl.commit(EntityTransactionImpl.java:116)
	... 10 more
Caused by: java.lang.IllegalStateException: Transaction is in state ROLLED_BACK.  There is no legal transition out of this state.
	at com.google.appengine.api.datastore.TransactionImpl.ensureTxnStarted(TransactionImpl.java:204)
	at com.google.appengine.api.datastore.TransactionImpl.commitAsync(TransactionImpl.java:80)
	at com.google.appengine.api.datastore.TransactionImpl.commit(TransactionImpl.java:76)
	at org.datanucleus.store.appengine.DatastoreTransaction.commit(DatastoreTransaction.java:61)
	at org.datanucleus.store.appengine.DatastoreXAResource.commit(DatastoreXAResource.java:88)
	at org.datanucleus.transaction.Transaction.commit(Transaction.java:149)
	at org.datanucleus.transaction.TransactionManager.commit(TransactionManager.java:95)
	at org.datanucleus.TransactionImpl.internalCommit(TransactionImpl.java:390)
	at org.datanucleus.TransactionImpl.commit(TransactionImpl.java:258)
	at org.datanucleus.jpa.EntityTransactionImpl.commit(EntityTransactionImpl.java:104)
	... 10 more
=== 2011-01-19 13:33:07,131 [izer] DEBUG JPADataSource - Committing transaction.
=== 2011-01-19 13:33:07,133 [izer] ERROR JPADataSource - Failed to commit transaction. Rolling back.
javax.persistence.RollbackException: Transaction failed to commit
	at org.datanucleus.jpa.EntityTransactionImpl.commit(EntityTransactionImpl.java:118)
	at org.datanucleus.store.appengine.jpa.DatastoreEntityTransactionImpl.commit(DatastoreEntityTransactionImpl.java:55)
	at com.isomorphic.jpa.EMFProviderLMT.commitTransaction(EMFProviderLMT.java:153)
	at com.isomorphic.jpa.EMF.commitTransaction(EMF.java:131)
	at com.isomorphic.jpa.JPADataSource.freeResources(JPADataSource.java:578)
	at com.isomorphic.datasource.DSRequest.freeResources(DSRequest.java:2771)
	at com.isomorphic.datasource.DSRequest.finalize(DSRequest.java:1758)
	at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
	at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:83)
	at java.lang.ref.Finalizer.access$100(Finalizer.java:14)
	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160)
Caused by: javax.persistence.PersistenceException: Transaction rolled back due to failure during commit
	at org.datanucleus.jpa.NucleusJPAHelper.getJPAExceptionForNucleusException(NucleusJPAHelper.java:283)
	at org.datanucleus.jpa.EntityTransactionImpl.commit(EntityTransactionImpl.java:116)
	... 10 more
Caused by: java.lang.IllegalStateException: Transaction is in state ROLLED_BACK.  There is no legal transition out of this state.
	at com.google.appengine.api.datastore.TransactionImpl.ensureTxnStarted(TransactionImpl.java:204)
	at com.google.appengine.api.datastore.TransactionImpl.commitAsync(TransactionImpl.java:80)
	at com.google.appengine.api.datastore.TransactionImpl.commit(TransactionImpl.java:76)
	at org.datanucleus.store.appengine.DatastoreTransaction.commit(DatastoreTransaction.java:61)
	at org.datanucleus.store.appengine.DatastoreXAResource.commit(DatastoreXAResource.java:88)
	at org.datanucleus.transaction.Transaction.commit(Transaction.java:149)
	at org.datanucleus.transaction.TransactionManager.commit(TransactionManager.java:95)
	at org.datanucleus.TransactionImpl.internalCommit(TransactionImpl.java:390)
	at org.datanucleus.TransactionImpl.commit(TransactionImpl.java:258)
	at org.datanucleus.jpa.EntityTransactionImpl.commit(EntityTransactionImpl.java:104)
	... 10 more

[atomatom]

I replaced my server side DSRequest.execute() calls with direct JPA queries and I'm not getting any more exceptions.

This route is probably better for me, since Advanced Criteria is not implemented for GAE/JPA.

[atomatom]

I thought I had side-stepped these exceptions, but they are still occurring if I rely on com.isomorphic.jpa.GAEJPADataSource. In particular, I have a form bound to a GAEJPADataSource that uses another GAEJPADataSource as an optionDataSource, and I get the above exception.

Although the exception does not appear to lose any data, my current workaround is to go through and remove GAEJPADataSource. My guess is this exception is caused by hitting the GAEJPADataSource too frequently. It makes me worry about using this datasource in production.

[alius]

Hi,

Just to be clear:
you have created your own DS on server;
from within this DS (on server) you create new DSRequest for GAEJPADataSource and execute it.

GAE-JPA and JPA DS'es do not release resources on execution to avoid LazyLoadingExceptions while traversing through managed entities. Normally resources released by RPCManager after HTTP response is completed.

To release resources manually you can (and have to) call:

newRequest.freeResources()

after request is executed.

BTW: in your log there is warning message:

[WARN] Request completed without committing or rolling back transaction with id 0.  Transaction will be rolled back.

This message is generated by Google engine. It tracks that transaction has been opened and not closed upon finishing request.

PS. Make sure that you understand Google engine's transaction mechanism:
http://code.google.com/appengine/docs/java/datastore/transactions.html

PS1. You can configure your application (in server.properties) to use EMFProviderNoTransactions.

Regards,
Alius.

[atomatom]

Thanks very much for the tips. I've decided against using DataSource on the server side. It really was not making much sense, and it's easier to write JPA queries. The GAEJPADataSource has limitations like no AdvancedCriteria implementation. Plus, how many layers of abstraction is healthy... :)

These exceptions still occur when using the GAEJPADataSource on the client side. Two cases seems to make it happen. 1) hitting the datasource frequently 2) using an optionDataSource (which I guess makes two data source fetches in a single request)

Switching to com.isomorphic.jpa.EMFProviderNoTransactions stops the exceptions. I presume at the cost of some speed, but this is likely negligible for my application.

One point, these exceptions are being caused from inside isomorphic code, so I'm not sure how my understanding of transactions will help. I'm pretty sure it's a bug/unhandled case in the smartgwtee server code. I was simply calling a fetch. The thing that I find strange is that a transaction is opened on a read/fetch. (my guess is that is the bug... )

[atomatom]

For anyone else, if you use EMFProviderNoTransactions you also need to modify your persistence.xml file to set non-transactional reads/writes.

<persistence-unit name="transactions-optional">
        <provider>org.datanucleus.store.appengine.jpa.DatastorePersistenceProvider</provider>
        <properties>
[b]            <property name="datanucleus.NontransactionalRead" value="true"/>
            <property name="datanucleus.NontransactionalWrite" value="true"/>
[/b]            <property name="datanucleus.ConnectionURL" value="appengine"/>
        </properties>
    </persistence-unit>

[Isomorphic]

If you're comfortable using direct JPA, that may make sense for read-only use cases, at least in the short span before we roll out AdvancedCriteria for JPA (then it will make less sense).

As far as transactions, you may want to again turn that off only for reads, or, the more efficient approach, use Queuing (see QuickStart guide) to combine multiple reads into one HTTP request.

If you see any server exceptions that suggest a bug in the SmartGWT EE server (so far you haven't provided anything that indicates this) we would very much like to see that.