Upgrade from SmartGWT 12.1p to 13.0p: DMI not found

[rle125]

SmartGWT 13.0p / TomEE 8.0.14 / Firefox 115

After upgrading from SmartGWT 12.1p to 13.0p, we get the following exception:

ERROR [com.isomorphic.servlet.DataSourceLoader] (https-jsse-nio-8443-exec-1) BaseServlet Global Exception

javax.servlet.ServletException: DataSource 'myDMI' failed to load due to an exception on the server:

null

See the server-side log for additional details.

I verified myDMI exists in the same location as when it was working w/12.1p.

I don't see any additional server-side log details either.

Is there a way to get more detail regarding this error?

[Isomorphic]

It does seem strange that you're getting this failure with no additional error logging.

Firstly - can we confirm your setup:

The error log looks like you are attempting to load a dataSource you created called "myDMI" (myDMI.ds.xml).

The server.properties configuration file governs where the SmartClient system will look for DataSources, via the property "project.datasources"
See this guide: https://smartclient.com/smartclient-...rceDeclaration

Can you confirm that project.datasources is configured to point to the correct location to pick up your ds.xml file?

How are you attempting to load the DataSource -are you hitting the DataSource loader URL directly or using a loadDS jsp tag or something else?

Are you having problems loading other dataSources? If you import something like the SupplyItem dataSource, does it get served correctly?

If you are able to load other dataSources, but not this one, we'd recommend you share that DataSource file with us directly. You can use the support@isomorphic.com email address for this to avoid leaking any sensitive content if necessary.

Regards
Isomorphic Software

[rle125]

project.datasources is defined in our server.properties:

project.datasources: $webRoot/ds, $webRoot/myApp/ds

DataSources are loaded thorugh JSP:

<script src="myApp/sc/DataSourceLoader?dataSource=myDMI"></script>

I don't believe any DataSources are being loaded.

Is there logging we can enable to get more information regarding the DataSource loading?

[Isomorphic]

The SmartClient server uses log4j for logging. This can be configured as described in the documentation here: https://smartclient.com/smartclient-....serverLogging

The default levels would typically show you some useful information - likely an error message and stack trace - when a dataSource is failing to load. It may be worth verifying whether you have modified the default settings, and consider decreasing the logging threshold for at least com.isomorphic.servlet.DataSourceLoader to get additional information.

If you're not able to load any dataSources this suggests a configuration issue perhaps.
We'd recommend you review the deployment process you took when upgrading to 13.0, and also review the full server logs to see if there are any other warnings that may indicate what's going on.

Regards
Isomorphic Software

[rle125]

I verified all of the things from the DataSource FAQ.

I was able to get more logging, but it mainly shows validation, all of which seemed OK.

Were there 3rd party JARs that need updating when upgrading from 12.1p to 13.0p? I only upgraded the smartgwt and isomorphic specific JARs.

Here is the full stacktrace:

WARN [com.isomorphic.jpa.JPADataSource] (https-jsse-nio-8443-exec-6) Failed to derive data source 'myDMI' for class myPackage.MyEntity

java.util.ConcurrentModificationException

at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:911)

at java.util.ArrayList$Itr.next(ArrayList.java:861)

at com.isomorphic.annotations.DataSourceAnnotations.getDataSourceConfigFromAnnotatedClass(DataSourceAnnotations.java:177)

at com.isomorphic.jpa.JPADSGenerator.getDataSourceConfigFromJPAClass(JPADSGenerator.java:69)

at com.isomorphic.jpa.JPADataSource.deriveDS(JPADataSource.java:438)

at com.isomorphic.jpa.JPADataSource.init(JPADataSource.java:380)

at com.isomorphic.datasource.DataSource.initialize(DataSource.java:1078)

at com.isomorphic.datasource.BasicDataSource.fromConfig(BasicDataSource.java:248)

at com.isomorphic.datasource.DataSource.fromConfig(DataSource.java:1053)

at com.isomorphic.datasource.DataSource.loadDS(DataSource.java:666)

at com.isomorphic.datasource.DataSource.forName(DataSource.java:568)

at com.isomorphic.datasource.PoolableDataSourceFactory.makeUnpooledObject(PoolableDataSourceFactory.java:127)

at com.isomorphic.datasource.PoolableDataSourceFactory.makeObject(PoolableDataSourceFactory.java:152)

at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)

at com.isomorphic.pool.PoolManager.borrowObject(PoolManager.java:90)

at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:204)

at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:121)

at com.isomorphic.servlet.DataSourceLoader.processDataSource(DataSourceLoader.java:232)

at com.isomorphic.servlet.DataSourceLoader.processRequest(DataSourceLoader.java:196)

at com.isomorphic.servlet.DataSourceLoader.doGet(DataSourceLoader.java:112)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:670)

at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:178)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:779)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

at org.apache.tomee.webservices.CXFJAXRSFilter.doFilter(CXFJAXRSFilter.java:88)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:177)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)

at org.apache.tomee.catalina.OpenEJBValve.invoke(OpenEJBValve.java:45)

at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:660)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)

at org.apache.tomee.catalina.OpenEJBSecurityListener$RequestCapturer.invoke(OpenEJBSecurityListener.java:97)

at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)

at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)

at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)

at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:891)

at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1784)

at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)

at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

at java.lang.Thread.run(Thread.java:750)

ERROR [com.isomorphic.servlet.DataSourceLoader] (https-jsse-nio-8443-exec-6) Exception while attempting to load a DataSource

java.lang.NullPointerException

at com.isomorphic.jpa.JPADataSource.init(JPADataSource.java:381)

at com.isomorphic.datasource.DataSource.initialize(DataSource.java:1078)

at com.isomorphic.datasource.BasicDataSource.fromConfig(BasicDataSource.java:248)

at com.isomorphic.datasource.DataSource.fromConfig(DataSource.java:1053)

at com.isomorphic.datasource.DataSource.loadDS(DataSource.java:666)

at com.isomorphic.datasource.DataSource.forName(DataSource.java:568)

at com.isomorphic.datasource.PoolableDataSourceFactory.makeUnpooledObject(PoolableDataSourceFactory.java:127)

at com.isomorphic.datasource.PoolableDataSourceFactory.makeObject(PoolableDataSourceFactory.java:152)

at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)

at com.isomorphic.pool.PoolManager.borrowObject(PoolManager.java:90)

at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:204)

at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:121)

at com.isomorphic.servlet.DataSourceLoader.processDataSource(DataSourceLoader.java:232)

at com.isomorphic.servlet.DataSourceLoader.processRequest(DataSourceLoader.java:196)

at com.isomorphic.servlet.DataSourceLoader.doGet(DataSourceLoader.java:112)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:670)

at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:178)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:779)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

at org.apache.tomee.webservices.CXFJAXRSFilter.doFilter(CXFJAXRSFilter.java:88)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:177)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)

at org.apache.tomee.catalina.OpenEJBValve.invoke(OpenEJBValve.java:45)

at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:660)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)

at org.apache.tomee.catalina.OpenEJBSecurityListener$RequestCapturer.invoke(OpenEJBSecurityListener.java:97)

at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)

at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)

at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)

at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:891)

at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1784)

at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)

at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

at java.lang.Thread.run(Thread.java:750)

ERROR [com.isomorphic.servlet.DataSourceLoader] (https-jsse-nio-8443-exec-6) BaseServlet Global Exception

javax.servlet.ServletException: DataSource 'myDMI' failed to load due to an exception on the server:

null

See the server-side log for additional details.

at com.isomorphic.servlet.DataSourceLoader.processRequest(DataSourceLoader.java:215)

at com.isomorphic.servlet.DataSourceLoader.doGet(DataSourceLoader.java:112)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:670)

at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:178)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:779)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

at org.apache.tomee.webservices.CXFJAXRSFilter.doFilter(CXFJAXRSFilter.java:88)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:177)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)

at org.apache.tomee.catalina.OpenEJBValve.invoke(OpenEJBValve.java:45)

at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:660)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)

at org.apache.tomee.catalina.OpenEJBSecurityListener$RequestCapturer.invoke(OpenEJBSecurityListener.java:97)

at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:687)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)

at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)

at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)

at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:891)

at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1784)

at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)

at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

at java.lang.Thread.run(Thread.java:750)

[Isomorphic]

As a basic part of installation, yes, please upgrade all related jars as part of the installation process, as well as following all other installation steps. Frequently, problems will occur just because a partial installation was done, and the deployment ends up with mutually inconsistent files.

Having done the full installation, if the problem still occurs, as far as these errors, just to cover the basics..

1) are you actually using EJB?

2) are you actually using JPA?

3) have you declared a JPA-managed bean and are you using dataSource.autoDeriveSchema to interact with it?

4) do you have any logic that is automatically executed at application startup?

On the last point, there are seeming concurrency problems. It's not clear if these are problems resulting from a more basic error (like mismatched file versions), however, if you have logic that is kicked off at servlet start, or anything like that, we would need to know about it. Any such logic needs to run after Init.go() in the SmartClient framework, or needs to call Init.go() itself.

===
Edit: In our eagerness to provide a timely response, our team posted a duplicate message on this thread. We apologize for any confusion this may have caused and have since removed the redundant response to keep the conversation clear and focused.

[rle125]

We use JPA annotated entity classes and JPADataSource in our ds.xml.

The application server starts clean and the browser is able to successfully load the datasources (DataSourceLoader?dataSource=myDMI).

I tried replacing all the JARs with the SmartGWT 13.0p distribution versions.

I made sure Init.go() is called first.

But we're still seeing the null Exception

I did notice that SmartGWT 13.0p uses hibernate3.jar but our application requires hibernate 4.2.17 (core, entity-manager, commons-annotations).

Are there any further troubleshooting steps you can suggest?

[Isomorphic]

OK, we'll assume an intact installation. However, we're still flying pretty blind here: don't have the .ds.xml, or the JPA annotations, or any knowledge about what myDMI does.

Just in terms of basics, the ConcurrentModificationException probably means two threads are involved. So what are the two threads? Concurrent requests against that DataSource? Or perhaps a background task?

Note that, because of DataSource pooling, it's not allowed to hold onto an instance of a DataSource that you retrieved via e.g. RPCManager.getDataSource(), since such DataSources are automatically returned to the pool at the end of the HttpRequest. So one common way to get ConcurrentModificationExceptions is to hang onto a DataSource and try to use it again in another thread.

Note: for completeness: you can get a ConcurrentModificationException with a single thread, by just modifying a List during iteration. But since this code has already run successfull once (the DataSource loading), it's probably a multi-thread issue.

[rle125]

I was thinking the ConcurrentModificationException was due to the "null" ServletException.

Is there a way to get more information about this ServletException?

2024-11-11 13:58:22,609 ERROR [com.isomorphic.servlet.DataSourceLoader] (https-jsse-nio-8443-exec-1) BaseServlet Global Exception
javax.servlet.ServletException: DataSource 'myDMI' failed to load due to an exception on the server:
null
See the server-side log for additional details.
    at com.isomorphic.servlet.DataSourceLoader.processRequest(DataSourceLoader.java:215)

[Isomorphic]

It looks like you've customized the log a bit, but with the default logging, the ConcurrentModification at the top would be the first thing that happened, chronologically. Specifically, these read to us like 3 distinct logs, each with different log levels (WARN, ERROR and ERROR):

1. The root error is a ConcurrentModificationException (happens first)

2. the second error is because the JPADataSource init wasn't expecting this weird exception and ran into an NPE because it assumed it was going to get a valid result from the call that blew up

3. the third error is the DataSourceLoader reporting that DataSource loading failed, due to the above. Normally there would be some info here, but an NPE was thrown for an unanticipated reason, so there isn't anything further to report

So, 3 questions:

1. are there two threads at this time? Because the more we look, the more this seems like Spring is dynamically modifying or generating annotations, possibly by replacing classes at runtime

2. [previously asked] are there two threads at this time? We've only got a partial log here; for evidence of another thread being active, you'd look further up the log. Note the log contains thread markers (e.g. https-jsse-nio-8443-exec-6) so you can discern threads. Of course, whatever other thread is operating, it might not log anything, so you might need to try a debugger or similar

3. you said above:

The application server starts clean and the browser is able to successfully load the datasources (DataSourceLoader?dataSource=myDMI).

.. but this logs from the first post clearly show that DataSourceLoader is crashing trying to load "myDMI". So which is it? Do the DS load the first time but not the second, or second time but not the first, or intermittently, or something else?

[rle125]

I sent the full log to support@isomorphic.com .

The server-side exception occurs immediately after login, and before the browser reports successful loading of the ds.xml files.

I've been commenting out code to help narrow down the issue and I noticed that some datasources do work.

There does appear to be multiple threads, but I'm not exactly sure what is triggering them since our code doesn't explicitly do anything with the datasources except configuration.

[Isomorphic]

Thanks for the full log (sent separately via email).

Regarding this code you previously posted:

     <script src="myApp/sc/DataSourceLoader?dataSource=myDMI"></script>

Do you have a separate <script src=> tag for every single DataSource you're loading? You don't need to do that: you can list them all in one tag. That's going to be far, far, far faster.

Note that, even if you're doing this, it doesn't explain the ConcurrentModificationException - it's perfectly fine to make several concurrent calls to DataSourceLoader, and that's done all the time. We still suspect Spring weirdness in the annotations layer. However, if moving to a single DataSourceLoader fixes the problem for you, then we'll have something we can reasonably look into.

Also still need at least #3 above answered in order to understand the basics of the problem.

[rle125]

Combining into a single <script src=> tag still results in the server-side exceptions.

As far as #3, I meant that when the server is started, its clean. The server-side exceptions occur after login, followed by the browser successfully loading the datasources.

Is there anything else we can provide that can be useful in troubleshooting?

[Isomorphic]

That's too bad, but note that you definitely want to keep the combined tag regardless. The log you showed had something like 9 concurrent threads, because each <script src=> tag is another request (that's an HTML thing, not under our control) so combining is going to be much faster.

Then, as far as troubleshooting, there are two things to look at, both predicated on the idea that Spring is still in the process of doing something odd as you make this request:

1. let the system sit for about 5 minutes post-startup before issuing a request. If that works, that's a big clue

2. if you can, attach a debugger and look at whether other threads are active when this error occurs

Meanwhile, we will take a look at the code here and try to understand what could possibly trigger such an except. We aren't seeing it in our autotests.

[Isomorphic]

Oh, and, as previously mentioned, we still don't have the .ds.xml or POJO, so we have no idea what annotations you're working with. It could be something not covered by our tests, in which case we might have a straightforward bug here.

You can email those as needed. It would be very helpful to even get a partial .ds.xml and the POJO with the annotations intact but the logic stripped, if that's necessary.