[rle125]

The latest 13.0p build resolved the errors.

I'll continue to test it out and report any issues we encounter.

Thanks for the great support!

[Isomorphic]

Thanks for sending through all the information you've sent over the last few days. After some additional experimenting and code analysis, we have been able to reproduce and fix the problem with the @Embedded field - the problem is actually in SmartClient code, not Hibernate. We have not been able to reproduce the second problem you reported (in post #20), but we have applied a speculative fix that we're reasonably confident will resolve that issue as well (it is only speculative in the sense that we can't say it fixes the issue because we can't repro the issue - we do have good reason to think it will fix the issue)

So please try your use case with tomorrow's 13.0 build and let us know how it goes.

Regards,
Isomorphic Software Support

[Isomorphic]

Could you please either post or email the stacktrace for the ConcurrentModificationException that you are still getting? It doesn't look like that exception could be thrown from the same line unless it is trying to process an @Embedded property, so we think you are seeing a different issue with the same symptom

[rle125]

Removing the @Embedded tag resolved the ConcurrentModificationException on 3 of the datasources that contained that annotation.

However, I get the same ConcurrentModificationException on another datasource, but without the @Embedded tag. I'll be sending the POJO and ds.xml for that datasource via email.

[rle125]

Yes, we were using Hibernate 4.2.17 successfully with SmartGWT Power 12.1p. Build date was 7/26/23.

I'll try removing the @Embedded tag. If it doesn't help I'll send the additional files.

[Isomorphic]

Following up on this:

Were you using Hibernate 4.2.17 with your 12.1 installation? If so, please let us know the exact build date of the SmartGWT 12.1 installation.

Secondly, your POJO contains many references to other application classes and we obviously have no idea what those classes look like. So we had to strip it down basically to field and function definitions, and having done so we do not see the ConcurrentModificationException you are getting (either with Hibernate 3 or Hibernate 4.2.17). However, from the location of the failure, we think the issue is with the @Embedded "ipDeviceInstance" field. It also seems fishy that the POJO class both extends IPDeviceInstance and declares an @Embedded field of that type. Does this observation help? For example, should that embedded IPDeviceInstance even be there?

We are now at a stage where we can't investigate further with the information we have. The next stage, if the above comments do not help you resolve things, would be to provide us with the IPDeviceInstance class, stripped down to just annotated fields and functions, so we can try again to repro the issue. But it really does look like an issue with Hibernate internal state, caused by a persistent class both extending a base class, and containing an embedded reference to an instance of that same base class

Regards,
Isomorphic Software Support

[rle125]

1. let the system sit for about 5 minutes post-startup before issuing a request. If that works, that's a big clue

Waited 30 minutes but still received the server-side exception.

2. if you can, attach a debugger and look at whether other threads are active when this error occurs

Will attempt this.

POJO and ds.xml sent via email.

[Isomorphic]

Oh, and, as previously mentioned, we still don't have the .ds.xml or POJO, so we have no idea what annotations you're working with. It could be something not covered by our tests, in which case we might have a straightforward bug here.

You can email those as needed. It would be very helpful to even get a partial .ds.xml and the POJO with the annotations intact but the logic stripped, if that's necessary.

[Isomorphic]

That's too bad, but note that you definitely want to keep the combined tag regardless. The log you showed had something like 9 concurrent threads, because each <script src=> tag is another request (that's an HTML thing, not under our control) so combining is going to be much faster.

Then, as far as troubleshooting, there are two things to look at, both predicated on the idea that Spring is still in the process of doing something odd as you make this request:

1. let the system sit for about 5 minutes post-startup before issuing a request. If that works, that's a big clue

2. if you can, attach a debugger and look at whether other threads are active when this error occurs

Meanwhile, we will take a look at the code here and try to understand what could possibly trigger such an except. We aren't seeing it in our autotests.

[rle125]

Combining into a single <script src=> tag still results in the server-side exceptions.

As far as #3, I meant that when the server is started, its clean. The server-side exceptions occur after login, followed by the browser successfully loading the datasources.

Is there anything else we can provide that can be useful in troubleshooting?

[Isomorphic]

Thanks for the full log (sent separately via email).

Regarding this code you previously posted:

     <script src="myApp/sc/DataSourceLoader?dataSource=myDMI"></script>

Do you have a separate <script src=> tag for every single DataSource you're loading? You don't need to do that: you can list them all in one tag. That's going to be far, far, far faster.

Note that, even if you're doing this, it doesn't explain the ConcurrentModificationException - it's perfectly fine to make several concurrent calls to DataSourceLoader, and that's done all the time. We still suspect Spring weirdness in the annotations layer. However, if moving to a single DataSourceLoader fixes the problem for you, then we'll have something we can reasonably look into.

Also still need at least #3 above answered in order to understand the basics of the problem.

[rle125]

I sent the full log to support@isomorphic.com .

The server-side exception occurs immediately after login, and before the browser reports successful loading of the ds.xml files.

I've been commenting out code to help narrow down the issue and I noticed that some datasources do work.

There does appear to be multiple threads, but I'm not exactly sure what is triggering them since our code doesn't explicitly do anything with the datasources except configuration.

[Isomorphic]

It looks like you've customized the log a bit, but with the default logging, the ConcurrentModification at the top would be the first thing that happened, chronologically. Specifically, these read to us like 3 distinct logs, each with different log levels (WARN, ERROR and ERROR):

1. The root error is a ConcurrentModificationException (happens first)

2. the second error is because the JPADataSource init wasn't expecting this weird exception and ran into an NPE because it assumed it was going to get a valid result from the call that blew up

3. the third error is the DataSourceLoader reporting that DataSource loading failed, due to the above. Normally there would be some info here, but an NPE was thrown for an unanticipated reason, so there isn't anything further to report

So, 3 questions:

1. are there two threads at this time? Because the more we look, the more this seems like Spring is dynamically modifying or generating annotations, possibly by replacing classes at runtime

2. [previously asked] are there two threads at this time? We've only got a partial log here; for evidence of another thread being active, you'd look further up the log. Note the log contains thread markers (e.g. https-jsse-nio-8443-exec-6) so you can discern threads. Of course, whatever other thread is operating, it might not log anything, so you might need to try a debugger or similar

3. you said above:

The application server starts clean and the browser is able to successfully load the datasources (DataSourceLoader?dataSource=myDMI).

.. but this logs from the first post clearly show that DataSourceLoader is crashing trying to load "myDMI". So which is it? Do the DS load the first time but not the second, or second time but not the first, or intermittently, or something else?

[rle125]

I was thinking the ConcurrentModificationException was due to the "null" ServletException.

Is there a way to get more information about this ServletException?

2024-11-11 13:58:22,609 ERROR [com.isomorphic.servlet.DataSourceLoader] (https-jsse-nio-8443-exec-1) BaseServlet Global Exception
javax.servlet.ServletException: DataSource 'myDMI' failed to load due to an exception on the server:
null
See the server-side log for additional details.
    at com.isomorphic.servlet.DataSourceLoader.processRequest(DataSourceLoader.java:215)

[Isomorphic]

OK, we'll assume an intact installation. However, we're still flying pretty blind here: don't have the .ds.xml, or the JPA annotations, or any knowledge about what myDMI does.

Just in terms of basics, the ConcurrentModificationException probably means two threads are involved. So what are the two threads? Concurrent requests against that DataSource? Or perhaps a background task?

Note that, because of DataSource pooling, it's not allowed to hold onto an instance of a DataSource that you retrieved via e.g. RPCManager.getDataSource(), since such DataSources are automatically returned to the pool at the end of the HttpRequest. So one common way to get ConcurrentModificationExceptions is to hang onto a DataSource and try to use it again in another thread.

Note: for completeness: you can get a ConcurrentModificationException with a single thread, by just modifying a List during iteration. But since this code has already run successfull once (the DataSource loading), it's probably a multi-thread issue.