Stackoverflow when loading datasource

[msule]

Hi,

We are using smartclient 12.1 version for deploying our webapp. We recently upgraded to this version and seeing stackoverflow when the UI loads.

ERROR LoadDSTag - Exception while attempting to process a loadDS tag.
java.lang.StackOverflowError: null
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:498) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:480) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.size(TreeTrie.java:309) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.ClassMatcher$ByPackage.size(ClassMatcher.java:181) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.ClassMatcher$ByPackageOrName.size(ClassMatcher.java:284) ~[jetty-util-12.0.16.jar:12.0.16]
at java.util.AbstractCollection.isEmpty(AbstractCollection.java:89) ~[?:?]
at org.eclipse.jetty.util.IncludeExcludeSet.isIncludedAndNotExcluded(IncludeExcludeSet.java:204) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.ClassMatcher.combine(ClassMatcher.java:810) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.ClassMatcher.match(ClassMatcher.java:759) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.ee8.webapp.WebAppContext.isServerClass(WebAppContext.java:705) ~[jetty-ee8-webapp-12.0.16.jar:12.0.16]
at org.eclipse.jetty.ee8.webapp.WebAppContext.isHiddenClass(WebAppContext.java:695) ~[jetty-ee8-webapp-12.0.16.jar:12.0.16]
at org.eclipse.jetty.ee8.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:393) ~[jetty-ee8-webapp-12.0.16.jar:12.0.16]
at java.lang.ClassLoader.loadClass(ClassLoader.java:1095) ~[?:?]
at org.apache.xerces.parsers.ObjectFactory.findProviderClass(Unknown Source) ~[xercesImpl.jar:?]
at org.apache.xerces.parsers.ObjectFactory.newInstance(Unknown Source) ~[xercesImpl.jar:?]
at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source) ~[xercesImpl.jar:?]
at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source) ~[xercesImpl.jar:?]
at org.apache.xerces.parsers.DOMParser.<init>(Unknown Source) ~[xercesImpl.jar:?]
at org.apache.xerces.parsers.DOMParser.<init>(Unknown Source) ~[xercesImpl.jar:?]
at org.apache.xerces.jaxp.DocumentBuilderImpl.<init>(Unknown Source) ~[xercesImpl.jar:?]
at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source) ~[xercesImpl.jar:?]
at com.isomorphic.xml.XML.parseXML(XML.java:295) ~[isomorphic_core_rpc.jar:?]
at com.isomorphic.xml.XML.parseXML(XML.java:152) ~[isomorphic_core_rpc.jar:?]
at com.isomorphic.xml.XML.toDSRecords(XML.java:498) ~[isomorphic_core_rpc.jar:?]
at com.isomorphic.xml.XML$RecordsFromXMLCache.loadObjectFromFile(XML.java:664) ~[isomorphic_core_rpc.jar:?]
at com.isomorphic.store.ProcessedFileCache.getObjectFromFile(ProcessedFileCache.java:147) ~[isomorphic_core_rpc.jar:?]
at com.isomorphic.xml.XML.loadCacheableDSRecords(XML.java:676) ~[isomorphic_core_rpc.jar:?]
at com.isomorphic.store.DataStructCache.loadInstance(DataStructCache.java:187) ~[isomorphic_core_rpc.jar:?]
at com.isomorphic.datasource.DataSource.loadDS(DataSource.java:569) ~[isomorphic_core_rpc.jar:?]
at com.isomorphic.datasource.DataSource.forName(DataSource.java:544) ~[isomorphic_core_rpc.jar:?]
at com.isomorphic.datasource.PoolableDataSourceFactory.makeUnpooledObject(PoolableDataSourceFactory.java:125) ~[isomorphic_core_rpc.jar:?]
at com.isomorphic.datasource.PoolableDataSourceFactory.makeObject(PoolableDataSourceFactory.java:150) ~[isomorphic_core_rpc.jar:?]
at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220) ~[commons-pool-1.6.jar:1.6]
at com.isomorphic.pool.PoolManager.borrowObject(PoolManager.java:85) ~[isomorphic_core_rpc.jar:?]
at

Java version: 17.0.14,17.0.14+7,IBM Corporation
Java VM: Eclipse OpenJ9 VM openj9-0.49.0,Eclipse OpenJ9
OS-System: Linux 4.18.0-553.56.1.el8_10.x86_64,amd64

Libraries being used

smartclient/12.1.2024/isomorphic_assembly.jar
smartclient/12.1.2024/isomorphic_compression.jar
smartclient/12.1.2024/isomorphic_core_rpc.jar
smartclient/12.1.2024/isomorphic_js_parser.jar
smartclient/12.1.2024/isomorphic_realtime_messaging.jar

The issue doesnt happen instantly on startup. It only happens after few days of usage. We have multiple datasources and its failing in loading the first data source. The stack size is 256kb but we also have tried with 1MB and the issue is still reproducible. Any help to identify is appreciated? We have to upgrade in a month and blocked because of this.

[Isomorphic]

This isn't a SmartClient issue, it's something wrong in Jetty. SmartClient is just asking for XML to be parsed and something blows up inside Jetty's internal classloading mechanism.

Note that even if the XML were invalid (it's not), the result of bad XML should be a parse error, not an infinite loop inside the Jetty classloader.

These are the likely causes:

1) you upgraded Jetty at the same time, and you've got multiple versions of Jetty on the classpath

or

2) you are hot deploying jars, and the hot deploy mechanism is unreliable if used too long

or

3) there's a legit Jetty bug and it's been fixed, so you just need to upgrade Jetty to the latest patched version

There are some less likely causes, like a JVM bug, but the above are where to start.

[msule]

We are seeing the stackoverflow error sometimes at the isomorphic jar level as well. But everytime the issue is happening , its trying to create an object using the

PoolManager.borrowObject(PoolManager.java:85) ~[isomorphic_core_rpc.jar:?] and this is unable to do it successfully.

I am unable to add the stacktrace file but basically the same code does recursion for 280+ times and fails .

[Isomorphic]

This is the same issue. Again you're seeing Jetty go into an infinite loop inside of Jetty's custom classloader logic. This cannot possibly be caused by SmartClient, since we do not have any classloader customizations. So you have to look to a cause within Jetty and/or your deployment or tools, as explained above.