Stackoverflow when loading datasource

[msule]

Hi,

We are using smartclient 12.1 version for deploying our webapp. We recently upgraded to this version and seeing stackoverflow when the UI loads.

ERROR LoadDSTag - Exception while attempting to process a loadDS tag.
java.lang.StackOverflowError: null
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:498) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:494) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.keySet(TreeTrie.java:480) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.TreeTrie.size(TreeTrie.java:309) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.ClassMatcher$ByPackage.size(ClassMatcher.java:181) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.ClassMatcher$ByPackageOrName.size(ClassMatcher.java:284) ~[jetty-util-12.0.16.jar:12.0.16]
at java.util.AbstractCollection.isEmpty(AbstractCollection.java:89) ~[?:?]
at org.eclipse.jetty.util.IncludeExcludeSet.isIncludedAndNotExcluded(IncludeExcludeSet.java:204) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.ClassMatcher.combine(ClassMatcher.java:810) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.util.ClassMatcher.match(ClassMatcher.java:759) ~[jetty-util-12.0.16.jar:12.0.16]
at org.eclipse.jetty.ee8.webapp.WebAppContext.isServerClass(WebAppContext.java:705) ~[jetty-ee8-webapp-12.0.16.jar:12.0.16]
at org.eclipse.jetty.ee8.webapp.WebAppContext.isHiddenClass(WebAppContext.java:695) ~[jetty-ee8-webapp-12.0.16.jar:12.0.16]
at org.eclipse.jetty.ee8.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:393) ~[jetty-ee8-webapp-12.0.16.jar:12.0.16]
at java.lang.ClassLoader.loadClass(ClassLoader.java:1095) ~[?:?]
at org.apache.xerces.parsers.ObjectFactory.findProviderClass(Unknown Source) ~[xercesImpl.jar:?]
at org.apache.xerces.parsers.ObjectFactory.newInstance(Unknown Source) ~[xercesImpl.jar:?]
at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source) ~[xercesImpl.jar:?]
at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source) ~[xercesImpl.jar:?]
at org.apache.xerces.parsers.DOMParser.<init>(Unknown Source) ~[xercesImpl.jar:?]
at org.apache.xerces.parsers.DOMParser.<init>(Unknown Source) ~[xercesImpl.jar:?]
at org.apache.xerces.jaxp.DocumentBuilderImpl.<init>(Unknown Source) ~[xercesImpl.jar:?]
at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source) ~[xercesImpl.jar:?]
at com.isomorphic.xml.XML.parseXML(XML.java:295) ~[isomorphic_core_rpc.jar:?]
at com.isomorphic.xml.XML.parseXML(XML.java:152) ~[isomorphic_core_rpc.jar:?]
at com.isomorphic.xml.XML.toDSRecords(XML.java:498) ~[isomorphic_core_rpc.jar:?]
at com.isomorphic.xml.XML$RecordsFromXMLCache.loadObjectFromFile(XML.java:664) ~[isomorphic_core_rpc.jar:?]
at com.isomorphic.store.ProcessedFileCache.getObjectFromFile(ProcessedFileCache.java:147) ~[isomorphic_core_rpc.jar:?]
at com.isomorphic.xml.XML.loadCacheableDSRecords(XML.java:676) ~[isomorphic_core_rpc.jar:?]
at com.isomorphic.store.DataStructCache.loadInstance(DataStructCache.java:187) ~[isomorphic_core_rpc.jar:?]
at com.isomorphic.datasource.DataSource.loadDS(DataSource.java:569) ~[isomorphic_core_rpc.jar:?]
at com.isomorphic.datasource.DataSource.forName(DataSource.java:544) ~[isomorphic_core_rpc.jar:?]
at com.isomorphic.datasource.PoolableDataSourceFactory.makeUnpooledObject(PoolableDataSourceFactory.java:125) ~[isomorphic_core_rpc.jar:?]
at com.isomorphic.datasource.PoolableDataSourceFactory.makeObject(PoolableDataSourceFactory.java:150) ~[isomorphic_core_rpc.jar:?]
at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220) ~[commons-pool-1.6.jar:1.6]
at com.isomorphic.pool.PoolManager.borrowObject(PoolManager.java:85) ~[isomorphic_core_rpc.jar:?]
at

Java version: 17.0.14,17.0.14+7,IBM Corporation
Java VM: Eclipse OpenJ9 VM openj9-0.49.0,Eclipse OpenJ9
OS-System: Linux 4.18.0-553.56.1.el8_10.x86_64,amd64

Libraries being used

smartclient/12.1.2024/isomorphic_assembly.jar
smartclient/12.1.2024/isomorphic_compression.jar
smartclient/12.1.2024/isomorphic_core_rpc.jar
smartclient/12.1.2024/isomorphic_js_parser.jar
smartclient/12.1.2024/isomorphic_realtime_messaging.jar

The issue doesnt happen instantly on startup. It only happens after few days of usage. We have multiple datasources and its failing in loading the first data source. The stack size is 256kb but we also have tried with 1MB and the issue is still reproducible. Any help to identify is appreciated? We have to upgrade in a month and blocked because of this.

[Isomorphic]

This isn't a SmartClient issue, it's something wrong in Jetty. SmartClient is just asking for XML to be parsed and something blows up inside Jetty's internal classloading mechanism.

Note that even if the XML were invalid (it's not), the result of bad XML should be a parse error, not an infinite loop inside the Jetty classloader.

These are the likely causes:

1) you upgraded Jetty at the same time, and you've got multiple versions of Jetty on the classpath

or

2) you are hot deploying jars, and the hot deploy mechanism is unreliable if used too long

or

3) there's a legit Jetty bug and it's been fixed, so you just need to upgrade Jetty to the latest patched version

There are some less likely causes, like a JVM bug, but the above are where to start.

[msule]

We are seeing the stackoverflow error sometimes at the isomorphic jar level as well. But everytime the issue is happening , its trying to create an object using the

PoolManager.borrowObject(PoolManager.java:85) ~[isomorphic_core_rpc.jar:?] and this is unable to do it successfully.

I am unable to add the stacktrace file but basically the same code does recursion for 280+ times and fails .

[Isomorphic]

This is the same issue. Again you're seeing Jetty go into an infinite loop inside of Jetty's custom classloader logic. This cannot possibly be caused by SmartClient, since we do not have any classloader customizations. So you have to look to a cause within Jetty and/or your deployment or tools, as explained above.

[msule]

What I have noticed is when the code hits the makeObject method it goes into recursion for this stack.

at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220) ~[commons-pool-1.6.jar:1.6] at com.isomorphic.pool.PoolManager.borrowObject(PoolManager.java:85) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:195) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.ValidationContext.getType(ValidationContext.java:412) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.ValidationContext.getType(ValidationContext.java:390) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.findDataSource(BasicDataSource.java:3212) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.getType(BasicDataSource.java:3132) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.logAttributeNotDeclaredMessage(BasicDataSource.java:2435) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.elementAsRecord(BasicDataSource.java:2100) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.toRecord(BasicDataSource.java:1663) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1647) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1608) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.DataSource.create(DataSource.java:2280) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:2624) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:2516) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.elementAsRecord(BasicDataSource.java:2240) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.toRecord(BasicDataSource.java:1663) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1622) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.DataSource.recordsFromXML(DataSource.java:2189) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.xml.XML.toDSRecords(XML.java:512) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.xml.XML.toDSRecords(XML.java:498) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.xml.XML$RecordsFromXMLCache.loadObjectFromFile(XML.java:664) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.store.ProcessedFileCache.getObjectFromFile(ProcessedFileCache.java:147) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.xml.XML.loadCacheableDSRecords(XML.java:676) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.store.DataStructCache.loadInstance(DataStructCache.java:187) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.DataSource.loadDS(DataSource.java:569) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.DataSource.forName(DataSource.java:544) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.PoolableDataSourceFactory.makeUnpooledObject(PoolableDataSourceFactory.java:125) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.PoolableDataSourceFactory.makeObject(PoolableDataSourceFactory.java:150) ~[isomorphic_core_rpc.jar:?] at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220) ~[commons-pool-1.6.jar:1.6] at com.isomorphic.pool.PoolManager.borrowObject(PoolManager.java:85) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:195) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.ValidationContext.getType(ValidationContext.java:412) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.ValidationContext.getType(ValidationContext.java:390) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.findDataSource(BasicDataSource.java:3212) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.getType(BasicDataSource.java:3132) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.logAttributeNotDeclaredMessage(BasicDataSource.java:2435) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.elementAsRecord(BasicDataSource.java:2100) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.toRecord(BasicDataSource.java:1663) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1647) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1608) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.DataSource.create(DataSource.java:2280) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:2624) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:2516) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.elementAsRecord(BasicDataSource.java:2240) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.toRecord(BasicDataSource.java:1663) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1622) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.DataSource.recordsFromXML(DataSource.java:2189) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.xml.XML.toDSRecords(XML.java:512) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.xml.XML.toDSRecords(XML.java:498) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.xml.XML$RecordsFromXMLCache.loadObjectFromFile(XML.java:664) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.store.ProcessedFileCache.getObjectFromFile(ProcessedFileCache.java:147) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.xml.XML.loadCacheableDSRecords(XML.java:676) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.store.DataStructCache.loadInstance(DataStructCache.java:187) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.DataSource.loadDS(DataSource.java:569) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.DataSource.forName(DataSource.java:544) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.PoolableDataSourceFactory.makeUnpooledObject(PoolableDataSourceFactory.java:125) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.PoolableDataSourceFactory.makeObject(PoolableDataSourceFactory.java:150) ~[isomorphic_core_rpc.jar:?] at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220) ~[commons-pool-1.6.jar:1.6] at com.isomorphic.pool.PoolManager.borrowObject(PoolManager.java:85) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:195) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.ValidationContext.getType(ValidationContext.java:412) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.ValidationContext.getType(ValidationContext.java:390) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.findDataSource(BasicDataSource.java:3212) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.getType(BasicDataSource.java:3132) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.logAttributeNotDeclaredMessage(BasicDataSource.java:2435) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.elementAsRecord(BasicDataSource.java:2100) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.toRecord(BasicDataSource.java:1663) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1647) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1608) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.DataSource.create(DataSource.java:2280) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:2624) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:2516) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.elementAsRecord(BasicDataSource.java:2240) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.toRecord(BasicDataSource.java:1663) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1622) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.DataSource.recordsFromXML(DataSource.java:2189) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.xml.XML.toDSRecords(XML.java:512) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.xml.XML.toDSRecords(XML.java:498) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.xml.XML$RecordsFromXMLCache.loadObjectFromFile(XML.java:664) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.store.ProcessedFileCache.getObjectFromFile(ProcessedFileCache.java:147) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.xml.XML.loadCacheableDSRecords(XML.java:676) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.store.DataStructCache.loadInstance(DataStructCache.java:187) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.DataSource.loadDS(DataSource.java:569) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.DataSource.forName(DataSource.java:544) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.PoolableDataSourceFactory.makeUnpooledObject(PoolableDataSourceFactory.java:125) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.PoolableDataSourceFactory.makeObject(PoolableDataSourceFactory.java:150) ~[isomorphic_core_rpc.jar:?] at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220) ~[commons-pool-1.6.jar:1.6] at com.isomorphic.pool.PoolManager.borrowObject(PoolManager.java:85) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:195) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.ValidationContext.getType(ValidationContext.java:412) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.ValidationContext.getType(ValidationContext.java:390) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.findDataSource(BasicDataSource.java:3212) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.getType(BasicDataSource.java:3132) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.logAttributeNotDeclaredMessage(BasicDataSource.java:2435) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.elementAsRecord(BasicDataSource.java:2100) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.toRecord(BasicDataSource.java:1663) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1647) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1608) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.DataSource.create(DataSource.java:2280) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:2624) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:2516) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.elementAsRecord(BasicDataSource.java:2240) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.toRecord(BasicDataSource.java:1663) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1622) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.DataSource.recordsFromXML(DataSource.java:2189) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.xml.XML.toDSRecords(XML.java:512) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.xml.XML.toDSRecords(XML.java:498) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.xml.XML$RecordsFromXMLCache.loadObjectFromFile(XML.java:664) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.store.ProcessedFileCache.getObjectFromFile(ProcessedFileCache.java:147) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.xml.XML.loadCacheableDSRecords(XML.java:676) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.store.DataStructCache.loadInstance(DataStructCache.java:187) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.DataSource.loadDS(DataSource.java:569) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.DataSource.forName(DataSource.java:544) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.PoolableDataSourceFactory.makeUnpooledObject(PoolableDataSourceFactory.java:125) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.PoolableDataSourceFactory.makeObject(PoolableDataSourceFactory.java:150) ~[isomorphic_core_rpc.jar:?] at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220) ~[commons-pool-1.6.jar:1.6] at com.isomorphic.pool.PoolManager.borrowObject(PoolManager.java:85) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:195) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:121) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.taglib.LoadDSTag.outputDSAsJS(LoadDSTag.java:130) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.taglib.LoadDSTag.outputJS(LoadDSTag.java:103) ~[isomorphic_core_rpc.jar:?] at com.isomorphic.taglib.LoadDSTag.doStartTag(LoadDSTag.java:59) ~[isomorphic_core_rpc.jar:?] at org.apache.jsp.home_jsp._jspx_meth_isomorphic_005floadDS_005f0(home_jsp.java:994) ~[?:?]

This stack is recurring 280+ times and finally the jetty classloading is called. The stack is now full with the recursion and jetty is unable to load anything. Also when I debugged the smartclient doesnt go through this loop in a working scenario. It doesnt hit code block when its working
if (null == latch.getPair()) {

try {

V obj = _factory.makeObject(key);

latch.setPair(new ObjectTimestampPair<V>(obj));

newlyCreated = true;

} finally {

if (!newlyCreated) {

// object cannot be created

synchronized (this) {

latch.getPool().decrementInternalProcessingCount();

// No need to reset latch - about to throw exception

}

allocate();

}

}

}

So for some reason its not able to initialize the latch and going into this loop of creating a object.

[Isomorphic]

This is still likely a Jetty ClassLoading issue. You have to ask: why is Jetty still reloading classes after 280 passes through the exact same methods? They should all be loaded already.

And if Jetty is confused and has a corrupted class cache, causing it to reload classes that are already loaded, that would wipe out static state (like caches!) causing this loop.

So the troubleshooting steps remain the same. You seem to have ignore those steps and tried something else, but if you actually want to fix this, you should be looking into what we suggested.

[msule]

1. We verified and dont see any conflicting Jetty version. Its all the expected jars in classpath.
2. No hot deploy of jars or code. We are running the instance for 3-4 days and all of a sudden system starts to fail with this error. We analysed the memory and we have enough. But since the stack doesnt come under the heap we are unable to understand why it would fail to create a object(if that is what the code is doing)
3. We reached out to Jetty as well and this is the summary of what they said.
"The vast bulk of the stack is from com.isomorphic.datasource and is recursing loading an instance, and has nothing to do with Jetty. The top of the stack is 41 frames as the Jetty classloader is asked to load a particular class. This is just a fairly normal traverse of a tree structure that has a depth of 41. We should not need to do it for an isEmpty check, and I am going to update our code to avoid doing so. However, those 41 frames are just the innocent bystanders of the huge stack recursion from com.isomorphic.datasource. If it was not those stacks, then it would be some other method on the 281st repetition.

So you need to raise an issue with com.isomorphic.datasource and/or look at your data structures for very deep recursive entries.

As I said, meanwhile I will update our TreeTrie to be more efficient, but it is not the cause of this problem."


So wrt the analysis done so far the code is trying to load our first datasource and throws this stackoverflow error.

[Isomorphic]

Jetty has not addressed the issue of why Jetty is continuing to load / reload classes in the middle of a recursive loop that is accessing the exact same classes and methods repeatedly, as we mentioned above.

Presumably you didn't ask them that question - you should do so now if not.

[Isomorphic]

Note that you can also turn on server-side logging (https://smartclient.com/smartclient-....serverLogging) to see which specific DataSources are being repeatedly reloaded.

You can also turn on classloader logging at the JVM level (note this would be a lot of output).

We expect you'll see a short loop involving the same set of DataSources, probably low-level internal ones. And you will see that Jetty is repeatedly re-loading the same classes over and over.

That breaks our caching of already-loaded DataSources by wiping out statically stored data, hence the loop.

Some information from AI on how to enable classloader tracing on a couple of different JVMs. The Jetty team might be able to give you a more effective tracing approach.

Add JVM classloading trace:

• HotSpot: -Xlog:class+load=info
• OpenJ9: -Xtrace:print=mt,trigger=method{java/lang/ClassLoader.loadClass}
  You’ll see class loads interleaved after many DS re-entries, not at the beginning.

[Isomorphic]

Just following up on this - did you fix the issue? It would be good to know if there is a now-fixed Jetty issue that we should warn other customers about.