In our application almost all our datasources are dynamic datasources. If we load such a datasource using DataSource.forName when another datasource is executing we get some problems, it's a bit hard to be specific there is a lot going on.
This is the deadlock we are running into
So the first thread is loading a dynamic datasource, while the second thread is executing a DMI call that uses DataSource.fromXml("Some dynamic datasource" , (at this line: at com.genohm.slims.server.dmi.specific.RequestableDMI$1.execute(RequestableDMI.java:52))
So one of the locks comes from a synchronized method in PoolableDataSourceFactory
My question here is, to my untrained eye it looks like the synchronized here is not needed since both isSchemaLess and dsRequest are ThreadLocals. Removing this would solve my deadlock problem. Do you think it's correct to remove the synchronized here?
The SmartGWT version we are using is SmartClient Version: v9.0p_2013-12-06/PowerEdition Deployment (built 2013-12-06), but the same "synchronized" is still present in the latest builds.
This is the deadlock we are running into
Code:
Found one Java-level deadlock: ============================= "btpool0-8": waiting to lock monitor 7fc61f024848 (object 7989bc4c0, a com.isomorphic.datasource.PoolableDataSourceFactory), which is held by "btpool0-5" "btpool0-5": waiting to lock monitor 7fc61f0248f0 (object 7e29882c8, a java.lang.Class), which is held by "btpool0-8" Java stack information for the threads listed above: =================================================== "btpool0-8": at com.isomorphic.datasource.PoolableDataSourceFactory.makeUnpooledObject(PoolableDataSourceFactory.java:129) - waiting to lock <7989bc4c0> (a com.isomorphic.datasource.PoolableDataSourceFactory) at com.isomorphic.datasource.PoolableDataSourceFactory.makeObject(PoolableDataSourceFactory.java:138) at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1179) at com.isomorphic.pool.PoolManager.borrowObject(PoolManager.java:84) at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:95) at com.isomorphic.datasource.ValidationContext.getType(ValidationContext.java:287) at com.isomorphic.datasource.BasicDataSource.findDataSource(BasicDataSource.java:2182) at com.isomorphic.datasource.BasicDataSource.getType(BasicDataSource.java:2069) at com.isomorphic.datasource.BasicDataSource.findElementType(BasicDataSource.java:1819) at com.isomorphic.datasource.BasicDataSource.elementAsRecord(BasicDataSource.java:1195) at com.isomorphic.datasource.BasicDataSource.toRecord(BasicDataSource.java:1107) at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1091) at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1052) at com.isomorphic.datasource.DataSource.create(DataSource.java:1280) at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:1619) at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:1566) at com.isomorphic.datasource.BasicDataSource.elementAsRecord(BasicDataSource.java:1399) at com.isomorphic.datasource.BasicDataSource.toRecord(BasicDataSource.java:1107) at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1066) at com.isomorphic.datasource.DataSource.recordsFromXML(DataSource.java:1236) at com.isomorphic.xml.XML.toDSRecords(XML.java:337) at com.isomorphic.xml.XML.toDSRecords(XML.java:328) at com.isomorphic.xml.XML.toDSRecords(XML.java:324) at com.isomorphic.xml.XML.toDSRecords(XML.java:316) at com.isomorphic.datasource.DataSource.fromXML(DataSource.java:606) at com.isomorphic.datasource.DataSource.fromXML(DataSource.java:575) at com.genohm.slims.server.datasource.generator.DataSourceConverter.convertToDs(DataSourceConverter.java:44) at com.genohm.slims.server.datasource.generator.DataSourceConverter.getDataSource(DataSourceConverter.java:36) at com.isomorphic.datasource.DataSource.getDynamicDataSource(DataSource.java:390) - locked <7e29882c8> (a java.lang.Class for com.isomorphic.datasource.DataSource) at com.isomorphic.datasource.DataSource.forName(DataSource.java:226) at com.isomorphic.datasource.DataSource.forName(DataSource.java:220) at com.isomorphic.datasource.DataSource.forName(DataSource.java:209) at com.isomorphic.datasource.DataSource.forName(DataSource.java:205) at com.genohm.slims.server.dmi.specific.RequestableDMI$1.execute(RequestableDMI.java:52) at com.genohm.slims.server.dmi.specific.RequestableDMI$1.execute(RequestableDMI.java:1) at com.genohm.slims.server.dmi.DSResponseUtil.wrapServiceCall(DSResponseUtil.java:144) at com.genohm.slims.server.dmi.specific.RequestableDMI.getAvailableRequestables(RequestableDMI.java:38) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at com.isomorphic.base.Reflection.adaptArgsAndInvoke(Reflection.java:973) at com.isomorphic.datasource.DataSourceDMI.execute(DataSourceDMI.java:416) at com.isomorphic.datasource.DataSourceDMI.execute(DataSourceDMI.java:64) at com.isomorphic.datasource.DSRequest.execute(DSRequest.java:2427) at com.isomorphic.servlet.IDACall.handleDSRequest(IDACall.java:215) at com.isomorphic.servlet.IDACall.processRPCTransaction(IDACall.java:172) at com.isomorphic.servlet.IDACall.processRequest(IDACall.java:137) at com.isomorphic.servlet.IDACall.doPost(IDACall.java:73) at javax.servlet.http.HttpServlet.service(HttpServlet.java:637) at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:152) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1097) at com.isomorphic.servlet.CompressionFilter.doFilter(CompressionFilter.java:260) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1088) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:322) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:184) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:155) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1088) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) at org.mortbay.jetty.handler.ContextHandler.__handle(ContextHandler.java:729) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.handler.RequestLogHandler.handle(RequestLogHandler.java:49) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:324) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505) at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:843) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:647) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:395) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:488) "btpool0-5": at com.isomorphic.datasource.DataSource.getDynamicDataSource(DataSource.java:363) - waiting to lock <7e29882c8> (a java.lang.Class for com.isomorphic.datasource.DataSource) at com.isomorphic.datasource.DataSource.forName(DataSource.java:226) at com.isomorphic.datasource.DataSource.forName(DataSource.java:220) at com.isomorphic.datasource.DataSource.forName(DataSource.java:209) at com.isomorphic.datasource.PoolableDataSourceFactory.makeUnpooledObject(PoolableDataSourceFactory.java:130) - locked <7989bc4c0> (a com.isomorphic.datasource.PoolableDataSourceFactory) at com.isomorphic.pool.PoolManager.borrowUnpooledObject(PoolManager.java:123) at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:93) at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:86) at com.isomorphic.servlet.DataSourceLoader.processRequest(DataSourceLoader.java:132) at com.isomorphic.servlet.DataSourceLoader.doGet(DataSourceLoader.java:96) at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:152) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1097) at com.isomorphic.servlet.CompressionFilter.doFilter(CompressionFilter.java:247) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1088) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:322) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116) at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87) at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334) at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:184) at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:155) at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237) at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1088) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181) at org.mortbay.jetty.handler.ContextHandler.__handle(ContextHandler.java:729) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.handler.RequestLogHandler.handle(RequestLogHandler.java:49) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:324) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505) at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:829) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:513) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380) at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:395) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:488)
So one of the locks comes from a synchronized method in PoolableDataSourceFactory
Code:
public synchronized Object makeUnpooledObject(Object key)
throws Exception
{
Boolean isSchemaless = (Boolean)this.isSchemaless.get();
return DataSource.forName(key.toString(), (DSRequest)this.dsRequest.get(), isSchemaless == null ? false : isSchemaless.booleanValue());
}
The SmartGWT version we are using is SmartClient Version: v9.0p_2013-12-06/PowerEdition Deployment (built 2013-12-06), but the same "synchronized" is still present in the latest builds.
Comment