Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
Clear All
new posts

    Deadlocks during loading of dynamic datasources

    In our application almost all our datasources are dynamic datasources. If we load such a datasource using DataSource.forName when another datasource is executing we get some problems, it's a bit hard to be specific there is a lot going on.

    This is the deadlock we are running into

    Code:
    Found one Java-level deadlock:
    =============================
    "btpool0-8":
      waiting to lock monitor 7fc61f024848 (object 7989bc4c0, a com.isomorphic.datasource.PoolableDataSourceFactory),
      which is held by "btpool0-5"
    "btpool0-5":
      waiting to lock monitor 7fc61f0248f0 (object 7e29882c8, a java.lang.Class),
      which is held by "btpool0-8"
    
    Java stack information for the threads listed above:
    ===================================================
    "btpool0-8":
    	at com.isomorphic.datasource.PoolableDataSourceFactory.makeUnpooledObject(PoolableDataSourceFactory.java:129)
    	- waiting to lock <7989bc4c0> (a com.isomorphic.datasource.PoolableDataSourceFactory)
    	at com.isomorphic.datasource.PoolableDataSourceFactory.makeObject(PoolableDataSourceFactory.java:138)
    	at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1179)
    	at com.isomorphic.pool.PoolManager.borrowObject(PoolManager.java:84)
    	at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:95)
    	at com.isomorphic.datasource.ValidationContext.getType(ValidationContext.java:287)
    	at com.isomorphic.datasource.BasicDataSource.findDataSource(BasicDataSource.java:2182)
    	at com.isomorphic.datasource.BasicDataSource.getType(BasicDataSource.java:2069)
    	at com.isomorphic.datasource.BasicDataSource.findElementType(BasicDataSource.java:1819)
    	at com.isomorphic.datasource.BasicDataSource.elementAsRecord(BasicDataSource.java:1195)
    	at com.isomorphic.datasource.BasicDataSource.toRecord(BasicDataSource.java:1107)
    	at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1091)
    	at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1052)
    	at com.isomorphic.datasource.DataSource.create(DataSource.java:1280)
    	at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:1619)
    	at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:1566)
    	at com.isomorphic.datasource.BasicDataSource.elementAsRecord(BasicDataSource.java:1399)
    	at com.isomorphic.datasource.BasicDataSource.toRecord(BasicDataSource.java:1107)
    	at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1066)
    	at com.isomorphic.datasource.DataSource.recordsFromXML(DataSource.java:1236)
    	at com.isomorphic.xml.XML.toDSRecords(XML.java:337)
    	at com.isomorphic.xml.XML.toDSRecords(XML.java:328)
    	at com.isomorphic.xml.XML.toDSRecords(XML.java:324)
    	at com.isomorphic.xml.XML.toDSRecords(XML.java:316)
    	at com.isomorphic.datasource.DataSource.fromXML(DataSource.java:606)
    	at com.isomorphic.datasource.DataSource.fromXML(DataSource.java:575)
    	at com.genohm.slims.server.datasource.generator.DataSourceConverter.convertToDs(DataSourceConverter.java:44)
    	at com.genohm.slims.server.datasource.generator.DataSourceConverter.getDataSource(DataSourceConverter.java:36)
    	at com.isomorphic.datasource.DataSource.getDynamicDataSource(DataSource.java:390)
    	- locked <7e29882c8> (a java.lang.Class for com.isomorphic.datasource.DataSource)
    	at com.isomorphic.datasource.DataSource.forName(DataSource.java:226)
    	at com.isomorphic.datasource.DataSource.forName(DataSource.java:220)
    	at com.isomorphic.datasource.DataSource.forName(DataSource.java:209)
    	at com.isomorphic.datasource.DataSource.forName(DataSource.java:205)
    	at com.genohm.slims.server.dmi.specific.RequestableDMI$1.execute(RequestableDMI.java:52)
    	at com.genohm.slims.server.dmi.specific.RequestableDMI$1.execute(RequestableDMI.java:1)
    	at com.genohm.slims.server.dmi.DSResponseUtil.wrapServiceCall(DSResponseUtil.java:144)
    	at com.genohm.slims.server.dmi.specific.RequestableDMI.getAvailableRequestables(RequestableDMI.java:38)
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    	at java.lang.reflect.Method.invoke(Method.java:597)
    	at com.isomorphic.base.Reflection.adaptArgsAndInvoke(Reflection.java:973)
    	at com.isomorphic.datasource.DataSourceDMI.execute(DataSourceDMI.java:416)
    	at com.isomorphic.datasource.DataSourceDMI.execute(DataSourceDMI.java:64)
    	at com.isomorphic.datasource.DSRequest.execute(DSRequest.java:2427)
    	at com.isomorphic.servlet.IDACall.handleDSRequest(IDACall.java:215)
    	at com.isomorphic.servlet.IDACall.processRPCTransaction(IDACall.java:172)
    	at com.isomorphic.servlet.IDACall.processRequest(IDACall.java:137)
    	at com.isomorphic.servlet.IDACall.doPost(IDACall.java:73)
    	at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
    	at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:152)
    	at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
    	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
    	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1097)
    	at com.isomorphic.servlet.CompressionFilter.doFilter(CompressionFilter.java:260)
    	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1088)
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:322)
    	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
    	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    	at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    	at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182)
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:184)
    	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:155)
    	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
    	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
    	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1088)
    	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360)
    	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
    	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
    	at org.mortbay.jetty.handler.ContextHandler.__handle(ContextHandler.java:729)
    	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java)
    	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
    	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    	at org.mortbay.jetty.handler.RequestLogHandler.handle(RequestLogHandler.java:49)
    	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    	at org.mortbay.jetty.Server.handle(Server.java:324)
    	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505)
    	at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:843)
    	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:647)
    	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211)
    	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380)
    	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:395)
    	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:488)
    "btpool0-5":
    	at com.isomorphic.datasource.DataSource.getDynamicDataSource(DataSource.java:363)
    	- waiting to lock <7e29882c8> (a java.lang.Class for com.isomorphic.datasource.DataSource)
    	at com.isomorphic.datasource.DataSource.forName(DataSource.java:226)
    	at com.isomorphic.datasource.DataSource.forName(DataSource.java:220)
    	at com.isomorphic.datasource.DataSource.forName(DataSource.java:209)
    	at com.isomorphic.datasource.PoolableDataSourceFactory.makeUnpooledObject(PoolableDataSourceFactory.java:130)
    	- locked <7989bc4c0> (a com.isomorphic.datasource.PoolableDataSourceFactory)
    	at com.isomorphic.pool.PoolManager.borrowUnpooledObject(PoolManager.java:123)
    	at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:93)
    	at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:86)
    	at com.isomorphic.servlet.DataSourceLoader.processRequest(DataSourceLoader.java:132)
    	at com.isomorphic.servlet.DataSourceLoader.doGet(DataSourceLoader.java:96)
    	at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
    	at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:152)
    	at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
    	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
    	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1097)
    	at com.isomorphic.servlet.CompressionFilter.doFilter(CompressionFilter.java:247)
    	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1088)
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:322)
    	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
    	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    	at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    	at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182)
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
    	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
    	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:184)
    	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:155)
    	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
    	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
    	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1088)
    	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360)
    	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
    	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
    	at org.mortbay.jetty.handler.ContextHandler.__handle(ContextHandler.java:729)
    	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java)
    	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
    	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    	at org.mortbay.jetty.handler.RequestLogHandler.handle(RequestLogHandler.java:49)
    	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
    	at org.mortbay.jetty.Server.handle(Server.java:324)
    	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505)
    	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:829)
    	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:513)
    	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211)
    	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380)
    	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:395)
    	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:488)
    So the first thread is loading a dynamic datasource, while the second thread is executing a DMI call that uses DataSource.fromXml("Some dynamic datasource" , (at this line: at com.genohm.slims.server.dmi.specific.RequestableDMI$1.execute(RequestableDMI.java:52))

    So one of the locks comes from a synchronized method in PoolableDataSourceFactory

    Code:
     public synchronized Object makeUnpooledObject(Object key)
        throws Exception
      {
        Boolean isSchemaless = (Boolean)this.isSchemaless.get();
        return DataSource.forName(key.toString(), (DSRequest)this.dsRequest.get(), isSchemaless == null ? false : isSchemaless.booleanValue());
      }
    My question here is, to my untrained eye it looks like the synchronized here is not needed since both isSchemaLess and dsRequest are ThreadLocals. Removing this would solve my deadlock problem. Do you think it's correct to remove the synchronized here?

    The SmartGWT version we are using is SmartClient Version: v9.0p_2013-12-06/PowerEdition Deployment (built 2013-12-06), but the same "synchronized" is still present in the latest builds.

    #2
    Upgrade to the latest nightly build, I had the same issue which was fixed a few weeks ago.

    Comment


      #3
      Just to concur - yes, it looks like you are running into an already-fixed synchronization issue, and you should update to a new build (where the code is a bit different as well).

      Comment


        #4
        I updated to the latests 4.0 function, and still have the same problem. The deadlock looks a bit different though

        Code:
        Found one Java-level deadlock:
        =============================
        "btpool0-5":
          waiting to lock monitor 7fa6cc889348 (object 7e2a3eb88, a java.lang.Class),
          which is held by "btpool0-4"
        "btpool0-4":
          waiting to lock monitor 7fa6d08043f0 (object 7e63cf150, a java.lang.Class),
          which is held by "btpool0-5"
        
        Java stack information for the threads listed above:
        
        "btpool0-5":
        	at com.isomorphic.datasource.DataSource.getDynamicDataSource(DataSource.java:363)
        	- waiting to lock <7e2a3eb88> (a java.lang.Class for com.isomorphic.datasource.DataSource)
        	at com.isomorphic.datasource.DataSource.forName(DataSource.java:226)
        	at com.isomorphic.datasource.DataSource.forName(DataSource.java:220)
        	at com.isomorphic.datasource.DataSource.forName(DataSource.java:209)
        	at com.isomorphic.datasource.PoolableDataSourceFactory.makeUnpooledObject(PoolableDataSourceFactory.java:130)
        	- locked <795629308> (a com.isomorphic.datasource.PoolableDataSourceFactory)
        	at com.isomorphic.datasource.PoolableDataSourceFactory.makeObject(PoolableDataSourceFactory.java:138)
        	at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1179)
        	at com.isomorphic.pool.ISCKeyedObjectPool.borrowObject(ISCKeyedObjectPool.java:106)
        	- locked <7941d8300> (a com.isomorphic.pool.ISCKeyedObjectPool)
        	at com.isomorphic.pool.PoolManager.borrowObject(PoolManager.java:84)
        	at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:135)
        	- locked <7e63cf150> (a java.lang.Class for com.isomorphic.datasource.DataSourceManager)
        	at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:86)
        	at com.isomorphic.servlet.DataSourceLoader.processRequest(DataSourceLoader.java:132)
        	at com.isomorphic.servlet.DataSourceLoader.doGet(DataSourceLoader.java:96)
        	at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
        	at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:152)
        	at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
        	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1097)
        	at com.isomorphic.servlet.CompressionFilter.doFilter(CompressionFilter.java:247)
        	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1088)
        	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:322)
        	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
        	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
        	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
        	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
        	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
        	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        	at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
        	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        	at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
        	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182)
        	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
        	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
        	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:184)
        	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:155)
        	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
        	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
        	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1088)
        	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360)
        	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
        	at org.mortbay.jetty.handler.ContextHandler.__handle(ContextHandler.java:729)
        	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java)
        	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
        	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        	at org.mortbay.jetty.handler.RequestLogHandler.handle(RequestLogHandler.java:49)
        	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        	at org.mortbay.jetty.Server.handle(Server.java:324)
        	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505)
        	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:829)
        	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:513)
        	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211)
        	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380)
        	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:395)
        	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:488)
        "btpool0-4":
        	at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:125)
        	- waiting to lock <7e63cf150> (a java.lang.Class for com.isomorphic.datasource.DataSourceManager)
        	at com.isomorphic.datasource.ValidationContext.getType(ValidationContext.java:287)
        	at com.isomorphic.datasource.ValidationContext.getType(ValidationContext.java:267)
        	at com.isomorphic.datasource.DataSource.getElementType(DataSource.java:1231)
        	at com.isomorphic.datasource.DataSource.recordsFromXML(DataSource.java:1247)
        	at com.isomorphic.xml.XML.toDSRecords(XML.java:337)
        	at com.isomorphic.xml.XML.toDSRecords(XML.java:328)
        	at com.isomorphic.xml.XML.toDSRecords(XML.java:324)
        	at com.isomorphic.xml.XML.toDSRecords(XML.java:316)
        	at com.isomorphic.datasource.DataSource.fromXML(DataSource.java:651)
        	at com.isomorphic.datasource.DataSource.fromXML(DataSource.java:635)
        	at com.isomorphic.datasource.DataSource.fromXML(DataSource.java:604)
        	at com.genohm.slims.server.datasource.generator.DataSourceConverter.convertToDs(DataSourceConverter.java:44)
        	at com.genohm.slims.server.datasource.generator.DataSourceConverter.getDataSource(DataSourceConverter.java:36)
        	at com.isomorphic.datasource.DataSource.getDynamicDataSource(DataSource.java:390)
        	- locked <7e2a3eb88> (a java.lang.Class for com.isomorphic.datasource.DataSource)
        	at com.isomorphic.datasource.DataSource.forName(DataSource.java:226)
        	at com.isomorphic.datasource.DataSource.forName(DataSource.java:220)
        	at com.isomorphic.datasource.DataSource.forName(DataSource.java:209)
        	at com.isomorphic.datasource.DataSource.forName(DataSource.java:205)
        	at com.genohm.slims.server.dmi.specific.RequestableDMI$1.execute(RequestableDMI.java:52)
        	at com.genohm.slims.server.dmi.specific.RequestableDMI$1.execute(RequestableDMI.java:1)
        	at com.genohm.slims.server.dmi.DSResponseUtil.wrapServiceCall(DSResponseUtil.java:144)
        	at com.genohm.slims.server.dmi.specific.RequestableDMI.getAvailableRequestables(RequestableDMI.java:38)
        	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        	at java.lang.reflect.Method.invoke(Method.java:597)
        	at com.isomorphic.base.Reflection.adaptArgsAndInvoke(Reflection.java:973)
        	at com.isomorphic.datasource.DataSourceDMI.execute(DataSourceDMI.java:416)
        	at com.isomorphic.datasource.DataSourceDMI.execute(DataSourceDMI.java:64)
        	at com.isomorphic.datasource.DSRequest.execute(DSRequest.java:2430)
        	at com.isomorphic.servlet.IDACall.handleDSRequest(IDACall.java:215)
        	at com.isomorphic.servlet.IDACall.processRPCTransaction(IDACall.java:172)
        	at com.isomorphic.servlet.IDACall.processRequest(IDACall.java:137)
        	at com.isomorphic.servlet.IDACall.doPost(IDACall.java:73)
        	at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
        	at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:152)
        	at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        	at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
        	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1097)
        	at com.isomorphic.servlet.CompressionFilter.doFilter(CompressionFilter.java:260)
        	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1088)
        	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:322)
        	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:116)
        	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
        	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
        	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
        	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
        	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        	at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
        	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        	at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
        	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        	at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:182)
        	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
        	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
        	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:184)
        	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:155)
        	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
        	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
        	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1088)
        	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:360)
        	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
        	at org.mortbay.jetty.handler.ContextHandler.__handle(ContextHandler.java:729)
        	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java)
        	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
        	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        	at org.mortbay.jetty.handler.RequestLogHandler.handle(RequestLogHandler.java:49)
        	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        	at org.mortbay.jetty.Server.handle(Server.java:324)
        	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:505)
        	at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:843)
        	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:647)
        	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:211)
        	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:380)
        	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:395)
        	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:488)
        
        Found 1 deadlock.

        Comment


          #5
          The original deadlock issue that JLivermore refers to seems like the same issue you're experiencing here, so please first assure yourself that you are definitely picking up the latest code and don't have anything hanging over from a previous build. If you're happy that this isn't the case, please provide more information to help us reproduce the problem - show the code that is executing when the deadlock occurs, as well as any important context, like the .ds.xml files or dynamically-generated XML that is involved, and any server-side DMI code that is involved with the deadlocking dataSources. If you don't want to post those things publicly, you can email them to support@isomorphic.com, referencing this thread.

          Thanks,
          Isomorphic Software Support

          Comment


            #6
            Actually, looking more closely at the thread dump we see what appears to be the problem. Your original post claimed that this line is calling DataSource.fromXML():
            Code:
            at com.genohm.slims.server.dmi.specific.RequestableDM I$1.execute(RequestableDMI.java:52)
            In fact, it is calling DataSource.forName(), which is not allowed (forName() is an internal API that should only be called by framework code). The supported way to obtain a DataSource instance by name is DataSourceManager.getDataSource()

            Comment


              #7
              Sorry for the confusion, i was indeed using DataSource.forName() switching it to DataSourceManager.getDataSource() indeed resolved my problem. Thanks!

              Comment

              Working...
              X