Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
Clear All
new posts

    Thread deadloc, Not able to figure out the RCA as we have a lot of DataSources and application has been live in production for many years but issues started few months back after upgrading to v12.0p_2019-05-17




    "http-nio-9079-exec-5" #107 daemon prio=5 os_prio=0 tid=0x00007f4598017000 nid=0x10222 waiting for monitor entry [0x00007f4527dfa000]
    java.lang.Thread.State: BLOCKED (on object monitor)
    at com.isomorphic.datasource.SimpleType.validateValue(SimpleType.java:91)
    - waiting to lock <0x00000005ce004548> (a com.isomorphic.datasource.SimpleType)
    at com.isomorphic.datasource.SimpleType.create(SimpleType.java:75)
    at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:2363)
    at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:2278)
    at com.isomorphic.datasource.BasicDataSource.elementAsRecord(BasicDataSource.java:2104)
    at com.isomorphic.datasource.BasicDataSource.toRecord(BasicDataSource.java:1476)
    at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1460)
    at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1421)
    at com.isomorphic.datasource.DataSource.create(DataSource.java:2141)
    at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:2363)
    at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:2278)
    at com.isomorphic.datasource.BasicDataSource.elementAsRecord(BasicDataSource.java:2020)
    at com.isomorphic.datasource.BasicDataSource.toRecord(BasicDataSource.java:1476)
    at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1435)
    at com.isomorphic.datasource.DataSource.recordsFromXML(DataSource.java:2054)
    at com.isomorphic.datasource.IsomorphicXML.toRecord(IsomorphicXML.java:97)
    at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1435)
    at com.isomorphic.datasource.DataSource.recordsFromXML(DataSource.java:2054)
    at com.isomorphic.taglib.WidgetXMLTag.toJS(WidgetXMLTag.java:148)
    at com.isomorphic.taglib.WidgetXMLTag.parseAndOutputErrors(WidgetXMLTag.java:125)
    at com.isomorphic.taglib.WidgetXMLTag.doAfterBody(WidgetXMLTag.java:78)
    at org.apache.jsp.home_jsp._jspx_meth_isomorphic_005fXML_005f0(home_jsp.java:2357)
    at org.apache.jsp.home_jsp._jspService(home_jsp.java:881)
    at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:443)
    at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:385)
    at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:329)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
    at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176)
    at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)
    at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)
    at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:389)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
    at com.ias.filter.CorsFilter.doFilterInternal(CorsFilter.java:29)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
    at com.isomorphic.servlet.CompressionFilter._doFilter(CompressionFilter.java:260)
    at com.isomorphic.servlet.BaseFilter.doFilter(BaseFilter.java:93)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
    at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:265)
    at org.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:107)
    at org.acegisecurity.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:72)
    at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
    at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
    at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
    at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:124)
    at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
    at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
    at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
    at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174)
    at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
    at org.acegisecurity.ui.logout.LogoutFilter.doFilter(LogoutFilter.java:110)
    at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
    at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
    at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
    at com.ias.dm.login.InternationalizationFilter.doFilterInternal(InternationalizationFilter.java:66)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
    at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
    at org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:149)
    at org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
    at com.ias.dispatchmanager.cas.authentication.SessionTimeoutFilter.doFilter(SessionTimeoutFilter.java:47)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
    at org.jasig.cas.client.session.SingleSignOutFilter.doFilter(SingleSignOutFilter.java:106)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:108)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:620)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349)
    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:784)
    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:802)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1410)
    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
    - locked <0x00000005ce165578> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)

    "http-nio-9079-exec-13":

    at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:167)

    - waiting to lock <0x00000005ce0009c0> (a java.lang.Class for com.isomorphic.datasource.DataSourceManager)

    at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:121)

    at com.isomorphic.datasource.DSRequest.getDataSource(DSRequest.java:2356)

    at com.isomorphic.datasource.DSRequest.setDataSourceName(DSRequest.java:2330)

    at com.isomorphic.datasource.DSRequest.<init>(DSRequest.java:436)

    at com.isomorphic.datasource.DataSource.fetchById(DataSource.java:6939)

    at com.isomorphic.velocity.StoredRecordHandler.fetchStoredRecord(StoredRecordHandler.java:188)

    at com.isomorphic.velocity.StoredRecordHandler.size(StoredRecordHandler.java:234)

    at java.util.HashMap.putMapEntries(HashMap.java:501)

    at java.util.HashMap.<init>(HashMap.java:490)

    at com.isomorphic.velocity.EditedRecordHandler.prepareCombinedRecord(EditedRecordHandler.java:91)

    at com.isomorphic.velocity.EditedRecordHandler.put(EditedRecordHandler.java:177)

    at com.isomorphic.datasource.ValidationContext.setResultingValue(ValidationContext.java:292)

    at com.isomorphic.util.DefaultValidators.validateField(DefaultValidators.java:192)

    at com.isomorphic.datasource.SimpleType.validateValue(SimpleType.java:101)

    - locked <0x00000005ce004548> (a com.isomorphic.datasource.SimpleType)

    at com.isomorphic.datasource.SimpleType.create(SimpleType.java:75)

    at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:2363)

    at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:2278)

    at com.isomorphic.datasource.BasicDataSource.elementAsRecord(BasicDataSource.java:2104)

    at com.isomorphic.datasource.BasicDataSource.toRecord(BasicDataSource.java:1476)

    at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1460)

    at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1421)

    at com.isomorphic.datasource.DataSource.create(DataSource.java:2141)

    at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:2363)

    at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:2278)

    at com.isomorphic.datasource.BasicDataSource.elementAsRecord(BasicDataSource.java:2020)

    at com.isomorphic.datasource.BasicDataSource.toRecord(BasicDataSource.java:1476)

    at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1435)

    at com.isomorphic.datasource.DataSource.recordsFromXML(DataSource.java:2054)

    at com.isomorphic.datasource.IsomorphicXML.toRecord(IsomorphicXML.java:97)

    at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1435)

    at com.isomorphic.datasource.DataSource.recordsFromXML(DataSource.java:2054)

    at com.isomorphic.taglib.WidgetXMLTag.toJS(WidgetXMLTag.java:148)

    at com.isomorphic.taglib.WidgetXMLTag.parseAndOutputErrors(WidgetXMLTag.java:125)

    at com.isomorphic.taglib.WidgetXMLTag.doAfterBody(WidgetXMLTag.java:78)

    at org.apache.jsp.home_jsp._jspx_meth_isomorphic_005fXML_005f0(home_jsp.java:2357)

    at org.apache.jsp.home_jsp._jspService(home_jsp.java:881)

    at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)

    at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:443)

    at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:385)

    at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:329)

    at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:230)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)

    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)

    at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:176)

    at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)

    at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)

    at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:389)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)

    at com.ias.filter.CorsFilter.doFilterInternal(CorsFilter.java:29)

    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)

    at com.isomorphic.servlet.CompressionFilter._doFilter(CompressionFilter.java:260)

    at com.isomorphic.servlet.BaseFilter.doFilter(BaseFilter.java:93)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)

    at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:265)

    at org.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:107)

    at org.acegisecurity.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:72)

    at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)

    at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)

    at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)

    at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:124)

    at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)

    at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)

    at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)

    at org.acegisecurity.ui.basicauth.BasicProcessingFilter.doFilter(BasicProcessingFilter.java:174)

    at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)

    at org.acegisecurity.ui.logout.LogoutFilter.doFilter(LogoutFilter.java:110)

    at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)

    at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)

    at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)

    at com.ias.dm.login.InternationalizationFilter.doFilterInternal(InternationalizationFilter.java:66)

    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)

    at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)

    at org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:149)

    at org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)

    at com.ias.dispatchmanager.cas.authentication.SessionTimeoutFilter.doFilter(SessionTimeoutFilter.java:47)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)

    at org.jasig.cas.client.session.SingleSignOutFilter.doFilter(SingleSignOutFilter.java:106)

    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)

    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)

    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:198)

    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:108)

    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)

    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)

    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)

    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:620)

    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)

    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349)

    at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:784)

    at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)

    at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:802)

    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1410)

    at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

    - locked <0x00000005ce32ec60> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)

    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

    at java.lang.Thread.run(Thread.java:748)




    Found 1 deadlock.


    #2
    What release were you using previously?

    Comment


      #3

      We had a build from 2013 before this: v90p_2013-08-17. To give you more context, we also moved from WebLogic to Tomcat and upgraded from Java 1.6 to Java 1.8 in the same release.

      Comment


        #4
        It's impossible to know what changed in 6 years to cause this problem, but it's common when upgrading, to encounter issues from longstanding incorrect usage, where there is no bug in the framework. We have no similar reports from any other customer, and our load tests are all passing. You could try it with the current patched release of 12.0. If that doesn't fix things, right now the options are:

        1. analyze further on your own, and see if you can come up with a test case showing that the framework deadlocks when all of your usage is clearly correct

        or

        2. buy a block of consulting hours so we can dive into your application code and try to figure out what's going on

        Comment

        Working...
        X