Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
Clear All
new posts

    Problem with Apache DBCP and maxed out pool

    Hi Isomorphic,

    it seems we do have a problem with current 12.0p (v12.0p_2019-03-23) and the shipped commons-dbcp-1.4.jar.
    It is likely that this is related to this thread and the DBCP issue you mention in #6 there.

    Release notes for unreleased Apache Commons DBCP 1.4.1 mention the bug that 1.2.2 claims to have fixed. It contains this fix which claims that the fix in 1.2.2 is incomplete. Now 1.4.1 is unreleased, so there is not much you can do here. I did ask for a release date, though. But I do assume that DBCP 2 (new namespace) fixes it as well, as this is in active development (3 releases in 2018, 1 in 2019).

    We do have the configuration in place that is said to cause the issue (positive minIdle and timeBetweenEvictionRunsMillis):
    Code:
    sql.defaultDatabase: Oracle
    sql.useAnsiJoins: true
    
    # Apache Commons DBCP Database connection pooling
    sql.pool.enabled: true
    [B]sql.pool.minIdle: 1[/B]
    sql.pool.maxActive: 10
    sql.pool.whenExhaustedAction: block
    sql.pool.testWhileIdle: true
    [B]sql.pool.timeBetweenEvictionRunsMillis: 15000[/B]
    sql.pool.minEvictableIdleTimeMillis: 29000
    
    sql.Oracle.database.type: oracle
    sql.Oracle.autoJoinTransactions: true
    sql.Oracle.database.supportsSQLLimit: false
    
    sql.Oracle.interface.credentialsInURL: true
    sql.Oracle.interface.type: dataSource
    
    sql.Oracle.pingTest: select 1 from dual
    
    sql.Oracle.driver: oracle.jdbc.pool.OracleDataSource
    sql.Oracle.driver.url: jdbc:oracle:thin:@xxxxxx:1521/DB11GXDB.localdomain
    sql.Oracle.driver.user: xxx
    sql.Oracle.driver.password: xxx
    
    sql.Oracle.driver.driverType: thin
    sql.Oracle.driver.networkProtocol: tcp
    sql.Oracle.driver.context:
    Regarding the issue itself:
    We run many instances of our application as own WAR files. These application run on the same Tomcat (9.0.17). Now if the issue happens, one application (server.xml <Host> entry) maxes out it connections. New requests will hang (stack trace follows). This in turn maxes out our Tomcat threads (maxThreads here) eventually, bringing all instances effectively down, as no more threads can be created.

    We restarted our application to prevent the "whole Tomcat down" issue from happening.
    At that time, we had many (~50) BLOCKED threads, most of them with a stack trace where the top line is like this (there are of course many different codepaths that use DB connections in the end, so the whole stack is always a bit different:
    Code:
    [B]Example 1:[/B]
    Stack Trace
    http-nio-8080-exec-76 [133] (BLOCKED)
       com.isomorphic.datasource.DataSourceManager.getDataSource line: 167 
       com.isomorphic.datasource.DataSourceManager.getDataSource line: 121 
       com.isomorphic.datasource.DSRequest.getDataSource line: 2356 
       com.isomorphic.datasource.DSRequest.setDataSourceName line: 2330 
       com.isomorphic.datasource.DSRequest.<init> line: 436 
       org.apache.jsp.Lms_jsp._jspService line: 396 
       org.apache.jasper.runtime.HttpJspBase.service line: 70 
       javax.servlet.http.HttpServlet.service line: 741 
       org.apache.jasper.servlet.JspServletWrapper.service line: 476 
       org.apache.jasper.servlet.JspServlet.serviceJspFile line: 385 
       org.apache.jasper.servlet.JspServlet.service line: 329 
       javax.servlet.http.HttpServlet.service line: 741 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 231 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       org.apache.tomcat.websocket.server.WsFilter.doFilter line: 53 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       com.isomorphic.servlet.CompressionFilter._doFilter line: 260 
       com.isomorphic.servlet.BaseFilter.doFilter line: 93 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       com.lmscompany.lms.server.ResponseHeaderFilter.doFilter line: 40 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter line: 126 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       org.apache.catalina.core.StandardWrapperValve.invoke line: 200 
       org.apache.catalina.core.StandardContextValve.invoke line: 96 
       org.apache.catalina.authenticator.AuthenticatorBase.invoke line: 607 
       org.apache.catalina.core.StandardHostValve.invoke line: 139 
       org.apache.catalina.valves.ErrorReportValve.invoke line: 92 
       org.apache.catalina.valves.AbstractAccessLogValve.invoke line: 678 
       org.apache.catalina.valves.rewrite.RewriteValve.invoke line: 282 
       org.apache.catalina.core.StandardEngineValve.invoke line: 74 
       org.apache.catalina.valves.RemoteIpValve.invoke line: 679 
       org.apache.catalina.valves.rewrite.RewriteValve.invoke line: 539 
       org.apache.catalina.core.StandardEngineValve.invoke line: 74 
       org.apache.catalina.valves.RemoteIpValve.invoke line: 679 
       org.apache.catalina.connector.CoyoteAdapter.service line: 343 
       org.apache.coyote.http11.Http11Processor.service line: 408 
       org.apache.coyote.AbstractProcessorLight.process line: 66 
       org.apache.coyote.AbstractProtocol$ConnectionHandler.process line: 834 
       org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun line: 1415 
       org.apache.tomcat.util.net.SocketProcessorBase.run line: 49 
       java.util.concurrent.ThreadPoolExecutor.runWorker line: 1149 
       java.util.concurrent.ThreadPoolExecutor$Worker.run line: 624 
       org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run line: 61 
       java.lang.Thread.run line: 748 
    
    [B]Example 2:[/B]
    Stack Trace
    http-nio-8080-exec-73 [130] (BLOCKED)
       com.isomorphic.datasource.DataSourceManager.getDataSource line: 167 
       com.isomorphic.datasource.DataSourceManager.getDataSource line: 121 
       com.isomorphic.datasource.DSRequest.getDataSource line: 2356 
       com.isomorphic.datasource.DSRequest.setDataSourceName line: 2330 
       com.isomorphic.datasource.DSRequest.<init> line: 436 
       com.lmscompany.lms.server.SendMail.writeActionMails line: 166 
       com.lmscompany.lms.server.SendMail.doGet line: 50 
       javax.servlet.http.HttpServlet.service line: 634 
       javax.servlet.http.HttpServlet.service line: 741 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 231 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       org.apache.tomcat.websocket.server.WsFilter.doFilter line: 53 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       com.isomorphic.servlet.CompressionFilter._doFilter line: 247 
       com.isomorphic.servlet.BaseFilter.doFilter line: 93 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter line: 126 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       org.apache.catalina.core.StandardWrapperValve.invoke line: 200 
       org.apache.catalina.core.StandardContextValve.invoke line: 96 
       org.apache.catalina.authenticator.AuthenticatorBase.invoke line: 490 
       org.apache.catalina.core.StandardHostValve.invoke line: 139 
       org.apache.catalina.valves.ErrorReportValve.invoke line: 92 
       org.apache.catalina.valves.AbstractAccessLogValve.invoke line: 678 
       org.apache.catalina.valves.rewrite.RewriteValve.invoke line: 282 
       org.apache.catalina.core.StandardEngineValve.invoke line: 74 
       org.apache.catalina.valves.RemoteIpValve.invoke line: 679 
       org.apache.catalina.valves.rewrite.RewriteValve.invoke line: 539 
       org.apache.catalina.core.StandardEngineValve.invoke line: 74 
       org.apache.catalina.valves.RemoteIpValve.invoke line: 679 
       org.apache.catalina.connector.CoyoteAdapter.service line: 343 
       org.apache.coyote.http11.Http11Processor.service line: 408 
       org.apache.coyote.AbstractProcessorLight.process line: 66 
       org.apache.coyote.AbstractProtocol$ConnectionHandler.process line: 834 
       org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun line: 1415 
       org.apache.tomcat.util.net.SocketProcessorBase.run line: 49 
       java.util.concurrent.ThreadPoolExecutor.runWorker line: 1149 
       java.util.concurrent.ThreadPoolExecutor$Worker.run line: 624 
       org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run line: 61 
       java.lang.Thread.run line: 748 
    
    [B]Example 3:[/B]
    Stack Trace
    http-nio-8080-exec-65 [122] (BLOCKED)
       com.isomorphic.datasource.DataSourceManager.getDataSource line: 167 
       com.isomorphic.datasource.DataSourceManager.getDataSource line: 121 
       com.isomorphic.datasource.DSRequest.getDataSource line: 2356 
       com.isomorphic.datasource.DSRequest.setDataSourceName line: 2330 
       com.isomorphic.datasource.DSRequest.<init> line: 436 
       com.lmscompany.lms.server.SendMail.writeActionMails line: 166 
       com.lmscompany.lms.server.SendMail.doGet line: 50 
       javax.servlet.http.HttpServlet.service line: 634 
       javax.servlet.http.HttpServlet.service line: 741 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 231 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       org.apache.tomcat.websocket.server.WsFilter.doFilter line: 53 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       com.isomorphic.servlet.CompressionFilter._doFilter line: 247 
       com.isomorphic.servlet.BaseFilter.doFilter line: 93 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter line: 126 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       org.apache.catalina.core.StandardWrapperValve.invoke line: 200 
       org.apache.catalina.core.StandardContextValve.invoke line: 96 
       org.apache.catalina.authenticator.AuthenticatorBase.invoke line: 490 
       org.apache.catalina.core.StandardHostValve.invoke line: 139 
       org.apache.catalina.valves.ErrorReportValve.invoke line: 92 
       org.apache.catalina.valves.AbstractAccessLogValve.invoke line: 678 
       org.apache.catalina.valves.rewrite.RewriteValve.invoke line: 282 
       org.apache.catalina.core.StandardEngineValve.invoke line: 74 
       org.apache.catalina.valves.RemoteIpValve.invoke line: 679 
       org.apache.catalina.valves.rewrite.RewriteValve.invoke line: 539 
       org.apache.catalina.core.StandardEngineValve.invoke line: 74 
       org.apache.catalina.valves.RemoteIpValve.invoke line: 679 
       org.apache.catalina.connector.CoyoteAdapter.service line: 343 
       org.apache.coyote.http11.Http11Processor.service line: 408 
       org.apache.coyote.AbstractProcessorLight.process line: 66 
       org.apache.coyote.AbstractProtocol$ConnectionHandler.process line: 834 
       org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun line: 1415 
       org.apache.tomcat.util.net.SocketProcessorBase.run line: 49 
       java.util.concurrent.ThreadPoolExecutor.runWorker line: 1149 
       java.util.concurrent.ThreadPoolExecutor$Worker.run line: 624 
       org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run line: 61 
       java.lang.Thread.run line: 748
    A few had a stack trace starting with this:
    Code:
    Stack Trace
    http-nio-8080-exec-29 [86] (BLOCKED)
       com.isomorphic.pool.ISCObjectPool.borrowObject line: 111 
       com.isomorphic.pool.PoolManager.borrowObject line: 102 
       com.isomorphic.pool.PoolManager.borrowObject line: 76 
       com.isomorphic.sql.SQLConnectionManager.getConnection line: 177 
       com.isomorphic.sql.SQLDriver.getTransformedResults line: 953 
       com.isomorphic.sql.SQLDriver.executeQuery line: 1394 
       com.isomorphic.sql.SQLDataSource.executeNativeQuery line: 585 
       com.isomorphic.sql.SQLDataSource.executeNativeQuery line: 579 
       com.isomorphic.sql.SQLDataSource.SQLExecute line: 1772 
       com.isomorphic.sql.SQLDataSource.processRequest line: 439 
       com.isomorphic.sql.SQLDataSource.executeFetch line: 384 
       com.lmscompany.lms.server.LMSSQLDataSource.executeFetch line: 149 
       com.isomorphic.datasource.DataSource.execute line: 2384 
       com.lmscompany.lms.server.LMSSQLDataSource.execute line: 191 
       com.isomorphic.application.AppBase.executeDefaultDSOperation line: 646 
       com.isomorphic.application.AppBase.executeAppOperation line: 547 
       com.isomorphic.application.AppBase.execute line: 490 
       com.isomorphic.datasource.DSRequest.execute line: 2835 
       com.lmscompany.lms.server.worker.T_FIELD_CONFIG.getData line: 94 
       com.lmscompany.lms.server.RefreshCycle.checkCacheAge line: 52 
       com.lmscompany.lms.server.worker.T_FIELD_CONFIG.getFieldConfigDataResponse line: 102 
       com.lmscompany.lms.server.RegisterDS.formatFieldMultiple line: 368 
       com.lmscompany.lms.server.RegisterDS.access$400 line: 52 
       com.lmscompany.lms.server.RegisterDS$3.getDataSource line: 251 
       com.isomorphic.datasource.DataSource.getDynamicDataSource line: 807 
       com.isomorphic.datasource.DataSource.forName line: 459 
       com.isomorphic.datasource.PoolableDataSourceFactory.makeUnpooledObject line: 134 
       com.isomorphic.pool.PoolManager.borrowUnpooledObject line: 143 
       com.isomorphic.datasource.DataSourceManager.getDataSource line: 169 
       com.isomorphic.datasource.DataSourceManager.getDataSource line: 121 
       com.isomorphic.datasource.BasicDataSource.getSuper line: 798 
       com.isomorphic.datasource.BasicDataSource.init line: 287 
       com.isomorphic.sql.SQLDataSource.init line: 257 
       com.isomorphic.datasource.DataSource.initialize line: 921 
       com.isomorphic.datasource.BasicDataSource.fromConfig line: 238 
       com.isomorphic.datasource.DataSource.fromConfig line: 906 
       com.isomorphic.datasource.DataSource.loadDS line: 534 
       com.isomorphic.datasource.DataSource.forName line: 462 
       com.isomorphic.datasource.PoolableDataSourceFactory.makeUnpooledObject line: 134 
       com.isomorphic.datasource.PoolableDataSourceFactory.makeObject line: 153 
       org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject line: 1220 
       com.isomorphic.pool.ISCKeyedObjectPool.borrowObject line: 106 
       com.isomorphic.pool.PoolManager.borrowObject line: 92 
       com.isomorphic.datasource.DataSourceManager.getDataSource line: 171 
       com.isomorphic.datasource.DataSourceManager.getDataSource line: 121 
       com.isomorphic.datasource.DSRequest.getDataSource line: 2356 
       com.isomorphic.datasource.DSRequest.decodeUploadedStrings line: 826 
       com.isomorphic.datasource.DSRequest.<init> line: 707 
       com.isomorphic.rpc.RPCManager.parseRequest line: 2526 
       com.isomorphic.rpc.RPCManager.<init> line: 447 
       com.lmscompany.lms.server.LMSIDACall.processRequest line: 48 
       com.isomorphic.servlet.IDACall._processRequest line: 119 
       com.isomorphic.servlet.IDACall.doPost line: 79 
       javax.servlet.http.HttpServlet.service line: 660 
       com.isomorphic.servlet.BaseServlet.service line: 176 
       javax.servlet.http.HttpServlet.service line: 741 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 231 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       org.apache.tomcat.websocket.server.WsFilter.doFilter line: 53 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       com.isomorphic.servlet.CompressionFilter._doFilter line: 260 
       com.isomorphic.servlet.BaseFilter.doFilter line: 93 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter line: 126 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       org.apache.catalina.core.StandardWrapperValve.invoke line: 200 
       org.apache.catalina.core.StandardContextValve.invoke line: 96 
       org.apache.catalina.authenticator.AuthenticatorBase.invoke line: 607 
       org.apache.catalina.core.StandardHostValve.invoke line: 139 
       org.apache.catalina.valves.ErrorReportValve.invoke line: 92 
       org.apache.catalina.valves.AbstractAccessLogValve.invoke line: 678 
       org.apache.catalina.valves.rewrite.RewriteValve.invoke line: 542 
       org.apache.catalina.core.StandardEngineValve.invoke line: 74 
       org.apache.catalina.valves.RemoteIpValve.invoke line: 679 
       org.apache.catalina.connector.CoyoteAdapter.service line: 343 
       org.apache.coyote.http11.Http11Processor.service line: 408 
       org.apache.coyote.AbstractProcessorLight.process line: 66 
       org.apache.coyote.AbstractProtocol$ConnectionHandler.process line: 834 
       org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun line: 1415 
       org.apache.tomcat.util.net.SocketProcessorBase.run line: 49 
       java.util.concurrent.ThreadPoolExecutor.runWorker line: 1149 
       java.util.concurrent.ThreadPoolExecutor$Worker.run line: 624 
       org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run line: 61 
       java.lang.Thread.run line: 748
    Does this (DataSourceManager.getDataSource:167 and ISCObjectPool.borrowObject:111) somehow relate to a potential problem with Apache Commons DBCP?

    Thank you & Best regards
    Blama

    #2
    Hi Isomorphic,

    this now happened again for another instance, and here all blocked threads are like this:
    Code:
    Stack Trace
    http-nio-8080-exec-15 [39] (BLOCKED)
       com.isomorphic.pool.ISCObjectPool.borrowObject line: 111
       com.isomorphic.pool.PoolManager.borrowObject line: 102
       com.isomorphic.pool.PoolManager.borrowObject line: 76
       com.isomorphic.sql.SQLConnectionManager.getConnection line: 177
       com.isomorphic.sql.SQLDriver.update line: 1160
       com.isomorphic.sql.SQLDriver.executeUpdate line: 1370
       com.isomorphic.sql.SQLDataSource.executeNativeUpdate line: 597
       com.isomorphic.sql.SQLDataSource.SQLExecute line: 1853
       com.isomorphic.sql.SQLDataSource.processRequest line: 439
       com.isomorphic.sql.SQLDataSource.executeUpdate line: 388
       com.lmscompany.lms.server.LMSSQLDataSource.executeUpdate line: 200
       com.isomorphic.datasource.DataSource.execute line: 2425
       com.lmscompany.lms.server.LMSSQLDataSource.execute line: 191
       com.isomorphic.application.AppBase.executeDefaultDSOperation line: 646
       com.isomorphic.application.AppBase.executeAppOperation line: 547
       com.isomorphic.application.AppBase.execute line: 490
       com.isomorphic.datasource.DSRequest.execute line: 2835
       com.lmscompany.lms.server.LogIncomingRequest.updateDSRequestStatus line: 168
       com.lmscompany.lms.server.LMSIDACall.handleDSRequest line: 221
       com.isomorphic.servlet.IDACall.processRPCTransaction line: 187
       com.lmscompany.lms.server.LMSIDACall.processRequest line: 109
       com.isomorphic.servlet.IDACall._processRequest line: 119
       com.isomorphic.servlet.IDACall.doPost line: 79
       javax.servlet.http.HttpServlet.service line: 660
       com.isomorphic.servlet.BaseServlet.service line: 176
       javax.servlet.http.HttpServlet.service line: 741
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 231
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166
       org.apache.tomcat.websocket.server.WsFilter.doFilter line: 53
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166
       com.isomorphic.servlet.CompressionFilter._doFilter line: 260
       com.isomorphic.servlet.BaseFilter.doFilter line: 93
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166
       org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter line: 126
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166
       org.apache.catalina.core.StandardWrapperValve.invoke line: 200
       org.apache.catalina.core.StandardContextValve.invoke line: 96
       org.apache.catalina.authenticator.AuthenticatorBase.invoke line: 607
       org.apache.catalina.core.StandardHostValve.invoke line: 139
       org.apache.catalina.valves.ErrorReportValve.invoke line: 92
       org.apache.catalina.valves.AbstractAccessLogValve.invoke line: 678
       org.apache.catalina.valves.rewrite.RewriteValve.invoke line: 542
       org.apache.catalina.core.StandardEngineValve.invoke line: 74
       org.apache.catalina.valves.RemoteIpValve.invoke line: 679
       org.apache.catalina.connector.CoyoteAdapter.service line: 343
       org.apache.coyote.http11.Http11Processor.service line: 408
       org.apache.coyote.AbstractProcessorLight.process line: 66
       org.apache.coyote.AbstractProtocol$ConnectionHandler.process line: 834
       org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun line: 1415
       org.apache.tomcat.util.net.SocketProcessorBase.run line: 49
       java.util.concurrent.ThreadPoolExecutor.runWorker line: 1149
       java.util.concurrent.ThreadPoolExecutor$Worker.run line: 624
       org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run line: 61
       java.lang.Thread.run line: 748
    and from the line "com.isomorphic.sql.SQLDriver.executeUpdate line: 1370" it is different, depending on the type of DB-action (update, select, ...).

    And for information: All the connections blocking the system are created in a very short time. Out application does not have much traffic, but this is creation time of the connections for #1, fetched from the DB:
    Code:
    10.05.19 12:16:43
    13.05.19 11:18:24
    13.05.19 11:18:28
    13.05.19 11:18:28
    13.05.19 11:18:29
    13.05.19 11:18:29
    13.05.19 11:18:29
    13.05.19 11:18:30
    13.05.19 11:18:31
    13.05.19 11:18:31
    13.05.19 11:18:31
    It are 11 lines - this and the fact that one connection is very old matches the configuration:
    10 connections: server.properties setting sql.pool.maxActive: 10
    1 connection: Tomcat JDBCRealm from server.xml

    Those 10 new connections in 7 seconds can't possible be from user interaction.

    Best regards
    Blama

    Comment


      #3
      DataSourceManager.get() does involve a DBCP pool. So it looks like you will need to turn off eviction runs until 1.4.1 comes out.

      Comment


        #4
        Hi Isomorphic,

        chances for a 1.4.1 are bad, see here, where they are suggesting to switch to 2.6.0.
        1.4 is more than 9 years old now and commons.dbcp2 is in active development since 2014 with ~2 releases a year.

        For now, I tried setting sql.pool.minIdle: 0 as they claim that positive minIdle and timeBetweenEvictionRunsMillis together will trigger the issue. I'll watch if this mitigates the problem successfully.

        Best regards
        Blama

        Comment


          #5
          Actually they seem to indicate the bug has been fixed, but not released, so perhaps there's hope. There's always the (painful) path of building it yourself from source.

          There's a not a lot we can do for you here in the immediate term, as DBCP2 is not just a minor version bump but a replacement library with major internal differences. We will of course switch over sometime soon, but we can't switch existing or nearly-out releases to the new library, not when the only problem with the existing library is a feature that, so far, not even one other user seems to need. Sorry that this is a problem for you.

          Comment


            #6
            Hi Isomorphic,

            I understand that it is not possible to do such a change in a 12.0p nightly. With nearly-out releases I assume you refer to 12.1d/p, which is of course bad from my point of view. So you are saying you plan a change for next next major release?
            IF the minIdle: 0 change fixes it for me, I'm happy with 1.4 as well. IF our planned switch Oracle->Postgres does invalidate the necessity of timeBetweenEvictionRunsMillis at all, I'm also happy with 1.4. So let's hope that this is the case. If not, building from source, which I already considered, might be a last resort.

            And while I'm a big fan of keeping proven solutions, I also think that it is a good idea if you updated all your used libraries at least to the state where you use a version that is in active development, so that jar-file.version bumping becomes possible if ever necessary.
            I did not check all Apache commons and other used libraries, but for some there are already versions with namespace changes out, so if there appears a bug in such an unmaintained version, there will be a problem like now.

            Best regards
            Blama

            Comment


              #7
              Hi

              We too would like to upgrade our DBCP dependency but we're blocked by the fact that SmartGWT depends on it.
              Possibly we can run both versions in parallel, we haven't tried that path yet (but I don't think it's ideal either).

              DBCP 1.4 is really old and there is no support on it anymore. JIRA tickets are referred to newer versions.

              So +1 for a dbcp upgrade.

              Thanks

              Comment


                #8
                You aren’t blocked, it’s a different package name.

                So it looks like the only problem continues to be the obscure bug involving eviction runs which they appear to have fixed (but not yet released) in 1.x. Let us know if we’ve missed something.

                Comment


                  #9
                  Just a FYI for everyone hitting this thread via search. We are now using JNDI and with this Tomcat's current built-in DBCP (see a few times here when you search for "commons"). This is working and also saving a connection needed otherwise for login.
                  Nevertheless I do think that it is a good idea to have current versions of the dependencies, that are not EOL, see #6.

                  Best regards
                  Blama

                  Comment


                    #10
                    Hi Isomorphic,

                    I had a look at the lib folder of 13.0d and saw that you bumped a few libs and also removed some and added others.
                    Unfortunately this one is not one of the changed ones. As I pointed out in #4, there won't be a new version of Apache Commons DBCP 1.4.

                    I saw you current readme.html, which still speaks of a minimum requirement of Java 1.5 - I think here it should be safe to say that new development should be using at least Java 8 (or even 11?), which would allow you to use current DBCP 2.7.0. You say similar things to your users and it's completely reasonable in both cases IMHO.

                    While we are no longer affected by the issue, as we use JNDI now with a more current version of DBCP (via tomcat-dbcp), that is automatically bumped with Tomcat versions (search for "dbcp" here), it would be nice for all other users, that use your default configuration, to be not affected by the issue.
                    Also - and possibly more important - it looks very bad when evaluating frameworks when the dependencies are very EOL.

                    Best regards
                    Blama

                    Comment


                      #11
                      Many of the largest customers out there prefer a completely stable environment, so bumping the minimum Java version can’t be done yet.

                      But yet again no one is blocked. It’s a different package name and you can control which one you use.

                      However if there is even one of other person being affected by this obscure eviction run issue, we would like to hear about that (just post here).

                      Comment

                      Working...
                      X