Problem with Apache DBCP and maxed out pool

[Blama]

Hi Isomorphic,

it seems we do have a problem with current 12.0p (v12.0p_2019-03-23) and the shipped commons-dbcp-1.4.jar.
It is likely that this is related to this thread and the DBCP issue you mention in #6 there.

Release notes for unreleased Apache Commons DBCP 1.4.1 mention the bug that 1.2.2 claims to have fixed. It contains this fix which claims that the fix in 1.2.2 is incomplete. Now 1.4.1 is unreleased, so there is not much you can do here. I did ask for a release date, though. But I do assume that DBCP 2 (new namespace) fixes it as well, as this is in active development (3 releases in 2018, 1 in 2019).

We do have the configuration in place that is said to cause the issue (positive minIdle and timeBetweenEvictionRunsMillis):

sql.defaultDatabase: Oracle
sql.useAnsiJoins: true

# Apache Commons DBCP Database connection pooling
sql.pool.enabled: true
[B]sql.pool.minIdle: 1[/B]
sql.pool.maxActive: 10
sql.pool.whenExhaustedAction: block
sql.pool.testWhileIdle: true
[B]sql.pool.timeBetweenEvictionRunsMillis: 15000[/B]
sql.pool.minEvictableIdleTimeMillis: 29000

sql.Oracle.database.type: oracle
sql.Oracle.autoJoinTransactions: true
sql.Oracle.database.supportsSQLLimit: false

sql.Oracle.interface.credentialsInURL: true
sql.Oracle.interface.type: dataSource

sql.Oracle.pingTest: select 1 from dual

sql.Oracle.driver: oracle.jdbc.pool.OracleDataSource
sql.Oracle.driver.url: jdbc:oracle:thin:@xxxxxx:1521/DB11GXDB.localdomain
sql.Oracle.driver.user: xxx
sql.Oracle.driver.password: xxx

sql.Oracle.driver.driverType: thin
sql.Oracle.driver.networkProtocol: tcp
sql.Oracle.driver.context:

Regarding the issue itself:
We run many instances of our application as own WAR files. These application run on the same Tomcat (9.0.17). Now if the issue happens, one application (server.xml <Host> entry) maxes out it connections. New requests will hang (stack trace follows). This in turn maxes out our Tomcat threads (maxThreads here) eventually, bringing all instances effectively down, as no more threads can be created.

We restarted our application to prevent the "whole Tomcat down" issue from happening.
At that time, we had many (~50) BLOCKED threads, most of them with a stack trace where the top line is like this (there are of course many different codepaths that use DB connections in the end, so the whole stack is always a bit different:

[B]Example 1:[/B]
Stack Trace
http-nio-8080-exec-76 [133] (BLOCKED)
   com.isomorphic.datasource.DataSourceManager.getDataSource line: 167 
   com.isomorphic.datasource.DataSourceManager.getDataSource line: 121 
   com.isomorphic.datasource.DSRequest.getDataSource line: 2356 
   com.isomorphic.datasource.DSRequest.setDataSourceName line: 2330 
   com.isomorphic.datasource.DSRequest.<init> line: 436 
   org.apache.jsp.Lms_jsp._jspService line: 396 
   org.apache.jasper.runtime.HttpJspBase.service line: 70 
   javax.servlet.http.HttpServlet.service line: 741 
   org.apache.jasper.servlet.JspServletWrapper.service line: 476 
   org.apache.jasper.servlet.JspServlet.serviceJspFile line: 385 
   org.apache.jasper.servlet.JspServlet.service line: 329 
   javax.servlet.http.HttpServlet.service line: 741 
   org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 231 
   org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
   org.apache.tomcat.websocket.server.WsFilter.doFilter line: 53 
   org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
   org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
   com.isomorphic.servlet.CompressionFilter._doFilter line: 260 
   com.isomorphic.servlet.BaseFilter.doFilter line: 93 
   org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
   org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
   com.lmscompany.lms.server.ResponseHeaderFilter.doFilter line: 40 
   org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
   org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
   org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter line: 126 
   org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
   org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
   org.apache.catalina.core.StandardWrapperValve.invoke line: 200 
   org.apache.catalina.core.StandardContextValve.invoke line: 96 
   org.apache.catalina.authenticator.AuthenticatorBase.invoke line: 607 
   org.apache.catalina.core.StandardHostValve.invoke line: 139 
   org.apache.catalina.valves.ErrorReportValve.invoke line: 92 
   org.apache.catalina.valves.AbstractAccessLogValve.invoke line: 678 
   org.apache.catalina.valves.rewrite.RewriteValve.invoke line: 282 
   org.apache.catalina.core.StandardEngineValve.invoke line: 74 
   org.apache.catalina.valves.RemoteIpValve.invoke line: 679 
   org.apache.catalina.valves.rewrite.RewriteValve.invoke line: 539 
   org.apache.catalina.core.StandardEngineValve.invoke line: 74 
   org.apache.catalina.valves.RemoteIpValve.invoke line: 679 
   org.apache.catalina.connector.CoyoteAdapter.service line: 343 
   org.apache.coyote.http11.Http11Processor.service line: 408 
   org.apache.coyote.AbstractProcessorLight.process line: 66 
   org.apache.coyote.AbstractProtocol$ConnectionHandler.process line: 834 
   org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun line: 1415 
   org.apache.tomcat.util.net.SocketProcessorBase.run line: 49 
   java.util.concurrent.ThreadPoolExecutor.runWorker line: 1149 
   java.util.concurrent.ThreadPoolExecutor$Worker.run line: 624 
   org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run line: 61 
   java.lang.Thread.run line: 748 

[B]Example 2:[/B]
Stack Trace
http-nio-8080-exec-73 [130] (BLOCKED)
   com.isomorphic.datasource.DataSourceManager.getDataSource line: 167 
   com.isomorphic.datasource.DataSourceManager.getDataSource line: 121 
   com.isomorphic.datasource.DSRequest.getDataSource line: 2356 
   com.isomorphic.datasource.DSRequest.setDataSourceName line: 2330 
   com.isomorphic.datasource.DSRequest.<init> line: 436 
   com.lmscompany.lms.server.SendMail.writeActionMails line: 166 
   com.lmscompany.lms.server.SendMail.doGet line: 50 
   javax.servlet.http.HttpServlet.service line: 634 
   javax.servlet.http.HttpServlet.service line: 741 
   org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 231 
   org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
   org.apache.tomcat.websocket.server.WsFilter.doFilter line: 53 
   org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
   org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
   com.isomorphic.servlet.CompressionFilter._doFilter line: 247 
   com.isomorphic.servlet.BaseFilter.doFilter line: 93 
   org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
   org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
   org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter line: 126 
   org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
   org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
   org.apache.catalina.core.StandardWrapperValve.invoke line: 200 
   org.apache.catalina.core.StandardContextValve.invoke line: 96 
   org.apache.catalina.authenticator.AuthenticatorBase.invoke line: 490 
   org.apache.catalina.core.StandardHostValve.invoke line: 139 
   org.apache.catalina.valves.ErrorReportValve.invoke line: 92 
   org.apache.catalina.valves.AbstractAccessLogValve.invoke line: 678 
   org.apache.catalina.valves.rewrite.RewriteValve.invoke line: 282 
   org.apache.catalina.core.StandardEngineValve.invoke line: 74 
   org.apache.catalina.valves.RemoteIpValve.invoke line: 679 
   org.apache.catalina.valves.rewrite.RewriteValve.invoke line: 539 
   org.apache.catalina.core.StandardEngineValve.invoke line: 74 
   org.apache.catalina.valves.RemoteIpValve.invoke line: 679 
   org.apache.catalina.connector.CoyoteAdapter.service line: 343 
   org.apache.coyote.http11.Http11Processor.service line: 408 
   org.apache.coyote.AbstractProcessorLight.process line: 66 
   org.apache.coyote.AbstractProtocol$ConnectionHandler.process line: 834 
   org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun line: 1415 
   org.apache.tomcat.util.net.SocketProcessorBase.run line: 49 
   java.util.concurrent.ThreadPoolExecutor.runWorker line: 1149 
   java.util.concurrent.ThreadPoolExecutor$Worker.run line: 624 
   org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run line: 61 
   java.lang.Thread.run line: 748 

[B]Example 3:[/B]
Stack Trace
http-nio-8080-exec-65 [122] (BLOCKED)
   com.isomorphic.datasource.DataSourceManager.getDataSource line: 167 
   com.isomorphic.datasource.DataSourceManager.getDataSource line: 121 
   com.isomorphic.datasource.DSRequest.getDataSource line: 2356 
   com.isomorphic.datasource.DSRequest.setDataSourceName line: 2330 
   com.isomorphic.datasource.DSRequest.<init> line: 436 
   com.lmscompany.lms.server.SendMail.writeActionMails line: 166 
   com.lmscompany.lms.server.SendMail.doGet line: 50 
   javax.servlet.http.HttpServlet.service line: 634 
   javax.servlet.http.HttpServlet.service line: 741 
   org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 231 
   org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
   org.apache.tomcat.websocket.server.WsFilter.doFilter line: 53 
   org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
   org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
   com.isomorphic.servlet.CompressionFilter._doFilter line: 247 
   com.isomorphic.servlet.BaseFilter.doFilter line: 93 
   org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
   org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
   org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter line: 126 
   org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
   org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
   org.apache.catalina.core.StandardWrapperValve.invoke line: 200 
   org.apache.catalina.core.StandardContextValve.invoke line: 96 
   org.apache.catalina.authenticator.AuthenticatorBase.invoke line: 490 
   org.apache.catalina.core.StandardHostValve.invoke line: 139 
   org.apache.catalina.valves.ErrorReportValve.invoke line: 92 
   org.apache.catalina.valves.AbstractAccessLogValve.invoke line: 678 
   org.apache.catalina.valves.rewrite.RewriteValve.invoke line: 282 
   org.apache.catalina.core.StandardEngineValve.invoke line: 74 
   org.apache.catalina.valves.RemoteIpValve.invoke line: 679 
   org.apache.catalina.valves.rewrite.RewriteValve.invoke line: 539 
   org.apache.catalina.core.StandardEngineValve.invoke line: 74 
   org.apache.catalina.valves.RemoteIpValve.invoke line: 679 
   org.apache.catalina.connector.CoyoteAdapter.service line: 343 
   org.apache.coyote.http11.Http11Processor.service line: 408 
   org.apache.coyote.AbstractProcessorLight.process line: 66 
   org.apache.coyote.AbstractProtocol$ConnectionHandler.process line: 834 
   org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun line: 1415 
   org.apache.tomcat.util.net.SocketProcessorBase.run line: 49 
   java.util.concurrent.ThreadPoolExecutor.runWorker line: 1149 
   java.util.concurrent.ThreadPoolExecutor$Worker.run line: 624 
   org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run line: 61 
   java.lang.Thread.run line: 748

A few had a stack trace starting with this:

Stack Trace
http-nio-8080-exec-29 [86] (BLOCKED)
   com.isomorphic.pool.ISCObjectPool.borrowObject line: 111 
   com.isomorphic.pool.PoolManager.borrowObject line: 102 
   com.isomorphic.pool.PoolManager.borrowObject line: 76 
   com.isomorphic.sql.SQLConnectionManager.getConnection line: 177 
   com.isomorphic.sql.SQLDriver.getTransformedResults line: 953 
   com.isomorphic.sql.SQLDriver.executeQuery line: 1394 
   com.isomorphic.sql.SQLDataSource.executeNativeQuery line: 585 
   com.isomorphic.sql.SQLDataSource.executeNativeQuery line: 579 
   com.isomorphic.sql.SQLDataSource.SQLExecute line: 1772 
   com.isomorphic.sql.SQLDataSource.processRequest line: 439 
   com.isomorphic.sql.SQLDataSource.executeFetch line: 384 
   com.lmscompany.lms.server.LMSSQLDataSource.executeFetch line: 149 
   com.isomorphic.datasource.DataSource.execute line: 2384 
   com.lmscompany.lms.server.LMSSQLDataSource.execute line: 191 
   com.isomorphic.application.AppBase.executeDefaultDSOperation line: 646 
   com.isomorphic.application.AppBase.executeAppOperation line: 547 
   com.isomorphic.application.AppBase.execute line: 490 
   com.isomorphic.datasource.DSRequest.execute line: 2835 
   com.lmscompany.lms.server.worker.T_FIELD_CONFIG.getData line: 94 
   com.lmscompany.lms.server.RefreshCycle.checkCacheAge line: 52 
   com.lmscompany.lms.server.worker.T_FIELD_CONFIG.getFieldConfigDataResponse line: 102 
   com.lmscompany.lms.server.RegisterDS.formatFieldMultiple line: 368 
   com.lmscompany.lms.server.RegisterDS.access$400 line: 52 
   com.lmscompany.lms.server.RegisterDS$3.getDataSource line: 251 
   com.isomorphic.datasource.DataSource.getDynamicDataSource line: 807 
   com.isomorphic.datasource.DataSource.forName line: 459 
   com.isomorphic.datasource.PoolableDataSourceFactory.makeUnpooledObject line: 134 
   com.isomorphic.pool.PoolManager.borrowUnpooledObject line: 143 
   com.isomorphic.datasource.DataSourceManager.getDataSource line: 169 
   com.isomorphic.datasource.DataSourceManager.getDataSource line: 121 
   com.isomorphic.datasource.BasicDataSource.getSuper line: 798 
   com.isomorphic.datasource.BasicDataSource.init line: 287 
   com.isomorphic.sql.SQLDataSource.init line: 257 
   com.isomorphic.datasource.DataSource.initialize line: 921 
   com.isomorphic.datasource.BasicDataSource.fromConfig line: 238 
   com.isomorphic.datasource.DataSource.fromConfig line: 906 
   com.isomorphic.datasource.DataSource.loadDS line: 534 
   com.isomorphic.datasource.DataSource.forName line: 462 
   com.isomorphic.datasource.PoolableDataSourceFactory.makeUnpooledObject line: 134 
   com.isomorphic.datasource.PoolableDataSourceFactory.makeObject line: 153 
   org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject line: 1220 
   com.isomorphic.pool.ISCKeyedObjectPool.borrowObject line: 106 
   com.isomorphic.pool.PoolManager.borrowObject line: 92 
   com.isomorphic.datasource.DataSourceManager.getDataSource line: 171 
   com.isomorphic.datasource.DataSourceManager.getDataSource line: 121 
   com.isomorphic.datasource.DSRequest.getDataSource line: 2356 
   com.isomorphic.datasource.DSRequest.decodeUploadedStrings line: 826 
   com.isomorphic.datasource.DSRequest.<init> line: 707 
   com.isomorphic.rpc.RPCManager.parseRequest line: 2526 
   com.isomorphic.rpc.RPCManager.<init> line: 447 
   com.lmscompany.lms.server.LMSIDACall.processRequest line: 48 
   com.isomorphic.servlet.IDACall._processRequest line: 119 
   com.isomorphic.servlet.IDACall.doPost line: 79 
   javax.servlet.http.HttpServlet.service line: 660 
   com.isomorphic.servlet.BaseServlet.service line: 176 
   javax.servlet.http.HttpServlet.service line: 741 
   org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 231 
   org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
   org.apache.tomcat.websocket.server.WsFilter.doFilter line: 53 
   org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
   org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
   com.isomorphic.servlet.CompressionFilter._doFilter line: 260 
   com.isomorphic.servlet.BaseFilter.doFilter line: 93 
   org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
   org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
   org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter line: 126 
   org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
   org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
   org.apache.catalina.core.StandardWrapperValve.invoke line: 200 
   org.apache.catalina.core.StandardContextValve.invoke line: 96 
   org.apache.catalina.authenticator.AuthenticatorBase.invoke line: 607 
   org.apache.catalina.core.StandardHostValve.invoke line: 139 
   org.apache.catalina.valves.ErrorReportValve.invoke line: 92 
   org.apache.catalina.valves.AbstractAccessLogValve.invoke line: 678 
   org.apache.catalina.valves.rewrite.RewriteValve.invoke line: 542 
   org.apache.catalina.core.StandardEngineValve.invoke line: 74 
   org.apache.catalina.valves.RemoteIpValve.invoke line: 679 
   org.apache.catalina.connector.CoyoteAdapter.service line: 343 
   org.apache.coyote.http11.Http11Processor.service line: 408 
   org.apache.coyote.AbstractProcessorLight.process line: 66 
   org.apache.coyote.AbstractProtocol$ConnectionHandler.process line: 834 
   org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun line: 1415 
   org.apache.tomcat.util.net.SocketProcessorBase.run line: 49 
   java.util.concurrent.ThreadPoolExecutor.runWorker line: 1149 
   java.util.concurrent.ThreadPoolExecutor$Worker.run line: 624 
   org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run line: 61 
   java.lang.Thread.run line: 748

Does this (DataSourceManager.getDataSource:167 and ISCObjectPool.borrowObject:111) somehow relate to a potential problem with Apache Commons DBCP?

Thank you & Best regards
Blama

[Blama]

Hi Isomorphic,

this now happened again for another instance, and here all blocked threads are like this:

Stack Trace
http-nio-8080-exec-15 [39] (BLOCKED)
   com.isomorphic.pool.ISCObjectPool.borrowObject line: 111
   com.isomorphic.pool.PoolManager.borrowObject line: 102
   com.isomorphic.pool.PoolManager.borrowObject line: 76
   com.isomorphic.sql.SQLConnectionManager.getConnection line: 177
   com.isomorphic.sql.SQLDriver.update line: 1160
   com.isomorphic.sql.SQLDriver.executeUpdate line: 1370
   com.isomorphic.sql.SQLDataSource.executeNativeUpdate line: 597
   com.isomorphic.sql.SQLDataSource.SQLExecute line: 1853
   com.isomorphic.sql.SQLDataSource.processRequest line: 439
   com.isomorphic.sql.SQLDataSource.executeUpdate line: 388
   com.lmscompany.lms.server.LMSSQLDataSource.executeUpdate line: 200
   com.isomorphic.datasource.DataSource.execute line: 2425
   com.lmscompany.lms.server.LMSSQLDataSource.execute line: 191
   com.isomorphic.application.AppBase.executeDefaultDSOperation line: 646
   com.isomorphic.application.AppBase.executeAppOperation line: 547
   com.isomorphic.application.AppBase.execute line: 490
   com.isomorphic.datasource.DSRequest.execute line: 2835
   com.lmscompany.lms.server.LogIncomingRequest.updateDSRequestStatus line: 168
   com.lmscompany.lms.server.LMSIDACall.handleDSRequest line: 221
   com.isomorphic.servlet.IDACall.processRPCTransaction line: 187
   com.lmscompany.lms.server.LMSIDACall.processRequest line: 109
   com.isomorphic.servlet.IDACall._processRequest line: 119
   com.isomorphic.servlet.IDACall.doPost line: 79
   javax.servlet.http.HttpServlet.service line: 660
   com.isomorphic.servlet.BaseServlet.service line: 176
   javax.servlet.http.HttpServlet.service line: 741
   org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 231
   org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166
   org.apache.tomcat.websocket.server.WsFilter.doFilter line: 53
   org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193
   org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166
   com.isomorphic.servlet.CompressionFilter._doFilter line: 260
   com.isomorphic.servlet.BaseFilter.doFilter line: 93
   org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193
   org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166
   org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter line: 126
   org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193
   org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166
   org.apache.catalina.core.StandardWrapperValve.invoke line: 200
   org.apache.catalina.core.StandardContextValve.invoke line: 96
   org.apache.catalina.authenticator.AuthenticatorBase.invoke line: 607
   org.apache.catalina.core.StandardHostValve.invoke line: 139
   org.apache.catalina.valves.ErrorReportValve.invoke line: 92
   org.apache.catalina.valves.AbstractAccessLogValve.invoke line: 678
   org.apache.catalina.valves.rewrite.RewriteValve.invoke line: 542
   org.apache.catalina.core.StandardEngineValve.invoke line: 74
   org.apache.catalina.valves.RemoteIpValve.invoke line: 679
   org.apache.catalina.connector.CoyoteAdapter.service line: 343
   org.apache.coyote.http11.Http11Processor.service line: 408
   org.apache.coyote.AbstractProcessorLight.process line: 66
   org.apache.coyote.AbstractProtocol$ConnectionHandler.process line: 834
   org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun line: 1415
   org.apache.tomcat.util.net.SocketProcessorBase.run line: 49
   java.util.concurrent.ThreadPoolExecutor.runWorker line: 1149
   java.util.concurrent.ThreadPoolExecutor$Worker.run line: 624
   org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run line: 61
   java.lang.Thread.run line: 748

and from the line "com.isomorphic.sql.SQLDriver.executeUpdate line: 1370" it is different, depending on the type of DB-action (update, select, ...).

And for information: All the connections blocking the system are created in a very short time. Out application does not have much traffic, but this is creation time of the connections for #1, fetched from the DB:

10.05.19 12:16:43
13.05.19 11:18:24
13.05.19 11:18:28
13.05.19 11:18:28
13.05.19 11:18:29
13.05.19 11:18:29
13.05.19 11:18:29
13.05.19 11:18:30
13.05.19 11:18:31
13.05.19 11:18:31
13.05.19 11:18:31

It are 11 lines - this and the fact that one connection is very old matches the configuration:
10 connections: server.properties setting sql.pool.maxActive: 10
1 connection: Tomcat JDBCRealm from server.xml

Those 10 new connections in 7 seconds can't possible be from user interaction.

Best regards
Blama

[Isomorphic]

DataSourceManager.get() does involve a DBCP pool. So it looks like you will need to turn off eviction runs until 1.4.1 comes out.

[Blama]

Hi Isomorphic,

chances for a 1.4.1 are bad, see here, where they are suggesting to switch to 2.6.0.
1.4 is more than 9 years old now and commons.dbcp2 is in active development since 2014 with ~2 releases a year.

For now, I tried setting sql.pool.minIdle: 0 as they claim that positive minIdle and timeBetweenEvictionRunsMillis together will trigger the issue. I'll watch if this mitigates the problem successfully.

Best regards
Blama

[Isomorphic]

Actually they seem to indicate the bug has been fixed, but not released, so perhaps there's hope. There's always the (painful) path of building it yourself from source.

There's a not a lot we can do for you here in the immediate term, as DBCP2 is not just a minor version bump but a replacement library with major internal differences. We will of course switch over sometime soon, but we can't switch existing or nearly-out releases to the new library, not when the only problem with the existing library is a feature that, so far, not even one other user seems to need. Sorry that this is a problem for you.

[Blama]

Hi Isomorphic,

I understand that it is not possible to do such a change in a 12.0p nightly. With nearly-out releases I assume you refer to 12.1d/p, which is of course bad from my point of view. So you are saying you plan a change for next next major release?
IF the minIdle: 0 change fixes it for me, I'm happy with 1.4 as well. IF our planned switch Oracle->Postgres does invalidate the necessity of timeBetweenEvictionRunsMillis at all, I'm also happy with 1.4. So let's hope that this is the case. If not, building from source, which I already considered, might be a last resort.

And while I'm a big fan of keeping proven solutions, I also think that it is a good idea if you updated all your used libraries at least to the state where you use a version that is in active development, so that jar-file.version bumping becomes possible if ever necessary.
I did not check all Apache commons and other used libraries, but for some there are already versions with namespace changes out, so if there appears a bug in such an unmaintained version, there will be a problem like now.

Best regards
Blama

[monk]

Hi

We too would like to upgrade our DBCP dependency but we're blocked by the fact that SmartGWT depends on it.
Possibly we can run both versions in parallel, we haven't tried that path yet (but I don't think it's ideal either).

DBCP 1.4 is really old and there is no support on it anymore. JIRA tickets are referred to newer versions.

So +1 for a dbcp upgrade.

Thanks

[Isomorphic]

You aren’t blocked, it’s a different package name.

So it looks like the only problem continues to be the obscure bug involving eviction runs which they appear to have fixed (but not yet released) in 1.x. Let us know if we’ve missed something.

[Blama]

Just a FYI for everyone hitting this thread via search. We are now using JNDI and with this Tomcat's current built-in DBCP (see a few times here when you search for "commons"). This is working and also saving a connection needed otherwise for login.
Nevertheless I do think that it is a good idea to have current versions of the dependencies, that are not EOL, see #6.

Best regards
Blama

[Blama]

Hi Isomorphic,

I had a look at the lib folder of 13.0d and saw that you bumped a few libs and also removed some and added others.
Unfortunately this one is not one of the changed ones. As I pointed out in #4, there won't be a new version of Apache Commons DBCP 1.4.

I saw you current readme.html, which still speaks of a minimum requirement of Java 1.5 - I think here it should be safe to say that new development should be using at least Java 8 (or even 11?), which would allow you to use current DBCP 2.7.0. You say similar things to your users and it's completely reasonable in both cases IMHO.

While we are no longer affected by the issue, as we use JNDI now with a more current version of DBCP (via tomcat-dbcp), that is automatically bumped with Tomcat versions (search for "dbcp" here), it would be nice for all other users, that use your default configuration, to be not affected by the issue.
Also - and possibly more important - it looks very bad when evaluating frameworks when the dependencies are very EOL.

Best regards
Blama

[Isomorphic]

Many of the largest customers out there prefer a completely stable environment, so bumping the minimum Java version can’t be done yet.

But yet again no one is blocked. It’s a different package name and you can control which one you use.

However if there is even one of other person being affected by this obscure eviction run issue, we would like to hear about that (just post here).