Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
Clear All
new posts

  • Blama
    started a topic Problem with Apache DBCP and maxed out pool

    Problem with Apache DBCP and maxed out pool

    Hi Isomorphic,

    it seems we do have a problem with current 12.0p (v12.0p_2019-03-23) and the shipped commons-dbcp-1.4.jar.
    It is likely that this is related to this thread and the DBCP issue you mention in #6 there.

    Release notes for unreleased Apache Commons DBCP 1.4.1 mention the bug that 1.2.2 claims to have fixed. It contains this fix which claims that the fix in 1.2.2 is incomplete. Now 1.4.1 is unreleased, so there is not much you can do here. I did ask for a release date, though. But I do assume that DBCP 2 (new namespace) fixes it as well, as this is in active development (3 releases in 2018, 1 in 2019).

    We do have the configuration in place that is said to cause the issue (positive minIdle and timeBetweenEvictionRunsMillis):
    Code:
    sql.defaultDatabase: Oracle
    sql.useAnsiJoins: true
    
    # Apache Commons DBCP Database connection pooling
    sql.pool.enabled: true
    sql.pool.minIdle: 1
    sql.pool.maxActive: 10
    sql.pool.whenExhaustedAction: block
    sql.pool.testWhileIdle: true
    sql.pool.timeBetweenEvictionRunsMillis: 15000
    sql.pool.minEvictableIdleTimeMillis: 29000
    
    sql.Oracle.database.type: oracle
    sql.Oracle.autoJoinTransactions: true
    sql.Oracle.database.supportsSQLLimit: false
    
    sql.Oracle.interface.credentialsInURL: true
    sql.Oracle.interface.type: dataSource
    
    sql.Oracle.pingTest: select 1 from dual
    
    sql.Oracle.driver: oracle.jdbc.pool.OracleDataSource
    sql.Oracle.driver.url: jdbc:oracle:thin:@xxxxxx:1521/DB11GXDB.localdomain
    sql.Oracle.driver.user: xxx
    sql.Oracle.driver.password: xxx
    
    sql.Oracle.driver.driverType: thin
    sql.Oracle.driver.networkProtocol: tcp
    sql.Oracle.driver.context:
    Regarding the issue itself:
    We run many instances of our application as own WAR files. These application run on the same Tomcat (9.0.17). Now if the issue happens, one application (server.xml <Host> entry) maxes out it connections. New requests will hang (stack trace follows). This in turn maxes out our Tomcat threads (maxThreads here) eventually, bringing all instances effectively down, as no more threads can be created.

    We restarted our application to prevent the "whole Tomcat down" issue from happening.
    At that time, we had many (~50) BLOCKED threads, most of them with a stack trace where the top line is like this (there are of course many different codepaths that use DB connections in the end, so the whole stack is always a bit different:
    Code:
    Example 1:
    Stack Trace
    http-nio-8080-exec-76 [133] (BLOCKED)
       com.isomorphic.datasource.DataSourceManager.getDataSource line: 167 
       com.isomorphic.datasource.DataSourceManager.getDataSource line: 121 
       com.isomorphic.datasource.DSRequest.getDataSource line: 2356 
       com.isomorphic.datasource.DSRequest.setDataSourceName line: 2330 
       com.isomorphic.datasource.DSRequest.<init> line: 436 
       org.apache.jsp.Lms_jsp._jspService line: 396 
       org.apache.jasper.runtime.HttpJspBase.service line: 70 
       javax.servlet.http.HttpServlet.service line: 741 
       org.apache.jasper.servlet.JspServletWrapper.service line: 476 
       org.apache.jasper.servlet.JspServlet.serviceJspFile line: 385 
       org.apache.jasper.servlet.JspServlet.service line: 329 
       javax.servlet.http.HttpServlet.service line: 741 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 231 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       org.apache.tomcat.websocket.server.WsFilter.doFilter line: 53 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       com.isomorphic.servlet.CompressionFilter._doFilter line: 260 
       com.isomorphic.servlet.BaseFilter.doFilter line: 93 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       com.lmscompany.lms.server.ResponseHeaderFilter.doFilter line: 40 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter line: 126 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       org.apache.catalina.core.StandardWrapperValve.invoke line: 200 
       org.apache.catalina.core.StandardContextValve.invoke line: 96 
       org.apache.catalina.authenticator.AuthenticatorBase.invoke line: 607 
       org.apache.catalina.core.StandardHostValve.invoke line: 139 
       org.apache.catalina.valves.ErrorReportValve.invoke line: 92 
       org.apache.catalina.valves.AbstractAccessLogValve.invoke line: 678 
       org.apache.catalina.valves.rewrite.RewriteValve.invoke line: 282 
       org.apache.catalina.core.StandardEngineValve.invoke line: 74 
       org.apache.catalina.valves.RemoteIpValve.invoke line: 679 
       org.apache.catalina.valves.rewrite.RewriteValve.invoke line: 539 
       org.apache.catalina.core.StandardEngineValve.invoke line: 74 
       org.apache.catalina.valves.RemoteIpValve.invoke line: 679 
       org.apache.catalina.connector.CoyoteAdapter.service line: 343 
       org.apache.coyote.http11.Http11Processor.service line: 408 
       org.apache.coyote.AbstractProcessorLight.process line: 66 
       org.apache.coyote.AbstractProtocol$ConnectionHandler.process line: 834 
       org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun line: 1415 
       org.apache.tomcat.util.net.SocketProcessorBase.run line: 49 
       java.util.concurrent.ThreadPoolExecutor.runWorker line: 1149 
       java.util.concurrent.ThreadPoolExecutor$Worker.run line: 624 
       org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run line: 61 
       java.lang.Thread.run line: 748 
    
    Example 2:
    Stack Trace
    http-nio-8080-exec-73 [130] (BLOCKED)
       com.isomorphic.datasource.DataSourceManager.getDataSource line: 167 
       com.isomorphic.datasource.DataSourceManager.getDataSource line: 121 
       com.isomorphic.datasource.DSRequest.getDataSource line: 2356 
       com.isomorphic.datasource.DSRequest.setDataSourceName line: 2330 
       com.isomorphic.datasource.DSRequest.<init> line: 436 
       com.lmscompany.lms.server.SendMail.writeActionMails line: 166 
       com.lmscompany.lms.server.SendMail.doGet line: 50 
       javax.servlet.http.HttpServlet.service line: 634 
       javax.servlet.http.HttpServlet.service line: 741 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 231 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       org.apache.tomcat.websocket.server.WsFilter.doFilter line: 53 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       com.isomorphic.servlet.CompressionFilter._doFilter line: 247 
       com.isomorphic.servlet.BaseFilter.doFilter line: 93 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter line: 126 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       org.apache.catalina.core.StandardWrapperValve.invoke line: 200 
       org.apache.catalina.core.StandardContextValve.invoke line: 96 
       org.apache.catalina.authenticator.AuthenticatorBase.invoke line: 490 
       org.apache.catalina.core.StandardHostValve.invoke line: 139 
       org.apache.catalina.valves.ErrorReportValve.invoke line: 92 
       org.apache.catalina.valves.AbstractAccessLogValve.invoke line: 678 
       org.apache.catalina.valves.rewrite.RewriteValve.invoke line: 282 
       org.apache.catalina.core.StandardEngineValve.invoke line: 74 
       org.apache.catalina.valves.RemoteIpValve.invoke line: 679 
       org.apache.catalina.valves.rewrite.RewriteValve.invoke line: 539 
       org.apache.catalina.core.StandardEngineValve.invoke line: 74 
       org.apache.catalina.valves.RemoteIpValve.invoke line: 679 
       org.apache.catalina.connector.CoyoteAdapter.service line: 343 
       org.apache.coyote.http11.Http11Processor.service line: 408 
       org.apache.coyote.AbstractProcessorLight.process line: 66 
       org.apache.coyote.AbstractProtocol$ConnectionHandler.process line: 834 
       org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun line: 1415 
       org.apache.tomcat.util.net.SocketProcessorBase.run line: 49 
       java.util.concurrent.ThreadPoolExecutor.runWorker line: 1149 
       java.util.concurrent.ThreadPoolExecutor$Worker.run line: 624 
       org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run line: 61 
       java.lang.Thread.run line: 748 
    
    Example 3:
    Stack Trace
    http-nio-8080-exec-65 [122] (BLOCKED)
       com.isomorphic.datasource.DataSourceManager.getDataSource line: 167 
       com.isomorphic.datasource.DataSourceManager.getDataSource line: 121 
       com.isomorphic.datasource.DSRequest.getDataSource line: 2356 
       com.isomorphic.datasource.DSRequest.setDataSourceName line: 2330 
       com.isomorphic.datasource.DSRequest.<init> line: 436 
       com.lmscompany.lms.server.SendMail.writeActionMails line: 166 
       com.lmscompany.lms.server.SendMail.doGet line: 50 
       javax.servlet.http.HttpServlet.service line: 634 
       javax.servlet.http.HttpServlet.service line: 741 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 231 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       org.apache.tomcat.websocket.server.WsFilter.doFilter line: 53 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       com.isomorphic.servlet.CompressionFilter._doFilter line: 247 
       com.isomorphic.servlet.BaseFilter.doFilter line: 93 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter line: 126 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       org.apache.catalina.core.StandardWrapperValve.invoke line: 200 
       org.apache.catalina.core.StandardContextValve.invoke line: 96 
       org.apache.catalina.authenticator.AuthenticatorBase.invoke line: 490 
       org.apache.catalina.core.StandardHostValve.invoke line: 139 
       org.apache.catalina.valves.ErrorReportValve.invoke line: 92 
       org.apache.catalina.valves.AbstractAccessLogValve.invoke line: 678 
       org.apache.catalina.valves.rewrite.RewriteValve.invoke line: 282 
       org.apache.catalina.core.StandardEngineValve.invoke line: 74 
       org.apache.catalina.valves.RemoteIpValve.invoke line: 679 
       org.apache.catalina.valves.rewrite.RewriteValve.invoke line: 539 
       org.apache.catalina.core.StandardEngineValve.invoke line: 74 
       org.apache.catalina.valves.RemoteIpValve.invoke line: 679 
       org.apache.catalina.connector.CoyoteAdapter.service line: 343 
       org.apache.coyote.http11.Http11Processor.service line: 408 
       org.apache.coyote.AbstractProcessorLight.process line: 66 
       org.apache.coyote.AbstractProtocol$ConnectionHandler.process line: 834 
       org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun line: 1415 
       org.apache.tomcat.util.net.SocketProcessorBase.run line: 49 
       java.util.concurrent.ThreadPoolExecutor.runWorker line: 1149 
       java.util.concurrent.ThreadPoolExecutor$Worker.run line: 624 
       org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run line: 61 
       java.lang.Thread.run line: 748
    A few had a stack trace starting with this:
    Code:
    Stack Trace
    http-nio-8080-exec-29 [86] (BLOCKED)
       com.isomorphic.pool.ISCObjectPool.borrowObject line: 111 
       com.isomorphic.pool.PoolManager.borrowObject line: 102 
       com.isomorphic.pool.PoolManager.borrowObject line: 76 
       com.isomorphic.sql.SQLConnectionManager.getConnection line: 177 
       com.isomorphic.sql.SQLDriver.getTransformedResults line: 953 
       com.isomorphic.sql.SQLDriver.executeQuery line: 1394 
       com.isomorphic.sql.SQLDataSource.executeNativeQuery line: 585 
       com.isomorphic.sql.SQLDataSource.executeNativeQuery line: 579 
       com.isomorphic.sql.SQLDataSource.SQLExecute line: 1772 
       com.isomorphic.sql.SQLDataSource.processRequest line: 439 
       com.isomorphic.sql.SQLDataSource.executeFetch line: 384 
       com.lmscompany.lms.server.LMSSQLDataSource.executeFetch line: 149 
       com.isomorphic.datasource.DataSource.execute line: 2384 
       com.lmscompany.lms.server.LMSSQLDataSource.execute line: 191 
       com.isomorphic.application.AppBase.executeDefaultDSOperation line: 646 
       com.isomorphic.application.AppBase.executeAppOperation line: 547 
       com.isomorphic.application.AppBase.execute line: 490 
       com.isomorphic.datasource.DSRequest.execute line: 2835 
       com.lmscompany.lms.server.worker.T_FIELD_CONFIG.getData line: 94 
       com.lmscompany.lms.server.RefreshCycle.checkCacheAge line: 52 
       com.lmscompany.lms.server.worker.T_FIELD_CONFIG.getFieldConfigDataResponse line: 102 
       com.lmscompany.lms.server.RegisterDS.formatFieldMultiple line: 368 
       com.lmscompany.lms.server.RegisterDS.access$400 line: 52 
       com.lmscompany.lms.server.RegisterDS$3.getDataSource line: 251 
       com.isomorphic.datasource.DataSource.getDynamicDataSource line: 807 
       com.isomorphic.datasource.DataSource.forName line: 459 
       com.isomorphic.datasource.PoolableDataSourceFactory.makeUnpooledObject line: 134 
       com.isomorphic.pool.PoolManager.borrowUnpooledObject line: 143 
       com.isomorphic.datasource.DataSourceManager.getDataSource line: 169 
       com.isomorphic.datasource.DataSourceManager.getDataSource line: 121 
       com.isomorphic.datasource.BasicDataSource.getSuper line: 798 
       com.isomorphic.datasource.BasicDataSource.init line: 287 
       com.isomorphic.sql.SQLDataSource.init line: 257 
       com.isomorphic.datasource.DataSource.initialize line: 921 
       com.isomorphic.datasource.BasicDataSource.fromConfig line: 238 
       com.isomorphic.datasource.DataSource.fromConfig line: 906 
       com.isomorphic.datasource.DataSource.loadDS line: 534 
       com.isomorphic.datasource.DataSource.forName line: 462 
       com.isomorphic.datasource.PoolableDataSourceFactory.makeUnpooledObject line: 134 
       com.isomorphic.datasource.PoolableDataSourceFactory.makeObject line: 153 
       org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject line: 1220 
       com.isomorphic.pool.ISCKeyedObjectPool.borrowObject line: 106 
       com.isomorphic.pool.PoolManager.borrowObject line: 92 
       com.isomorphic.datasource.DataSourceManager.getDataSource line: 171 
       com.isomorphic.datasource.DataSourceManager.getDataSource line: 121 
       com.isomorphic.datasource.DSRequest.getDataSource line: 2356 
       com.isomorphic.datasource.DSRequest.decodeUploadedStrings line: 826 
       com.isomorphic.datasource.DSRequest.<init> line: 707 
       com.isomorphic.rpc.RPCManager.parseRequest line: 2526 
       com.isomorphic.rpc.RPCManager.<init> line: 447 
       com.lmscompany.lms.server.LMSIDACall.processRequest line: 48 
       com.isomorphic.servlet.IDACall._processRequest line: 119 
       com.isomorphic.servlet.IDACall.doPost line: 79 
       javax.servlet.http.HttpServlet.service line: 660 
       com.isomorphic.servlet.BaseServlet.service line: 176 
       javax.servlet.http.HttpServlet.service line: 741 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 231 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       org.apache.tomcat.websocket.server.WsFilter.doFilter line: 53 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       com.isomorphic.servlet.CompressionFilter._doFilter line: 260 
       com.isomorphic.servlet.BaseFilter.doFilter line: 93 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter line: 126 
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193 
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166 
       org.apache.catalina.core.StandardWrapperValve.invoke line: 200 
       org.apache.catalina.core.StandardContextValve.invoke line: 96 
       org.apache.catalina.authenticator.AuthenticatorBase.invoke line: 607 
       org.apache.catalina.core.StandardHostValve.invoke line: 139 
       org.apache.catalina.valves.ErrorReportValve.invoke line: 92 
       org.apache.catalina.valves.AbstractAccessLogValve.invoke line: 678 
       org.apache.catalina.valves.rewrite.RewriteValve.invoke line: 542 
       org.apache.catalina.core.StandardEngineValve.invoke line: 74 
       org.apache.catalina.valves.RemoteIpValve.invoke line: 679 
       org.apache.catalina.connector.CoyoteAdapter.service line: 343 
       org.apache.coyote.http11.Http11Processor.service line: 408 
       org.apache.coyote.AbstractProcessorLight.process line: 66 
       org.apache.coyote.AbstractProtocol$ConnectionHandler.process line: 834 
       org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun line: 1415 
       org.apache.tomcat.util.net.SocketProcessorBase.run line: 49 
       java.util.concurrent.ThreadPoolExecutor.runWorker line: 1149 
       java.util.concurrent.ThreadPoolExecutor$Worker.run line: 624 
       org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run line: 61 
       java.lang.Thread.run line: 748
    Does this (DataSourceManager.getDataSource:167 and ISCObjectPool.borrowObject:111) somehow relate to a potential problem with Apache Commons DBCP?

    Thank you & Best regards
    Blama

  • Isomorphic
    replied
    You arenít blocked, itís a different package name.

    So it looks like the only problem continues to be the obscure bug involving eviction runs which they appear to have fixed (but not yet released) in 1.x. Let us know if weíve missed something.

    Leave a comment:


  • monk
    replied
    Hi

    We too would like to upgrade our DBCP dependency but we're blocked by the fact that SmartGWT depends on it.
    Possibly we can run both versions in parallel, we haven't tried that path yet (but I don't think it's ideal either).

    DBCP 1.4 is really old and there is no support on it anymore. JIRA tickets are referred to newer versions.

    So +1 for a dbcp upgrade.

    Thanks

    Leave a comment:


  • Blama
    replied
    Hi Isomorphic,

    I understand that it is not possible to do such a change in a 12.0p nightly. With nearly-out releases I assume you refer to 12.1d/p, which is of course bad from my point of view. So you are saying you plan a change for next next major release?
    IF the minIdle: 0 change fixes it for me, I'm happy with 1.4 as well. IF our planned switch Oracle->Postgres does invalidate the necessity of timeBetweenEvictionRunsMillis at all, I'm also happy with 1.4. So let's hope that this is the case. If not, building from source, which I already considered, might be a last resort.

    And while I'm a big fan of keeping proven solutions, I also think that it is a good idea if you updated all your used libraries at least to the state where you use a version that is in active development, so that jar-file.version bumping becomes possible if ever necessary.
    I did not check all Apache commons and other used libraries, but for some there are already versions with namespace changes out, so if there appears a bug in such an unmaintained version, there will be a problem like now.

    Best regards
    Blama

    Leave a comment:


  • Isomorphic
    replied
    Actually they seem to indicate the bug has been fixed, but not released, so perhaps there's hope. There's always the (painful) path of building it yourself from source.

    There's a not a lot we can do for you here in the immediate term, as DBCP2 is not just a minor version bump but a replacement library with major internal differences. We will of course switch over sometime soon, but we can't switch existing or nearly-out releases to the new library, not when the only problem with the existing library is a feature that, so far, not even one other user seems to need. Sorry that this is a problem for you.

    Leave a comment:


  • Blama
    replied
    Hi Isomorphic,

    chances for a 1.4.1 are bad, see here, where they are suggesting to switch to 2.6.0.
    1.4 is more than 9 years old now and commons.dbcp2 is in active development since 2014 with ~2 releases a year.

    For now, I tried setting sql.pool.minIdle: 0 as they claim that positive minIdle and timeBetweenEvictionRunsMillis together will trigger the issue. I'll watch if this mitigates the problem successfully.

    Best regards
    Blama

    Leave a comment:


  • Isomorphic
    replied
    DataSourceManager.get() does involve a DBCP pool. So it looks like you will need to turn off eviction runs until 1.4.1 comes out.

    Leave a comment:


  • Blama
    replied
    Hi Isomorphic,

    this now happened again for another instance, and here all blocked threads are like this:
    Code:
    Stack Trace
    http-nio-8080-exec-15 [39] (BLOCKED)
       com.isomorphic.pool.ISCObjectPool.borrowObject line: 111
       com.isomorphic.pool.PoolManager.borrowObject line: 102
       com.isomorphic.pool.PoolManager.borrowObject line: 76
       com.isomorphic.sql.SQLConnectionManager.getConnection line: 177
       com.isomorphic.sql.SQLDriver.update line: 1160
       com.isomorphic.sql.SQLDriver.executeUpdate line: 1370
       com.isomorphic.sql.SQLDataSource.executeNativeUpdate line: 597
       com.isomorphic.sql.SQLDataSource.SQLExecute line: 1853
       com.isomorphic.sql.SQLDataSource.processRequest line: 439
       com.isomorphic.sql.SQLDataSource.executeUpdate line: 388
       com.lmscompany.lms.server.LMSSQLDataSource.executeUpdate line: 200
       com.isomorphic.datasource.DataSource.execute line: 2425
       com.lmscompany.lms.server.LMSSQLDataSource.execute line: 191
       com.isomorphic.application.AppBase.executeDefaultDSOperation line: 646
       com.isomorphic.application.AppBase.executeAppOperation line: 547
       com.isomorphic.application.AppBase.execute line: 490
       com.isomorphic.datasource.DSRequest.execute line: 2835
       com.lmscompany.lms.server.LogIncomingRequest.updateDSRequestStatus line: 168
       com.lmscompany.lms.server.LMSIDACall.handleDSRequest line: 221
       com.isomorphic.servlet.IDACall.processRPCTransaction line: 187
       com.lmscompany.lms.server.LMSIDACall.processRequest line: 109
       com.isomorphic.servlet.IDACall._processRequest line: 119
       com.isomorphic.servlet.IDACall.doPost line: 79
       javax.servlet.http.HttpServlet.service line: 660
       com.isomorphic.servlet.BaseServlet.service line: 176
       javax.servlet.http.HttpServlet.service line: 741
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 231
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166
       org.apache.tomcat.websocket.server.WsFilter.doFilter line: 53
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166
       com.isomorphic.servlet.CompressionFilter._doFilter line: 260
       com.isomorphic.servlet.BaseFilter.doFilter line: 93
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166
       org.apache.catalina.filters.HttpHeaderSecurityFilter.doFilter line: 126
       org.apache.catalina.core.ApplicationFilterChain.internalDoFilter line: 193
       org.apache.catalina.core.ApplicationFilterChain.doFilter line: 166
       org.apache.catalina.core.StandardWrapperValve.invoke line: 200
       org.apache.catalina.core.StandardContextValve.invoke line: 96
       org.apache.catalina.authenticator.AuthenticatorBase.invoke line: 607
       org.apache.catalina.core.StandardHostValve.invoke line: 139
       org.apache.catalina.valves.ErrorReportValve.invoke line: 92
       org.apache.catalina.valves.AbstractAccessLogValve.invoke line: 678
       org.apache.catalina.valves.rewrite.RewriteValve.invoke line: 542
       org.apache.catalina.core.StandardEngineValve.invoke line: 74
       org.apache.catalina.valves.RemoteIpValve.invoke line: 679
       org.apache.catalina.connector.CoyoteAdapter.service line: 343
       org.apache.coyote.http11.Http11Processor.service line: 408
       org.apache.coyote.AbstractProcessorLight.process line: 66
       org.apache.coyote.AbstractProtocol$ConnectionHandler.process line: 834
       org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun line: 1415
       org.apache.tomcat.util.net.SocketProcessorBase.run line: 49
       java.util.concurrent.ThreadPoolExecutor.runWorker line: 1149
       java.util.concurrent.ThreadPoolExecutor$Worker.run line: 624
       org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run line: 61
       java.lang.Thread.run line: 748
    and from the line "com.isomorphic.sql.SQLDriver.executeUpdate line: 1370" it is different, depending on the type of DB-action (update, select, ...).

    And for information: All the connections blocking the system are created in a very short time. Out application does not have much traffic, but this is creation time of the connections for #1, fetched from the DB:
    Code:
    10.05.19 12:16:43
    13.05.19 11:18:24
    13.05.19 11:18:28
    13.05.19 11:18:28
    13.05.19 11:18:29
    13.05.19 11:18:29
    13.05.19 11:18:29
    13.05.19 11:18:30
    13.05.19 11:18:31
    13.05.19 11:18:31
    13.05.19 11:18:31
    It are 11 lines - this and the fact that one connection is very old matches the configuration:
    10 connections: server.properties setting sql.pool.maxActive: 10
    1 connection: Tomcat JDBCRealm from server.xml

    Those 10 new connections in 7 seconds can't possible be from user interaction.

    Best regards
    Blama

    Leave a comment:

Working...
X