Announcement

**Isomorphic** · 12 Mar 2014, 09:52

So again, step 1 is to look at whether the server sent a response and what it was.

This is something you'd do with Firebug or a similar tool (basically press F12 in whatever browser).

As far as the ThreadLocal warnings, it looks like Spring or another framework you are using might be leaking ThreadLocals. We don't create any ThreadLocals of the types shown in your logs.

**JLivermore** · 12 Mar 2014, 10:14

I see 3 calls to IDACall that are pending and never seem to come back?

http://127.0.0.1:8080/tzw/com.anstca.tz.Main/sc/IDACall?isc_rpc=1&isc_v=v9.1p_2014-03-11&isc_xhr=1
http://127.0.0.1:8080/tzw/com.anstca.tz.Main/sc/IDACall?isc_rpc=1&isc_v=v9.1p_2014-03-11&isc_xhr=1
http://127.0.0.1:8080/tzw/com.anstca.tz.Main/sc/IDACall?isc_rpc=1&isc_v=v9.1p_2014-03-11&isc_xhr=1

**Isomorphic** · 12 Mar 2014, 10:18

Calls to DataSource.load() do not go to IDACall. You previously said you had isolated this to a DataSource.load() call...

You didn't share any details of what tool you're using or how you determined they "never seem to come back", but you might be looking at an unrelated issue (or a non-issue).

**JLivermore** · 12 Mar 2014, 10:22

In IE-11, F-12 brings up the browser debugging tools, and in there there is a network tab that shows network activity.

Correct, from what I see in the server log, after the server delivers the XML DataSource from the DSGenerator, everything seems to stop.

Aside from the 3 IDACall(s), all requests appear to be coming back.

**JLivermore** · 12 Mar 2014, 10:30

attached is a screen shot of the network requests including the images downloaded after the timeout.

**JLivermore** · 12 Mar 2014, 10:35

here is the screen shot again...

Attached Files

**Isomorphic** · 12 Mar 2014, 10:40

OK.. so obviously you need to know what those requests are and what happened on the server.

The IE tool can show you the contents of the request. The RPC tab of the Developer Console can also show you those requests and will reveal more information (like what component triggered them).

Then you want to look at the server-side logs for those requests. You posted some other logs earlier, but there don't seem to be any logs from SmartGWT about those requests. This might suggest they never get to SmartGWT at all, which could happen if you have a misfiring Filter servlet, broken authentication or security system, or similar.

**JLivermore** · 13 Mar 2014, 06:04

Looking at the SmartClient Dev Console tracking RPCs I'm seeing a common request timing out. The URL is an IDACall of Type DSRequest. Looking at the timestamp of the request lines up with these messages in the server log. The messages below looks problematic, does 'Unsupported with Accept-Encoding header' mean a failure of the request from the client on the server?

Code:

=== 2014-03-13 09:53:54,738 [ec-1] INFO  RequestContext - URL: '/tzw/com.anstca.tz.Main/sc/IDACall', User-Agent: 'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko': Unsupported with Accept-Encoding header
=== 2014-03-13 09:53:55,731 [ec-2] INFO  RequestContext - URL: '/tzw/com.anstca.tz.Main/rpc/pivottableremoteservice', User-Agent: 'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko': Unsupported with Accept-Encoding header
=== 2014-03-13 09:53:56,246 [c-10] INFO  RequestContext - URL: '/tzw/com.anstca.tz.Main/sc/IDACall', User-Agent: 'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko': Unsupported with Accept-Encoding header
=== 2014-03-13 09:53:56,525 [ec-8] INFO  RequestContext - URL: '/tzw/com.anstca.tz.Main/sc/IDACall', User-Agent: 'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko': Unsupported with Accept-Encoding header

Here is the DSRequest for the suspected first message above

Code:

{
    dataSource:"pivot_table_report_launcher_list", 
    operationType:"fetch", 
    data:{
        UserID:"233", 
        FactTableID:"POF"
    }, 
    textMatchStyle:"exact", 
    showPrompt:false, 
    oldValues:{
        UserID:"233", 
        FactTableID:"POF"
    }, 
    requestId:"pivot_table_report_launcher_list$6278", 
    fallbackToEval:false, 
    lastClientEventThreadCode:"TMR4", 
    bypassCache:true
}

is there an email address I can send a screen shot of the DevConsole's RPC tab to? it's a large image, and your image limitations won't let me post the screen shot. I tried emailing support@isomorphic.com and got a bounce back?

**JLivermore** · 13 Mar 2014, 08:29

The previous posting probably doesn't provide much info. Hopefully this one will...

I think I have narrowed this down. Looking at the server logs and correlating the Time Sent in the DevConsole's RPC tab with the timestamps in the server log, it appears that the server gets hung when we're expecting the RPCManager on the server to make a Spring call.

In the same login session, I have one case where a DSRequest is made with a predefined XML datasource called 'pivot_table_report_launcher_list' works fine. And then a second one hangs.

Code:

<?xml version="1.0" encoding="UTF-8"?>
<DataSource
    ID="pivot_table_report_launcher_list"
    serverType="generic">
    <fields>
        <field name="reportID"   type="integer"  primaryKey="true" />
        <field name="reportName"   type="text"  />
    </fields>

    <operationBindings>
        <binding operationType="fetch" serverMethod="fetchReportsForUserIDFactTableID">
            <serverObject lookupStyle="spring" bean="pivotTableServices"/>
        </binding>
    </operationBindings>

</DataSource>

I've managed to strip out all other messages in the log to clearly see the specific thread assigned to the call:

Code:

=====================================
Successful pivot_table_report_launcher_list
RPC Time Sent: 12:01:54:304
=====================================
=== 2014-03-13 12:01:54,305 [ec-3] INFO  RequestContext - URL: '/tzw/com.anstca.tz.Main/sc/IDACall', User-Agent: 'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko': Unsupported with Accept-Encoding header
=== 2014-03-13 12:01:57,636 [ec-3] DEBUG XML - Parsed XML from (in memory stream): 1ms
=== 2014-03-13 12:01:57,637 [ec-3] DEBUG RPCManager - Processing 1 requests.
=== 2014-03-13 12:01:57,637 [ec-3] DEBUG DSRequest - Caching instance 320 of DS pivot_table_report_launcher_list from DSRequest.getDataSource()
=== 2014-03-13 12:01:57,638 [ec-3] DEBUG RPCManager - Request #1 (DSRequest) payload: {
    criteria:{
        UserID:"233",
        FactTableID:"POF"
    },
    operationConfig:{
        dataSource:"pivot_table_report_launcher_list",
        operationType:"fetch",
        textMatchStyle:"exact"
    },
    appID:"builtinApplication",
    operation:"pivot_table_report_launcher_list_fetch",
    oldValues:{
        UserID:"233",
        FactTableID:"POF"
    }
}
2014-03-13 12:01:57,638 DEBUG [com.anstca.tz.server.post.mvc.servlet.TZWCall] - <==> TZWCall DataSourceName:pivot_table_report_launcher_list>
=== 2014-03-13 12:01:57,638 [ec-3] DEBUG DeclarativeSecurity - Processing security checks for DataSource null, field null
=== 2014-03-13 12:01:57,639 [ec-3] DEBUG DeclarativeSecurity - DataSource pivot_table_report_launcher_list is not in the pre-checked list, processing...
=== 2014-03-13 12:01:57,640 [ec-3] INFO  ServerObject - DMI on Spring bean: pivotTableServices
2014-03-13 12:01:57,640 DEBUG [com.anstca.tz.server.floater.pivottable.service.PivotTableServicesImpl] - <PivotTableServicesImpl.fetchReports(DSRequest) criteria:[{UserID=233, FactTableID=POF}]>
2014-03-13 12:01:57,640 DEBUG [com.anstca.tz.server.floater.pivottable.dao.PivotTableDaoJdbcImpl] - <clientID:233 factTableID:POF>
2014-03-13 12:01:57,640 DEBUG [com.anstca.tz.server.floater.pivottable.dao.PivotTableDaoJdbcImpl] - <getFactTableSimpleJdbcTemplate factTableKey:tradeZoomDataSource>
=== 2014-03-13 12:01:57,645 [ec-3] INFO  DSResponse - DSResponse: List with 5 items
2014-03-13 12:01:57,645 DEBUG [com.anstca.tz.server.floater.pivottable.service.PivotTableServicesImpl] - <PivotTableServicesImpl.fetchReports(DSRequest) results:5>
=== 2014-03-13 12:01:57,645 [ec-3] DEBUG RPCManager - Content type for RPC transaction: text/plain; charset=UTF-8
=== 2014-03-13 12:01:57,645 [ec-3] DEBUG RPCManager - DMI response, dropExtraFields: true
=== 2014-03-13 12:01:57,646 [ec-3] INFO  Compression - /tzw/com.anstca.tz.Main/sc/IDACall: 470 -> 274 bytes


=====================================
Failed pivot_table_report_launcher_list
RPC Time Sent: 12:02:31:565
=====================================
=== 2014-03-13 12:02:31,566 [c-10] INFO  RequestContext - URL: '/tzw/com.anstca.tz.Main/sc/IDACall', User-Agent: 'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko': Unsupported with Accept-Encoding header
=== 2014-03-13 12:02:32,328 [c-10] DEBUG XML - Parsed XML from (in memory stream): 1ms

It appears that the RPCManager is not making the Spring call in the second instance, thus nothing gets sent back to the client and then we have the timeout.

Is there a way to enable more detailed debug output from the server side for SmartGWT?

**Isomorphic** · 13 Mar 2014, 13:30

SmartGWT uses standard log4j, so you can modify the log4.isc.config.xml file we ship with the product to enable more logging.

Note that the call to Spring is probably not the problem. Before attempting to contact Spring, there would have been a few more logs, like this one (from the successful log):

=== 2014-03-13 12:01:57,637 [ec-3] DEBUG RPCManager - Processing 1 requests.

Although additional logging output might help, probably the best thing is to attach a debugger to the JVM and look at the hung thread.

**JLivermore** · 14 Mar 2014, 04:57

How many instances of com.isomorphic.rpc.RPCManager would you expect to see in singer server? In my development instance, I'm seeing 5 instances of:

com.isomorphic.rpc.Manager
com.isomorphic.appliation.AppBase

Is this ok?

**JLivermore** · 14 Mar 2014, 05:12

Here is a thread dump after the server hangs:

Code:

Full thread dump

"http-bio-8080-exec-7@4698" daemon prio=5 tid=0x2b nid=NA waiting for monitor entry
  java.lang.Thread.State: BLOCKED
	 blocks http-bio-8080-exec-10@5113
	 waiting for http-bio-8080-exec-10@5113 to release lock on <0x1551> (a com.isomorphic.pool.ISCKeyedObjectPool)
	  at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:937)
	  at com.isomorphic.pool.PoolManager.borrowObject(PoolManager.java:84)
	  at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:95)
	  at com.isomorphic.datasource.ValidationContext.getType(ValidationContext.java:299)
	  at com.isomorphic.datasource.ValidationContext.getType(ValidationContext.java:279)
	  at com.isomorphic.datasource.DataSource.getElementType(DataSource.java:1204)
	  at com.isomorphic.datasource.DataSource.recordsFromXML(DataSource.java:1220)
	  at com.isomorphic.xml.XML.toDSRecords(XML.java:337)
	  at com.isomorphic.xml.XML.toDSRecords(XML.java:328)
	  at com.isomorphic.xml.XML.toDSRecords(XML.java:324)
	  at com.isomorphic.xml.XML.toDSRecords(XML.java:316)
	  at com.isomorphic.datasource.DataSource.fromXML(DataSource.java:608)
	  at com.isomorphic.datasource.DataSource.fromXML(DataSource.java:577)
	  at com.anstca.tz.server.floater.pivottable.ds.PivotTableTreeDSGenerator.getDataSource(PivotTableTreeDSGenerator.java:37)
	  at com.isomorphic.datasource.DataSource.getDynamicDataSource(DataSource.java:392)
	  - locked <0xd05> (a java.lang.Class)
	  at com.isomorphic.datasource.DataSource.forName(DataSource.java:228)
	  at com.isomorphic.datasource.DataSource.forName(DataSource.java:222)
	  at com.isomorphic.datasource.DataSource.forName(DataSource.java:211)
	  at com.isomorphic.datasource.PoolableDataSourceFactory.makeUnpooledObject(PoolableDataSourceFactory.java:130)
	  - locked <0x1553> (a com.isomorphic.datasource.PoolableDataSourceFactory)
	  at com.isomorphic.pool.PoolManager.borrowUnpooledObject(PoolManager.java:123)
	  at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:93)
	  at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:86)
	  at com.isomorphic.datasource.DSRequest.getDataSource(DSRequest.java:2220)
	  at com.isomorphic.datasource.DSRequest.decodeUploadedStrings(DSRequest.java:789)
	  at com.isomorphic.datasource.DSRequest.<init>(DSRequest.java:676)
	  at com.isomorphic.rpc.RPCManager.parseRequest(RPCManager.java:2249)
	  at com.isomorphic.rpc.RPCManager.<init>(RPCManager.java:309)
	  at com.isomorphic.rpc.RPCManager.<init>(RPCManager.java:289)
	  at com.isomorphic.servlet.IDACall.processRequest(IDACall.java:132)
	  at com.isomorphic.servlet.IDACall.doPost(IDACall.java:73)
	  at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
	  at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:152)
	  at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
	  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
	  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
	  at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
	  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
	  at com.isomorphic.servlet.CompressionFilter.doFilter(CompressionFilter.java:260)
	  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
	  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
	  at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
	  at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:183)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
	  at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
	  at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:343)
	  at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:260)
	  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
	  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
	  at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
	  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
	  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
	  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
	  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
	  at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
	  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
	  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
	  at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
	  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
	  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:409)
	  at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1044)
	  at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
	  at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
	  - locked <0x1554> (a org.apache.tomcat.util.net.SocketWrapper)
	  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	  at java.lang.Thread.run(Thread.java:744)

"http-bio-8080-exec-10@5113" daemon prio=5 tid=0x35 nid=NA waiting for monitor entry
  java.lang.Thread.State: BLOCKED
	 blocks http-bio-8080-exec-7@4698
	 blocks http-bio-8080-exec-3@4217
	 waiting for http-bio-8080-exec-7@4698 to release lock on <0x1553> (a com.isomorphic.datasource.PoolableDataSourceFactory)
	  at com.isomorphic.datasource.PoolableDataSourceFactory.makeUnpooledObject(PoolableDataSourceFactory.java:129)
	  at com.isomorphic.datasource.PoolableDataSourceFactory.makeObject(PoolableDataSourceFactory.java:138)
	  at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:964)
	  - locked <0x1551> (a com.isomorphic.pool.ISCKeyedObjectPool)
	  at com.isomorphic.pool.PoolManager.borrowObject(PoolManager.java:84)
	  at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:95)
	  at com.isomorphic.datasource.ValidationContext.getType(ValidationContext.java:299)
	  at com.isomorphic.datasource.ValidationContext.getType(ValidationContext.java:279)
	  at com.isomorphic.datasource.DataSource.getElementType(DataSource.java:1204)
	  at com.isomorphic.datasource.DataSource.recordsFromXML(DataSource.java:1220)
	  at com.isomorphic.datasource.DataSource.recordsFromXML(DataSource.java:1274)
	  at com.isomorphic.datasource.ListType.toRecord(ListType.java:53)
	  at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1158)
	  at com.isomorphic.datasource.DataSource.create(DataSource.java:1282)
	  at com.isomorphic.datasource.BasicDataSource.validateFieldValue(BasicDataSource.java:1768)
	  at com.isomorphic.datasource.BasicDataSource.elementAsRecord(BasicDataSource.java:1503)
	  at com.isomorphic.datasource.BasicDataSource.toRecord(BasicDataSource.java:1199)
	  at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1158)
	  at com.isomorphic.datasource.DataSource.recordsFromXML(DataSource.java:1238)
	  at com.isomorphic.xml.XML.toDSRecords(XML.java:337)
	  at com.isomorphic.xml.XML.toDSRecords(XML.java:328)
	  at com.isomorphic.xml.XML.toDSRecords(XML.java:324)
	  at com.isomorphic.xml.XML.toDSRecords(XML.java:316)
	  at com.isomorphic.rpc.RPCManager.parseRequest(RPCManager.java:2184)
	  at com.isomorphic.rpc.RPCManager.<init>(RPCManager.java:309)
	  at com.isomorphic.rpc.RPCManager.<init>(RPCManager.java:289)
	  at com.isomorphic.servlet.IDACall.processRequest(IDACall.java:132)
	  at com.isomorphic.servlet.IDACall.doPost(IDACall.java:73)
	  at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
	  at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:152)
	  at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
	  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
	  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
	  at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
	  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
	  at com.isomorphic.servlet.CompressionFilter.doFilter(CompressionFilter.java:260)
	  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
	  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
	  at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
	  at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:183)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
	  at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
	  at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:343)
	  at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:260)
	  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
	  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
	  at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
	  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
	  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
	  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
	  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
	  at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
	  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
	  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
	  at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
	  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
	  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:409)
	  at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1044)
	  at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
	  at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
	  - locked <0x1552> (a org.apache.tomcat.util.net.SocketWrapper)
	  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	  at java.lang.Thread.run(Thread.java:744)

"http-bio-8080-exec-3@4217" daemon prio=5 tid=0x27 nid=NA waiting for monitor entry
  java.lang.Thread.State: BLOCKED
	 waiting for http-bio-8080-exec-10@5113 to release lock on <0x1551> (a com.isomorphic.pool.ISCKeyedObjectPool)
	  at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:937)
	  at com.isomorphic.pool.PoolManager.borrowObject(PoolManager.java:84)
	  at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:95)
	  at com.isomorphic.datasource.ValidationContext.getType(ValidationContext.java:299)
	  at com.isomorphic.datasource.ValidationContext.getType(ValidationContext.java:279)
	  at com.isomorphic.datasource.DataSource.getElementType(DataSource.java:1204)
	  at com.isomorphic.datasource.DataSource.recordsFromXML(DataSource.java:1220)
	  at com.isomorphic.xml.XML.toDSRecords(XML.java:337)
	  at com.isomorphic.xml.XML.toDSRecords(XML.java:328)
	  at com.isomorphic.xml.XML.toDSRecords(XML.java:324)
	  at com.isomorphic.xml.XML.toDSRecords(XML.java:316)
	  at com.isomorphic.rpc.RPCManager.parseRequest(RPCManager.java:2184)
	  at com.isomorphic.rpc.RPCManager.<init>(RPCManager.java:309)
	  at com.isomorphic.rpc.RPCManager.<init>(RPCManager.java:289)
	  at com.isomorphic.servlet.IDACall.processRequest(IDACall.java:132)
	  at com.isomorphic.servlet.IDACall.doPost(IDACall.java:73)
	  at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
	  at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:152)
	  at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
	  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
	  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
	  at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
	  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
	  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
	  at com.isomorphic.servlet.CompressionFilter.doFilter(CompressionFilter.java:260)
	  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
	  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
	  at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:118)
	  at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:84)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:113)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:103)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:113)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:146)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:54)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:45)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilter(BasicAuthenticationFilter.java:150)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:183)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.session.ConcurrentSessionFilter.doFilter(ConcurrentSessionFilter.java:125)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
	  at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:342)
	  at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:192)
	  at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:160)
	  at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:343)
	  at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:260)
	  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
	  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
	  at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
	  at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
	  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
	  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
	  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
	  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
	  at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
	  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
	  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:100)
	  at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
	  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
	  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:409)
	  at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1044)
	  at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
	  at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
	  - locked <0x1556> (a org.apache.tomcat.util.net.SocketWrapper)
	  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	  at java.lang.Thread.run(Thread.java:744)

"main@1" prio=5 tid=0x1 nid=NA runnable
  java.lang.Thread.State: RUNNABLE
	  at java.net.DualStackPlainSocketImpl.accept0(DualStackPlainSocketImpl.java:-1)
	  at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
	  at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
	  at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:198)
	  - locked <0x14fb> (a java.net.SocksSocketImpl)
	  at java.net.ServerSocket.implAccept(ServerSocket.java:530)
	  at java.net.ServerSocket.accept(ServerSocket.java:498)
	  at org.apache.catalina.core.StandardServer.await(StandardServer.java:452)
	  at org.apache.catalina.startup.Catalina.await(Catalina.java:778)
	  at org.apache.catalina.startup.Catalina.start(Catalina.java:724)
	  at sun.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-1)
	  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
	  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	  at java.lang.reflect.Method.invoke(Method.java:606)
	  at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:322)
	  at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:456)

"RMI TCP Accept-0@1145" daemon prio=5 tid=0x10 nid=NA runnable
  java.lang.Thread.State: RUNNABLE
	  at java.net.DualStackPlainSocketImpl.accept0(DualStackPlainSocketImpl.java:-1)
	  at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
	  at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
	  at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:198)
	  - locked <0x14f6> (a java.net.SocksSocketImpl)
	  at java.net.ServerSocket.implAccept(ServerSocket.java:530)
	  at java.net.ServerSocket.accept(ServerSocket.java:498)
	  at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
	  at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388)
	  at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360)
	  at java.lang.Thread.run(Thread.java:744)

"http-bio-8080-exec-9@4700" daemon prio=5 tid=0x2d nid=NA waiting
  java.lang.Thread.State: WAITING
	  at sun.misc.Unsafe.park(Unsafe.java:-1)
	  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
	  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
	  at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	  at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
	  at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
	  at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
	  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	  at java.lang.Thread.run(Thread.java:744)

"http-bio-8080-exec-8@4699" daemon prio=5 tid=0x2c nid=NA waiting
  java.lang.Thread.State: WAITING
	  at sun.misc.Unsafe.park(Unsafe.java:-1)
	  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
	  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
	  at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	  at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
	  at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
	  at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
	  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	  at java.lang.Thread.run(Thread.java:744)

"http-bio-8080-exec-6@4697" daemon prio=5 tid=0x2a nid=NA waiting
  java.lang.Thread.State: WAITING
	  at sun.misc.Unsafe.park(Unsafe.java:-1)
	  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
	  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
	  at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	  at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
	  at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
	  at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
	  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	  at java.lang.Thread.run(Thread.java:744)

"http-bio-8080-exec-4@4693" daemon prio=5 tid=0x28 nid=NA waiting
  java.lang.Thread.State: WAITING
	  at sun.misc.Unsafe.park(Unsafe.java:-1)
	  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
	  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
	  at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	  at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
	  at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
	  at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
	  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	  at java.lang.Thread.run(Thread.java:744)

"http-bio-8080-exec-2@4136" daemon prio=5 tid=0x26 nid=NA waiting
  java.lang.Thread.State: WAITING
	  at sun.misc.Unsafe.park(Unsafe.java:-1)
	  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
	  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
	  at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	  at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
	  at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
	  at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
	  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	  at java.lang.Thread.run(Thread.java:744)

"http-bio-8080-exec-1@4135" daemon prio=5 tid=0x25 nid=NA waiting
  java.lang.Thread.State: WAITING
	  at sun.misc.Unsafe.park(Unsafe.java:-1)
	  at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
	  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2043)
	  at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	  at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:104)
	  at org.apache.tomcat.util.threads.TaskQueue.take(TaskQueue.java:32)
	  at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
	  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	  at java.lang.Thread.run(Thread.java:744)

"ajp-bio-8009-Acceptor-0@1971" daemon prio=5 tid=0x1a nid=NA runnable
  java.lang.Thread.State: RUNNABLE
	  at java.net.DualStackPlainSocketImpl.accept0(DualStackPlainSocketImpl.java:-1)
	  at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
	  at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
	  at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:198)
	  - locked <0x14f3> (a java.net.SocksSocketImpl)
	  at java.net.ServerSocket.implAccept(ServerSocket.java:530)
	  at java.net.ServerSocket.accept(ServerSocket.java:498)
	  at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
	  at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:219)
	  at java.lang.Thread.run(Thread.java:744)

"http-bio-8080-Acceptor-0@1966" daemon prio=5 tid=0x18 nid=NA runnable
  java.lang.Thread.State: RUNNABLE
	  at java.net.DualStackPlainSocketImpl.accept0(DualStackPlainSocketImpl.java:-1)
	  at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
	  at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
	  at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:198)
	  - locked <0x14f4> (a java.net.SocksSocketImpl)
	  at java.net.ServerSocket.implAccept(ServerSocket.java:530)
	  at java.net.ServerSocket.accept(ServerSocket.java:498)
	  at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
	  at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:219)
	  at java.lang.Thread.run(Thread.java:744)

"RMI TCP Accept-1099@1088" daemon prio=5 tid=0xf nid=NA runnable
  java.lang.Thread.State: RUNNABLE
	  at java.net.DualStackPlainSocketImpl.accept0(DualStackPlainSocketImpl.java:-1)
	  at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
	  at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
	  at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:198)
	  - locked <0x14f7> (a java.net.SocksSocketImpl)
	  at java.net.ServerSocket.implAccept(ServerSocket.java:530)
	  at java.net.ServerSocket.accept(ServerSocket.java:498)
	  at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388)
	  at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360)
	  at java.lang.Thread.run(Thread.java:744)

"RMI TCP Accept-0@1008" daemon prio=5 tid=0xe nid=NA runnable
  java.lang.Thread.State: RUNNABLE
	  at java.net.DualStackPlainSocketImpl.accept0(DualStackPlainSocketImpl.java:-1)
	  at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
	  at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
	  at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:198)
	  - locked <0x14f8> (a java.net.SocksSocketImpl)
	  at java.net.ServerSocket.implAccept(ServerSocket.java:530)
	  at java.net.ServerSocket.accept(ServerSocket.java:498)
	  at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:388)
	  at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:360)
	  at java.lang.Thread.run(Thread.java:744)

"RMI Scheduler(0)@1597" daemon prio=5 tid=0x13 nid=NA waiting
  java.lang.Thread.State: WAITING
	  at sun.misc.Unsafe.park(Unsafe.java:-1)
	  at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
	  at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
	  at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1090)
	  at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:807)
	  at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1068)
	  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130)
	  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	  at java.lang.Thread.run(Thread.java:744)

"Thread-7@2301" daemon prio=5 tid=0x20 nid=NA runnable
  java.lang.Thread.State: RUNNABLE
	  at sun.nio.ch.Iocp.getQueuedCompletionStatus(Iocp.java:-1)
	  at sun.nio.ch.Iocp.access$300(Iocp.java:46)
	  at sun.nio.ch.Iocp$EventHandlerTask.run(Iocp.java:333)
	  at java.lang.Thread.run(Thread.java:744)

"C3P0PooledConnectionPoolManager[identityToken->1hgeunv901hgryby113r58p|6fe2ffc3]-HelperThread-#5@5443" daemon prio=5 tid=0x3c nid=NA waiting
  java.lang.Thread.State: WAITING
	  at java.lang.Object.wait(Object.java:-1)
	  at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)

"C3P0PooledConnectionPoolManager[identityToken->1hgeunv901hgryby113r58p|6fe2ffc3]-HelperThread-#4@5441" daemon prio=5 tid=0x3b nid=NA waiting
  java.lang.Thread.State: WAITING
	  at java.lang.Object.wait(Object.java:-1)
	  at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)

"C3P0PooledConnectionPoolManager[identityToken->1hgeunv901hgryby113r58p|6fe2ffc3]-HelperThread-#3@5445" daemon prio=5 tid=0x3a nid=NA waiting
  java.lang.Thread.State: WAITING
	  at java.lang.Object.wait(Object.java:-1)
	  at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)

"C3P0PooledConnectionPoolManager[identityToken->1hgeunv901hgryby113r58p|6fe2ffc3]-HelperThread-#2@5447" daemon prio=5 tid=0x39 nid=NA waiting
  java.lang.Thread.State: WAITING
	  at java.lang.Object.wait(Object.java:-1)
	  at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)

"C3P0PooledConnectionPoolManager[identityToken->1hgeunv901hgryby113r58p|6fe2ffc3]-HelperThread-#1@5446" daemon prio=5 tid=0x38 nid=NA waiting
  java.lang.Thread.State: WAITING
	  at java.lang.Object.wait(Object.java:-1)
	  at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)

"C3P0PooledConnectionPoolManager[identityToken->1hgeunv901hgryby113r58p|6fe2ffc3]-HelperThread-#0@5444" daemon prio=5 tid=0x37 nid=NA waiting
  java.lang.Thread.State: WAITING
	  at java.lang.Object.wait(Object.java:-1)
	  at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)

"C3P0PooledConnectionPoolManager[identityToken->1hgeunv901hgryby113r58p|44b63da]-HelperThread-#5@4735" daemon prio=5 tid=0x34 nid=NA waiting
  java.lang.Thread.State: WAITING
	  at java.lang.Object.wait(Object.java:-1)
	  at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)

"C3P0PooledConnectionPoolManager[identityToken->1hgeunv901hgryby113r58p|44b63da]-HelperThread-#4@4734" daemon prio=5 tid=0x33 nid=NA waiting
  java.lang.Thread.State: WAITING
	  at java.lang.Object.wait(Object.java:-1)
	  at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)

"C3P0PooledConnectionPoolManager[identityToken->1hgeunv901hgryby113r58p|44b63da]-HelperThread-#3@4733" daemon prio=5 tid=0x32 nid=NA waiting
  java.lang.Thread.State: WAITING
	  at java.lang.Object.wait(Object.java:-1)
	  at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)

"C3P0PooledConnectionPoolManager[identityToken->1hgeunv901hgryby113r58p|44b63da]-HelperThread-#2@4732" daemon prio=5 tid=0x31 nid=NA waiting
  java.lang.Thread.State: WAITING
	  at java.lang.Object.wait(Object.java:-1)
	  at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)

"C3P0PooledConnectionPoolManager[identityToken->1hgeunv901hgryby113r58p|44b63da]-HelperThread-#1@4731" daemon prio=5 tid=0x30 nid=NA waiting
  java.lang.Thread.State: WAITING
	  at java.lang.Object.wait(Object.java:-1)
	  at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)

"C3P0PooledConnectionPoolManager[identityToken->1hgeunv901hgryby113r58p|44b63da]-HelperThread-#0@4730" daemon prio=5 tid=0x2f nid=NA waiting
  java.lang.Thread.State: WAITING
	  at java.lang.Object.wait(Object.java:-1)
	  at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:683)

"GC Daemon@1701" daemon prio=2 tid=0x15 nid=NA waiting
  java.lang.Thread.State: WAITING
	  at java.lang.Object.wait(Object.java:-1)
	  at sun.misc.GC$Daemon.run(GC.java:117)

"http-bio-8080-exec-5@4694" daemon prio=5 tid=0x29 nid=NA runnable
  java.lang.Thread.State: RUNNABLE
	  at java.net.SocketInputStream.socketRead0(SocketInputStream.java:-1)
	  at java.net.SocketInputStream.read(SocketInputStream.java:152)
	  at java.net.SocketInputStream.read(SocketInputStream.java:122)
	  at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:516)
	  at org.apache.coyote.http11.InternalInputBuffer.fill(InternalInputBuffer.java:501)
	  at org.apache.coyote.http11.Http11Processor.setRequestLineReadTimeout(Http11Processor.java:173)
	  at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:950)
	  at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
	  at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:315)
	  - locked <0x1555> (a org.apache.tomcat.util.net.SocketWrapper)
	  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	  at java.lang.Thread.run(Thread.java:744)

"ajp-bio-8009-AsyncTimeout@1972" daemon prio=5 tid=0x1b nid=NA sleeping
  java.lang.Thread.State: TIMED_WAITING
	  at java.lang.Thread.sleep(Thread.java:-1)
	  at org.apache.tomcat.util.net.JIoEndpoint$AsyncTimeout.run(JIoEndpoint.java:151)
	  at java.lang.Thread.run(Thread.java:744)

"http-bio-8080-AsyncTimeout@1968" daemon prio=5 tid=0x19 nid=NA sleeping
  java.lang.Thread.State: TIMED_WAITING
	  at java.lang.Thread.sleep(Thread.java:-1)
	  at org.apache.tomcat.util.net.JIoEndpoint$AsyncTimeout.run(JIoEndpoint.java:151)
	  at java.lang.Thread.run(Thread.java:744)

"ContainerBackgroundProcessor[StandardEngine[Catalina]]@1958" daemon prio=5 tid=0x17 nid=NA sleeping
  java.lang.Thread.State: TIMED_WAITING
	  at java.lang.Thread.sleep(Thread.java:-1)
	  at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1508)
	  at java.lang.Thread.run(Thread.java:744)

"C3P0PooledConnectionPoolManager[identityToken->1hgeunv901hgryby113r58p|6fe2ffc3]-AdminTaskTimer@5442" daemon prio=5 tid=0x36 nid=NA waiting
  java.lang.Thread.State: WAITING
	  at java.lang.Object.wait(Object.java:-1)
	  at java.util.TimerThread.mainLoop(Timer.java:552)
	  at java.util.TimerThread.run(Timer.java:505)

"C3P0PooledConnectionPoolManager[identityToken->1hgeunv901hgryby113r58p|44b63da]-AdminTaskTimer@4723" daemon prio=5 tid=0x2e nid=NA waiting
  java.lang.Thread.State: WAITING
	  at java.lang.Object.wait(Object.java:-1)
	  at java.util.TimerThread.mainLoop(Timer.java:552)
	  at java.util.TimerThread.run(Timer.java:505)

"Finalizer@2588" daemon prio=8 tid=0x3 nid=NA waiting
  java.lang.Thread.State: WAITING
	  at java.lang.Object.wait(Object.java:-1)
	  at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
	  at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
	  at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:189)

"Reference Handler@5354" daemon prio=10 tid=0x2 nid=NA waiting
  java.lang.Thread.State: WAITING
	  at java.lang.Object.wait(Object.java:-1)
	  at java.lang.Object.wait(Object.java:503)
	  at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)

"Attach Listener@5352" daemon prio=5 tid=0x5 nid=NA runnable
  java.lang.Thread.State: RUNNABLE

"Signal Dispatcher@5353" daemon prio=9 tid=0x4 nid=NA runnable
  java.lang.Thread.State: RUNNABLE

**JLivermore** · 14 Mar 2014, 07:02

Another interesting bit of information, I cannot get the app to lock up using Chrome. It locks up easily using IE-11.

**Isomorphic** · 14 Mar 2014, 12:50

Great, the thread dump reveals a deadlock and we think we see how to correct it.

IE vs Chrome is just changing the timing of the requests and this just happens to make the deadlock more or less likely.

In the meantime, you might want to look over your code to make better use of Queuing (see QuickStart Guide). It looks like you have a lot of concurrent requests to the server just from a single browser, which would be more efficient if queued.

We'll let you know when we've corrected the deadlock.

**JLivermore** · 14 Mar 2014, 13:50

Thanks for the pointer to Queuing.

Announcement

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment