Exception while attempting to load a DataSource NullPointerException

[stonebranch2]

Hi Isomorphic,

We have a situation where a client is using a pass-through URL, and due to what would appear to be some kind of configuration issue, when the client accesses our web application using that pass-through URL, the client experiences a failure loading data sources and all subsequence access for other users is impacted.

We understand that if the incoming request is somehow corrupted, it will not function properly, however, our concern is that users who otherwise have been accessing the web application fine can no longer access it due to datasource loading issues.

SmartClient Version: v10.1p_2016-04-28/Pro Deployment (built 2016-04-28)

In the smartgwt.log, we see the following NullPointerException.

2016-05-11-15:28:05:487 -0500  ERROR [http-nio-8444-exec-7] Exception while attempting to load a DataSource
java.lang.NullPointerException
at org.apache.coyote.http11.InternalNioOutputBuffer.flushBuffer(InternalNioOutputBuffer.java:234)
at org.apache.coyote.http11.InternalNioOutputBuffer.addToBB(InternalNioOutputBuffer.java:189)
at org.apache.coyote.http11.InternalNioOutputBuffer.commit(InternalNioOutputBuffer.java:177)
at org.apache.coyote.http11.AbstractHttp11Processor.action(AbstractHttp11Processor.java:741)
at org.apache.coyote.Response.action(Response.java:181)
at org.apache.coyote.Response.sendHeaders(Response.java:352)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:335)
at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:317)
at org.apache.catalina.connector.CoyoteWriter.flush(CoyoteWriter.java:94)
at com.isomorphic.servlet.DataSourceLoader.processRequest(DataSourceLoader.java:287)
at com.isomorphic.servlet.DataSourceLoader.doGet(DataSourceLoader.java:107)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:162)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at xxx.xxx.xxx.xxx.CacheControlFilter.doFilter(CacheControlFilter.java:46)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:617)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1527)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1484)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
2016-05-11-15:28:05:488 -0500  ERROR [http-nio-8444-exec-7] BaseServlet Global Exception
java.lang.NullPointerException
at org.apache.coyote.http11.InternalNioOutputBuffer.flushBuffer(InternalNioOutputBuffer.java:234)
at org.apache.coyote.http11.InternalNioOutputBuffer.addToBB(InternalNioOutputBuffer.java:189)
at org.apache.coyote.http11.InternalNioOutputBuffer.access$000(InternalNioOutputBuffer.java:41)
at org.apache.coyote.http11.InternalNioOutputBuffer$SocketOutputBuffer.doWrite(InternalNioOutputBuffer.java:320)
at org.apache.coyote.http11.filters.IdentityOutputFilter.doWrite(IdentityOutputFilter.java:93)
at org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:256)
at org.apache.coyote.Response.doWrite(Response.java:503)
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:388)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:426)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:342)
at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:317)
at org.apache.catalina.connector.Response.flushBuffer(Response.java:510)
at org.apache.catalina.connector.ResponseFacade.flushBuffer(ResponseFacade.java:318)
at com.isomorphic.servlet.DataSourceLoader.processRequest(DataSourceLoader.java:295)
at com.isomorphic.servlet.DataSourceLoader.doGet(DataSourceLoader.java:107)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:162)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at xxx.xxx.xxx.xxx.CacheControlFilter.doFilter(CacheControlFilter.java:46)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:617)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1527)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1484)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)

We have also seen the following in the smartgwt.log.

2016-05-11-15:37:24:585 -0500  ERROR [http-nio-8444-exec-10] BaseServlet Global Exception
javax.servlet.ServletException: DataSource 'null' failed to load due to an exception on the server:
No 'dataSource' parameter provided
See the server-side log for additional details.
    at com.isomorphic.servlet.DataSourceLoader.processRequest(DataSourceLoader.java:292)
    at com.isomorphic.servlet.DataSourceLoader.doGet(DataSourceLoader.java:107)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
    at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:162)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at xxx.xxx.xxx.xxx.CacheControlFilter.doFilter(CacheControlFilter.java:46)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:502)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:142)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:617)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:668)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1527)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1484)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:745)

In the Tomcat localhost_access_log, we see a number of the following; note the GGET, and ET, etc.

xxx.xx.xx.xxx - - [11/May/2016:09:05:20 -0500] "GET /xxx/xxx/sc/DataSourceLoader?dataSource=<a number of data sources> HTTP/1.1" 400 -
xxx.xx.xx.xxx - - [11/May/2016:09:05:20 -0500] "GGET /xxx/xxx/sc/DataSourceLoader?dataSource=<a number of data sources> HTTP/1.1" 501 1147
xxx.xx.xx.xxx - - [11/May/2016:09:05:20 -0500] "ET /xxx/xxx/xxx.nocache.js HTTP/1.1" 501 1143

This is not something we can replicate in-house, so unfortunately, what I have provided is all I can for now.

Thanks

[Isomorphic]

What's a "pass-through URL"? Are you talking about Apache AJP? A reverse proxy setup?

Looking at the "GGET" and "ET" in the logs, it looks like whatever is "passing through" the request is intermittently dropping bytes and adding others, which would explain your NPE. SmartGWT is out of the loop at that point; we've given all the data to the servlet API and called stream.flush() to have it delivered to the browser. The fact that the underlying Tomcat engine actually *crashes* strongly suggests that whatever it's talking to has done something very very wrong on the wire, at a very low level - like actually sending a misframed TCP packet or similar.

[stonebranch2]

What's a "pass-through URL"? Are you talking about Apache AJP? A reverse proxy setup?

Tivoli junction