Large Data Export CSV

[jkroll]

Version:
SmartClient Version: v9.1p_2014-09-09/PowerEdition Deployment

Browsers:
IE, Firefox

Error Logs:
Not sure if I am looking at smtart gwt logs or not but there is not information outside of isc.showConsole() that has anything to do with SmartGWT (This could be a place to start in terms of more information)

Current Setup:
Using a list grid with advanced filters so end user can filter data and then export at a later time once they are happy with their filters.
Using the showcase examples for the export with streamResults(true)

Problem:
After 64secs or 49,401kb have been downloaded from browser (225K records of the 1.1 million in the database) the file stops as if it has completed the download or some "file size limit" has been reached. I know the file is not completed because the last line is truncated (uncompleted)

I have played around with weblogic connection time outs so I am not convinced it is a timeout issue but still open to all ideas.

I am not able to find an exception anywhere and I have looked at the debugging guides but I can't seem to get any more information out of the system.

Questions:

Does anyone have any example code for implementing a large csv export using StreamResults(true) flag?
Any ideas for debugging would be helpful as well.

[Isomorphic]

There's no file size / output limit imposed by SmartGWT, and nothing more than what you see in the sample is necessary. We suspect a configuration setting is necessary in some intervening layer - WebLogic, or other layers like Spring or your webserver, if present.

Getting your server logs working should be priority #1 since this probably contains an exception which would hint at the source of the problem.

At the moment, it sounds like you have no server logs at all - nothing that says "com.isomorphic."? Did you skip the installation steps related to log4j?

[jkroll]

Thank you for your response it allowed us to focus purely on getting the logs with better information. Looks like we have an out of memory exception but I was under the assumption that streaming was supposed to prevent that. The streaming flag does allow the application not to completely die and only the export gracefully die (which is great). Any ideas on what configuration we are failing to use or if you know of a way to get better information for debugging?

=== 2015-11-18 10:59:36,918 [ng)'] DEBUG StreamingResponseIterator - Processing record: 200000
=== 2015-11-18 10:59:38,582 [ng)'] DEBUG StreamingResponseIterator - Processing record: 210000
=== 2015-11-18 10:59:39,962 [ng)'] DEBUG StreamingResponseIterator - Processing record: 220000
=== 2015-11-18 10:59:41,472 [ng)'] DEBUG SQLTransaction - getConnection() found transactional connection for Oracle (connection is null)
=== 2015-11-18 10:59:41,473 [ng)'] DEBUG SQLDriver - Freeing SQLDriver dbConnection 628809091
=== 2015-11-18 10:59:41,473 [ng)'] DEBUG SQLConnectionManager - About to close ISCPoolableConnection with hashcode "628809091"
=== 2015-11-18 10:59:41,473 [ng)'] DEBUG PoolableSQLConnectionFactory - Executing pingTest 'select 1 from dual' on connection 628809091
=== 2015-11-18 10:59:41,475 [ng)'] ERROR IDACall - Top-level servlet error:
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2367)
at java.io.CharArrayWriter.write(CharArrayWriter.java:122)
at java.io.Writer.write(Writer.java:157)
at com.isomorphic.rpc.DataExport.write(DataExport.java:678)
at com.isomorphic.rpc.DataExport.exportResultSet(DataExport.java:588)
at com.isomorphic.rpc.DataExport.exportResultSet(DataExport.java:459)
at com.isomorphic.rpc.RPCManager.completeResponse(RPCManager.java:1099)
at com.isomorphic.rpc.RPCManager.send(RPCManager.java:636)
at com.isomorphic.servlet.IDACall.processRPCTransaction(IDACall.java:172)
at com.isomorphic.servlet.IDACall.processRequest(IDACall.java:137)
at com.isomorphic.servlet.IDACall.doPost(IDACall.java:73)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:751)
at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:152)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:844)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:242)
at weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:216)
at weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:132)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:352)
at weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:235)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3284)
at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3254)
at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
at weblogic.servlet.provider.WlsSubjectHandle.run(WlsSubjectHandle.java:57)
at weblogic.servlet.internal.WebAppServletContext.doSecuredExecute(WebAppServletContext.java:2163)
at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2089)
at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2074)
at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1512)
at weblogic.servlet.provider.ContainerSupportProviderImpl$WlsRequestExecutor.run(ContainerSupportProviderImpl.java:254)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

[Isomorphic]

Streaming greatly reduces the amount of memory required for exporting, so that only a constant amount of memory should be required for an arbitrarily large export.

You haven't mentioned how much memory you've allocated for your VM - it may be that you don't have a problem with exports taking up unlimited amounts of memory, instead you've just provided too little memory to accommodate even the constant amount needed for streaming. You could both try upping the memory and watching the behavior during a large export to see if there seems to be a pattern of unlimited growth.

If there is, there are a couple of places it could be coming from:

1. something buffering the output, such as a third-party system for implementing HTTP compression. The compression system that comes with SmartGWT automatically avoids compressing streaming results for this reason

2. bad JDBC behavior: perhaps the JDBC driver is retaining rows in memory even though SmartGWT code is careful to drop references to already-exported rows, so that they become garbage

If you see the telltale pattern of boundlessly increasing memory, a heap analyzer tool could show you where it is.