SmartClient Version: v9.1p_2014-03-18/Pro Deployment (built 2014-03-18)
Hi
I'm using a server side data source (com.isomorphic.SQL.DataSource) to fetch a large amount of data for processing. I would like to stream the results in order to minimize server side heap use. Looking at your documentation I thought that calling setStreamResults(true) on the request would as stated in the documentation: "If true, cause results to be streamed one record at a time." I'm running on MySQL 5.0.77 which supports unbufferd queries so I was guessing/hoped that underneath the hood setStreamResults was enabling this behavior on the underlying jdbc statement ?
SO my question is should I expect this to work or am I doing something wrong ? I saw a Isomorphic response from 2006 to a similar question that stated: "streamResults can be used to enable streaming for the default exports. It has no effect when your code is directly accessing the results, except of course that it causes the problem you just described."
I implemented the following however when executed it ran out of heap space on the server when running large queries. jvisualvm clearly showed heap space growing and then exceeding the max, also note this code work fine for smaller queries
[code]
DSRequest ds = DataSourceManger.getDataSource("myds");
DSRequest dsr = new DSRequest("myds","fetch")
dsr.setStreamResults(true);
dsr.setAdvancedCriteria(mycrit);
dsr.shouldStreamResults(); // this method is not documented , behaves the same with or with out it ?
logger.info("calling ds.execute");
DSResponse res = ds.execute(dsr);
// THREW OutOfMemoryError before next stmt
logger.inf("call completed");
StreamingResponseIterator iter = (StreamingResponseIterator) res.getData()
// iterate over date here ......
[/code/
mysd.ds.xml has the serverType set to sql, I read somewhere that only SQLDataSources honored the setStreamResults
<DataSource ID='myds" dbName="mydb" serverType="sql" etc.
<fields> .. </fields>
</DataSource>
Thanks in advance
Hi
I'm using a server side data source (com.isomorphic.SQL.DataSource) to fetch a large amount of data for processing. I would like to stream the results in order to minimize server side heap use. Looking at your documentation I thought that calling setStreamResults(true) on the request would as stated in the documentation: "If true, cause results to be streamed one record at a time." I'm running on MySQL 5.0.77 which supports unbufferd queries so I was guessing/hoped that underneath the hood setStreamResults was enabling this behavior on the underlying jdbc statement ?
SO my question is should I expect this to work or am I doing something wrong ? I saw a Isomorphic response from 2006 to a similar question that stated: "streamResults can be used to enable streaming for the default exports. It has no effect when your code is directly accessing the results, except of course that it causes the problem you just described."
I implemented the following however when executed it ran out of heap space on the server when running large queries. jvisualvm clearly showed heap space growing and then exceeding the max, also note this code work fine for smaller queries
[code]
DSRequest ds = DataSourceManger.getDataSource("myds");
DSRequest dsr = new DSRequest("myds","fetch")
dsr.setStreamResults(true);
dsr.setAdvancedCriteria(mycrit);
dsr.shouldStreamResults(); // this method is not documented , behaves the same with or with out it ?
logger.info("calling ds.execute");
DSResponse res = ds.execute(dsr);
// THREW OutOfMemoryError before next stmt
logger.inf("call completed");
StreamingResponseIterator iter = (StreamingResponseIterator) res.getData()
// iterate over date here ......
[/code/
mysd.ds.xml has the serverType set to sql, I read somewhere that only SQLDataSources honored the setStreamResults
<DataSource ID='myds" dbName="mydb" serverType="sql" etc.
<fields> .. </fields>
</DataSource>
Thanks in advance
Comment