Approach for the download of large export from database

[mattcecile]

SmartClient Version: v8.2p_2013-03-28/PowerEdition Deployment (built 2013-03-28)

I'm looking for general approach and advice here please.

From the client, we need to download a file from the server. This file will be created on the server side by fetching rows from a table in the database and returning the result to the client.

This is straightforward except in the case where the size of the data selected is large (100k rows or more). In this case, the execute(dsRequest) throws an out of memory exception and fails.

I've been unable to find an example or discussion of this problem. Using setStartRow and setEndRow seems to cause a complete fetch of the table (from the SQL trace and execution times involved). DSRequest.setStreamResults looked promising but just causes a null pointer exception when used and I could find no examples of its use.

Thanks for any direction you can offer.

Matt Cecile

[Isomorphic]

There's currently built-in support for exporting large results to CSV without requiring the entire dataset to be in memory. This is what the setStreamResults() API is related to.

Other kinds of exports, such as Excel (via POI), cannot be similarly achieved because the POI library itself imposes a restriction that the entire generated spreadsheet must be in memory at once.

Currently, to do a custom export of many thousands of rows, you would need to execute several *separate* dsRequests to fetch different ranges of rows. And of course, you need to carefully ensure that however you are *outputting* the data is actually being written to the network and freed from server memory as the export proceeds.

If you are looking for either supported APIs for streaming data from a DSResponse (something like a DSResponse.nextRecord() API) or just having the export operation written for you (if it's a typical type of export) consider Feature Sponsorship.