CubeGrid: How do I limit the data for export to avoid STATUS_MAX_POST_SIZE_EXCEEDED?

[pankaj.jain]

Hi Isomorphic,

While evaluating SmartGWT 6.0 for CubeGrid, we noticed that when we export large dataset from CubeGrid, we get error: STATUS_MAX_POST_SIZE_EXCEEDED.
However, to resolve this, we want to limit the export size from the CubeGrid.

The base export code (almost same as from the showcase)

Button exportButton = new Button("Export");
exportButton.addClickHandler(new ClickHandler() {
    @Override
    public void onClick(ClickEvent event) {
        DSRequest dsRequestProperties = new DSRequest();
        dsRequestProperties.setExportAs(ExportFormat.CSV);
        dsRequestProperties.setExportFilename(exportFileName.toString());
        dsRequestProperties.setExportDisplay(ExportDisplay.DOWNLOAD);
        cubeGrid.exportClientData(dsRequestProperties);
    }
});

Ideally we would like to limit the data based on the size (Maybe allow exports only till export filesize is < 5 MB). Is this even possible?
The other option is to limit the amount of data (rows and columns) being exported!

For limiting the data, we tried using

dsRequestProperties.setEndRow(50000);

which did not work!

How do I go about limiting the data to be exported when the user tries to export a large data set?

[Isomorphic]

For exportClientData(), the only data that goes to the server is data that has already been loaded on the client (this is also why setEndRow() does nothing - no new query is performed). For this type of export, the exported data can't realistically grow to a size that you wouldn't want to allow the user to export. Most likely, your server is configured for a relatively small post size, and you should just increase it.

For exportData() (server-based export), you could indeed set an endRow to limit the export size.

[pankaj.jain]

Hi,
I again started looking at this functionality, and we are still facing issues when we set maxPostSize="-1" in our Tomcat (conf/server.xml).
Basically, the browser (Chrome) hangs for a while (5-10 mins approx, not exactly timed it), before returning full control.
However, there is no error message popup or any export done.

The smartgwt log complains about an empty RPCManager
Also, the export size seems to be around ~100MB
Could it be that there is a timeout due to the lag when passing the huge data from client to the server?

smartgwt.log

=== 2017-01-04 05:49:50,184 [55-6] INFO  RequestContext - URL: '/mi/MarketIntelligence/sc/IDACall', User-Agent: 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36': Safari with Accept-Encoding header
=== 2017-01-04 05:49:50,184 [55-6] DEBUG IDACall - Header Name:Value pair: host:ltxl0801.sgdcelab.sabre.com:2200
=== 2017-01-04 05:49:50,184 [55-6] DEBUG IDACall - Header Name:Value pair: cache-control:max-age=0
=== 2017-01-04 05:49:50,184 [55-6] DEBUG IDACall - Header Name:Value pair: origin:http://ltxl0801.sgdcelab.sabre.com:2200
=== 2017-01-04 05:49:50,184 [55-6] DEBUG IDACall - Header Name:Value pair: upgrade-insecure-requests:1
=== 2017-01-04 05:49:50,184 [55-6] DEBUG IDACall - Header Name:Value pair: user-agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.87 Safari/537.36
=== 2017-01-04 05:49:50,184 [55-6] DEBUG IDACall - Header Name:Value pair: content-type:application/x-www-form-urlencoded
=== 2017-01-04 05:49:50,184 [55-6] DEBUG IDACall - Header Name:Value pair: accept:text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
=== 2017-01-04 05:49:50,184 [55-6] DEBUG IDACall - Header Name:Value pair: dnt:1
=== 2017-01-04 05:49:50,184 [55-6] DEBUG IDACall - Header Name:Value pair: referer:http://ltxl0801.sgdcelab.sabre.com:2200/mi-dev/
=== 2017-01-04 05:49:50,184 [55-6] DEBUG IDACall - Header Name:Value pair: accept-encoding:gzip, deflate
=== 2017-01-04 05:49:50,184 [55-6] DEBUG IDACall - Header Name:Value pair: accept-language:en-US,en;q=0.8
=== 2017-01-04 05:49:50,184 [55-6] DEBUG IDACall - Header Name:Value pair: cookie:iPlanetDirectoryPro=AQIC5wM2LY4SfczJ-kkWi6Tq2Q_L6CR_2lCMudzrhDM-F9A.*AAJTSQACMDEAAlNLABQtNjYzNjM0MDY5NDE5NDEzMzMyMA..*; portalOpenSSOToken=AQIC5wM2LY4Sfcyr2a2vSt8QcrFuqg2uwDNoN4IIb9nCT7I.*AAJTSQACMDMAAlMxAAIwMQ..*; ssolbcookie=01; isc_cState=ready; GLog=%7B%0A%20%20%20%20trackRPC%3Afalse%0A%7D
=== 2017-01-04 05:49:50,184 [55-6] DEBUG IDACall - Header Name:Value pair: uid:ul000123
=== 2017-01-04 05:49:50,184 [55-6] DEBUG IDACall - Header Name:Value pair: x-forwarded-for:10.163.216.20
=== 2017-01-04 05:49:50,184 [55-6] DEBUG IDACall - Header Name:Value pair: x-forwarded-host:ltxl0801.sgdcelab.sabre.com:2200
=== 2017-01-04 05:49:50,184 [55-6] DEBUG IDACall - Header Name:Value pair: x-forwarded-server:ltxl0801.sgdcelab.sabre.com
=== 2017-01-04 05:49:50,184 [55-6] DEBUG IDACall - Header Name:Value pair: connection:Keep-Alive
=== 2017-01-04 05:49:50,184 [55-6] DEBUG IDACall - Header Name:Value pair: [B]content-length:102448195[/B]
=== 2017-01-04 05:49:50,184 [55-6] DEBUG IDACall - session exists: null
=== 2017-01-04 05:49:50,184 [55-6] DEBUG IDACall - remote user: null
[B]=== 2017-01-04 05:50:26,888 [55-6] WARN  RPCManager - Unexpected empty RPCManager transaction: POST'd data appears to have been removed from the request before the server framework received it. This may be due to application / server settings restricting maximum POST / file upload size, or due to security software on your server, browser or network that erroneously blocked the request, or due to bugs in servlet filters.[/B]
=== 2017-01-04 05:50:26,889 [55-6] INFO  Compression - /mi/MarketIntelligence/sc/IDACall: 93 -> 101 bytes

Just to let you know, the cube renders without any issue on the browser, and with hardly any delay (20-30 seconds).
Also, realistically, our application will allow users to generate cube for a much larger data set, so it's very much possible that the export size can get to few hundred GBs!
Maybe we should abandon client side export totally for our cube functionality and look at server side export?
What do you suggest?

(SmartGWT 6.0, build dt. 2016/12/31)

Thanks!

[Isomorphic]

100MB is too large of an export and since it's taking 5+ minutes, you are probably experiencing an HTTP timeout, which is typically 4 minutes.

As we covered previously, the size of the export is proportional to the amount of data that was loaded. We wouldn't expect the export to get to 100MB unless you had loaded 100s of thousands of records - how many are you actually loading?

Assuming you've actually loaded just a reasonable amount, we'd like to see the actual data posted to the server, to understand how it got so large. Maybe you've applied settings that multiply the size of the export (such as lots of per-cell styling) - this will be evident from the posted data.

[Isomorphic]

To clarify, we don't need to see the entire 100MB export that's causing a problem. Assuming that all your exports are much larger than the data loaded, a much smaller example is what we're looking for.