Export with remote service call

[ssequeira]

Hello.
I have searched in the forums and I haven't found such a post so far.
I hope this post is not a duplicate of some other.

The scenario is the following:
I am using a DataSource of the generic type.
When a fetch is executed the criteria passed on the request is used to call a remote service, from the server. The request needs to go through the server as I need to secure and filter the criteria sent from client to server before calling the remote service. I.e. the remote service cannot be directly called from the client.
The data is then displayed on a ListGrid.
The remote service supports pagination, so everything works alright.

What I want to do also is support the export functionality, when dsRequest.shouldStreamResults() evaluates to true.
The following code depicts how the DSResponse is constructed after the remote call is done. The remote call results in the List<MyData>, which is used to construct the DSResponse. I am not showing pagination related code for simplicity reasons.

public DSResponse getDsResponse(List<MyData> data, DSRequest dsRequest)  throws Exception {
	DSResponse response = new DSResponse(dsRequest.getDataSource());
	response.setData(createFromList(data));
	response.setSuccess();
	return response;
}

public List<Object> createFromList(List<MyData> data)  {
	List<Map<String,Object> dsData = new ArrayList<Map<String,Object>>(data.size());
	for(MyData d : data) {
		dsData.add(d.toDsResponseEntry());
	}
	return dsData;
}

private static class MyData extends HashMap<String,Object> {

	public Map<String,Object> toDsResponseEntry() {
		return this;
	}
}

This works pretty well for the normal fetch operation. But on the streaming export situation it does not work.
The problem is that when dsResponse.hasNextRecord() is called on the response object it always evaluates to false, although dsResponse.getData() returns the correct data (list with n entries).

I have noticed that in a SQLDataSource, the object returned from dsResponse.getData() is StreamingResponseIterator. I have tried to use it to no avail.

The solution that I have found was to:

public class RemoteDSResponse extends DSResponse {

    public RemoteDSResponse(DataSource dataSource) {
        super(dataSource);
    }

    @Override
    public boolean hasNextRecord() {
        Iterator<?> data = (Iterator<?>) getData();
        return data.hasNext();
    }
    
    @Override
    public Object nextRecordAsObject() throws StreamingResponseException {
        Iterator<?> data = (Iterator<?>) getData();
        return data.next();
    }

Using an Iterator as data on the DSResponse and overriding those methods works.

The question I have is:
Is this the correct approach to tackle this problem, or is there an obvious way that I am missing?

Using smartgwt power 3.1-p20130725

[Isomorphic]

That's a bit of clever reverse-engineering; unfortunately those are not documented methods, although the approach you've shown here may well make sense as the recommended approach for implementing streaming data once those methods *are* documented.

Are you actually getting any kind of benefit from streaming, or are you basically loading the whole dataset into the application server up front anyway? Because if you are loading the whole dataset at once, you can just turn streaming off and use setData() and the performance will be the same, and you won't be using undocumented methods.

[ssequeira]

For now all data is loaded up front due to restrictions on the remote service. Later on these restrictions will be removed when the service is able to provide data more efficiently, and in this case it might make sense to use the streaming approach, as the dataset might get considerably bigger.

It would be a great help if this, or other, standard approach and related APIs could be documented.

Thank you.

[Isomorphic]

We plan to document these APIs in the future, probably in 4.1, and it's very likely that the recommended approach will be to do exactly what you show here.