Binding CSV to listgrid(datagrid)

[Vinod.Narayankar]

We have scenario in which we are pulling data from database in offfline mode and exporting in "CSV"(Excel) . Once the user clicks on the CSV(Excel) links. we need to show the data in list grid.

Is it possible to do this?
does the filtering works on list grid works??

Can you please share some sample code for this?

[Isomorphic]

It's not entirely clear from your description how you are intending to have this work. Is the intention that the user uploads a .csv file to the SmartClient app, at which point the app starts to work with the data? In this case, are you intending to import the data to a server-side dataSource (backed by a database table, for example)?
Or is the intention to simply have a SmartClient component (listGrid) issue fetches directly against a URL that serves up .csv formatted responses?

[Vinod.Narayankar]

Binding CSV to listgrid(datagrid)

User will login to our SMARTGWT application and he will pull some data in offline mode. In Back grond we will create CSV file using Jasper software as per users request. Once the file is created it will appear as link in smartGWT application . As soon as User clicks on the link,we want to read the CSV and bind to listGRID.

Here intention is to bind the data to listgrid. we don't want to store this data in database.

[Isomorphic]

Is the file on the client or server? As GWT is compiled to JS, which doesn't have local file access in a browser, I'll assume it's on the server.

In that case, check out the documentation for RPCRequest.setServerOutputAsString(), which should allow you to get the CSV content back to your client. At that point, you could use a client-only DataSource and the DataSource.recordsFromText() API to get working Records that can be installed in the ListGrid via ListGrid.applyRecordData().

An Example of the last part is here: http://smartclient.com/smartgwt/show..._grid_category

[Vinod.Narayankar]

Binding CSV to listgrid(datagrid)

I am doing the below

DataSource dataSource = new DataSource();
TextImportSettings settings = new TextImportSettings();
settings.setHasHeaderLine(true);
settings.setFieldSeparator(",");
settings.setLineSeparator("\n\r");
Record[] records = dataSource.recordsFromText(response.getDataAsString(), settings);
RecordList recordList = new RecordList(records);

This part of the code works -
response.getDataAsString(), settings
i get the entire data from excel as strings where the columns are separated by ","

But the output from this does not get stored into the
Record[] records

If the string gets stored into records then it should be straight forward to display it in the listgrid using
applyRecordData(recordList);

[Isomorphic]

recordsFromText is a tested, working method, so the most likely cause of problems here would be something like a mismatch between your expected data format, and what's actually in the text response. This may be as simple as a difference in the specified separator character vs the actual separator ("\n" vs "\r\n", for example).

You can probably debug this further by modifying your test case to simply create a hardcoded string in Java code that matches your expected output, and use the recordsFromText method on that.

If this doesn't give you enough information to get things working, please feel free to post a small runnable test case showing your DataSource definition, the response text, and your code to parse it into records and we will take a look on our end

Regards
Isomorphic Software

[Warren.Menezes]

Hi Isomorphic

Our main aim is to load data from a csv file that is stored on our sever into the listgrid. The list grid will only contain the data in the csv which the user can filter, sort and further play around with within our application. We do not want to load the data from the csv file into a database table.

In order to achieve this task, I have created a custom DataSource and I have overriden the executeFetch method to read the csv file and return the data. We are only interested in reading the data and not performing any write operations on the data.
This seems to work fine for me with a small csv file . But I have concerns about loading a large csv file. I am also worried about the java heap size if I load all the records of the csv file into memory.

Is this the right approach or is there something easier in the smartGWT framework that can help me achieve the same task.

I want to be able to perform all the listgrid operations like paging, filtering, sorting, grouping, freezing columns without any problems.

Thanks

[Isomorphic]

The previous answers seem to cover all aspects of this question comprehensively. There are both client and server approaches - see above.

[Warren.Menezes]

I have tried out the approach you have mentioned above.

I have done the following:

rpcRequest.setServerOutputAsString(true);
rpcRequest.setActionURL(csvFile);

In the callback I do:

Record[] records = dataSource.recordsFromText(dsResponse.getDataAsString(), settings);
listGrid.setData(records);

This works without a problem for me. All the data from the csv appears in the listgrid. My only concern is how do I handle large csv files with about 2lakhs records.
Right now I tried this with a small csv file. When I try a larger one with about 1lakhs rows I get an OutOfMemoryError exception. This is in Debug Mode

Exception in thread "Thread-1" java.lang.OutOfMemoryError: Java heap space
	at java.lang.StringCoding$StringEncoder.encode(StringCoding.java:232)
	at java.lang.StringCoding.encode(StringCoding.java:272)
	at java.lang.String.getBytes(String.java:946)
	at com.google.gwt.dev.protobuf.CodedOutputStream.writeStringNoTag(CodedOutputStream.java:342)
	at com.google.gwt.dev.protobuf.CodedOutputStream.writeString(CodedOutputStream.java:181)
	at com.google.gwt.dev.shell.remoteui.RemoteMessageProto$Message$Request$ViewerRequest$LogData.writeTo(RemoteMessageProto.java:2496)
	at com.google.gwt.dev.protobuf.CodedOutputStream.writeMessageNoTag(CodedOutputStream.java:367)
	at com.google.gwt.dev.protobuf.CodedOutputStream.writeMessage(CodedOutputStream.java:209)
	at com.google.gwt.dev.shell.remoteui.RemoteMessageProto$Message$Request$ViewerRequest$AddLogEntry.writeTo(RemoteMessageProto.java:3321)
	at com.google.gwt.dev.protobuf.CodedOutputStream.writeMessageNoTag(CodedOutputStream.java:367)
	at com.google.gwt.dev.protobuf.CodedOutputStream.writeMessage(CodedOutputStream.java:209)
	at com.google.gwt.dev.shell.remoteui.RemoteMessageProto$Message$Request$ViewerRequest.writeTo(RemoteMessageProto.java:4328)
	at com.google.gwt.dev.protobuf.CodedOutputStream.writeMessageNoTag(CodedOutputStream.java:367)
	at com.google.gwt.dev.protobuf.CodedOutputStream.writeMessage(CodedOutputStream.java:209)
	at com.google.gwt.dev.shell.remoteui.RemoteMessageProto$Message$Request.writeTo(RemoteMessageProto.java:5847)
	at com.google.gwt.dev.protobuf.CodedOutputStream.writeMessageNoTag(CodedOutputStream.java:367)
	at com.google.gwt.dev.protobuf.CodedOutputStream.writeMessage(CodedOutputStream.java:209)
	at com.google.gwt.dev.shell.remoteui.RemoteMessageProto$Message.writeTo(RemoteMessageProto.java:9877)
	at com.google.gwt.dev.protobuf.AbstractMessageLite.writeDelimitedTo(AbstractMessageLite.java:83)
	at com.google.gwt.dev.shell.remoteui.MessageTransport$PendingRequest.send(MessageTransport.java:112)
	at com.google.gwt.dev.shell.remoteui.MessageTransport$4.run(MessageTransport.java:340)
	at java.lang.Thread.run(Thread.java:662)

I have the following questions:
1. Is there any limit on the number of rows in the csv file while using this approach?
2. How can I achieve a paging mechanism where records are fetched from the csv only when required?
3. Will filtering, grouping, summary fields work with this approach?

[Isomorphic]

That out of memory exception indicates that GWT has run out of memory in Development Mode. The actual browser environment (compiled mode) will have different and higher limits.

Still, that's too many records to load client-side, even in Chrome.

Keeping the entire file in server memory might be OK if only one user at a time can have such a file in memory on the server. If multiple users can, it's likely the server will run out of memory too.

As we explained before, a temporary SQL table is a good approach, especially if the data is shared between multiple users. In this case, because you can use SQLDataSource to access the data, all sorting and filtering behaviors will work normally.

If you prefer something less efficient, you could load CSV data from a file every time, but you will also need to implement your own logic for filtering and sorting.