Importing CSV into a listgrid

[Vittorio Paternostro]

OK, that's really quite bizarre. Firefox clearly recognizes that it has a form with an <input type="file"> and that it should post a multipart message with a file in it, and it just simply does not include the file. We've never heard of this behavior.

1. do you have this behavior with any other browser? Chrome has similar developer tools that will allow you to see the file

2. do you have this behavior if you use the live Showcase "File Upload" sample?

3. do you have this behavior if you choose some other file? Perhaps a small one?

Unfortunately I don't have the time to investigate further. My Chrome plugin is temporarily out. The live showcase with the image works fine. I can see the binary data in Firebug. The file I'm trying to upload is very small
I'm making the batch uploader work and get the data from the grid.

[Blama]

Hi Blama,

I'm uploading a CSV file containing my data, how you would you create Records as you suggested?

Thought about it. Records might be the wrong approach, as it is client side. You'll want to work with the data on the server. So it is going to be something like this:

//Have your row parsed before, then use the data like this:
		DSRequest myReq = new DSRequest("myDS", DataSource.OP_ADD) {
			{
				HashMap<String, Object> valueMap = new HashMap<String, Object>() {
					{
						put("MYPK", 1234);
						put("FIELD", "foo");
					}
				};
				setValues(valueMap);
			}
		};
		DSResponse myResp = dsRequest.execute();
		if (myResp.statusIsSuccess()) {
			// ....
		}

Please note, that this is *serverside*, so you will include packages like

import com.isomorphic.datasource.DSRequest;
import com.isomorphic.datasource.DSResponse;
import com.isomorphic.datasource.DataSource;

and not from com.smartgwt.client.*.

Best regards,
Blama

[Isomorphic]

OK, that's really quite bizarre. Firefox clearly recognizes that it has a form with an <input type="file"> and that it should post a multipart message with a file in it, and it just simply does not include the file. We've never heard of this behavior.

1. do you have this behavior with any other browser? Chrome has similar developer tools that will allow you to see the file

2. do you have this behavior if you use the live Showcase "File Upload" sample?

3. do you have this behavior if you choose some other file? Perhaps a small one?

[Vittorio Paternostro]

It's unclear what part of the Firebug interface you've grabbed information from - what you've posted actually does suggest a file upload, but isn't completely definitive.

In the most recent Firefox/Firebug, if you look at the request in the "Net" panel and select the request for the upload, you should be able to see raw binary data in the "Post" tab. This would be definitive. You can also check the Content-Length in the "Headers" tab (you want the *Request* Content-Length header) and compare that to the server log.

The Firebug details I posted are the two parts (file
_transaction and Source) coming from inspecting the POST in the Net>Post tab, of course. And as I mentioned and as you can see, the file data and name are empty:

file_transaction
<transaction xmlns:xsi="http://www.w3.org/2000/10/XMLSchema-instance" xsi:type="xsd:Object"><transactionNum xsi:type="xsd:long">6</transactionNum><operations xsi:type="xsd:List"><elem xsi:type="xsd:Object"><criteria xsi:type="xsd:Object"></criteria><values xsi:type="xsd:Object"></values><operationConfig xsi:type="xsd:Object"><dataSource>updateDataSource</dataSource><operationType>update</operationType></operationConfig><componentId>isc_DynamicForm_7</componentId><appID>builtinApplication</appID><operation>updateDataSource_update</operation><oldValues xsi:type="xsd:Object"></oldValues></elem></operations><jscallback>if (!(new RegExp(&quot;^(\\d{1,3}\\.){3}\\d{1,3}$&quot;).test(document.domain))) {while (!window.isc &amp;&amp; document.domain.indexOf(&quot;.&quot;) != -1 ) { try { parent.isc; break} catch (e) {document.domain = document.domain.replace(/.*?\./, &quot;&quot;)}}}parent.isc.Comm.hiddenFrameReply(6,results)</jscallback></transaction>

Source
-----------------------------68784638123650836141556675 Content-Disposition: form-data; name="file"; filename="" Content-Type: application/octet-stream

As we mentioned, misbehaving filter servlets or intervening systems like proxies might strip the file out of the request before SmartGWT server code even receives it. So we're trying to help you figure out whether the request is being tampered with and if so, and what subsystem is doing it.

Thank you for trying to help me out...
I don't think there is any filter servlet or intervening system, a different project runs on the same dev mode in the same Intellij and uses uploadForm.submitForm() fine. It generate a similar POST and I can see the data in the same Firebug section. The file goes to a HttpServlet that gets the data in req.getInputStream().

[Isomorphic]

It's unclear what part of the Firebug interface you've grabbed information from - what you've posted actually does suggest a file upload, but isn't completely definitive.

The server logs likewise suggest that Firefox was attempting to post a file (Content-Type is multipart/form-data) but the file is clearly gone by the time SmartGWT gets the request (Content-Length 1295).

In the most recent Firefox/Firebug, if you look at the request in the "Net" panel and select the request for the upload, you should be able to see raw binary data in the "Post" tab. This would be definitive. You can also check the Content-Length in the "Headers" tab (you want the *Request* Content-Length header) and compare that to the server log.

As we mentioned, misbehaving filter servlets or intervening systems like proxies might strip the file out of the request before SmartGWT server code even receives it. So we're trying to help you figure out whether the request is being tampered with and if so, and what subsystem is doing it.

[Vittorio Paternostro]

Hi Blama,

I'm uploading a CSV file containing my data, how you would you create Records as you suggested?

[Blama]

A ListGrid contains Records. DataSources work with DSRequests containing Records.
So you could create Records on your own, create a DSRequest for them and send the request to the DataSource.

Best regards,
Blama

[Vittorio Paternostro]

The problem with the first two calls is that your binary field is called "image" and you are passing some other values for fieldName.

Thanks for the answer.
I did a bad copy an paste.

FileItem imageItem = new FileItem("file");

ISCFileItem file = dsRequest.getUploadedFile("file");

<DataSource ID="updateDataSource" serverType="generic">
	<serverObject lookupStyle="new" className="com.dotomi.zisa.beans.VirtualTableBean" methodName="add"/>

	<fields>
		<field name="file" type="binary"/>
	</fields>
</DataSource>

the file is empty.

I checked with Firebug the POST and I do not see the file I'm uploading:

file	
_transaction	<transaction xmlns:xsi="http://www.w3.org/2000/10/XMLSchema-instance" xsi:type="xsd:Object"><transactionNum xsi:type="xsd:long">6</transactionNum><operations xsi:type="xsd:List"><elem xsi:type="xsd:Object"><criteria xsi:type="xsd:Object"></criteria><values xsi:type="xsd:Object"></values><operationConfig xsi:type="xsd:Object"><dataSource>updateDataSource</dataSource><operationType>update</operationType></operationConfig><componentId>isc_DynamicForm_7</componentId><appID>builtinApplication</appID><operation>updateDataSource_update</operation><oldValues xsi:type="xsd:Object"></oldValues></elem></operations><jscallback>if (!(new RegExp(&quot;^(\\d{1,3}\\.){3}\\d{1,3}$&quot;).test(document.domain))) {while (!window.isc &amp;&amp; document.domain.indexOf(&quot;.&quot;) != -1 ) { try { parent.isc; break} catch (e) {document.domain = document.domain.replace(/.*?\./, &quot;&quot;)}}}parent.isc.Comm.hiddenFrameReply(6,results)</jscallback></transaction>

Source
-----------------------------68784638123650836141556675 Content-Disposition: form-data; name="file"; filename="" Content-Type: application/octet-stream

Note that, as shown in samples, binding to a DataSource with a binary field and calling saveData() is all you need to do to upload a file.

That's why I'm trying to use it.

2. check for filter servlets, security-related plugins in your application server, or other layers that might have intercepted the binary file before it reaches SmartGWT server code.

It's in dev mode in Intellij.

3. post the server logs. Just a note that this is always required when reporting a server-side issue

the logs:

=== 2014-01-16 09:51:09,097 [l0-4] DEBUG IDACall - Header Name:Value pair: Host:vpaternostro-lux.dotomi.com:8888
=== 2014-01-16 09:51:09,097 [l0-4] DEBUG IDACall - Header Name:Value pair: User-Agent:Mozilla/5.0 (X11; Linux x86_64; rv:25.0) Gecko/20100101 Firefox/25.0
=== 2014-01-16 09:51:09,097 [l0-4] DEBUG IDACall - Header Name:Value pair: Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
=== 2014-01-16 09:51:09,097 [l0-4] DEBUG IDACall - Header Name:Value pair: Accept-Language:en-US,en;q=0.5
=== 2014-01-16 09:51:09,097 [l0-4] DEBUG IDACall - Header Name:Value pair: Accept-Encoding:gzip, deflate
=== 2014-01-16 09:51:09,097 [l0-4] DEBUG IDACall - Header Name:Value pair: Referer:http://vpaternostro-lux.dotomi.com:8888/zisa/zisa.html?gwt.codesvr=vpaternostro-lux.dotomi.com:9997
=== 2014-01-16 09:51:09,097 [l0-4] DEBUG IDACall - Header Name:Value pair: Cookie:JSESSIONID=bg5fspzhmivr; DotomiUser=874701979913113657$0$136977626$$1; DotomiNet=2$DjQqblZ1THBLB21bBAV6VAFHKDNEGQNGC19tVlFAYHxmfw8BAg1CXRZSBwIZJRQWJA1ZfWhhWnZRIgFAaV0XVzMdb1F9ewR7AEQAXGlUW09neGd2CA%3D%3D; SS_MID=17cfb219-4164-4dd3-bb87-3e0783de3fbdhq8k0f2t; ss_cid=1c4075e9-3cfa-4de3-ab1f-05199c032ca4; __utma=158422191.1922713510.1389304471.1389304471.1389304471.1; __utmz=158422191.1389304471.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); dtm_token=AQENIpHR_AxROAEJKx3bAQEG0QE; isc_cState=ready; crowd.token_key=R1aejnO3JN07H3tBCoJXWg00; JSESSIONID=bg5fspzhmivr
=== 2014-01-16 09:51:09,097 [l0-4] DEBUG IDACall - Header Name:Value pair: Connection:keep-alive
=== 2014-01-16 09:51:09,097 [l0-4] DEBUG IDACall - Header Name:Value pair: Content-Type:multipart/form-data; boundary=---------------------------4950107451227722678174011772
=== 2014-01-16 09:51:09,097 [l0-4] DEBUG IDACall - Header Name:Value pair: Content-Length:1295
=== 2014-01-16 09:51:09,097 [l0-4] DEBUG IDACall - session exists: bg5fspzhmivr
=== 2014-01-16 09:51:09,097 [l0-4] DEBUG IDACall - remote user: null
=== 2014-01-16 09:51:09,116 [l0-4] DEBUG XML - Parsed XML from (in memory stream): 2ms
=== 2014-01-16 09:51:09,117 [l0-4] DEBUG RPCManager - Processing 1 requests.
=== 2014-01-16 09:51:09,117 [l0-4] DEBUG DSRequest - Caching instance 13 of DS updateDataSource from DSRequest.getDataSource()
=== 2014-01-16 09:51:09,118 [l0-4] DEBUG RPCManager - Request #1 (DSRequest) payload: {
    criteria:{
    },
    values:{
    },
    operationConfig:{
        dataSource:"updateDataSource",
        operationType:"update"
    },
    componentId:"isc_DynamicForm_7",
    appID:"builtinApplication",
    operation:"updateDataSource_update",
    oldValues:{
    }
}
=== 2014-01-16 09:51:09,118 [l0-4] INFO  IDACall - Performing 1 operation(s)
=== 2014-01-16 09:51:09,118 [l0-4] DEBUG DeclarativeSecurity - Processing security checks for DataSource null, field null
=== 2014-01-16 09:51:09,118 [l0-4] DEBUG DeclarativeSecurity - DataSource updateDataSource is not in the pre-checked list, processing...
=== 2014-01-16 09:51:09,119 [l0-4] INFO  DSResponse - DSResponse: List with 0 items
=== 2014-01-16 09:51:09,119 [l0-4] DEBUG RPCManager - Content type for RPC transaction: text/html; charset=UTF-8
=== 2014-01-16 09:51:09,119 [l0-4] DEBUG RPCManager - DMI response, dropExtraFields: false
=== 2014-01-16 09:51:09,120 [l0-4] INFO  Compression - /zisa/sc/IDACall: 648 -> 405 bytes

This is taking too much time. I'll start working on a solution with BatchUploader. Is there a way to parse the uploaded CSV content into something I can cycle through without sending the data to an additional grid that shares the datasource like in the example?

[Isomorphic]

Those are the correct APIs to use to interact with the uploaded file (as covered in the Binary Fields overview). It's not an especially complicated API :)

The problem with the first two calls is that your binary field is called "image" and you are passing some other values for fieldName. getUploadedFiles() however will work regardless - if that's null, the things to check are:

1. use a tool like Firebug or Chrome's developer tools to look at the request to verify there's actually a file in it. Note that, as shown in samples, binding to a DataSource with a binary field and calling saveData() is all you need to do to upload a file.

2. check for filter servlets, security-related plugins in your application server, or other layers that might have intercepted the binary file before it reaches SmartGWT server code.

3. post the server logs. Just a note that this is always required when reporting a server-side issue

[Vittorio Paternostro]

In particular, your mention of a servlet suggests that you were not aware of how upload is typically done (no need to create a servlet) and you merely mentioned that you were "thinking" about sending data as POJOs, not that this was a requirement.

I'm asking YOU what's the simplest ways to do it, since the tipically done must be not typical since there is no example of it on Google. A servlet with form submit() and request.getInputStream() is one of the ways to do it that we have already implemented. But there must be a simpler way with some RPC call where I can process datafrom an uploaded file and process a callback response.

The FileUpload example uses an image to put data on a grid and does not suggest how to read the uploaded file on the server.
I've tried a similar dynamic form with the same saveData() and on the server side the following methods on the DSRequest instance:
ISCFileItem file = dsRequest.getUploadedFile("file");
ISCFileItem file1 = dsRequest.getUploadedFile("title");
List file2 = dsRequest.getUploadedFiles();
Map map = dsRequest.getValues();
RequestContext context = dsRequest.getRequestContext();

and they are not null. I do not see the uploaded file on the request. How can I get it?

As far as how to get some POJOs after uploading and parsing a CSV, the previously mentioned APIs - dsRequest.addRelatedUpdates() and dsRequest.setProperty() - are both ways to deliver data to the client which could then be used to populate POJOs on the client-side (we're just guessing they are client-side POJOs - you didn't specify).

I will create my Pojos and DTOs and put to the response, I just need to read the file. But it does not seem to be in the request object with the file uploader widget!

We don't actually have enough information to create any kind of sample from this, but again for your reference, if you want to have a particular use case prototyped for you, you should take a look at our Consulting our Jump Start commercial services - your Support service is oriented around addressing product flaws and general how-to questions, and is not a service you can use to just have some arbitrary use case actually coded for you.

I don't need you to do my job, i can implement it with Apache POI to import/export Excel files by myself. You should provide examples how to use FileUpload and Batch Upload to do with text since those examples are very situation specific. It's impossible that to upload and process just a text file you have to go through such complications.

<DataSource ID="updateDataSource" serverType="generic">
	<serverObject lookupStyle="new" className="com.dotomi.zisa.beans.VirtualTableBean" methodName="add"/>
	<fields>
		<field name="image" type="binary"/>
	</fields>
</DataSource>

//same as the FileUpload sample. 
		ButtonItem saveItem = new ButtonItem("save", "Save");
		saveItem.addClickHandler(new com.smartgwt.client.widgets.form.fields.events.ClickHandler() {
			public void onClick(com.smartgwt.client.widgets.form.fields.events.ClickEvent event) {
				DataSource dataSource = DataSource.get("updateDataSource");
				uploadForm.setDataSource(dataSource);
				if (!uploadForm.hasErrors()) {
					uploadForm.saveData(new com.smartgwt.client.data.DSCallback() {
						public void execute(DSResponse response, Object data, DSRequest request) {
							uploadForm.editNewRecord();
						}
					});
				}
			}
		});

    // can't find the file data on the request
	public static DSResponse add(com.isomorphic.datasource.DSRequest dsRequest) throws Exception { [...]

[Isomorphic]

This reply is not useful. You didn't read my post.

Actually, we did, and we would recommend you re-read your own post - you will find the reasons why you got the reply you did.

In particular, your mention of a servlet suggests that you were not aware of how upload is typically done (no need to create a servlet) and you merely mentioned that you were "thinking" about sending data as POJOs, not that this was a requirement.

As far as the BatchUploader sample, it doesn't seem relevant since you apparently need POJOs as the output of a CSV upload, and not a fully orchestrated review-and-save interaction like the BatchUploader offers. However, for your reference, there's nothing in the BatchUploader sample that is specific to a Hibernate DataSource. The Hibernate DataSource shown in that sample can be plug-replaced with any other DataSource.

As far as how to get some POJOs after uploading and parsing a CSV, the previously mentioned APIs - dsRequest.addRelatedUpdates() and dsRequest.setProperty() - are both ways to deliver data to the client which could then be used to populate POJOs on the client-side (we're just guessing they are client-side POJOs - you didn't specify).

We don't actually have enough information to create any kind of sample from this, but again for your reference, if you want to have a particular use case prototyped for you, you should take a look at our Consulting our Jump Start commercial services - your Support service is oriented around addressing product flaws and general how-to questions, and is not a service you can use to just have some arbitrary use case actually coded for you.

[Vittorio Paternostro]

You can declare a separate dataSourceField.exportTitle if you have placed styling in DataSourceField.title. Or you can move the styling to listGridField.title so it is not in the DataSource.

This is useful, thanks.

You can use a FileItem to send a CSV to the server. There is no need to write a servlet - see the several examples under Upload / Download for sample code.

You haven't mentioned the volume of data involved. If the volume is large, the best approach is probably a temporary SQL table or similar server-side structure to hold the merged dataset, which can then be accessed by a DataSource just like any other SQL table.

If the data volume is smaller, one approach would be to send all the changes back via the server-side API DSResponse.addRelatedUpdate(), which would cause all the changes to be reflected in a client-side grid. Other properties you might want to use in custom logic, such as the total rows affected, could be communicated via DSResponse.setProperty().

This reply is not useful. You didn't read my post.
I have mentioned the examples that you linked. And explained that for what I need to do and they are not helpful.

I do not want to use a temp database table since there is no direct data to compare in the database.

As I said clearly, I do not want to changes the values updating the grid directly. I need to have the values in Pojos and call my own save mechanism.

I'm asking you if you could please provide an example of FileUpload or BatchUploader where I can get and manipulate the data from the CSV that uses a normal datasource (in my case a dynamic one), not the Hibernate one as in the BatchUploader example.

Even if I don't give the user the possibility to make a confirmation as mentioned above, I need to report to them a summary of what has changed.

[Isomorphic]

You can declare a separate dataSourceField.exportTitle if you have placed styling in DataSourceField.title. Or you can move the styling to listGridField.title so it is not in the DataSource.

You can use a FileItem to send a CSV to the server. There is no need to write a servlet - see the several examples under Upload / Download for sample code.

You haven't mentioned the volume of data involved. If the volume is large, the best approach is probably a temporary SQL table or similar server-side structure to hold the merged dataset, which can then be accessed by a DataSource just like any other SQL table.

If the data volume is smaller, one approach would be to send all the changes back via the server-side API DSResponse.addRelatedUpdate(), which would cause all the changes to be reflected in a client-side grid. Other properties you might want to use in custom logic, such as the total rows affected, could be communicated via DSResponse.setProperty().

[Vittorio Paternostro]

Hi all,

I download data from a listgrid into a csv file with exportData.
Very simple and very effective. Except that is putting in the file the listgridfield titles and not the names. The titles have a <span> with CSS style to make some of them bold. How do I put the names instead?

    private void downloadCsv() {
        DSRequest dsRequestProperties = new DSRequest();
        dsRequestProperties.setExportAs(ExportFormat.CSV);
        dsRequestProperties.setExportDisplay(ExportDisplay.DOWNLOAD);
        List<String> gridFields = MainPanel.getGrid().getFieldsNames();
        dsRequestProperties.setExportFields(gridFields.toArray(new String[0]));
        MainPanel.getGrid().exportData(dsRequestProperties);
    }

Now the bigger question:

After the user makes some modifications in the csv file in Excel (I'm using CSV since XLS uploading is not yet supported), I'd like to reimport the data into the grid by uploading the same file. I do not need the user to review the changes in an additional grid after the validation as in the BatchUploader sample.

I would like to make following:
- let the user upload a csv file, verifying the file extension (probably client side?)
- get the data from the file and, for the records where primary key and fields match, update the data in the grid. I was thinking sending the file to a servlet with a FileItem in a dynamic form, parse it into Pojos, send back to the client the data in any data structure and programmatically doing the match client side
- At the callback, prompt the user with a confirmation window with a summary of how many records are going to be updated, and at user confirmation, save the changes reusing my saving mechanism (the editing is not with datasource save, it's done in a dynamic form editor that bulk saves the changes sending data to the backend by RPC and refetches the grid at callback).

The grid has a dynamic datasource. In the BatchUploader sample the records are added to the grid. Mine is an update of records. Also, it's using a Hibernate datasource while I have a dynamic datasource. In the same example, I can't find the servlet that handles the request (specifically, the exampleTransactionManagerController).

Can you please indicate the simpler way to do what I need with some example code?
Is it possible to use a FileItem in a dynamic form to send the uloaded file to a servlet, parse the data into Pojos, and send them back to the client for validation and processing like I want to do?

Thanks,
Vittorio

=========== ENVIRONMENT DETAILS ===========

smartgwt.version:4.1.d20130712
SmartClient Version: SNAPSHOT_v9.1d_2013-07-12/PowerEdition Deployment (built 2013-07-12)

Firefox 25.0