Announcement

**Blama** · 8 Nov 2017, 12:31

Hi Isomorphic,

issue #2 (minor), same setup, but

Code:

public DSResponse upload(DSRequest dsRequest, HttpServletRequest servletRequest) throws Exception {
        return new DSResponse(dsRequest.getDataSource()).setSuccess();
}

For the request from #1, the result is:

Code:

<response>
<status>
0
</status>
<queueStatus>
0
</queueStatus>
<startRow>
-1
</startRow>
<endRow>
-1
</endRow>
<totalRows>
-1
</totalRows>
<queueStatus>
0
</queueStatus>
<data />
 </response>

Please note that <queueStatus> is in there twice.

Best regards
Blama

**Blama** · 8 Nov 2017, 12:49

Hi Isomorphic,

issue #3, same request:

Code:

package com.smartgwt.sample.server.listener;

import java.io.Reader;
import java.io.StringReader;
import java.io.StringWriter;
import java.util.List;
import java.util.Map;

import javax.servlet.http.HttpServletRequest;

import com.isomorphic.datasource.DSRequest;
import com.isomorphic.datasource.DSResponse;
import com.isomorphic.datasource.DataSource;
import com.isomorphic.tools.DataImport;
import com.isomorphic.tools.DataImport.ImportFormat;

public class EmployeesUpload {
    public DSResponse upload(DSRequest dsRequest, HttpServletRequest servletRequest) throws Exception {

        @SuppressWarnings("unchecked")
        Map<String, Object> requestValues = dsRequest.getValues();
        Reader jsonReader = new StringReader(convertToCSV(requestValues));

        DataImport dataImporter = new DataImport(ImportFormat.CSV, ";");
        dataImporter.setPopulateDisplayFields(true);

        @SuppressWarnings("unchecked")
        List<Map<String, Object>> dataImporterResult = dataImporter.importDataSourceRecords(jsonReader, "employeesUpload");

        DSRequest addRequest = new DSRequest("employeesUpload", DataSource.OP_ADD, dsRequest.getRPCManager());
        addRequest.setValues(dataImporterResult);
        DSResponse addResponse = addRequest.execute();
        return addResponse;
    }

    /*
        private String convertToJSON(Map<String, Object> requestValues) {
            // json-simple-1.1.1.jar
            // https://code.google.com/archive/p/json-simple/
            JSONObject jsonFinalObject = new JSONObject();
            for (Map.Entry<String, Object> entry : requestValues.entrySet()) {
                jsonFinalObject.put(entry.getKey(), entry.getValue());
            }
            return jsonFinalObject.toJSONString();
        }

        private String convertToXML(Map<String, Object> requestValues) {
            StringWriter sw = new StringWriter();
            sw.append("<record>\r\n");
            for (Map.Entry<String, Object> entry : requestValues.entrySet()) {
                sw.append("\t<" + entry.getKey() + ">" + entry.getValue() + "</" + entry.getKey() + ">\r\n");
            }
            sw.append("</record>");
            return sw.toString();
        }
    */
    private String convertToCSV(Map<String, Object> requestValues) {
        StringWriter csvHeader = new StringWriter();
        StringWriter csvData = new StringWriter();
        for (Map.Entry<String, Object> entry : requestValues.entrySet()) {
            String cleanKey = entry.getKey().replace(";", "").replace("\"", "");
            String cleanValue = entry.getValue().toString().replace(";", "").replace("\"", "");
            csvHeader.append(cleanKey + ";");
            csvData.append(cleanValue + ";");
        }
        return csvHeader.toString() + "\r\n" + csvData.toString();
    }
}

Response:

Code:

<response>
<status>
0
</status>
<queueStatus>
0
</queueStatus>
<startRow>
-1
</startRow>
<endRow>
-1
</endRow>
<totalRows>
-1
</totalRows>
<queueStatus>
0
</queueStatus>
<queueStatus>
0
</queueStatus>
<data>
<record>
<Salary>
12345.0
</Salary>
<ReportsTo>
4
</ReportsTo>
<Gender>
male
</Gender>
<EmployeeId>
100002
</EmployeeId>
<Job>
Developer
</Job>
<ReportsToName>
Charles Madigen
</ReportsToName>
 </record>
</data>
 </response>

Issues:

As you can see, the queueStatus is now present three times.
Generally speaking this is working, but IMHO the workaround with CSV should not be necessary. DataImport-methods should have an overload that take Map<String, Object> and do the conversion displayName->ID.
When using JSON or XML, what is the expected structure? Could you add it to the docs? You can see my convertToXML() and convertToJSON(), but the DataImport-docs don't say anything about the enclosing tag or similar.
Additionally, XML is mentioned as possibility besides XML/CSV in the class description, but never in the methods.

DataImport contains methods that can be used to import data from a test data file to the List of Maps format commonly used for DataSource records (importToRows()), or directly into a DataSource (importToDataSource()).
By default the input file is expected to contain comma-delimited data like a .csv file, but JSON and XML are also supported.
Imported data may be transformed during import, for details search SmartClient Reference for "dataSourceField.importStrategy".

As enhancement: It would be great if it were possible to directly add data to a DataSource via a ARC request using the String-to-ID lookup and not to have to call DataImport oneself in a boilerplate method. If this is not a good idea as default for all ADDs for fields with importStrategy="display", then perhaps as additional attribute for an ADD or UPDATE operation binding.

Best regards
Blama

**Blama** · 27 Nov 2017, 00:44

Hi Isomorphic,

those now sat quite a while. Do you have an opinion on those?
Especially the "queueStatus is present multiple times" is a bug IMHO. As I gave a customer access to the API now, I expect the question why this is to come up very soon now.

Thank you & Best regards
Blama

**Isomorphic** · 27 Nov 2017, 06:16

These issues are assigned. It's definitely a bug that queueStatus would appear twice; we haven't looked deeply enough into the others to say yet.

**Blama** · 6 Dec 2017, 06:21

Hi Isomorphic,

do you have an update on this one?

Thank you & Best regards
Blama

**Isomorphic** · 6 Dec 2017, 15:34

No updates yet, they will be posted here when we have them.

**Isomorphic** · 15 Dec 2017, 04:05

Issues #1 and #2 are fixed and are available for download in nightly builds since Dec 15 (today). Issues from #3 are in progress, thank you for your patience.

**Blama** · 30 Dec 2017, 02:03

Hi Isomorphic,

I can see that #1 and #2 are fixed using v11.1p_2017-12-27. Do you have any further information on #3?

Thank you & Best regards
Blama

**Isomorphic** · 3 Jan 2018, 05:10

Apologies for the delay, we are addressing issues you listed in #3 (same order as in your post):

fixed
this suggestion is in progress and will be reported here when it's done
we've added XML and JSON formats to the DataImport docs with the reference to format descriptions and samples in Smartclient Reference
taken into consideration

**Blama** · 15 Jan 2018, 00:34

Hi Isomorphic,

thanks. I can see that #3.1 is fixed and also the advice for #3.3 in the SmartGWT docs to look here.

I'll wait for an update of #3.2 and perhaps #3.4 then.

Best regards
Blama

**Blama** · 15 Oct 2018, 11:35

Hi Isomorphic,

#3 point 2 is indeed necessary, more a bug than an enhancement if you want to upload data that includes linefeeds (\r, \n, \r\n). If you do this now like convertToCSV() in #3 this will fail if you have linefeeds (because it seems like a new data row in the CSV). This is the one where you write "in progress" in #10.
Linefeeds are perfectly fine in data IMHO and must be converted to <br> at display time if you want to respect them in a label. This is also working as expected when entering them in a TextAreaItem. But they don't work if you need to use DataImport.

Best regards
Blama

**Isomorphic** · 18 Oct 2018, 01:37

Just to clarify if we understand #12 correctly. Please answer following questions.

Are you talking about linefeeds inside field values? And the way DataImport is not working is that it treats those as new lines, which splits data into rows incorrectly and fails, right? We expect such linefeeds to be escaped. Also, do you get these values from some kind of tool? If so, we'd like to know what it is. Thank you.

Regarding #3.2 (providing record as a Map instead of Reader to DataImport): do you expect that if we implement it, then this problem would automatically go away? Sorry for delay, we *are* going to implement this and will update this thread when we do.

**Blama** · 18 Oct 2018, 02:08

Hi Isomorphic,

answering in different order:

Originally posted by Isomorphic View Post

Regarding #3.2 (providing record as a Map instead of Reader to DataImport): do you expect that if we implement it, then this problem would automatically go away?

Yes, expecting it to go away, because there is no escaping whatsoever, but only Java Map and no escaping or CSV/JSON/XML encode on my side and no CSV/JSON/XML decode on your side needed.

Originally posted by Isomorphic View Post

Are you talking about linefeeds inside field values? And the way DataImport is not working is that it treats those as new lines, which splits data into rows incorrectly and fails, right? We expect such linefeeds to be escaped. Also, do you get these values from some kind of tool? If so, we'd like to know what it is. Thank you.

Yes, linefeeds in the field values. Data is self generated. I have a (pretty dumbed down, JSON would be better, but (perhaps) more difficult to generate for websites with forms) mail format that I import:

Code:

fieldname: value1;
fieldname2: value2;
fieldname3: value3;

(with the requirement of no ; and : in the data). But for ease of mind, lets just assume it were JSON.
I then build this from it:

Code:

fieldname;fieldname2;fieldname3
"value1";"value2";"value3"

and feed this to DataImport as CSV. DataImport is needed, as some of the fields need to go though DataImport's fK-value ->ID conversion. Otherwise I'd just use plain ADD.
As you can see, I do have "value1" with quotes, which is what you mean by "We expect such linefeeds to be escaped" right?

So an 1-row import with linefeeds could look like this:

Code:

fieldname;fieldname2;fieldname3
"value1";"Some long data
in
field 2";"value3"

.

Or do you mean I should generated this instead by "We expect such linefeeds to be escaped"

Code:

fieldname;fieldname2;fieldname3
"value1";"Some long data\r\nin\r\nfield 2";"value3"

, so that it is always exactly 1 CSV row per data row?

Then I'd also might have problems with "\" (escape it to "\\"?).
This would also work for me for now and the change is easy on my side. But the correct way is using Java Map IMHO. Of course I could also build JSON or XML instead of CSV from the data, which is probably better, now that I think about it, but still, Java Map would be the one with basically no code needed.

Best regards
Blama

**Blama** · 18 Oct 2018, 02:14

FYI: Edited the post. As meta problem it seems that phpBB (or whatever runs the forum) has problems with \\ (backslashes) in data, at least when editing posts :)

Announcement

6.1p DataImport importToDataSource issue in combination with RESTHandler

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment