Severe problem with DataImport.importDataSourceRecords() after switching to current nightly

[Blama]

Hi Isomorphic,

it seems I have a problem with DataImport.importDataSourceRecords() that was not there before. I'm using v10.1p_2017-09-09 now and was using some older version (Aug 22, but not sure) before.
This is my row:

                                List<Map<String, Object>> dataImporterResult = dataImporter.importDataSourceRecords(emailContentReader, columnRemap,
                                        importDS);

Even though my columnRemap is OK and fits to my emailContentReader-content, not all fields get populated.
Does this already mean something to you? I'll try to narrow it down, but any suggestion is appreciated.

This is an important one for me.

Best regards
Blama

[Blama]

Hi Isomorphic,

1st observation: It does not happen with v10.1p_2017-08-20 using the very same code.

Best regards
Blama

[Blama]

Hi Isomorphic,

2nd observation: It does happen with v10.1p_2017-09-02 using the very same code.

Best regards
Blama

[Blama]

Hi Isomorphic,

2nd observation: It also does happen with v10.1p_2017-08-26 using the very same code.
There are no versions between 08-22 and 08-26. Perhaps you can find the offending part now even without a testcase, which I assume will be difficult to create.

The issue is that some fields are not in the dataImporterResult, even though they should be as fieldnames, remap and data fit perfectly.

Best regards
Blama

[Isomorphic]

This is most likely caused by new behaviour of DSField.getTitle() API, which now returns localized field title for localized field. Can you take this into consideration when providing columnRemap for DataImport? Let us know please how it worked for you. Thank you.

[Blama]

Hi Isomorphic,

I did not roll out versions newer than 08-20 where I have the import, but would like to do so now.
Therefore I revisited this thread.
Actually I do not think that it is related to your DSField.getTitle()-changes, at least I do not rely on the field-title. Perhaps you do rely on it internally and need changes there?

I'm using this call:

dataImporter.importDataSourceRecords(emailContentReader, [B]columnRemap,[/B] importDS); // emailContent is in CSV format

I explicitly give a mapping CSV-name to fieldName (the correct name-Attribute in .ds.xml)
Additionally I also set the uploadFieldName to the CSV's column name in ds.xml via DynamicDSGenerator.
So I really don't think the title-change here should affect me.

But I do wonder about this:
uploadFieldName-docs say:

Used by the BatchUploader to map a field in an upload file to this dataSourceField. This is only necessary if the dataSourceField's name and title differ from the name of the field in the upload file (Smart GWT will automatically map upload fields using the dataSourceField's title, if possible, if it does not get a direct match on field name).

So in my opinion I'm using two mechanisms to set the import name, so that no field title is needed here.
1) Using columnRemap should be enough to use the ds.xml field's name-Attribute.
2) If ignored, having uploadFieldName at all and having it set to the CSV-column title should also eliminate the need for column-to-title guessing/matching.

Best regards
Blama

[Isomorphic]

Regardless of using those two mechanisms, the title is still used as a fallback, as the docs say.

At the moment, we have no other theory as to why the behavior seems to be changed for you - nothing in this part of the code has changed in quite a while, and all related autotests are passing.

So please look into the possibility that your import was actually relying on field titles, and if that doesn't appear to be the issue, we will need a test case.

[Blama]

Hi Isomorphic.

OK, will do. Can you describe the mechanism you are using a bit?

• What has the highest preference? name, uploadFieldName, title?
• What is columnRemap compared to? Also title? This would make no sense IMHO, as the developer here explicitly maps towards a name.
• Is this per-field or does the 1st field set mechanism that is also going to be used for the following fields?

Thank you & Best regards
Blama

[Blama]

Hi Isomorphic,

I'm getting nearer to the root cause.
I'm not sure if this helps you, but it seems the problem is in this call (v11.1p_2017-10-11):

DataImport.java:

/* 1377 */     if ((rows != null) && (columnRemap != null)) {
/* 1378 */       rows = DataTools.remapRows(rows, columnRemap, keepNonRemappedColumns);
/*      */     }
/* 1380 */     return rows;
/*      */   }

Before the call, every key in rows has a corresponding entry with matching value in columnRemap, meaning rows.key = columnRemap.value.
After the call, rows contains fewer entries than before. I don't have reasoning for this.

Unfortunately JD Eclipse does not display DataTools.remapRows() correctly, which makes debugging a hassle, as the lines do not match and the current position is not the one Eclipse displays.

You are saying "nothing in this part of the code has changed in quite a while", but class DataTools is in isomorphic_core_rpc.jar and class DataImport is in isomorphic_tools.jar. Perhaps there was a change in the latter one?
Can you have a look?

Best regards
Blama

PS:
For future reference, the thread you mention in #5 is this one.

[Blama]

This is the code in question:

/*      */   public static Map remapRow(Map row, Map remap, boolean keepNonRemapped)
/*      */   {
/* 1971 */     if (remap == null) return row;
/* 1972 */     Map newRow = new HashMap();
/* 1973 */     for (Iterator e = row.keySet().iterator(); e.hasNext();) {
/* 1974 */       Object oldKey = e.next();
/* 1975 */       Object newKey = remap.get(oldKey);
/* 1976 */       Object data = row.get(oldKey);
/* 1977 */       if (newKey == null) {
/* 1978 */         if (keepNonRemapped) newRow.put(oldKey, data);
/*      */       } else {
/* 1980 */         newRow.put(newKey, data);
/*      */       }
/*      */     }
/* 1983 */     return newRow;
/*      */   }

To me it looks reasonable and the only reason to loose entries would be a a combination of newKey==null (which can't be the case IMHO) and keepNonRemapped=false (which is the case).

Do you have an idea?

Best regards
Blama

[Blama]

Hi Isomorphic,

please disregard #9 and #10.

Best regards
Blama

[Blama]

Having a closer look at #9 and #10 I realized that the there are entries in columnRemap for every key in rows-entry, but they are not necessarily the key in columnRemap, sometimes they are already the columnRemap-value.
--> The map is mixed with already translated values and still to be translated values.

This happens because you are correct in a strange way with your sentence "So please look into the possibility that your import was actually relying on field titles" in #7.

See this excerpt from List importToRows(Reader in, Map columnRemap, Map translators, DataSource dataSource, DSRequest dsRequest):

/* 1088 */         for (int i = 0; i < columns.size(); i++) {
/* 1089 */           [B]String column = (String)columns.get(i);[/B]
/* 1090 */           boolean exist = false;
/* 1091 */           if ((columnRemapTitles != null) && (columnRemapTitles.size() != 0)) {
/* 1092 */             String columnTitle = "";
/* 1093 */             for (Iterator e = columnRemapTitles.keySet().iterator(); e.hasNext();) {
/* 1094 */               columnTitle = (String)e.next();
/* 1095 */               [B]if (columnTitle.equalsIgnoreCase(column)) {[/B]
/* 1096 */                 exist = true;
/*      */               }
/*      */             }
/*      */             
/* 1100 */             [B]if (exist) columnsCopy.add(columnRemapTitles.get(columnTitle)); //this line mixes it all up[/B]
/*      */           }
/* 1102 */           if (!exist)
/*      */           {
/* 1104 */             String columnName = "";
/* 1105 */             for (Iterator e = linkedColumnRemap.keySet().iterator(); e.hasNext();) {
/* 1106 */               columnName = (String)e.next();
/* 1107 */               if (columnName.equalsIgnoreCase(column)) {
/* 1108 */                 exist = true;
/*      */               }
/*      */             }
/*      */             
/* 1112 */             if (exist) columnsCopy.add(columnName);
/*      */           }

As you can see, it 1st compares to columnTitle and if it matches adds the fieldName to columnsCopy. Before the title-change, this was never successful (because of the format you used) and I kinda relied on this.
Now it can be successful if the name of the data-field matches the title of the .ds.xml.

I'm wondering if all this should happen in a setting where I give the columnRemap parameter?!
The docs state:

An optional columnRemap can be provided to translate between column names in CSV/TSV or property names in JSON to the [B]field names[/B] of the DataSource. Use null as a Map value to cause data for a column to be discarded.

If [B]no [/B]columnRemap is provided, column names will be matched to DataSource fields by comparing to both the field name and field title, ignoring letter case. Any column name that isn't matched to a DataSource field is discarded.

IMHO the code shown above should not be run, if a column remap is provided.

I'm not perfectly sure here, but seeing my mixed map and this code makes the source of the error clear now.
This might have been a bug the whole time which now came to the surface after the field-title change in the other thread.
Right now the only solutions I see are:

• You change some code here because it seems to disagree with the docs.
• I remove the field-titles from my .ds.xml (any title could be used in some constellation as CSV-column name and result in this error). I would not like to do this, as later on the validators use the field title for the user-facing error message.

Best regards
Blama

[Blama]

Hi Isomorphic,

here a testcase with latest 6.1p:
Upload.java:

package com.smartgwt.sample.server.listener;

import java.io.IOException;
import java.io.Reader;
import java.io.StringReader;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import com.isomorphic.tools.DataImport;
import com.isomorphic.tools.DataImport.ImportFormat;

public class Upload extends HttpServlet {
    private static final long serialVersionUID = -536025913450167992L;

    @Override
    public void doGet(HttpServletRequest servletRequest, HttpServletResponse servletResponse) throws ServletException, IOException {
        try {
            String csvContent = "Manager;EmployeeName;Gender;ID";
            csvContent += "\r\n";
            csvContent += "Charles Madigan;New colleague;male;1001";
            csvContent += "\r\n";

            Reader emailContentReader = new StringReader(csvContent);
            DataImport dataImporter = new DataImport(ImportFormat.CSV, ";");
            dataImporter.setPopulateDisplayFields(true);

            String importDS = "employees";
            Map<String, String> columnRemap = new LinkedHashMap<String, String>();
            columnRemap.put("Manager", "ReportsTo"); // Manager IS the title of the field with name="ReportsTo"
            columnRemap.put("EmployeeName", "Name");
            columnRemap.put("Gender", "Gender");
            columnRemap.put("ID", "EmployeeId");

            List<Map<String, Object>> dataImporterResult = dataImporter.importDataSourceRecords(emailContentReader, columnRemap, importDS);
            if (dataImporterResult == null)
                System.out.println("Must not be null");
            else if (dataImporterResult.get(0).size() != 4)
                System.out.println("4 entries expected, found " + dataImporterResult.get(0).size());
        } catch (Exception e) {
            System.out.println(e.getMessage());
        }
    }
}

web.xml addition:

    <servlet>
        <servlet-name>Upload</servlet-name>
        <servlet-class>com.smartgwt.sample.server.listener.Upload</servlet-class>
    </servlet>
    <servlet-mapping>
        <servlet-name>Upload</servlet-name>
        <url-pattern>/Upload</url-pattern>
    </servlet-mapping>

Best regards
Blama

[Blama]

Hi Isomorphic,

using v11.1p_2017-08-08 (old, before the getFieldTitle()-change) I put a breakpoint here in importToRows(Reader in, Map columnRemap, Map translators, DataSource dataSource, DSRequest dsRequest):

/* 1088 */         for (int i = 0; i < columns.size(); i++) {
/* 1089 */           String column = (String)columns.get(i);
/* 1090 */           boolean exist = false;
/* 1091 */           if ((columnRemapTitles != null) && (columnRemapTitles.size() != 0)) {
/* 1092 */             String columnTitle = "";
/* 1093 */             for (Iterator e = columnRemapTitles.keySet().iterator(); e.hasNext();) {
/* 1094 */               columnTitle = (String)e.next();
/* 1095 */               if (columnTitle.equalsIgnoreCase(column)) {
/* 1096 */                 exist = true;
/*      */               }
/*      */             }
/*      */            
/* 1100 */             if (exist) columnsCopy.add([B]columnRemapTitles[/B].get(columnTitle));
/*      */           }

This is my .ds.xml (only one field using 18n-titles):

<DataSource [B]xmlns:fmt="lmscompany/fmt"[/B]
    ID="employees"
    serverType="sql"
    tableName="employeeTable"
    recordName="employee"
    testFileName="/examples/shared/ds/test_data/employees.data.xml"
    titleField="Name"
>
[B]<fmt:bundle basename="com.smartgwt.sample.server.listener.DSXMLResources-utf8" encoding="utf-8" />[/B]
    <fields>
        <field name="userOrder"       title="userOrder"       type="integer"  canEdit="false"    hidden="true"/>
        <field name="Name"            title="Name"            type="text"     length="128"/>
        <field name="EmployeeId"      title="Employee ID"     type="integer"  primaryKey="true"  required="true"/>
        <field name="ReportsTo"       type="integer"  required="true"
               foreignKey="employees.EmployeeId"  rootValue="1" detail="true">
[B]<title><fmt:message key="ReportsTo" /></title>[/B]
        </field>
        <field name="Job"             title="Title"           type="text"     length="128"/>
        <field name="Email"           title="Email"           type="text"     length="128"/>
        <field name="EmployeeType"    title="Employee Type"   type="text"     length="40"/>
        <field name="EmployeeStatus"  title="Status"          type="text"     length="40"/>
        <field name="Salary"          title="Salary"          type="float"/>
        <field name="OrgUnit"         title="Org Unit"        type="text"     length="128"/>
        <field name="Gender"          title="Gender"          type="text"     length="7">
            <valueMap>
                <value>male</value>
                <value>female</value>
            </valueMap>
        </field>
        <field name="MaritalStatus"   title="Marital Status"  type="text"     length="10">
            <valueMap>
                <value>married</value>
                <value>single</value>
            </valueMap>
        </field>
    </fields>
</DataSource>

Content in DSXMLResources-utf8.properties:

ReportsTo = MyManager

This is the content of columnRemapTitles in the code:

{userOrder=userOrder, Name=Name, Employee ID=EmployeeId, [B][Manager, i18n message: key=ReportsTo, message=MyManager]=ReportsTo[/B], Title=Job, Email=Email, Employee Type=EmployeeType, Status=EmployeeStatus, Salary=Salary, Org Unit=OrgUnit, Gender=Gender, Marital Status=MaritalStatus}

IMHO this shows the issue and I do still think that this matching should not run if using a columnRemap - in this case the developer should be responsible for mapping csvColumnName --> dsxmlFieldName.
With the old code this never was a problem (only when using title-fmt-i18n, which I'm doing everywhere) as your old getFieldTitle() did return something that never would match a CSV-column name.
With the new code this can and will happen, as of course I try to name field titles reasonably and the user does this with his or her CSV column names as well.

Best regards
Blama

[Blama]

Hi Isomorphic,

please note that I edited the .ds.xml from #14. I removed title="Manager", which led to the field having a title-array (unwanted).
Comparing the testcase in 13+14 in a current version and a version from 08-20 and before will show the issue (the final dataImporterResult in Upload.java has 4 entries with the older version and 3 with the newer one. It should be 4 in both cases.)
Also make sure that the DSXMLResources-utf8.properties filename matches your request-language.

Best regards
Blama