[Isomorphic]

This is a long thread with several statements and backtracks..

Can you re-explain in concise form what you think is a bug here? We don't need another dive into the source code, we need a brief explanation of what the problematic result is for you.

We're not understanding how the strategy of using the field title as a fallback is an issue for you. Are you ending up picking up columns in the import data that used to be dropped? Something else?

[Blama]

Hi Isomorphic,

you can see the issue if you compare the results for the testcase I provided. Test with an early August 6.1p nightly and with a current 6.1p nightly.

The problem is that if the title of the CSV file matches the title of a .ds.xml field, some internal conversion interferes with columnRemap.
This interference finally results in dropped columns from the DataImport-call.

In my opinion, if columnRemap is provided, the internal conversion should not happen.

The problem existed all the time, but it was not visible, as DSField.getTitle() never matched my CSV-title because DSField.getTitle() returned this stange map-format.
Now after the change, it can match the title.
In column remap I convert towards fieldName, not title, so accidentally matching towards title is a problem here.

Best regards
Blama

[Blama]

Hi Isomorphic,

did you reproduce the problem here? Currently, the problem here is the only thing stopping me from changing from 5.1p Simplicity based skin to 6.1p Tahoe based skin with it's more modern design and it's filterOperator support.
Also it's stopping me from switching to newer 5.1p. Both is not a problem for me currently as there were no bugs fixed recently that really affected me, but it would be nice to know what the status is anyway.

Best regards
Blama

[Isomorphic]

This is reproduced and we'll update this thread as soon as we have the fix.

[Blama]

Hi Isomorphic,

do you have an ETA for this one? I think this is the only one stopping me from switching to 6.1p.

Best regards
Blama

[Isomorphic]

Apologies for the delay. This is fixed across all versions from 5.1 and will be available for download in nightly builds since Nov 1 (tomorrow). Please let us know how it worked for you.

[Blama]

Hi Isomorphic,

this is working for me in the testcase. I'll try in my application as well.

Best regards
Blama

[Blama]

Hi Isomorphic,

sorry, it is not working - behavior is the same, also in the testcase (retested with current v11.1p_2017-11-08).
Please try the testcase, but make sure that the java row for the field in question matches the same in the .properties-translation file, so e.g.:

[B]Upload.java:[/B]
columnRemap.put("MyManager", "ReportsTo"); // MyManager IS the title of the field with name="ReportsTo"

[B]DSXMLResources-utf8.properties:[/B]
ReportsTo = MyManager

With this I still get 3 entries in the final map where I'd expect 4.

Best regards
Blama

[Blama]

Hi Isomorphic,

I got it reproduced in a BuiltInsDS-based testcase with all my application .ds.xml. I'm trying to strip this down a bit and will send you the result as eMail to support@isomorphic.com.

Best regards
Blama

[Blama]

Hi Isomorphic,

actually I got it striped down to a forums size and also without sensitive information (using v11.1p_2017-11-08):

Upload.java (register in web.xml):

package com.smartgwt.sample.server.listener;

import java.io.IOException;
import java.io.Reader;
import java.io.StringReader;
import java.util.LinkedHashMap;
import java.util.List;
import java.util.Map;

import javax.servlet.ServletException;
import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import com.isomorphic.tools.DataImport;
import com.isomorphic.tools.DataImport.ImportFormat;

public class Upload extends HttpServlet {
    private static final long serialVersionUID = -536025913450167992L;

    @Override
    public void doGet(HttpServletRequest servletRequest, HttpServletResponse servletResponse) throws ServletException, IOException {
        try {
            String csvContent = "Anzahl_Mitarbeiter;Firma;Bemerkungen;Straße;Branche;Anrede;Name;Land;Ort;Leadtemperatur;E-mail;Telefon;Vorname;PLZ";
            csvContent += "\r\n";
            csvContent += "20;ACME Inc.;Some comment;Main street 2;Aerospace;Mr;Doe;Germany;Munich;heiß;test@example.com;0123 45678;John;80803";
            csvContent += "\r\n";

            Reader emailContentReader = new StringReader(csvContent);
            DataImport dataImporter = new DataImport(ImportFormat.CSV, ";");
            dataImporter.setPopulateDisplayFields(true);

            String importDS = "V_LEADUPLOAD";
            Map<String, String> columnRemap = new LinkedHashMap<String, String>();
            columnRemap.put("Firma", "CUST_NAME");
            columnRemap.put("Anrede", "CUST_CONTACTPERS_GENDER_SN");
            columnRemap.put("Vorname", "CUST_CONTACTPERS_GIVENNAME");
            columnRemap.put("Name", "CUST_CONTACTPERS_SURNAME");
            columnRemap.put("Telefon", "CUST_CONTACTPERS_TELEPHONE");
            columnRemap.put("E-mail", "CUST_CONTACTPERS_EMAIL");
            columnRemap.put("Straße", "CUST_ADDRESS_STREET");
            columnRemap.put("PLZ", "CUST_ADDRESS_ZIPCODE");
            columnRemap.put("Ort", "CUST_ADDRESS_CITY");
            columnRemap.put("Land", "CUST_ADDRESS_COUNTRY_ID");
            columnRemap.put("Bemerkungen", "CREATORCOMMENT");
            columnRemap.put("Leadtemperatur", "MD_LEADTEMPERATURE_ID");
            columnRemap.put("Anzahl_Mitarbeiter", "MD_CUSTOMER_NUMBEROFEMPLOYEES");
            columnRemap.put("Branche", "INDUSTRY1_ID");

            List<Map<String, Object>> dataImporterResult = dataImporter.importDataSourceRecords(emailContentReader, columnRemap, importDS);
            if (dataImporterResult == null)
                servletResponse.getWriter().append("dataImporterResult is null.");
            else {
                for (Map.Entry<String, Object> entry : dataImporterResult.get(0).entrySet()) {
                    String msg = entry.getKey() + ": " + ((entry.getValue() == null) ? "null" : entry.getValue().toString());
                    servletResponse.getWriter().append(msg + "\r\n");
                    System.out.println(msg);
                }
            }
        } catch (Exception e) {
            System.out.println(e.getMessage());
        }
    }
}

V_LEADUPLOAD.ds.xml:

<DataSource xmlns="lmscompany/ds" xmlns:fmt="lmscompany/fmt" dbName="Oracle" tableName="V_LEAD_CURRENT" ID="V_LEADUPLOAD" serverType="sql">
    <fmt:bundle basename="com.smartgwt.sample.server.listener.DSXMLResources-utf8" encoding="utf-8" />
    <fields>
        <!-- CONTACT-related fields -->
        <field name="CUST_NAME" uploadFieldName="Company" length="80" type="text" escapeHTML="true">
            <title><fmt:message key="company" /></title>
        </field>
        <field name="CUST_DIVISION" uploadFieldName="Division" length="50" type="text" escapeHTML="true">
            <title><fmt:message key="division" /></title>
        </field>
        <field name="CUST_BRANCH" uploadFieldName="Branch" length="50" type="text" escapeHTML="true">
            <title><fmt:message key="branch" /></title>
        </field>
        <field name="CUST_DEPARTMENT" uploadFieldName="Department" length="50" type="text" escapeHTML="true">
            <title><fmt:message key="department" /></title>
        </field>
        <field name="CUST_URL" uploadFieldName="URL" length="100" type="text" escapeHTML="true">
            <title><fmt:message key="url" /></title>
        </field>
        <field name="CUST_CONTACTPERS_SOURCEUID" uploadFieldName="Kontaktperson-ID" length="50" type="text">
            <title><fmt:message key="contactPersonIDSource" /></title>
        </field>

        <field name="CUST_CONTACTPERS_GENDER_SN" importStrategy="key" uploadFieldName="CUST_CONTACTPERS_GENDER_SN" displayField="CUST_CONTACTPERS_GENDER_SNNAME"
            foreignKey="T_SETTINGS_IMPORT_FOA_MAP.TRANSLATION">
        </field>
        <field name="CUST_CONTACTPERS_GENDER_SNNAME" includeFrom="T_SETTINGS_IMPORT_FOA_MAP.NAME" />

        <field name="CUST_CONTACTPERS_TITLE" uploadFieldName="Titel" length="15" type="text" escapeHTML="true">
            <title><fmt:message key="title" /></title>
        </field>

        <field name="CUST_CONTACTPERS_GIVENNAME" uploadFieldName="Vorname" length="30" type="text" escapeHTML="true">
            <title><fmt:message key="givenName" /></title>
        </field>

        <field name="CUST_CONTACTPERS_SURNAME" uploadFieldName="Name" length="30" type="text" escapeHTML="true">
            <title><fmt:message key="name" /></title>
        </field>

        <field name="CUST_CONTACTPERS_POSITION" uploadFieldName="Position" length="100" type="text" escapeHTML="true">
            <title><fmt:message key="positionOfContactPerson" /></title>
        </field>

        <field name="CUST_CONTACTPERS_TELEPHONE" uploadFieldName="Telefonnummer" length="50" type="text" escapeHTML="true">
            <title><fmt:message key="telephoneNumber" /></title>
        </field>

        <field name="CUST_CONTACTPERS_MOBILE" uploadFieldName="Mobil" length="50" type="text" escapeHTML="true">
            <title><fmt:message key="mobile" /></title>
        </field>

        <field name="CUST_CONTACTPERS_FAX" uploadFieldName="Fax" length="50" type="text" escapeHTML="true">
            <title><fmt:message key="fax" /></title>
        </field>

        <field name="CUST_CONTACTPERS_EMAIL" uploadFieldName="eMail" length="100" type="emailType" escapeHTML="true">
            <title><fmt:message key="email" /></title>
        </field>

        <field name="CUST_ADDRESS_STREET" uploadFieldName="Str." length="50" type="text" escapeHTML="true">
            <title><fmt:message key="street" /></title>
        </field>

        <field name="CUST_ADDRESS_ADDRESSLINE1" uploadFieldName="Zusatz 1" length="50" type="text" escapeHTML="true">
            <title><fmt:message key="additional1" /></title>
        </field>

        <field name="CUST_ADDRESS_ZIPCODE" uploadFieldName="PLZ" length="10" type="text" escapeHTML="true">
            <title><fmt:message key="zipCode" /></title>
        </field>
        <field name="CUST_ADDRESS_CITY" uploadFieldName="Stadt" length="50" type="text" escapeHTML="true">
            <title><fmt:message key="city" /></title>
        </field>

        <field name="CUST_ADDRESS_COUNTRY_ID" importStrategy="key" uploadFieldName="ADDRESS_COUNTRY_ID" displayField="CUST_ADDRESS_COUNTRY_NAME"
            foreignKey="T_COUNTRY.ID">
            <title><fmt:message key="country" /></title>
        </field>
        <field name="CUST_ADDRESS_COUNTRY_NAME" includeFrom="T_COUNTRY.SHORTNAME_DE" />

        <field name="CREATORCOMMENT" uploadFieldName="Kommentar" length="2000" type="text" escapeHTML="true">
            <title><fmt:message key="comment" /></title>
        </field>

        <field name="INDUSTRY1_ID" importStrategy="key" customSelectExpression="''" uploadFieldName="INDUSTRY1_ID" displayField="INDUSTRY1_NAME"
            foreignKey="T_INDUSTRY.ID" relatedTableAlias="industry1">
        </field>
        <field name="INDUSTRY1_NAME" includeFrom="T_INDUSTRY.NAME" includeVia="INDUSTRY1_ID" />
    </fields>
</DataSource>

T_INDUSTRY.ds.xml:

<DataSource xmlns="lmscompany/ds" xmlns:fmt="lmscompany/fmt" dbName="Oracle" tableName="T_INDUSTRY" ID="T_INDUSTRY" serverType="sql">
    <fmt:bundle basename="com.smartgwt.sample.server.listener.DSXMLResources-utf8" encoding="utf-8" />
    <fields>
        <field primaryKey="true" hidden="true" name="ID" type="sequence" />
        <field hidden="true" name="TENANT_ID" type="integer" canEdit="false" />
        <field name="SHORTNAME" length="40" type="text" escapeHTML="true" required="true">
            <title><fmt:message key="shortname" /></title>
        </field>
        <field name="NAME" length="80" type="text" escapeHTML="true" required="true">
            <title><fmt:message key="name" /></title>
        </field>
    </fields>
</DataSource>

DSXMLResources-utf8.properties:

name = Name
givenName = Vorname

This is the result I'm getting in the browser:

[B]INDUSTRY1_NAME: Doe[/B]
CUST_ADDRESS_COUNTRY_NAME: Germany
CUST_ADDRESS_CITY: Munich
CUST_ADDRESS_COUNTRY_ID: null
CUST_CONTACTPERS_EMAIL: test@example.com
INDUSTRY1_ID: null
CUST_ADDRESS_STREET: Main street 2
CREATORCOMMENT: Some comment
CUST_CONTACTPERS_GENDER_SNNAME: Mr
CUST_CONTACTPERS_GIVENNAME: John
CUST_CONTACTPERS_GENDER_SN: null
CUST_CONTACTPERS_TELEPHONE: 0123 45678
CUST_ADDRESS_ZIPCODE: 80803
CUST_NAME: ACME Inc.

As you can see, there was some error with the INDUSTRY1_NAME and CUST_CONTACTPERS_NAME fields, where the data "jumped" from one field to another.
In my application, I found this as the now-empty field CUST_CONTACTPERS_NAME is mandatory and therefore the test import did not succeed.

• As you can see I'm putting 14 data fields in.
• Normally, I'd expect to get 17 back (3 extra for loopkup fields INDUSTRY1_NAME, CUST_ADDRESS_COUNTRY_NAME and CUST_CONTACTPERS_GENDER_SNNAME).
• As MD_LEADTEMPERATURE_ID and MD_CUSTOMER_NUMBEROFEMPLOYEES are not in the .ds.xml, so can be ignored (I included them to test also this behavior, you can see this also in the log).
• So it's down to 15 expected in the result, but it is 14 only in the Browser output, with CUST_CONTACTPERS_SURNAME being the missing field.

As INDUSTRY1_NAME (which gets the unexpected value "Doe") is an included field and has a name of "NAME" in it's .ds.xml, whereas the title of CUST_CONTACTPERS_NAME is "Name", I assume that this is somehow the issue.

Best regards
Blama

[Isomorphic]

To #23, since we checked lots of use cases, somehow we missed original issue you reported. This will be fixed shortly, but before that we'll reproduce/retest issues you described in #25. Thank you for your patience.

[Blama]

Hi Isomorphic,

right now I have problems recreating the issue from #23, which confuses me a bit. Can you really reproduce the problem here? Then something with my sample here is wrong right now, but I did start fresh IMHO.
Perhaps your fix in #21 did fix it as I reported in #22. I don't know what I did to see #23.

#25 is definitely open though and has perhaps a different reason than the original report, as it seems to be related to the 2nd DS.

The original testcase did not cover this, the one in #25 does.

Best regards
Blama

[Isomorphic]

Yes, we did reproduce #23 and it is fixed, although its availability in nightly builds will be confirmed in separate post later today.

Regarding #25. There are 2 fields in your setup (INDUSTRY1_NAME and CUST_CONTACTPERS_SURNAME), which are configured to use the same fmt message from the same bundle. So they end up having the same title "Name", which is not supported and currently causes unexpected behaviour. What do you expect to happen when you import data includes a field named "Name" which effectively points to 2 different fields?

[Blama]

Hi Isomorphic,

my use case is to upload data to a DataSource. That datasource also has (ID/Number)-based foreignKeys to other DataSources.
These DataSources (main, and linked ones) can have fields with the same name/title, as here for the String "Name" which can denote the Name of a person or the Name of an Industry.
As this happens quite a lot and I'm perfectly fine to use always the same translation here, I have only one properties file for this. Otherwise things like "name" need to be translated several times, which is redundant. If I'd ever want a different translation, I'd choose a different fmt-key here.

To the field names/titles:
Please note that the main DS (of course) does not have duplicate name attributes. This is a logical requirement you make in the docs. Of course this makes sense. This is not true for title attributes (you don't make the requirement and it would not make sense either, as explained above).

I do have this in my Java code:

columnRemap.put("Firma", "CUST_NAME");

This IMHO makes it clear where the data should go. Note that the line is not

columnRemap.put("Firma", [B]"Name"[/B]);

which really would be ambiguous given the DataSource.

I wrote in #6:

So in my opinion I'm using two mechanisms to set the import name, so that no field title is needed here.
1) Using columnRemap should be enough to use the ds.xml field's name-Attribute.
2) If ignored, having uploadFieldName at all and having it set to the CSV-column title should also eliminate the need for column-to-title guessing/matching.

You wrote in #7

Regardless of using those two mechanisms, the title is still used as a fallback, as the docs say.

I agree that it might be useful as a fallback. But if using columnRemap, IMHO the developer is fully responsible for providing entries for all fields in the data.
Definitely all the data fields given as key in columnRemap should not be subject to fallback mechanisms.

And actually this is also what you write in the docs for importDataSourceRecords().

An optional columnRemap can be provided to translate between column names in CSV/TSV or property names in JSON to the field names of the DataSource. Use null as a Map value to cause data for a column to be discarded.

If no columnRemap is provided, column names will be matched to DataSource fields by comparing to both the field name and field title, ignoring letter case. Any column name that isn't matched to a DataSource field is discarded.

To me the 2nd sentence translates to "don't use guessing/fallback if columnRemap is provided".

I believe that this is the root cause of the problem: Even if columnRemap is given, you normalize on title (translate columnRemap to titles), where you can't expect titles to be unique in a DataSource.
I don't know if the correct fix is

• don't use any fallback if using columnRemap or
• don't use fallback for data fields in columnRemap, use fallback (based on a combination of title, name and uploadFieldName) for others

but I think it is one of those.

Best regards
Blama

[Isomorphic]

This makes sense, we've made changes that you suggested. If a field is present in columnRemap, its title will be ignored, fields that are not re-mapped will continue using their titles.

Note, that accordingly to these new rules your setup also needs a change. You need to add INDUSTRY1_NAME field to columnRemap to exclude its title from the process, otherwise it will grab "Name" column from import data before the remapping takes place. Something like this:

columnRemap.put("excluded",[b] "INDUSTRY1_NAME"[/b]);
// or this:
// columnRemap.put("INDUSTRY1_NAME", [b]"INDUSTRY1_NAME"[/b]);

These changes will be available for download in nightly builds since Nov 16 (tomorrow). Let us know please how it worked for you.