BatchUploader key-lookup for values-to-null lookup does not return null

[Blama]

Hi Isomorphic,

I have an issue with BatchUploader with importStrategy="display" where the correct null-value is not set when the look leads to a value being null. Instead the original value is used.
This issue happens in my application using v10.1p_2016-11-08 as well.

Please see this BuiltInDS based test case:

BuiltInDS.html change:

DataSourceLoader?dataSource=employees,employeesUpload,batchUpload

BuiltInDS.java:

package com.smartgwt.sample.client;

import com.google.gwt.core.client.EntryPoint;
import com.smartgwt.client.core.KeyIdentifier;
import com.smartgwt.client.data.AdvancedCriteria;
import com.smartgwt.client.data.Criterion;
import com.smartgwt.client.data.DataSource;
import com.smartgwt.client.types.OperatorId;
import com.smartgwt.client.util.Page;
import com.smartgwt.client.util.PageKeyHandler;
import com.smartgwt.client.util.SC;
import com.smartgwt.client.widgets.BatchUploader;
import com.smartgwt.client.widgets.IButton;
import com.smartgwt.client.widgets.Window;
import com.smartgwt.client.widgets.events.ClickEvent;
import com.smartgwt.client.widgets.events.ClickHandler;
import com.smartgwt.client.widgets.form.fields.SelectItem;
import com.smartgwt.client.widgets.grid.ListGrid;
import com.smartgwt.client.widgets.grid.ListGridField;
import com.smartgwt.client.widgets.layout.VLayout;

public class BuiltInDS implements EntryPoint {
    private VLayout mainLayout;
    private IButton recreateBtn;

    public void onModuleLoad() {
        KeyIdentifier debugKey = new KeyIdentifier();
        debugKey.setCtrlKey(true);
        debugKey.setKeyName("D");

        Page.registerKey(debugKey, new PageKeyHandler() {
            public void execute(String keyName) {
                SC.showConsole();
            }
        });

        mainLayout = new VLayout(20);
        mainLayout.setWidth100();
        mainLayout.setHeight100();

        recreateBtn = new IButton("Recreate");
        recreateBtn.addClickHandler(new ClickHandler() {
            @Override
            public void onClick(ClickEvent event) {
                recreate();
            }
        });
        mainLayout.addMember(recreateBtn);
        recreate();
        mainLayout.draw();
    }

    private void recreate() {
        Window w = new Window();
        w.setWidth("95%");
        w.setHeight("95%");
        w.setMembersMargin(0);
        w.setModalMaskOpacity(70);
        w.setTitle("BatchUploader key-lookup for values -> null lookup cases");
        w.setShowMinimizeButton(false);
        w.setIsModal(true);
        w.setShowModalMask(true);
        w.centerInPage();

        BatchUploader batchUploader = new BatchUploader();
        {
            batchUploader.setWidth100();
            batchUploader.setUploadDataSource(DataSource.get("employeesUpload"));

            ListGridField nameToManagerIdLGF = new ListGridField("NameToManagerId");
            ListGridField jobLGF = new ListGridField("Job");
            ListGridField employeeManagerLGF = new ListGridField("EmployeeManager");
            
            SelectItem managerSelect = new SelectItem("EmployeeId");
            managerSelect.setOptionDataSource(DataSource.get("employees"));
            managerSelect.setAllowEmptyValue(true);
            nameToManagerIdLGF.setEditorProperties(managerSelect);

            batchUploader.setGridFields(nameToManagerIdLGF, jobLGF, employeeManagerLGF);
        }

        ListGrid editUsersLG = new ListGrid(DataSource.get("employees"));
        {
            editUsersLG.setAutoFetchData(false);
            editUsersLG.setShowDetailFields(true);
            editUsersLG.setCanEdit(true);
            editUsersLG.setHeight("25%");
            editUsersLG.setShowFilterEditor(true);
            editUsersLG.setAllowFilterExpressions(true);

            ListGridField employeeIdLGF = new ListGridField("EmployeeId");
            ListGridField nameLGF = new ListGridField("Name");
            ListGridField reportsToLGF = new ListGridField("ReportsTo");
            ListGridField jobLGF = new ListGridField("Job");

            editUsersLG.setFields(employeeIdLGF, nameLGF, reportsToLGF, jobLGF);
            editUsersLG.fetchData(new AdvancedCriteria(new Criterion("Name", OperatorId.STARTS_WITH, "Ch")));
        }

        w.addItem(batchUploader);
        w.addItem(editUsersLG);
        w.show();
    }
}

employeesUpload.ds.xml:

<DataSource xmlns="lmscompany/ds" ID="employeesUpload" serverType="sql" tableName="employeeTable" recordName="employee" useAnsiJoins="true">
    <fields>
        <field name="Job" title="Title" type="text" length="128">
            <validators>
                <validator type="serverCustom">
                    <serverObject lookupStyle="new" className="com.smartgwt.sample.server.listener.ValidatorTrue" />
                </validator>
            </validators>
        </field>

        <field name="NameToManagerId" importStrategy="display" uploadFieldName="Manager" displayField="EmployeeManager" title="Name to Manager" foreignKey="employees.ReportsTo">
            <validators>
                <validator type="hasRelatedRecord" />
                <validator type="serverCustom">
                    <serverObject lookupStyle="new" className="com.smartgwt.sample.server.listener.ValidatorTrue" />
                </validator>
            </validators>
        </field>
        <field name="EmployeeManager" includeFrom="employees.Name" includeVia="NameToManagerId" />        
    </fields>
</DataSource>

employees.ds.xml:

<DataSource
    ID="employees"
    serverType="sql"
    tableName="employeeTable"
    recordName="employee"
    testFileName="/examples/shared/ds/test_data/employees.data.xml"
    titleField="Name"
    useAnsiJoins="true"
>
    <fields>
        <field name="userOrder"       title="userOrder"       type="integer"  canEdit="false"    hidden="true"/>
        <field name="Name"            title="Name"            type="text"     length="128"/>
        <field name="EmployeeId"      title="Employee ID"     type="integer"  primaryKey="true"  required="true"/>
        <field name="ReportsTo"       title="Manager"         type="integer"
               foreignKey="employees.EmployeeId" joinType="outer" />
        <field name="Job"             title="Title"           type="text"     length="128"/>
        <field name="Email"           title="Email"           type="text"     length="128"/>
        <field name="EmployeeType"    title="Employee Type"   type="text"     length="40"/>
        <field name="EmployeeStatus"  title="Status"          type="text"     length="40"/>
        <field name="Salary"          title="Salary"          type="float"/>
        <field name="OrgUnit"         title="Org Unit"        type="text"     length="128"/>
        <field name="Gender"          title="Gender"          type="text"     length="7">
            <valueMap>
                <value>male</value>
                <value>female</value>
            </valueMap>
        </field>
        <field name="MaritalStatus"   title="Marital Status"  type="text"     length="10">
            <valueMap>
                <value>married</value>
                <value>single</value>
            </valueMap>
        </field>
    </fields>
</DataSource>

ValidatorTrue.java:

package com.lmscompany.lms.server.worker.validator;

import java.util.Map;

import javax.servlet.http.HttpServletRequest;

import com.isomorphic.datasource.DataSource;
import com.isomorphic.datasource.Validator;

public class ValidatorTrue {

    public boolean condition(Object value, Validator validator, String fieldName, @SuppressWarnings("rawtypes") Map record, DataSource ds,
            HttpServletRequest httpServletRequest) throws Exception {
        return true;
    }
}

TestdataImportEmployees.csv.txt:

Manager,Job
Charles Madigen,foo
Tamara Kane,bar

Steps to reproduce:

1. Start sample
2. Edit "Charles Madigan" in lower ListGrid to have no Manager (set reportsTo to null)
3. Upload the file, see this:

This is not expected, but logical, as the return of the Upload request is:

{
    affectedRows:0,
    data:{
        discardedColumns:[
        ],
        gridRows:[
            {
                Job:"foo",
[B]               NameToManagerId:"Charles Madigen"[/B]
            },
            {
                EmployeeManager:"Tamara Kane",
                Job:"bar",
                NameToManagerId:"4"
            }
        ],
        errors:[
            {
                errors:{
                    NameToManagerId:[
                        "Related record does not exist"
                    ]
                },
                rowNum:0
            }
        ]
    },
    invalidateCache:false,
    isDSResponse:true,
    operationType:"add",
    queueStatus:0,
    status:0
}

In the server log you can see this expected line:

=== 2016-11-16 19:40:37,452 [9-36] INFO  SQLDriver - [builtinApplication.null] Executing SQL query on 'HSQLDB' using connection '2099790644': SELECT employeeTable.userOrder, employeeTable.Name, employeeTable.EmployeeId, employeeTable.ReportsTo, employeeTable.Job, employeeTable.Email, employeeTable.EmployeeType, employeeTable.EmployeeStatus, employeeTable.Salary, employeeTable.OrgUnit, employeeTable.Gender, employeeTable.MaritalStatus FROM employeeTable WHERE (LOWER(employeeTable.Name)='charles madigen')
[B]=== 2016-11-16 19:40:37,454 [9-36] INFO  DSResponse - DSResponse: List with 1 items[/B]

In this case, where the lookup leads to a null value (as opposed to: No row was found for the given value), I 'd expect this as result:

        gridRows:[
            {
[B]               EmployeeManager:"Charles Madigen", [/B]
                Job:"foo",
[B]               NameToManagerId: null       (or this row missing)[/B]
            },
            {
                EmployeeManager:"Tamara Kane",
                Job:"bar",
                NameToManagerId:"4"
            }
        ],

The hasRelatedRecord-validator would not be executed for the "Charles Madigan" row, just like the validation passes, when you change the value to null via the ListGrid editing:



This is an important one for me.

Best regards
Blama

[Isomorphic]

This is fixed and will be available for download in nightly builds since Nov 23 (tomorrow). Let us know please how this worked for you.

[Blama]

Hi Isomorphic,

this is fixed for me using v10.1p_2016-11-25 in both the sample and my application.

Thank you
Blama