BatchUpload validation does not use client (or browser) language when validating

[Blama]

Hi Isomorphic,

it seems that BatchUploader validation does not use the GWT-Client (I'd prefer this) or browser language.

Please see the online BatchUploader sample using ?locale=de (now on v11.0p_2016-11-04). My browser is also configured to prefer German language (HTTP Header in all requests: "Accept-Language:de-DE,de;q=0.8,en;q=0.6,en-US;q=0.4").
If you right click the ListGrid headers you'll see that the client is indeed using German locale.

Steps to reproduce:

1. Open the sample with the link
2. Download the supplyItem.csv file
3. Remove the "Audio" Category value in the 1st data row to create an error
4. Upload the file
5. See that the server-sent error message is "Field is required" (German version expected here)
6. Update the 2nd data row in the GUI and remove the "Audio"-value there as well
7. See that the client-generated validation error is "Feld ist obligatorisch" (as expected)

Video:




Perhaps the solution is to sent the GWT language to the server in the operationId=setAttributes request?

This is a minor one for me, but happens in my application (v10.1p_2016-11-03) as well.

Best regards
Blama

[Isomorphic]

Server-side default validators are actually not locale-responsive by default, since normally client-side validation stops any of these validators from being triggered - they are just there to stop forged requests.

As with overriding the default messages when you don't like them, you can declare your own errorMessage, which can be localized via normal .ds.xml localization approaches.

For the "required" validation in particular this means explicitly declaring a validator of type "required" rather than using the required=true shorthand.

[Blama]

Hi Isomorphic,

thank for the hint on this one. It does seem though that there is a problem with the BatchUpload itself when not using the shorthand, as <validator type="required" /> is not called for null-values in the upload, which is it's very function.
See this testcase (using v10.1p_2016-11-08):

BuiltInDS.html change:

DataSourceLoader?dataSource=employees,employeesUpload,batchUpload

BuiltInDS.java:

package com.smartgwt.sample.client;

import com.google.gwt.core.client.EntryPoint;
import com.smartgwt.client.core.KeyIdentifier;
import com.smartgwt.client.data.DataSource;
import com.smartgwt.client.util.Page;
import com.smartgwt.client.util.PageKeyHandler;
import com.smartgwt.client.util.SC;
import com.smartgwt.client.widgets.BatchUploader;
import com.smartgwt.client.widgets.IButton;
import com.smartgwt.client.widgets.Window;
import com.smartgwt.client.widgets.events.ClickEvent;
import com.smartgwt.client.widgets.events.ClickHandler;
import com.smartgwt.client.widgets.form.fields.SelectItem;
import com.smartgwt.client.widgets.grid.ListGridField;
import com.smartgwt.client.widgets.layout.VLayout;

public class BuiltInDS implements EntryPoint {
    private VLayout mainLayout;
    private IButton recreateBtn;

    public void onModuleLoad() {
        KeyIdentifier debugKey = new KeyIdentifier();
        debugKey.setCtrlKey(true);
        debugKey.setKeyName("D");

        Page.registerKey(debugKey, new PageKeyHandler() {
            public void execute(String keyName) {
                SC.showConsole();
            }
        });

        mainLayout = new VLayout(20);
        mainLayout.setWidth100();
        mainLayout.setHeight100();

        recreateBtn = new IButton("Recreate");
        recreateBtn.addClickHandler(new ClickHandler() {
            @Override
            public void onClick(ClickEvent event) {
                recreate();
            }
        });
        mainLayout.addMember(recreateBtn);
        recreate();
        mainLayout.draw();
    }

    private void recreate() {
        Window w = new Window();
        w.setWidth("95%");
        w.setHeight("95%");
        w.setMembersMargin(0);
        w.setModalMaskOpacity(70);
        w.setTitle("type=\"required\" validator does not fire on upload");
        w.setShowMinimizeButton(false);
        w.setIsModal(true);
        w.setShowModalMask(true);
        w.centerInPage();

        BatchUploader batchUploader = new BatchUploader();
        batchUploader.setWidth100();
        batchUploader.setUploadDataSource(DataSource.get("employeesUpload"));

        ListGridField nameLGF = new ListGridField("Name");

        ListGridField managerLGF = new ListGridField("ReportsTo");
        SelectItem si = new SelectItem("ReportsTo");
        si.setOptionDataSource(DataSource.get("employees"));
        si.setValueField("EmployeeId");
        si.setDisplayField("Name");
        si.setSortField("Name");
        managerLGF.setEditorProperties(si);

        ListGridField genderLGF = new ListGridField("Gender");

        batchUploader.setGridFields(nameLGF, genderLGF, managerLGF);

        w.addItem(batchUploader);
        w.show();
    }
}

employeesUpload.ds.xml:

<DataSource ID="employeesUpload" serverType="sql" tableName="employeeTable" recordName="employee" useAnsiJoins="true">
    <fields>
        <field name="EmployeeId" primaryKey="true" hidden="true" type="sequence" />

        <field name="Name" uploadFieldName="Name" title="Name" type="text" length="128">
            <validators>
                <validator type="isUnique" caseSensitive="true" />
            </validators>
        </field>

        <field name="ReportsTo" uploadFieldName="ReportsTo" displayField="ReportsToName" title="Manager" required="true" importStrategy="display"
            foreignKey="employees.EmployeeId" relatedTableAlias="relatedReportsTo">
            <validators>
                <validator type="hasRelatedRecord" errorMessage="Unknown ReportsTo" />
            </validators>
        </field>
        <field name="ReportsToName" includeFrom="employees.Name" includeVia="ReportsTo" />

        <field name="Gender" uploadFieldName="Gender" title="Gender" type="text" length="7">
            <validators>
                <validator type="required" />
            </validators>
            <valueMap>
                <value>male</value>
                <value>female</value>
            </valueMap>
        </field>
    </fields>
</DataSource>

TestdataImportEmployees.csv.txt:

Name,ReportsTo,Gender
Anna Sun,Charles Madigen,
Mike Sun,Tamara Kane,male

Video:



Also after the change, when the validation error is displayed you'll see that the error does not come from the server (the validation request passes without errors), but from the client.
So it seems that the server validation for <validator type="required" /> is broken somehow.

Best regards
Blama

[Blama]

Hi Isomorphic,

could you reproduce the issue here?

Thank you & Best regards
Blama

[Blama]

Hi Isomorphic,

any update on this one?

Thank you & Best regards
Blama

[Isomorphic]

This is fixed and will be available for download in nightly builds since Nov 24 (today).

Note that declaring "required" validator is no longer necessary, you may set the DataSource.requiredMessage or/and DSField.requiredMessage to customize error message for required field.

[Blama]

Hi Isomorphic,

thanks for the fix. I do think though that it introduced some serious other error (in v10.1p_2016-11-24). I'm using i18n using fmt-tags for Validators, DataSourceFields, etc.
Now on page load, DataSourceLoader fails with this exception:

=== 2016-11-24 10:46:15,566 [ec-7] ERROR DataSourceLoader - Exception while attempting to load a DataSource
java.lang.ClassCastException: com.isomorphic.util.LocaleMessage cannot be cast to java.lang.String
    at com.isomorphic.datasource.BasicDataSource.buildFieldData(BasicDataSource.java:1063)
    at com.isomorphic.datasource.BasicDataSource.init(BasicDataSource.java:254)
    at com.isomorphic.sql.SQLDataSource.init(SQLDataSource.java:272)
    at com.isomorphic.datasource.DataSource.initialize(DataSource.java:936)
    at com.isomorphic.datasource.BasicDataSource.fromConfig(BasicDataSource.java:183)
    at com.isomorphic.datasource.DataSource.fromConfig(DataSource.java:921)
    at com.isomorphic.datasource.DataSource.loadDS(DataSource.java:540)
    at com.isomorphic.datasource.DataSource.forName(DataSource.java:468)
    at com.isomorphic.datasource.PoolableDataSourceFactory.makeUnpooledObject(PoolableDataSourceFactory.java:134)
    at com.isomorphic.datasource.PoolableDataSourceFactory.makeObject(PoolableDataSourceFactory.java:150)
    at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
    at com.isomorphic.pool.ISCKeyedObjectPool.borrowObject(ISCKeyedObjectPool.java:106)
    at com.isomorphic.pool.PoolManager.borrowObject(PoolManager.java:92)
    at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:135)
    at com.isomorphic.datasource.DataSourceManager.getDataSource(DataSourceManager.java:86)
    at com.isomorphic.servlet.DataSourceLoader.processRequest(DataSourceLoader.java:186)
    at com.isomorphic.servlet.DataSourceLoader.doGet(DataSourceLoader.java:107)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:622)
    at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:162)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.lmscompany.lms.server.MimeTypeFilter.doFilter(MimeTypeFilter.java:23)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.isomorphic.servlet.CompressionFilter._doFilter(CompressionFilter.java:260)
    at com.isomorphic.servlet.BaseFilter.doFilter(BaseFilter.java:88)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:614)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:521)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Unknown Source)

Looking at the line it seems that this is about getting the error message for a validator - the area this fix affects.

A DS definition here looks like this:

<DataSource xmlns="lmscompany/ds" xmlns:fmt="lmscompany/fmt" dbName="Oracle" tableName="T_CATEGORY1" ID="T_CATEGORY1" serverType="sql"
    serverConstructor="com.lmscompany.lms.server.LMSSQLDataSource">
    <fmt:bundle basename="com.lmscompany.lms.server.i18n.DSXMLResources-utf8" encoding="utf-8" />
    <fields>
        <field primaryKey="true" hidden="true" name="ID" type="sequence" />
...
...
        <field name="NAME" length="60" type="text" escapeHTML="true" required="true">
            <title><fmt:message key="name" /></title>
            <validators>
                <validator type="isUnique">
                    <errorMessage><fmt:message key="validatorNameInUse" /></errorMessage>
                </validator>
            </validators>
        </field>
...
...

Best regards
Blama

[Isomorphic]

This is fixed as well and should be available for download since Nov 25 (tomorrow). Please tell us how it worked for you to close this out, thanks!

[Blama]

Hi Isomorphic,

using v10.1p_2016-11-25 the DataSourceLoader Error is fixed, but the same issue persists in other cases:

Both of these have a cast-to-String issue:

        <field name="MD_SOURCEUID" uploadFieldName="QuellID" length="50" type="text" escapeHTML="true" required="true">
            <title><fmt:message key="leadIDSource" /></title>
            <requiredMessage><fmt:message key="validatorSourceUIDRequired" /></requiredMessage>
            <validators>
                <validator type="serverCustom">
                    <serverObject lookupStyle="new" className="com.lmscompany.lms.server.worker.validator.ValidatorTrue" />
                </validator>
            </validators>
        </field>

        <field name="MD_SOURCEUID" uploadFieldName="QuellID" length="50" type="text" escapeHTML="true">
            <title><fmt:message key="leadIDSource" /></title>
            <validators>
                <validator type="required">
                    <errorMessage><fmt:message key="validatorSourceUIDRequired" /></errorMessage>
                </validator>
                <validator type="serverCustom">
                    <serverObject lookupStyle="new" className="com.lmscompany.lms.server.worker.validator.ValidatorTrue" />
                </validator>
            </validators>
        </field>

W.r.t. your post #6, which of the two options is the one I should use? Are you deprecating validator type="required" with #6?

The error stack on BatchUpload with an empty value for a field as described above is:

=== 2016-11-25 15:59:16,268 [ec-2] WARN  RequestContext - dsRequest.execute() failed: 
java.lang.ClassCastException: com.isomorphic.util.LocaleMessage cannot be cast to java.lang.String
    at com.isomorphic.datasource.BasicDataSource.checkRequired(BasicDataSource.java:2286)
    at com.isomorphic.datasource.BasicDataSource.checkStructure(BasicDataSource.java:2227)
    at com.isomorphic.datasource.BasicDataSource.toRecord(BasicDataSource.java:1455)
    at com.isomorphic.datasource.BasicDataSource.toRecords(BasicDataSource.java:1365)
    at com.isomorphic.datasource.DataSource.validate(DataSource.java:3550)
    at com.isomorphic.tools.DataImport.validateDSRows(DataImport.java:477)
    at com.isomorphic.tools.BatchUpload.validateUploadData(BatchUpload.java:301)
    at com.lmscompany.lms.server.worker.BatchUploadDMI.batchUpload(BatchUploadDMI.java:47)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at com.isomorphic.base.Reflection.adaptArgsAndInvoke(Reflection.java:964)
    at com.isomorphic.datasource.DataSourceDMI.execute(DataSourceDMI.java:414)
    at com.isomorphic.datasource.DataSourceDMI.execute(DataSourceDMI.java:64)
    at com.isomorphic.datasource.DSRequest.execute(DSRequest.java:2699)
    at com.isomorphic.servlet.IDACall.handleDSRequest(IDACall.java:230)
    at com.lmscompany.lms.server.LMSIDACall.handleDSRequest(LMSIDACall.java:182)
    at com.isomorphic.servlet.IDACall.processRPCTransaction(IDACall.java:187)
    at com.lmscompany.lms.server.LMSIDACall.processRequest(LMSIDACall.java:92)
    at com.isomorphic.servlet.IDACall._processRequest(IDACall.java:119)
    at com.isomorphic.servlet.IDACall.doPost(IDACall.java:79)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
    at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:162)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at com.isomorphic.servlet.CompressionFilter._doFilter(CompressionFilter.java:260)
    at com.isomorphic.servlet.BaseFilter.doFilter(BaseFilter.java:88)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:212)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:106)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:614)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:141)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
    at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:616)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:521)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1096)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:674)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
    at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Unknown Source)

Best regards
Blama

[Blama]

Hi Isomorphic,

the original issue from #3 is fixed in the sample and in my application using v10.1p_2016-11-25 if I use a normal non-localized error message string.
Only the cast issue from #9 is still open.

Best regards
Blama

[Isomorphic]

This is fixed as well in nightlies since Nov 26.

No, validator type="required" is not deprecated, required/requiredMessage is just another way and there's no difference which to use.