serverCustom validator not called for update to NULL / hasRelatedRecord bug

[Blama]

Hi Isomorphic,

please see this test case based on 5.1p (v10.1p_2016-06-09) which shows a serverCustom validator problem related to the one you fixed here and a problem, where a hasRelatedRecord-validator tries to search for a null-record and fails.

BuiltInDS.java:

package com.smartgwt.sample.client;

import com.google.gwt.core.client.EntryPoint;
import com.smartgwt.client.Version;
import com.smartgwt.client.core.KeyIdentifier;
import com.smartgwt.client.data.DataSource;
import com.smartgwt.client.util.Page;
import com.smartgwt.client.util.PageKeyHandler;
import com.smartgwt.client.util.SC;
import com.smartgwt.client.widgets.BatchUploader;
import com.smartgwt.client.widgets.IButton;
import com.smartgwt.client.widgets.Window;
import com.smartgwt.client.widgets.events.ClickEvent;
import com.smartgwt.client.widgets.events.ClickHandler;
import com.smartgwt.client.widgets.grid.ListGridField;
import com.smartgwt.client.widgets.layout.VLayout;

public class BuiltInDS implements EntryPoint {
    private VLayout mainLayout;
    private IButton recreateBtn;

    public void onModuleLoad() {
        KeyIdentifier debugKey = new KeyIdentifier();
        debugKey.setCtrlKey(true);
        debugKey.setKeyName("D");

        Page.registerKey(debugKey, new PageKeyHandler() {
            public void execute(String keyName) {
                SC.showConsole();
            }
        });

        mainLayout = new VLayout(20);
        mainLayout.setWidth100();
        mainLayout.setHeight100();

        recreateBtn = new IButton("Recreate");
        recreateBtn.addClickHandler(new ClickHandler() {
            @Override
            public void onClick(ClickEvent event) {
                recreate();
            }
        });
        mainLayout.addMember(recreateBtn);
        recreate();
        mainLayout.draw();
    }

    private void recreate() {
        Window w = new Window();
        w.setWidth("95%");
        w.setHeight("95%");
        w.setMembersMargin(0);
        w.setModalMaskOpacity(70);
        w.setTitle(" (" + Version.getVersion() + "/" + Version.getSCVersionNumber() + ")");
        w.setTitle("TITLE" + w.getTitle());
        w.setShowMinimizeButton(false);
        w.setIsModal(true);
        w.setShowModalMask(true);
        w.centerInPage();

        final BatchUploader bu = new BatchUploader();
        bu.setWidth(500);
        bu.setUploadDataSource(DataSource.get("children"));
        ListGridField name = new ListGridField("Name");
        ListGridField parent = new ListGridField("Parent");
        ListGridField favItem = new ListGridField("FavItem");
        bu.setGridFields(name, parent, favItem);
        w.addItem(bu);
        w.show();
    }
}

BuiltInDS.html addition:

<script src="builtinds/sc/DataSourceLoader?dataSource=batchUpload,supplyItem,animals,employees,children"></script>

children.ds.xml:

<DataSource xmlns="lmscompany/ds" ID="children" serverType="sql" tableName="foobar">
    <fields>
        <field name="Name" type="text" length="20" required="true" />
        <field name="Parent" editorType="ComboBoxItem" importStrategy="display" displayField="Employees_Name" uploadFieldName="Parent" foreignKey="employees.EmployeeId">
            <validators>
                <validator type="serverCustom">
                    <serverObject lookupStyle="new" className="com.smartgwt.sample.server.listener.EitherOr" />
                    <errorMessage>$errorMessage</errorMessage>
                </validator>
                <validator type="hasRelatedRecord" />
            </validators>
        </field>
        <field name="Employees_Name" includeFrom="employees.Name" />

        <field name="FavItem" editorType="ComboBoxItem" importStrategy="display" displayField="supplyItem_itemName" uploadFieldName="FavItem" foreignKey="supplyItem.itemID">
            <validators>
                <validator type="serverCustom">
                    <serverObject lookupStyle="new" className="com.smartgwt.sample.server.listener.EitherOr" />
                    <errorMessage>$errorMessage</errorMessage>
                </validator>
                <validator type="hasRelatedRecord" />
            </validators>
        </field>
        <field name="supplyItem_itemName" includeFrom="supplyItem.itemName" />
    </fields>
</DataSource>

Testfile.csv:

"Name","FavItem","Parent"
"Joe","Glue UHU Super Glue 3ml","Charles Madigen"
"Anna","",""
"Sophia","","Ralph Brogan"
"Carol","File Half Arch Board Foolscap Black",""

If you upload the file, it looks like this:
As you can see, the eitherOr Validator was not fired for row 2. This is bug 1.



Now remove all Parent/Fav Items so that it looks like this:
Again the EitherOr-validator is not fired.



After commit, you'll see this log in the developer console:
As you can see, for the rows 1,3,4 you edited, there is a validation error from the EitherOr-validator (OK) and one from the hasRelatedRecord-validator (not OK, this is bug 2). The failure for row 2 is expected.
The SQL fired for the hasRelatedRecord-validator error messages are:

SELECT supplyItem.itemID, supplyItem.itemName, supplyItem.SKU, supplyItem.description, supplyItem.category, supplyItem.units, supplyItem.unitCost, supplyItem.inStock, supplyItem.nextShipment FROM supplyItem WHERE ([B]supplyItem.itemID IS NULL[/B])
SELECT employeeTable.userOrder, employeeTable.Name, employeeTable.EmployeeId, employeeTable.ReportsTo, employeeTable.Job, employeeTable.Email, employeeTable.EmployeeType, employeeTable.EmployeeStatus, employeeTable.Salary, employeeTable.OrgUnit, employeeTable.Gender, employeeTable.MaritalStatus FROM employeeTable WHERE ([B]employeeTable.EmployeeId IS NULL[/B])

This is wrong, hasRelatedRecord should not fire for a null-entry.

Both these fire-or-not validator problems are related to the one you fixed here.

This is an important one for me.

Best regards
Blama

[Blama]

Hi Isomorphic,

can you reproduce this one?

Best regards
Blama

[Isomorphic]

Apologies for the delay. Yes, this is reproduced and the fix is on the way.

[Blama]

Hi Isomorphic,

any update on this one?

Thank you & Best regards
Blama

[Isomorphic]

Apologies for the delay and thank you for your patience.

We made changes addressing your issues:
1. hasRelatedRecord validator will no longer be executed for NULL values.
2. new API was added DataSource.validate(Map data, ErrorReport errors) to allow adding custom validation errors on top of builtin validation errors.

Builtin per-field validation cannot be used in your use case, cause EitherOr custom validator is based on multiple fields. You should subclass from the DataSource (SQLDataSource or which you are using) and override new API to add this kind of validation. See javadocs for details.

These changes are available in nightly builds since July 4 (today). Let us know please how this worked for you.

[Blama]

Hi Isomorphic,

I adjusted my testcase to your new API. Unfortunately it does not work as expected, yet. I'm using v10.1p_2016-07-08 and did these changes:

SQLDataSourceChildren.java + referencing it in children.ds.xml's serverConstructor:

package com.smartgwt.sample.server.listener;

import java.util.Map;

import com.isomorphic.log.Logger;
import com.isomorphic.sql.SQLDataSource;
import com.isomorphic.util.ErrorReport;

public class SQLDataSourceChildren extends SQLDataSource {
    private static final long serialVersionUID = 3994104605640773331L;

    @Override
    public ErrorReport validate(Map data, ErrorReport errors) throws Exception {
        Logger log = new Logger("com.isomorphic." + EitherOr.class.getName());
        log.info("Called SQLDataSource validation");

        Object favItem = data.get("FavItem");
        Object parent = data.get("Parent");
        log.info("parent:" + (parent != null ? parent.toString() : "NULL") + ", " + "favItem:"
                + (favItem != null ? favItem.toString() : "NULL"));
        if ((favItem == null && parent == null) || (favItem != null && parent != null)) {
            ErrorReport er = new ErrorReport();
            er.addError("FavItem", "Only one of FavItem, Parent may be used");
            er.addError("Parent", "Only one of FavItem, Parent may be used");
            return er;
        }
        return null;
    }
}

EitherOr.java:

package com.smartgwt.sample.server.listener;

import java.util.Map;

import com.isomorphic.datasource.DataSource;
import com.isomorphic.datasource.Validator;
import com.isomorphic.log.Logger;

public class EitherOr {

    public boolean condition(Object value, Validator validator, String fieldName, @SuppressWarnings("rawtypes") Map record, DataSource ds)
            throws Exception {
        // Logger log = new Logger("com.isomorphic." + EitherOr.class.getName());
        //
        // Object favItem = record.get("FavItem");
        // Object parent = record.get("Parent");
        // log.info("Called for field \"" + fieldName + "\", value \"" + (value == null ? "NULL" : value.toString()) + "\"");
        // if ((favItem == null && parent == null) || (favItem != null && parent != null)) {
        // validator.addErrorMessageVariable("errorMessage", "Exactly one field of Parent and FavItem must be filled");
        // return false;
        // }
        if (ds.validate(record, null) != null)
            return false;
        return true;
    }
}

Problems:

• EitherOr validator is still needed in .ds.xml, otherwise the new code is not called.
• On upload, row1 is correctly verified, on change of row3 or 4 to both-fields-with-values, the error is "$errorMessage", not some text
• On upload, row2 does not show errors, but it should. Also the validation is not triggered here.
• For the testfile, if you put a breakpoint in the overridded class, it is hit 5 times, three times for row1 (opposed to the expected two times). Why is that?

Best regards
Blama

[Isomorphic]

1. FIXED in nightly since Jul 16. This was a bug and is fixed, EitherOr validator should not be in ds.xml and new code will be always called during batch upload.
2. Couldn't reproduce.
3. The fact that validation is not always triggered is considered to be an issue, but it will probably take a while.
4. Couldn't reproduce. Could be side effect of having EitherOr validator in ds.xml.

[Blama]

Hi Isomorphic,

I retested using the same testcase (without the call to EitherOr in .ds.xml) and v10.1p_2016-07-18.

My observations:

1. On start the errors show as expected (OK)
2. On start, a breakpoint in SQLDataSourceChildren is hit 8 times. Shouldn't it be 4, once per row?
3. On change of any row, there is a validate request from the client, but a breakpoint in SQLDataSourceChildren is not hit (=class and its overridden ErrorReport validate(Map data, ErrorReport errors) not used?) and therefore no error is returned.

Best regards
Blama


PS: Updated children.ds.xml for completeness:

<DataSource serverConstructor="com.smartgwt.sample.server.listener.SQLDataSourceChildren" ID="children" serverType="sql" tableName="foobar">
    <fields>
        <field name="Name" type="text" length="20" required="true" />
        <field name="Parent" editorType="ComboBoxItem" importStrategy="display" displayField="Employees_Name" uploadFieldName="Parent" foreignKey="employees.EmployeeId">
            <validators>
                <validator type="hasRelatedRecord" />
            </validators>
        </field>
        <field name="Employees_Name" includeFrom="employees.Name" />

        <field name="FavItem" editorType="ComboBoxItem" importStrategy="display" displayField="supplyItem_itemName" uploadFieldName="FavItem" foreignKey="supplyItem.itemID">
            <validators>
                <validator type="hasRelatedRecord" />
            </validators>
        </field>
        <field name="supplyItem_itemName" includeFrom="supplyItem.itemName" />
    </fields>
</DataSource>

[Isomorphic]

As we mentioned, issue 3 will take a while to look at. We'll check whether your breakpoint is being hit too often at the same time.

[Blama]

Hi Isomorphic,

do you have an ETA for this?

Thank you & Best regards
Blama

[Isomorphic]

Not a very specific one. It will probably take around 2 weeks to look at and find a way to resolve.

[Blama]

OK, thank you. That's enough for me as a ballpark figure.

Best regards
Blama

[Blama]

Hi Isomorphic,

any news on this one?

Thank you & Best regards
Blama

[Isomorphic]

Hi Blama,

sorry for the delay, we made some changes to address these issues. You may download Aug 11 (or later) nightly and give it a shot. validate(Map, ErrorReport) should be always called now. Let us know please how it worked for you.

We are still working on this though and there is a possibility of more changes in this area in the near future. We'll update this thread if this happen to be the case.

Thank you

[Blama]

Hi Isomorphic,

thank you for the fix.
The testcase (using v10.1p_2016-08-11) is working now for me, this means:

• ErrorReport validate called 4 times on startup
• ErrorReport validate called for every field change
• Results are as expected

Very minor (and most likely no issue for me):
If you double click the 1st ComboBoxItem to change it, do so and hit Tab to change the 2nd one, validation is immediately called even tough row-editing is not finished.
There is no validateOnChange or similar set in the client java code.

Best regards
Blama