DynamicForm: Bug with unneeded errorenous fetchMissingValueReply for HiddenItem IE(?)

[Blama]

Hi Isomorphic,

I have this strange error in my application using v10.0p_2015-05-21 DynamicForm which seems only to happen in IE10 (tested with Win7 VM from modern.ie). It does not happen with IE11/FF26/GC43, at least I could not reproduce and I only get end-user reports for this issue from IE10 users.

I can reproduce in my application about 20% of the time, so it seems to be somehow timing related. Here are the erroneous requests from the Developer Console (the three last requests from the screenshot "My_application.png"). I'd not expect any of these because the FormItem used is a HiddenItem, but if they are there, the requests for ID:"" and ID:"Loading..." are definitely wrong.
Most of the time there is only the one request for the ID-value, sometimes two requests for "" and ID-value, sometimes three for "" and ID-value and "Loading...".

Main query (good):

{
    dataSource:"V_LEAD_CURRENT", 
    operationType:"fetch", 
    operationId:"fetchAddress", 
    componentId:"isc_AddEditLeadAddress_2_0", 
    data:{
        operator:"equals", 
        fieldName:"LEAD_ID", 
        value:844
    }, 
    textMatchStyle:"exact", 
    callback:{
        target:[DynamicForm ID:isc_AddEditLeadAddress_2_0], 
        methodName:"fetchDataReply"
    }, 
    showPrompt:true, 
    oldValues:{
        operator:"equals", 
        fieldName:"LEAD_ID", 
        value:844
    }, 
    requestId:"V_LEAD_CURRENT$62766", 
    useStrictJSON:true, 
    fallbackToEval:true, 
    lastClientEventThreadCode:"MUP0", 
    bypassCache:true
}

The three requests, where a maximum of one (then #2, but best zero of these IMHO) is needed:
Request 1:

{
    dataSource:"T_LEAD", 
    operationType:"fetch", 
    componentId:"isc_AddEditLeadAddress_2_0", 
    data:{
        ID:""
    }, 
    textMatchStyle:"exact", 
    callback:{
        target:[HiddenItem ID:isc_HiddenItem_8 name:LEAD_ID], 
        methodName:"fetchMissingValueReply"
    }, 
    showPrompt:false, 
    oldValues:{
        ID:""
    }, 
    requestId:"T_LEAD$62768", 
    internalClientContext:{
        newValue:"", 
        filterLocally:{
        }, 
        targetField:"ID", 
        fetchingMissingValues:{
            "Loading...":true
        }
    }, 
    useStrictJSON:true, 
    fallbackToEval:true, 
    componentContext:"LEAD_ID", 
    lastClientEventThreadCode:"TMR0", 
    bypassCache:true
}

Request 2:

{
    dataSource:"T_LEAD", 
    operationType:"fetch", 
    componentId:"isc_AddEditLeadAddress_2_0", 
    data:{
        ID:844
    }, 
    textMatchStyle:"exact", 
    callback:{
        target:[HiddenItem ID:isc_HiddenItem_8 name:LEAD_ID], 
        methodName:"fetchMissingValueReply"
    }, 
    showPrompt:false, 
    oldValues:{
        ID:844
    }, 
    requestId:"T_LEAD$62769", 
    internalClientContext:{
        newValue:844, 
        filterLocally:{
        }, 
        targetField:"ID", 
        fetchingMissingValues:{
            "":true
        }
    }, 
    useStrictJSON:true, 
    fallbackToEval:true, 
    componentContext:"LEAD_ID", 
    lastClientEventThreadCode:"TMR1", 
    bypassCache:true
}

Request 3:

{
    dataSource:"T_LEAD", 
    operationType:"fetch", 
    componentId:"isc_AddEditLeadAddress_2_0", 
    data:{
        ID:"Loading..."
    }, 
    textMatchStyle:"exact", 
    callback:{
        target:[HiddenItem ID:isc_HiddenItem_8 name:LEAD_ID], 
        methodName:"fetchMissingValueReply"
    }, 
    showPrompt:false, 
    oldValues:{
        ID:"Loading..."
    }, 
    requestId:"T_LEAD$62770", 
    internalClientContext:{
        newValue:"Loading...", 
        filterLocally:{
        }, 
        targetField:"ID", 
        fetchingMissingValues:{
        }
    }, 
    useStrictJSON:true, 
    fallbackToEval:true, 
    componentContext:"LEAD_ID", 
    lastClientEventThreadCode:"TMR4", 
    bypassCache:true
}

This is my real application field definition:
V_LEAD_CURRENT.ds.xml:

<field primaryKey="true" foreignKey="T_LEAD.ID" name="LEAD_ID" title="LEAD_ID" type="integer" hidden="true" required="true" />

T_LEAD.ds.xml:

<field primaryKey="true" hidden="true" name="ID" type="sequence" foreignKey="V_LEAD_CURRENT.LEAD_ID" joinType="outer" />

The V_LEAD_CURRENT.ds.xml does not have any includeFroms to the T_LEAD-datasource, but I don't think this is important.

This testcase resembles my application setup in the best possible way, but shows the error in a different place (but both in IE10/IE11, compared to my application where it only happens in IE10).
In the testcase the unneeded request for "Loading..." is issued when data for the main query is returned (1st button in the sample), while in my application this is the other way around.

BuiltInDS.java:

package com.smartgwt.sample.client;

import com.google.gwt.core.client.EntryPoint;
import com.smartgwt.client.core.KeyIdentifier;
import com.smartgwt.client.data.AdvancedCriteria;
import com.smartgwt.client.data.Criterion;
import com.smartgwt.client.data.DSCallback;
import com.smartgwt.client.data.DSRequest;
import com.smartgwt.client.data.DSResponse;
import com.smartgwt.client.data.DataSource;
import com.smartgwt.client.types.OperatorId;
import com.smartgwt.client.util.PageKeyHandler;
import com.smartgwt.client.util.Page;
import com.smartgwt.client.util.SC;
import com.smartgwt.client.widgets.IButton;
import com.smartgwt.client.widgets.events.ClickEvent;
import com.smartgwt.client.widgets.events.ClickHandler;
import com.smartgwt.client.widgets.form.DynamicForm;
import com.smartgwt.client.widgets.form.fields.ComboBoxItem;
import com.smartgwt.client.widgets.form.fields.FormItem;
import com.smartgwt.client.widgets.form.fields.HiddenItem;
import com.smartgwt.client.widgets.form.fields.SelectItem;
import com.smartgwt.client.widgets.layout.HLayout;
import com.smartgwt.client.widgets.layout.VLayout;

public class BuiltInDS implements EntryPoint {
	private VLayout vL;
	private TestForm tF;
	private HLayout hL;

	public void onModuleLoad() {
		KeyIdentifier debugKey = new KeyIdentifier();
		debugKey.setCtrlKey(true);
		debugKey.setKeyName("D");

		Page.registerKey(debugKey, new PageKeyHandler() {
			public void execute(String keyName) {
				SC.showConsole();
			}
		});

		vL = new VLayout(5);
		vL.setPadding(20);
		vL.setWidth100();
		vL.setHeight100();
		tF = new TestForm(194);

		hL = new HLayout(5);
		IButton recreate1 = new IButton("Recreate with ID=194 (record)");
		recreate1.setWidth(150);
		recreate1.addClickHandler(new ClickHandler() {
			@Override
			public void onClick(ClickEvent event) {
				vL.removeChild(tF);
				tF.markForRedraw();
				tF = new TestForm(194);
				vL.addMember(tF, 0);
			}
		});
		IButton recreate2 = new IButton("Recreate with ID=25 (no record)");
		recreate2.setWidth(150);
		recreate2.addClickHandler(new ClickHandler() {
			@Override
			public void onClick(ClickEvent event) {
				vL.removeChild(tF);
				tF.markForRedraw();
				tF = new TestForm(25);
				vL.addMember(tF, 0);
			}
		});

		hL.addMembers(recreate1, recreate2);
		vL.addMembers(tF, hL);
		vL.draw();
	}

	private class TestForm extends DynamicForm {
		HiddenItem EmployeeId;
		Integer idToFetch;

		public TestForm(Integer idToFetch) {
			super();
			this.idToFetch = idToFetch;
			setDataSource(DataSource.get("employees"));
			setAutoFetchData(false);
			setFetchOperation("willDefaultToFetchOnServer");

			EmployeeId = new HiddenItem("EmployeeId");
			// EmployeeId.setFetchMissingValues(true);
			final FormItem Name = new FormItem("Name");
			final ComboBoxItem ReportsTo = new ComboBoxItem("ReportsTo");
			ReportsTo.setOptionDataSource(DataSource.get("employees"));
			ReportsTo.setDisplayField("Name");
			ReportsTo.setSortField("Name");
			final FormItem Job = new FormItem("Job");
			final FormItem Email = new FormItem("Email");
			final FormItem EmployeeType = new FormItem("EmployeeType");
			final FormItem EmployeeStatus = new FormItem("EmployeeStatus");
			final FormItem Salary = new FormItem("Salary");
			final FormItem OrgUnit = new FormItem("OrgUnit");
			final SelectItem Gender = new SelectItem("Gender");

			setFields(EmployeeId, Name, ReportsTo, Job, Email, EmployeeType, EmployeeStatus, Salary, OrgUnit, Gender);
			fetchData(new AdvancedCriteria(new Criterion("EmployeeId", OperatorId.EQUALS, idToFetch)), new DSCallback() {
				@Override
				public void execute(DSResponse dsResponse, Object data, DSRequest dsRequest) {
					if (Helper.oneRow(dsResponse)) { // Detail row found
						// Nothing to do
					} else if (Helper.zeroRows(dsResponse)) { // No detail row found
						EmployeeId.setValue(TestForm.this.idToFetch); // Only 1 record with lifeSpan=25 (Myrmecophaga tridactyla), so it is "kind of a PK"
						// In my real application this is used for add/update of the form.
						// This fails validation then because of the value not being a number ("" or "Loading...")
						TestForm.this.rememberValues();
					} else {
						SC.say("There was an error retriving the record details");
					}
				}
			});
		}
	}

	public abstract static class Helper {
		public static boolean oneRow(DSResponse dsr) {
			return (dsr != null && dsr.getStatus() == DSResponse.STATUS_SUCCESS && dsr.getTotalRows() == 1);
		}

		public static boolean zeroRows(DSResponse dsr) {
			return (dsr != null && dsr.getStatus() == DSResponse.STATUS_SUCCESS && dsr.getTotalRows() == 0);
		}

		public static boolean isSuccess(DSResponse dsr) {
			return (dsr != null && dsr.getStatus() == DSResponse.STATUS_SUCCESS);
		}
	}
}

employees.ds.xml:

<DataSource ID="employees" serverType="sql" tableName="employeeTable" recordName="employee" testFileName="/examples/shared/ds/test_data/employees.data.xml"
	titleField="Name" useAnsiJoins="true">
	<fields>
		<field name="userOrder" title="userOrder" type="integer" canEdit="false" hidden="true" />
		<field name="Name" title="Name" type="text" length="128" />
		<field primaryKey="true" foreignKey="animals.lifeSpan" name="EmployeeId" title="EmployeeId" type="integer" hidden="true" required="true" />
		<field name="ReportsTo" title="Manager" type="integer" required="true" foreignKey="employee.EmployeeId" />
		<field name="Job" title="Title" type="text" length="128" />
		<field name="Email" title="Email" type="text" length="128" />
		<field name="EmployeeType" title="Employee Type" type="text" length="40" />
		<field name="EmployeeStatus" title="Status" type="text" length="40" />
		<field name="Salary" title="Salary" type="float" />
		<field name="OrgUnit" title="Org Unit" type="text" length="128" />
		<field name="Gender" title="Gender" type="text" length="7">
			<valueMap>
				<value>male</value>
				<value>female</value>
			</valueMap>
		</field>
		<field name="MaritalStatus" title="Marital Status" type="text" length="10">
			<valueMap>
				<value>married</value>
				<value>single</value>
			</valueMap>
		</field>
	</fields>
</DataSource>

animals.ds.xml:

<DataSource ID="animals" serverType="sql" tableName="animals" testFileName="animals.data.xml" useAnsiJoins="true">
	<fields>
		<field name="commonName" title="Animal" type="text" />
		<field name="scientificName" title="Scientific Name" type="text" required="true" />
		<field primaryKey="true" hidden="true" name="lifeSpan" type="sequence" foreignKey="employees.EmployeeId" joinType="outer" />
		<field name="status" title="Endangered Status" type="text">
			<valueMap>
				<value>Threatened</value>
				<value>Endangered</value>
				<value>Not Endangered</value>
				<value>Not currently listed</value>
				<value>May become threatened</value>
				<value>Protected</value>
			</valueMap>
		</field>
		<field name="diet" title="Diet" type="text" />
		<field name="information" title="Interesting Facts" type="text" length="1000" />
		<field name="picture" title="Picture" type="image" detail="true" imageURLPrefix="/isomorphic/system/reference/inlineExamples/tiles/images/" />
	</fields>
</DataSource>

At startup issued bad request for me using Win7 IE10 and v10.0p_2015-05-21 (#94 in the BuiltInDS.png screenshot):

{
    dataSource:"animals", 
    operationType:"fetch", 
    componentId:"isc_BuiltInDS_TestForm_29", 
    data:{
        lifeSpan:"Loading..."
    }, 
    textMatchStyle:"exact", 
    callback:{
        target:[HiddenItem ID:isc_HiddenItem_59 name:EmployeeId], 
        methodName:"fetchMissingValueReply"
    }, 
    showPrompt:false, 
    oldValues:{
        lifeSpan:"Loading..."
    }, 
    requestId:"animals$62769", 
    internalClientContext:{
        newValue:"Loading...", 
        filterLocally:{
        }, 
        targetField:"lifeSpan", 
        fetchingMissingValues:{
        }
    }, 
    useStrictJSON:true, 
    fallbackToEval:true, 
    componentContext:"EmployeeId", 
    lastClientEventThreadCode:"TMR8", 
    bypassCache:true
}

To summarize:

• It only happens in IE10 in my application (from my testing/user reports)
• In the testcase, where the fetchMissingValueReply-request happens slightly different, it happens for me in Win7 IE10 and Win 8.1 IE11
• I don't think there should be any fetchMissingValueReply-requests for HiddenItems
• If you think the fetchMissingValueReply-request is needed for a HiddenItem, it should be for the ID only and not for "" or "Loading..."

Best regards
Blama

[Isomorphic]

We just this past week made a series of ComboBoxItem changes designed to catch some edge case interactions that could explain this requests. Can you see if you can still reproduce this intermittent issue with the latest patches?

[Blama]

Hi Isomorphic,

I'll retest now, but please note that I'm talking about a HiddenItem.

Best regards
Blama

[Isomorphic]

Yes, at least part of the ComboBoxItem fix was in the base FormItem code, which is shared with HiddenItem.

[Blama]

Hi Isomorphic,

the error from the testcase seems to be fixed with v10.0p_2015-06-17 in IE10 and IE11. I did not test in my application, though. I will do so tomorrow.

Best regards
Blama

[Blama]

Hi Isomorphic,

one question remains: Why is there a fetchMissingValueReply-request for HiddenItems anyway?
This request is still there, but the one for "Loading..." is gone.

Best regards
Blama

[Isomorphic]

Hi Blama,
The default for the (documented) FormItem.fetchMissingValues property is true.
The default behavior in this case is that if there's a displayField (as well as a valueField) and the item is bound to an optionDataSource we issue a fetch to pick up the display field value so we can display the appropriate user-consumable value to the user.

However we agree this default doesn't make much sense in the context of a hidden item. Usually this wouldn't arise as you'd be unlikely to explicitly set a displayField on a HiddenItem - but this can of course come up as a result of settings on the DataSourceField, etc.

Anyway, for now you can turn this off in your app by setting this property (fetchMissingValues) explicitly to false on the item in question, but we'll also toggle the default so it is false for hidden items.

Regards
Isomorphic Software

[Blama]

Hi Isomorphic,

Anyway, for now you can turn this off in your app by setting this property (fetchMissingValues) explicitly to false on the item in question.

I will do so. This would have been the next experiment on my hotfix list anyway.

but we'll also toggle the default so it is false for hidden items.

Can you update here again once you made the change?

Best regards
Blama

[Isomorphic]

We've made the change to the default value in the 5.1d branch.
The next nightly build (June 20 or above) will contain this change.

We'll leave it as is in 5.0p since this is very simple to address at the application level and we don't want to introduce a (admittedly very obscure) backwards compatibility concern for anyone who may be relying on the existing default.

Regards
Isomorphic Software