Client side validation stops working

[andrerichards]

Hi,

SmartGWT Power Version: v5.0p
SmartClient Version: v10.0p_2014-12-21/PowerEdition Deployment (built 2014-12-21)
GWT Version: 2.5.1
Application server: Jetty & Tomcat
Server JVM (Oracle 1.8 / JRockit 1.6)
RDBMS: Microsoft SQL Server 2005
Environment: Compiled & Development mode
Browser versions: Any browser


I would like to ask assistance with a possible server side regression between SmartGWT Power v3.1p & v5.0p

In short, after a specific set of user actions (DMI calls to the server), client side validation starts failing throughout the application, until such time as the application server is restarted, then everything works again as normal, for a while, before it happens again.

So although the symptom we experience is client side, we suspect a server side cause, as when we restart the application server (Tomcat / Jetty) the client side validation works again. Note that merely reloading the page in the browser does not make the client side validation work again, the app server need to be restarted.

The problem is reproducible as outlined in an example app that I’ve attached, a bare bone application that can reproduce the problem, as well as a link to a video showing the steps necessary to reproduce. Note these steps in the video need to be performed in that specific order otherwise it might not be able to be reproduced.



Video: http://youtu.be/ZsAVzOR0B0w

Full sample app source: https://drive.google.com/a/hti-systems.co.za/file/d/0BxOXjU0SoiO5WUpUVUpKSVA5RVk




There seems to be many of these occurrences throughout our application and it is very, very difficult to identify the location of the problems in our code as there are no errors logged on the server, and no Javascript errors reported in the browser. Randomly, we are notified by our users that client side validation throughout the application have started failing.


We’re at a point where we are desperately in need of your assistance, as we’ll need to roll back to v3.1p which works perfectly with exactly the same code, if we can’t get this resolved.



GWT EntryPoint:

public class MyApp implements EntryPoint {

	@Override
	public void onModuleLoad() {

		Validation validation = new Validation();
		validation.show();

	}

}

Validation View

public class Validation extends VLayout {
	
	private DataSource ds;
	private DynamicForm form = new DynamicForm();
	
	public Validation(){
		super();
		setWidth100();
		setHeight100();
		setPadding(50);
		setBackgroundColor("#f5f5f5");
		setMembersMargin(20);
	}
	
	@Override
	protected void onInit(){
		createDataSource();
		createGrid();
		createForm();
		createButtons();
	}
	
	private void createButtons(){



		IButton someDataSourceDMIAddButton = new IButton("Step 1) Some DataSource DMI add");
		someDataSourceDMIAddButton.setWidth(220);
		someDataSourceDMIAddButton.setAlign(Alignment.LEFT);
		someDataSourceDMIAddButton.addClickHandler(new com.smartgwt.client.widgets.events.ClickHandler() {

			@Override
			public void onClick(com.smartgwt.client.widgets.events.ClickEvent event) {
				DSRequest dsRequest = new DSRequest();
				dsRequest.setOperationId("someDataSourceDMIAdd");
				Record record = new Record();
				record.setAttribute("Memo", "Generated memo");
				DataSource.get("Memo").addData(record, new DSCallback() {
					@Override
					public void execute(DSResponse response, Object rawData, DSRequest request) {
						SC.say("Done. Proceed to step 2");
					}
				}, dsRequest);
			}
		});



		IButton someFetchReturningNoResultsButton = new IButton("Step 2) Some DMI fetch returning 0 results");
		someFetchReturningNoResultsButton.setWidth(220);
		someFetchReturningNoResultsButton.setAlign(Alignment.LEFT);
		someFetchReturningNoResultsButton.addClickHandler(new com.smartgwt.client.widgets.events.ClickHandler() {

			@Override
			public void onClick(com.smartgwt.client.widgets.events.ClickEvent event) {

				DSRequest dsRequest = new DSRequest();
				dsRequest.setOperationId("someDataSourceDMIFetch");

				DataSource.get("DxInvoice").fetchData(new Criteria(), new DSCallback() {
					@Override
					public void execute(DSResponse response, Object rawData, DSRequest request) {
						SC.say("Done. Proceed to step 3");
					}
				}, dsRequest);
			}
		});



		IButton reloadButton = new IButton("Step 3) Refresh browser");
		reloadButton.setWidth(220);
		reloadButton.setAlign(Alignment.LEFT);
		reloadButton.addClickHandler(new com.smartgwt.client.widgets.events.ClickHandler() {

			@Override
			public void onClick(com.smartgwt.client.widgets.events.ClickEvent event) {
				Window.Location.reload();
			}
		});


		Label label = new Label("Step 4) Try save a memo longer than 10");



		VLayout buttons = new VLayout();
		buttons.addMember(someDataSourceDMIAddButton);
		buttons.addMember(someFetchReturningNoResultsButton);
		buttons.addMember(reloadButton);
		buttons.addMember(label);
		addMember(buttons);
	}
	
	private void createForm(){
		TextAreaItem crud = new TextAreaItem("Memo", "Memo");
		ButtonItem save = new ButtonItem("Save", "Save Memo (max length 1000)");
		save.setWidth(200);
		save.addClickHandler(new ClickHandler() {
			
			@Override
			public void onClick(ClickEvent event) {
				form.validate();
				
				if (!form.hasErrors())
					form.saveData();
			}
		});
		
		form.setDataSource(ds);
		form.setBackgroundColor("#FFFFFF");
		form.setFields(crud, save);
		addMember(form);
	}
	
	private void createGrid(){
		ListGridField Memo = new ListGridField("Memo", "Memo", 3000);

		ListGrid grid = new ListGrid();
		grid.setCanEdit(true);
		grid.setDataSource(ds);
		grid.setAutoFetchData(true);
		grid.setFields(Memo);
		grid.setWrapCells(true);
		grid.setFixedRecordHeights(false);
		addMember(grid);		
	}
	
	private void createDataSource(){
		ds = DataSource.get("Memo", new RequestTransformer() {
			
			@Override
			protected Object transformRequest(DSRequest dsRequest) {
				return dsRequest.getData();
			}
		}, new ResponseTransformer() {
			
			@Override
			protected void transformResponse(DSResponse response, DSRequest request,Object data) {
				if ((response.getData() != null) && (response.getData().length > 0))
					form.editRecord(response.getData()[0]);
			}
		}); 
	}
	
}

GWT Module file:

<?xml version="1.0" encoding="UTF-8"?>
<module rename-to='myapp'>

	<inherits name="com.google.gwt.user.User" />
	<inherits name="com.google.gwt.place.Place" />
	<inherits name="com.google.gwt.activity.Activity" />
	<inherits name="com.google.gwt.i18n.I18N" />
	<inherits name="com.google.gwt.i18n.CldrLocales" />
	<inherits name="com.google.gwt.core.Core" />

	
	 <inherits name="com.smartgwt.tools.SmartGwtTools"/>
    <inherits name="com.smartgwtpower.SmartGwtPowerNoScript"/>
    <inherits name="com.smartgwtpower.tools.Tools"/>
	<inherits name="com.google.gwt.logging.Logging" />
	<inherits name="com.google.gwt.http.HTTP" />
	<inherits name="com.google.gwt.json.JSON" />

	<extend-property name="locale" values="en" />
	<set-configuration-property name="locale.useragent" value="Y"/>
	<source path='client' />
	<source path='shared' />
	<entry-point class='com.example.client.MyApp' />
	
	<set-configuration-property name='xsiframe.failIfScriptTag' value='FALSE'/>

	<set-property name="gwt.logging.logLevel" value="INFO" />
	<set-property name="gwt.logging.enabled" 	value="TRUE" />
	<set-property name="gwt.logging.developmentModeHandler" value="ENABLED" />
	<set-property name="gwt.logging.systemHandler" value="DISABLED" />
	<set-property name="gwt.logging.popupHandler" value="DISABLED" />
	<set-property name="gwt.logging.consoleHandler" value="ENABLED"/>
	<set-property name="gwt.logging.firebugHandler" value="ENABLED" />
	<set-property name="gwt.logging.simpleRemoteHandler" value="ENABLED" />

</module>

Host page:

<!DOCTYPE HTML>
<html>
<head>
    <title>MyApp</title>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <meta name="gwt:property" content="locale=x_Y">
    <meta http-equiv="X-UA-Compatible" content="IE=Edge">

<body bottommargin="0" leftmargin="0" rightmargin="0" topmargin="0" marginheight="0" marginwidth="0">

<iframe id="__gwt_historyFrame" style="width:0;height:0;border:0"></iframe>

<!--add loading indicator while the app is being loaded-->
<div id="loadingWrapper">
<div id="loading">
    <div class="loadingIndicator">
        MyApp<br/>
        <span id="loadingMsg">Loading styles and images...</span>
    </div>
</div>
</div>


<script>
var isomorphicDir="myapp/sc/";
</script>


<script type="text/javascript">document.getElementById('loadingMsg').innerHTML = 'Loading Core API...';</script>

<!--include the SC Core API-->
<script src='myapp/sc/modules/ISC_Core.js'></script>
<script type="text/javascript">document.getElementById('loadingMsg').innerHTML = 'Loading UI Components...';</script>
<script src='myapp/sc/modules/ISC_Foundation.js'></script>
<script src='myapp/sc/modules/ISC_Containers.js'></script>
<script src='myapp/sc/modules/ISC_Grids.js'></script>
<script src='myapp/sc/modules/ISC_Forms.js'></script>
<script src='myapp/sc/modules/ISC_RichTextEditor.js'></script>
<script src='myapp/sc/modules/ISC_Calendar.js'></script>
<script type="text/javascript">document.getElementById('loadingMsg').innerHTML = 'Loading Data API...';</script>
<script src='myapp/sc/modules/ISC_DataBinding.js'></script>
<script src='myapp/sc/modules/ISC_Drawing.js'></script>
<script src='myapp/sc/modules/ISC_Charts.js'></script>

<!--load the datasources-->
<script src="myapp/sc/DataSourceLoader?dataSource=DxInvoice,Memo,DxReservation"></script>


<!--include the application JS-->
<script type="text/javascript">document.getElementById('loadingMsg').innerHTML = 'Loading modules<br>Please wait...';</script>
<script type="text/javascript" src="myapp/myapp.nocache.js"></script>
<script type="text/javascript" src="myapp/sc/skins/Enterprise/load_skin.js"></script>


</body>
</html>

Datasource - DxInvoice

<DataSource
	schema="dbo"
	dbName="MyApp"
	tableName="DxInvoice"
	ID="DxInvoice"
	dataSourceVersion="1"
	serverType="sql"
	logActionHistory="true"
>
	<fields>
		<field primaryKey="true" name="ID" type="sequence" hidden="true"></field>
		<field name="RESERVATION_ID" type="integer" foreignKey="DxReservation.ID" required="true" title="Reservation"></field>
		<field name="ResStatus" includeFrom="DxReservation.Status"/>
		<field name="Status" length="10" type="text" required="true"></field>
	</fields>
	<operationBindings> 
		<operationBinding operationType="fetch" operationId="someDataSourceDMIFetch" serverMethod="fetchme" outputs="Status">
            <serverObject className="com.example.server.DSInvoice" />
        </operationBinding>	
	</operationBindings>
</DataSource>

Datasource - DxReservation

<DataSource
	schema="dbo"
	dbName="MyApp"
	tableName="DxReservation"
	ID="DxReservation"
	dataSourceVersion="1"
	serverType="sql"
	logActionHistory="true"
>
	<fields>
		<field primaryKey="true" name="ID" type="sequence" hidden="true"></field>
		<field name="Status" length="10" type="text" hidden="true"></field>
	</fields>

</DataSource>

Datasource - Memo

<DataSource
	schema="dbo"
	dbName="MyApp"
	tableName="DxMemo"
	ID="Memo"
	dataSourceVersion="1"
	serverType="sql"
	logActionHistory="true"
>
	<fields>
		<field primaryKey="true" name="IDMEMO" type="sequence" hidden="true"></field>
		<field name="Memo" length="1000" type="text" required="true"></field>
	</fields>

    <operationBindings>
        <operationBinding operationType="add" operationId="someDataSourceDMIAdd" serverMethod="someDataSourceDMIAdd">
            <serverObject className="com.example.server.DSMemo"/>
        </operationBinding>
    </operationBindings>

</DataSource>

ServerObject - DSMemo

public class DSMemo {

	public static DSResponse someDataSourceDMIAdd(DSRequest dsRequest) throws Exception {

		DSRequest dsRequestReservation = new DSRequest();
		dsRequestReservation.setOperationType("add");
		dsRequestReservation.setDataSourceName("DxReservation");
		dsRequestReservation.setFieldValue("Status", "Individual");
		dsRequestReservation.execute();

		return dsRequest.execute();

	}
}

ServerObject DSInvoice:

public class DSInvoice {

	public DSResponse fetchme(DSRequest dsRequest) throws Exception {
		DSResponse resp = new DSResponse();

		List invoiceResult = new ArrayList();
		resp.setData(invoiceResult);
		return resp;  
	} 
}

Server.properties:

webRoot: __AUTODETECT__
gwtModuleName: myapp

isomorphicPathRootRelative: $gwtModuleName/sc

sql.defaultDatabase: MyApp

project.datasources: $webRoot/ds
project.ui: $webRoot/shared/ui
project.apps: $webRoot/shared/app

RPCManager.enabledBuiltinMethods: getPdfObject, xmlToJS, uploadProgressCheck, exportClientData, downloadClientExport, setAttributes, getLogNames, getLogEntries, getLogThresholds, setLogThreshold, getAvailableScriptEngines

sql.MyApp.driver.name: jdbc/MYAPP
sql.MyApp.database.type: sqlserver
sql.MyApp.interface.type: jndi
sql.MyApp.useGlobalIdentity: true
sql.useAnsiJoins: true
sql.pool.enabled: false

web.xml:

<web-app xmlns="http://java.sun.com/xml/ns/j2ee" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd"
         version="2.4">



    <!-- Default page to serve -->
    <welcome-file-list>
        <welcome-file>MyApp.html</welcome-file>
    </welcome-file-list>




    <!-- The IDACall servlet handles all Built-in DataSource operations -->
    <servlet>
        <servlet-name>IDACall</servlet-name>
        <servlet-class>com.isomorphic.servlet.IDACall</servlet-class>
        <!-- <servlet-class>com.isomorphic.servlet.IDACall</servlet-class> -->
    </servlet>

    <!-- The DataSourceLoader servlet returns Javascript representations of
        the dataSources whose ID's are passed to it - it is an alternative to using
        the <loadDS> JSP tag -->
    <servlet>
        <servlet-name>DataSourceLoader</servlet-name>
        <servlet-class>com.isomorphic.servlet.DataSourceLoader</servlet-class>
    </servlet>

    <!-- The FileDownload servlet downloads static files, like a webserver -->
    <servlet>
        <servlet-name>FileDownload</servlet-name>
        <servlet-class>com.isomorphic.servlet.FileDownload</servlet-class>
    </servlet>



    <!-- The PreCache servlet initializes when the servlet engine starts up
        and pre-loads data need for all client requests. This is optional, and improves
        performance of the first few page requests. PreCache cannot be invoked by
        a browser, because there is no "servlet-mapping" defined for it. -->

    <!-- PreCache is disabled by default because of an issue when debugging
        the project for the first time. However, it can be safely enabled when compiling
        with the GWT compiler prior to deployment. See the README.txt for more information. -->

    <!-- <servlet> <servlet-name>PreCache</servlet-name> <servlet-class>com.isomorphic.servlet.PreCache</servlet-class>
        <load-on-startup>2</load-on-startup> </servlet> -->

    <!-- RPCManager uses this URL by default for Built-in DataSource operations -->
    <servlet-mapping>
        <servlet-name>IDACall</servlet-name>
        <url-pattern>/myapp/sc/IDACall/*</url-pattern>
    </servlet-mapping>


    <!-- DataSourceLoader requests -->
    <servlet-mapping>
        <servlet-name>DataSourceLoader</servlet-name>
        <url-pattern>/myapp/sc/DataSourceLoader</url-pattern>
    </servlet-mapping>

    <!-- Use FileDownload servlet to download all static content that's part
        of the skin, such as image files, so we can set Expires headers and other
        cache control directives. In a production deployment, you'd want to use a
        webserver such as Apache to do this. -->
    <servlet-mapping>
        <servlet-name>FileDownload</servlet-name>
        <url-pattern>/myapp/sc/skins/*</url-pattern>
    </servlet-mapping>

    <!-- serve ISC modules compressed, with expires headers -->
    <servlet-mapping>
        <servlet-name>FileDownload</servlet-name>
        <url-pattern>/myapp/sc/system/modules/*</url-pattern>
    </servlet-mapping>

    <!-- serve ISC development modules compressed, with expires headers -->
    <servlet-mapping>
        <servlet-name>FileDownload</servlet-name>
        <url-pattern>/myapp/sc/system/development/*</url-pattern>
    </servlet-mapping>

    <!-- server skin assets with expires headers -->
    <servlet-mapping>
        <servlet-name>FileDownload</servlet-name>
        <url-pattern>/myapp/sc/system/reference/skin/*</url-pattern>
    </servlet-mapping>

    <!-- General config -->
    <session-config>
        <session-timeout>10</session-timeout>
    </session-config>

    <jsp-config>
        <!-- Isomorphic JSP tags -->
        <taglib>
            <taglib-uri>isomorphic</taglib-uri>
            <taglib-location>/WEB-INF/iscTaglib.xml</taglib-location>
        </taglib>
    </jsp-config>

    <mime-mapping>
        <extension>manifest</extension>
        <mime-type>text/cache-manifest</mime-type>
    </mime-mapping>

    <resource-ref>
        <description>DB Connection Pooling</description>
        <res-ref-name>jdbc/MYAPP</res-ref-name>
        <res-type>javax.sql.DataSource</res-type>
        <res-auth>Container</res-auth>
    </resource-ref>
</web-app>

Please let me know if you need any more information.

[Isomorphic]

It's good to see that you've finally taken *some* of the necessary steps we kept reminding you of in your other thread.

There is at least one bug we can spot just from glancing at this code: setDataSourceName() is not a supported or documented API, and you should instead be using the DSRequest constructor that takes a DataSource name. This may seem innocuous, but the method you're calling is unsupported for a reason - it manipulates the DataSource pool, and could somehow create the effect you're seeing.

If this doesn't happen to be the problem, we'll remind you again of the steps you need to take:

1. you need to update to the latest patched version

2. you need to create a minimal test case. What you've posted here is a big pile of code which depends on a SQL Server DB with table definitions and initial data we don't have, various other dependencies, incomplete imports, etc. If you really think there's a bug in SmartGWT itself, and not a weird issue with your deployment environment, demonstrate it by creating a minimal test case that just uses stock SDK resources.

[andrerichards]

Thank you for your reply.

I've implemented your suggestions:

- replaced method setDataSourceName() with the constructor
- removed any & all dependencies not included in the SDK
- replaced MS SQL requirement with HSQLDB and automatically create required schema etc.

The problem still continued exactly as before with the above changes.

I then downloaded the latest nightly build of SmartGWT v5.0p and indeed, I can confirm that at least this occurrence of the validation problem that we were able to reliably reproduce till now, is fixed in the latest nightly of SmartGWT v5.0p.

This is great news and I'm very happy that it is working now!

However, do you perhaps have release notes for SmartGWT v5.0? I've checked that in the SDK the release_notes.html only goes up until v3.1

Have the release notes location maybe changed?


Thank you

[Isomorphic]

There's a changelog file, but looking at a changelog is typically a pointless activity. Whatever was going on with your app may or may not have actually been resolved by a SmartGWT bugfix - the symptoms you reported (especially in your other thread) were bizarre enough that it might have been "fixed" because of framework code changes that, for example, may happen to avoid triggering a Tomcat bug, rather than fixing any SmartGWT bug per se.

Not to mention that we, being intimately familiar with the framework, did not recognize any recent framework bugfix as specifically resolving your issue. Working from the changelog, you would have been extremely unlikely to spot the relevant fix (again, if it was even a fix).

To avoid effort next time, just take our advice about trying the latest patched version as the *first* step, rather than the last.

[andrerichards]

Understood, thanks.

In terms of our own QA before rolling out the latest patched SDK, it would help tremendously if we have an indication of what other changes might possibly have an effect on our own code base.

Would it be possible to send us the change-log between version:

v10.0p_2014-12-21/PowerEdition Deployment (built 2014-12-21)

and

v10.0p_2015-05-12/PowerEdition Deployment (built 2015-05-12)


Thank you

[Isomorphic]

No, sorry, we don't publish such lists at the moment.

However, if you were planning to try to "focus your testing" by reading the changelog, you could replace that effort with reading tea leaves or sacrificing goats, which are similarly effective.

All kidding aside, when we are constantly working around the latest browser & mobile bugs, there is no reasonable way to go from a changelog to areas of the UI or code that need extra attention. We couldn't do that; you have much much less context.

Instead, focus your attention on robust automated testing so that you can update to the latest patched version with a minimum of manual effort.

[Blama]

Hello,

andrerichards reported in post #3 that the problem persists with the old version even after fixing the usage bugs Isomorphic pointed out.

Isomorpic said in post #4 Not to mention that we, being intimately familiar with the framework, did not recognize any recent framework bugfix as specifically resolving your issue..

As the issue itself seems severe and unexplainable, it might be good to know the root cause (hoping that it is one single change!).

If andrerichards could determine the nightly the issue went away (should be a download/compile/test cycle of approx 8 versions (log2(5 month worth of nightlys)), Isomorphic could verify if the old internal usage pattern also exists in other parts of the framework.

But as I'm not affected by this, this is just a suggestion.

Best regards
Blama

[andrerichards]

For completeness sake I did as Blama suggested and was finally able to narrow down the builds to the exact one where our problem went away.

The problem still existed up to the patched build of 2015-04-19 and was resolved ever since the next build which was 2015-04-25.

I've also uploaded my stripped down, no dependencies, minimal test case to the following URL in the event that it is needed:
https://drive.google.com/a/hti-systems.co.za/file/d/0BxOXjU0SoiO5cjduX3M2NExVbms/view?usp=sharing

As I've mentioned, our problem is now resolved, but as Blama mentioned, maybe this information might assist in the event Isomorphic wants to investigate this specific usage pattern further.


Thank you

[Isomorphic]

Since you took the time to isolate further, we looked closely at the changes in that time range, but, we don't see anything plausibly responsible.

Bear in mind, a framework change totally unrelated to the area of validation - something that causes one less image fetch or something like that - could mean that we are no longer triggering a timing bug in third-party software.

So the answer is not necessarily in changes to SmartGWT, and in fact, since this is the one and only report of this kind across a huge number of deployments, it seems very likely that the root cause was some kind of innocuous behavior change in SmartGWT combined with bugs in other software.

[andrerichards]

Agreed.

Just for info, I've isolated further, and by replacing only the isomorphic_core_rpc.jar with the one in build 2015-04-25, the problem went away.

Cheers

[Blama]

You mean the one in 2015-04-25, don't you?

[andrerichards]

Sorry, I fixed my typo in my previous post.

The 2015-04-19 build is the last build that showcased the problem.

By replacing isomorphic_core_rpc.jar in 2015-04-19 with the one from build 2015-04-25, the problem went away.