Need help with problem related to DataSource Fetch with incorrect criteria

Browsers: IE11, FF, Chrome
SmartGWT Version: SmartClient Version: v10.0p_2015-11-16/LGPL Development Only (built 2015-11-16)

I need help trying to isolate a problem I am seeing with my DataSource Fetch operation getting being called with the wrong criteria. I've tried narrowing it down in a small sample and cannot repro it there (I am only seeing this in the scope of our full product).

I have a ListGrid where I am setting the criteria (see attachment JSON1.txt - this was produced by getting the criteria object from the grid and converting it to JSON). This grid is attached to a DataSource in which we override the TransformRequest method because we have to do things like transform the criteria syntax to match that supported on the REST URL to our server).

However, when our code gets the criteria from the request and tries to convert it, the criteria seems to have been corrupted. If you look at the attachment JSON2.txt you will see the JSON representation of the criteria coming in via the request. Note that in the grid criteria in JSON1.txt the first test on job_dueDate has a "NOT". Note then that the criteria in the Fetch request (JSON2.txt) is missing the "NOT" (which totally changes the result set).

We first experienced this problem when we rolled our SmartGWT jars from the from the 2015-06-02 build to the 2015-10-31 build. If I just switch are jars back to the 2015/06/02 build everything works fine.

I have tried reproducing this in a small sample (since our app grid/datasource interaction is very large, complicated, and intertwined with many other services provided by our system). I've attached the sample (MatchNoneGridFilterIssue.java). However, the sample code is behaving a lot differently than our application. The attached file JSON3.txt shows the criteria on the grid and the file JSON4.txt shows the criteria passed to the Fetch. In the sample, they are identical and even the one on the Grid has been transformed significantly from the criteria that was actually set on the Grid. In addition, in both forms the dates are still in RelativeDate format. In our application, the grid criteria matches what as set on the grid and dates are in RelativeDate format, but the Fetch criteria is changed significantly, the "NOT" has been dropped, and the RelativeDates have been converted to an absolute Date.

I don't know why the sample is behaving so differently than our application code and I don't know why our application is dropping the NOT and the sample isn't.

I could use help finding out what is causing the issue. We are currently working with you on other grid issues that puts us in a position of needing to roll to newer releases to pick up fixes you are doing for us, but with this issue we cannot roll to the latest stuff because it is breaking other things in our product that we cannot find a way to work around.

[Isomorphic]

From a quick look we're going to need some more information to get to the bottom of this one. It doesn't seem very surprising that the sample behaves differently from your real app - you're building an AdvancedCriteria object (using JSON), and applying it to the grid via setCriteria and then getting it back via getCriteria, whereas in your application you're (presumably) applying the criteria to the grid in a similar way, but when you're getting the (modified / corrupted) criterion you're in another code flow - looking at the criteria property of a dsRequest passed to transform request.

Can we recommend you try to build a sample using the following approach:

1) Create a new DataSource with similar fields (or a subset thereof) to those in your app. ClientOnly would probably be fine for this.
2) Create a grid bound to the dataSource with similar field definitions to those in your app.
3) Apply the criteria to the grid however this happens in your app - so if you're calling 'setCriteria' or 'fetchData' on the grid and passing in an AdvancedCriteria object, do that - or if the user is creating the criteria via some interaction, provide a similar UI.
4) In the dataSource, override transformResponse to display the criteria object passed in from the dataSourceRequest -- the same place you're viewing the (broken) criteria in your app.

It seems you shouldn't need more scaffolding than that to reproduce the problem - it doesn't seem like you need to actually populate the grid, with data - just re-run the code flow from populating the grid's criteria to picking it back up from the request object.

Hopefully this'll reproduce the same issue (and be runnable by us), so we can take a proper look.

Thanks
Isomorphic Software

I'll give it a try, but unfortunately it is not nearly as easy as it sounds. Our product goes through literally 1000's of lines of code to convert our criteria which is stored in a custom syntax into one of your advanced criteria objects. I'll work on this. In the mean time, I have narrowed it down to the fact that whatever changed happened between the builds you have uploaded for 2015-07-25 (which works fine) and 2015-08-08 (which does not). There are no builds uploaded between those two date that would allow me to narrow it down farther.

[Isomorphic]

I'll give it a try, but unfortunately it is not nearly as easy as it sounds. Our product goes through literally 1000's of lines of code to convert our criteria which is stored in a custom syntax into one of your advanced criteria objects.

It seems like this isn't likely to be necessary to reproduce in the test case. It seems like you have a clear moment when you have visible, known criteria which are correct, applied to the grid. So presumably you can start there by simply creating the criteria explicitly in your test case and applying them to the grid. If we understand your description correctly - the problem occurs later in the application flow when these criteria are actually sent to the dataSource and then picked up by the transformRequest method. If so, that's the part you need to duplicate in the test case.

Of course if it isn't that simple, it's possible that you have a problem in the thousands of lines of code that manipulate the criteria - something that used to work and now doesn't. If that's the case, isolating what that step is is going to be key to identifying the problem.

I'll work on this. In the mean time, I have narrowed it down to the fact that whatever changed happened between the builds you have uploaded for 2015-07-25 (which works fine) and 2015-08-08 (which does not). There are no builds uploaded between those two date that would allow me to narrow it down farther.

We will probably need to see the problem. We could audit all code changes between those dates and look for anything that changed in the area, but we'd be somewhat blindly speculating as to why it should effect your particular app in the way it does so that's likely to be an inefficient way to get to the root of this issue.

I've attached the sample using our actual code trimmed down as much as was easy to do. The sample now produces the defect on the initial Fetch (the sample is still hooked to a DataSource that is looking for a server so it then fails with a transport error, but that can be ignored since this occurs after the problem is seen. The GWT log messages output the criteria in JSON as it looks after we create the criteria object, after we set it on the grid, and then what the fetch actually receives. I see the issue in all of the builds from 2015/08/08 onward.

Ooops, missed one of the files you need....

[Isomorphic]

Thanks - we've got this assigned to a developer for investigation. We'll let you know as soon as we have anything for you.

[Isomorphic]

This is fixed and will be available in the next (2015-11-19) nightly build.

Please let us know whether your issue is gone when you try it out.

That fixed it. Thank you for all of the help!