Two consecutive fetches with the same filters produce different criteria

[kimf]

Hi,

We are long-time SmartGWT users and bumped into a nasty little problem.

SmartGWT version: SmartClient Version: SC_SNAPSHOT-2011-08-02/Pro Deployment (built 2011-08-02) (The latest 2.5 Pro build)
Browsers: Re-producable in all the browsers we've tested on: IE8-9, Chrome, Firefox.

We've got a ListGrid which lists invoices. The invoices are fetched based on the ListGrid's filters.

ListGrid invoiceGrid = new ListGrid();
invoiceGrid.setShowFilterEditor(true);
invoiceGrid.setDataSource(LedgerInvoiceListRestDS.getInstance());
invoiceGrid.setAutoFetchData(false);
invoiceGrid.setDataFetchMode(FetchMode.BASIC);

ResultSet resultSet = new ResultSet();
resultSet.setUseClientFiltering(false);
invoiceGrid.setDataProperties(resultSet);

ListGridField entryDate = new ListGridField("entryDate", "Entry date");
entryDate.setFilterEditorType(new MiniDateRangeItem());
entryDate.setType(ListGridFieldType.DATE);
ListGridField invoicingDate = new ListGridField("invoicingDate", "Invoicing date");
invoicingDate.setFilterEditorType(new MiniDateRangeItem());
invoicingDate.setType(ListGridFieldType.DATE);

// And a bunch of other fields

invoiceGrid.setFields(entryDate, invoicingDate);

1. User enters the tab. The grid does not load data yet.
2. User clicks the Invoicing Date filter field. Date range picker dialog opens.
3. User picks a "from date" from the calendar component (2013-03-05) and exits the prompt by clicking OK.
4. User clicks the Entry Date filter field. Date range picker dialog opens.
5. User selects "from date" as Today from the Relative date combobox.
6. User selects "to date" as Tomorrow from the Relative date combobox.

7. User initiates fetch by clicking the filter button or by pressing Enter.

First request

{
    "dataSource":"ledgerInvoiceList", 
    "operationType":"fetch", 
    "componentId":"isc_MyListGrid_4", 
    "data":{
        "operator":"and", 
        "criteria":[
            {
                "operator":"and", 
                "criteria":[
                    {
                        "fieldName":"invoicingDate", 
                        "operator":"greaterOrEqual", 
                        "value":"2013-03-05"
                    }
                ]
            }, 
            {
                "operator":"and", 
                "criteria":[
                    {
                        "fieldName":"dueDate", 
                        "operator":"greaterOrEqual", 
                        "value":"2013-03-04T22:00:00"
                    }, 
                    {
                        "fieldName":"dueDate", 
                        "operator":"lessOrEqual", 
                        "value":"2013-04-05T20:59:59"
                    }
                ]
            }
        ], 
        "csrfToken":"BpNZxu6m1kC0lz7By3KgtQgk9E7owN4GvtOe4zVesAQ", 
        "isc_metaDataPrefix":"_", 
        "isc_dataFormat":"xml"
    }, 
    "sortBy":[
        "-invoiceNumber"
    ], 
    "textMatchStyle":"substring", 
    "resultSet":[ResultSet ID:isc_ResultSet_4 (created by: isc_MyListGrid_4)], 
    "callback":{
        "caller":[ResultSet ID:isc_ResultSet_4 (created by: isc_MyListGrid_4)], 
        "methodName":"fetchRemoteDataReply"
    }, 
    "willHandleError":true, 
    "showPrompt":true, 
    "prompt":"Haetaan tietoa...", 
    "clientContext":{
        "requestIndex":1
    }, 
    "requestId":"ledgerInvoiceList$62714"
}

Invalid set of invoices is returned.
User does not change any values in the filters and clicks the filter button.

Second request

{
    "dataSource":"ledgerInvoiceList", 
    "operationType":"fetch", 
    "componentId":"isc_MyListGrid_4", 
    "data":{
        "operator":"and", 
        "criteria":[
            {
                "fieldName":"invoicingDate", 
                "operator":"greaterOrEqual", 
                "value":"2013-03-05"
            }, 
            {
                "fieldName":"dueDate", 
                "operator":"greaterOrEqual", 
                "value":"2013-03-04T22:00:00"
            }, 
            {
                "fieldName":"dueDate", 
                "operator":"lessOrEqual", 
                "value":"2013-04-05T20:59:59"
            }
        ], 
        "csrfToken":"BpNZxu6m1kC0lz7By3KgtQgk9E7owN4GvtOe4zVesAQ", 
        "isc_metaDataPrefix":"_", 
        "isc_dataFormat":"xml"
    }, 
    "sortBy":[
        "-invoiceNumber"
    ], 
    "textMatchStyle":"substring", 
    "resultSet":[ResultSet ID:isc_ResultSet_4 (created by: isc_MyListGrid_4)], 
    "callback":{
        "caller":[ResultSet ID:isc_ResultSet_4 (created by: isc_MyListGrid_4)], 
        "methodName":"fetchRemoteDataReply"
    }, 
    "willHandleError":true, 
    "showPrompt":true, 
    "prompt":"Haetaan tietoa...", 
    "clientContext":{
        "requestIndex":2
    }, 
    "requestId":"ledgerInvoiceList$62715"
}

The criteria is in a different form. This time it is in a form which our backend understands. The invoices are returned correctly.

Problem 1: Any ideas what might be causing this?

Problem 2: The weird values in the date fields are another problem (2013-04-05T20:59:59 and 2013-03-04T22:00:00). I bet they've got something to do with the values we set into the DateUtil.

DateUtil.setShortDateDisplayFormatter(new FinnishDateFormat(true));
DateUtil.setNormalDateDisplayFormatter(new FinnishDateFormat(false));
DateUtil.setShortDatetimeDisplayFormatter(new FinnishDateFormat(false));
DateUtil.setDateInputFormatter(new FinnishDateFormat(false));
DateUtil.setDefaultDisplayTimezone("+02:00");
DateUtil.setAdjustForDST(true);

// FinnishDateFormat is formatting with these formats:
DateTimeFormat dtf = DateTimeFormat.getFormat("dd.MM.yyyy HH:mm:ss");
DateTimeFormat df = DateTimeFormat.getFormat("dd.MM.yyyy");

Are we doing something wrong?

Thanks

[kimf]

One of our testers noticed that if they submit a simple search which does not contain any date filters as the first search after opening the page, the filters work as expected. The date filters can then be added to the query and the criteria is produced as expected (no nested criterias).

Why are there nested criterias when there shouldn't be any real need for nesting?

[kimf]

Another case:

1. Tab opened.
2. Searched with simple criteria. TextField invoiceNumber, "5" as value.
3. Data block of the request looks like this (Valid):

    "data":{
        "invoiceNumber":"15", 
        "csrfToken":"V5hGAItIkdwcPrSVzmsu_jk0L6kVS8w6P1CEV7d1V3Y", 
        "isc_metaDataPrefix":"_", 
        "isc_dataFormat":"xml"
    }

4. Cleared the invoiceNumber filter value.
5. Added filter entryDate >= 2013-03-14. (From date value picked from the calendar component)
6. Data block of the request looks like this (Invalid nested criteria):

    "data":{
        "operator":"and", 
        "criteria":[
            {
                "operator":"and", 
                "criteria":[
                    {
                        "fieldName":"entryDate", 
                        "operator":"greaterOrEqual", 
                        "value":"2013-03-14"
                    }
                ]
            }
        ], 
        "csrfToken":"V5hGAItIkdwcPrSVzmsu_jk0L6kVS8w6P1CEV7d1V3Y", 
        "isc_metaDataPrefix":"_", 
        "isc_dataFormat":"xml"
    },

7. Add the invoiceNumber = 15 filter again.
8. Data block of the request looks like this (Valid again):

    "data":{
        "operator":"and", 
        "criteria":[
            {
                "fieldName":"invoiceNumber", 
                "operator":"iContains", 
                "value":"15"
            }, 
            {
                "fieldName":"entryDate", 
                "operator":"greaterOrEqual", 
                "value":"2013-03-14"
            }
        ], 
        "csrfToken":"V5hGAItIkdwcPrSVzmsu_jk0L6kVS8w6P1CEV7d1V3Y", 
        "isc_metaDataPrefix":"_", 
        "isc_dataFormat":"xml"
    },

After that we can add as many filter parameters as we like and it does not produce nested criterion.

[Isomorphic]

These are all valid criteria, and they are also equivalent.

You may find that 3.1 (which you should upgrade to regardless) reliably produces simplified criteria.

However, note that it would not be a bug if the behavior is the same - you should just upgrade your backend logic to reliably deal with AdvancedCriteria rather than just certain limited cases.

[kimf]

The answer is something I expected.

How many levels of nested criterion should the backend be able to handle? I haven't seen this documented anywhere.

Does SmartGWT server provide logic that could be used in our plain JavaEE backend for parsing the AdvancedCriteria?

[kimf]

Oh, the AdvancedCriteria handling feature is not even available in the Pro version. :|

[Isomorphic]

There's no limit to nesting with AdvancedCriteria. Generally speaking, one writes a recursive algorithm rather than hard coding two or three levels..

Pro will parse the AdvancedCriteria and provide them to you as a nested structure for execution by your own logic (see DSRequest.getAdvancedCriteria).

Power and above will automatically execute AdvancedCriteria against SQL, Hibernate or JPA.

[kimf]

Okay, thanks. We'll modify the backend logic.
One thing needs to be clarified. Can there be many criteria elements on the same level of depth?

Eg. Now we've seen this:

    "data":{
        "operator":"and", 
        "criteria":[
            {
                "operator":"and", 
                "criteria":[
                    {
                        "fieldName":"entryDate", 
                        "operator":"greaterOrEqual", 
                        "value":"2013-03-14"
                    }
                ]
            }
        ], 
        "csrfToken":"V5hGAItIkdwcPrSVzmsu_jk0L6kVS8w6P1CEV7d1V3Y", 
        "isc_metaDataPrefix":"_", 
        "isc_dataFormat":"xml"
    },

.. but could there also be cases like this? (bolded part is made up):

    "data":{
        "operator":"and", 
        "criteria":[
            {
                "operator":"and", 
                "criteria":[
                    {
                        "fieldName":"entryDate", 
                        "operator":"greaterOrEqual", 
                        "value":"2013-03-14"
                    }
                ],
                [B]"criteria":[
                    {
                        "fieldName":"entryDate", 
                        "operator":"lessOrEqual", 
                        "value":"2013-04-14"
                    }
                ],
                "criteria":[
                    {
                        "fieldName":"invoiceNumber", 
                        "operator":"iContains", 
                        "value":"12345"
                    }
                ][/B]
            }
        ], 
        "csrfToken":"V5hGAItIkdwcPrSVzmsu_jk0L6kVS8w6P1CEV7d1V3Y", 
        "isc_metaDataPrefix":"_", 
        "isc_dataFormat":"xml"
    },

We'll be in touch regarding the license upgrade.

Thanks.

[Isomorphic]

No, that can't happen.

It's actually also not valid JSON, or at any rate, the last criteria property in a given object would be the only value present in the object.