AdvancedCriteria Date format inconsistent.

[stonebranch1]

In testing out my filtering using Date fields (you previously addressed an issue with parsing/formatting of DateTime for me http://forums.smartclient.com/showthread.php?t=30218) I found one case where the stored value is not adhering to my format. When I use the between operator it is storing the dates in the ISO 8601 format rather than my defined format.

    "criteria":[
        {
            "fieldName":"tdate", 
            "operator":"equals", 
            "value":"2014-10-13"
        }, 
        {
            "operator":"iBetweenInclusive", 
            "fieldName":"tdate", 
            "start":"2014-12-01T05:00:00.000", 
            "end":"2015-01-01T04:59:59.999"
        }, 
        {
            "fieldName":"tdate", 
            "operator":"notEqual", 
            "value":"2014-05-31"
        }, 
        {
            "fieldName":"tdate", 
            "operator":"lessThan", 
            "value":"2016-05-01"
        }, 
        {
            "fieldName":"tdate", 
            "operator":"lessOrEqual", 
            "value":"2015-06-04"
        }, 
        {
            "fieldName":"tdate", 
            "operator":"greaterThan", 
            "value":"2014-05-01"
        }, 
        {
            "fieldName":"tdate", 
            "operator":"greaterOrEqual", 
            "value":"2014-05-13"
        }, 
        {
            "operator":"isNull", 
            "fieldName":"tdate"
        }, 
        {
            "operator":"notNull", 
            "fieldName":"tdate"
        }
    ]

Is this because the Between is wanting to set the start of day and end of day of the two dates respectively? Is it required to have those times included since this is just a Date field or should the values still be formatted using my specified formatter (in this case yyyy-MM-dd)?

[Isomorphic]

What are we looking at here? Server logs? RestDataSource capture? JSONEncoder serialization?

Note that none of the above representations would be affected by settings on DateUtil, if that's what you were expecting. DateUtil settings affect only values shown to end users.

[stonebranch1]

Sorry, this is the saved Filter criteria created by the FilterBuilder.

We allow creation and saving of different filters via the FilterBuilder and this is the saved filter for each type of filter operation we allow. (see attached image)

[Isomorphic]

That didn't actually answer the question - how was this particular JSON serialization obtained?

[stonebranch1]

It is persisted as a string via:

AdvancedCriteria criteria = filterBuilder.getCriteria();
String filter = criteria.toJSON();

Can you please confirm if you see the same in your tests or your environment?

[Isomorphic]

So again, the formatters you install on DateUtil affect user-visible values only and would not be expected to affect JSON serialization.

The values shown for betweenInclusive are as expected - this is how we generate criteria that meets the user's expectation of what it means to pick dates between a given day and another: that it should include everything from the beginning of the first day to the end of the second.

[stonebranch1]

Which is what I asked at the very beginning about the times.

"Is this because the Between is wanting to set the start of day and end of day of the two dates respectively?"

Thanks.

[Isomorphic]

Yeah, we saw that, but we first had to clear up the confusion about not following your "defined" format (and course know where this string was from!).

[stonebranch1]

Understood, thanks for clearing it up.

[stonebranch1]

Is there anyway I can control the JSON format for the Between filter operator to only have the Date as the stored value?

Reason I ask because it causes problems with timezones for our particular application. I will attempt to explain...

Users in our system are defined to be in a specific TimeZone. So for example Joe can be defined to be in Central TZ and Jane can be defined to be in Pacific TZ. (Not my preference but the legacy application supports this)

To handle this we send all the Dates to the server as String (yyyy-MM-dd) so that no timezone information is contained and hence a Date is the same Date regardless of what timezone the user or browser is in. For DateTimes we send Date objects back/forth but we format/parse all our DateTimes to show TimeZone so that there is never any ambiguity w/r to the DateTime and we always display the DateTime in the User's specified TimeZone.

So a DateTime of say 2014-07-01 at 11:00 Central is stored in UTC as 2014-07-01 16:00:00. When this DateTime is displayed in the UI for Joe, he will see "2014-07-01 11:00:00 -0500" regardless of the browser timezone, so even if Joe is on a system in Pacific TimeZone, he will still see his DateTimes in Central Time) and Jane will see "2014-07-01 09:00:00 -0700" (again regardless of browser TimeZone)

This is all working fine and dandy until we come to filtering. The Date field filtering is working OK except for the Between since the Between is adding the Time information. Soon as Time information is added, TimeZone comes into play.

The TimeZone being used to set the time value for a Date Between criteria appears to be using our User's timezone probably due to our parsing/formatting which is ok (Joe's Between filter shows T05:00 and Jane's filter shows T07:00 on between date filters).

However, I am seeing very strange behaviour when the user and their system are different timezones. Seems like the browser timezone is coming into play somehow. Jane who is say on an Eastern TimeZone system, sets a between Date to say July 31 & Dec 31 and when the filter is loaded up in the FilterBuilder, the filter shows 2 different dates of Jan 1 on the input field and Jan 2 on the display/label field (see attached image) where the filter has the value of:

{
    "_constructor":"AdvancedCriteria", 
    "operator":"and", 
    "criteria":[
        {
            "operator":"iBetweenInclusive", 
            "fieldName":"tdate", 
            "start":"2014-07-31T07:00:00.000", 
            "end":"2015-01-01T07:59:59.999"
        }
    ]
}

Likewise, say Joe is running on a system in Pacific time and he creates the same filter resulting in a filter defined as below, his start dates are incorrectly shown as both July 30th & July 29th even though the actual Date is July 31.

{
    "_constructor":"AdvancedCriteria", 
    "operator":"and", 
    "criteria":[
        {
            "operator":"iBetweenInclusive", 
            "fieldName":"tdate", 
            "start":"2014-07-31T05:00:00.000", 
            "end":"2015-01-01T05:59:59.999"
        }
    ]
}

Since our custom datasource handles the filtering we would be fine if we could have Time portion not actually stored in the JSON format of a Date. Remember I am only talking about Date types here w/r the Between and the Filter ( not DateTime fields) and the documentation also implies there should not be any time information:

When using the Smart GWT server framework, "date" values are automatically transmitted with year, month and day preserved and time value ignored.

When sent or received in XML or JSON, date field values should be serialized in the XML Schema date format - YYYY-MM-DD - are expected to be received in the same format. Any time value present for a "date" field is ignored.

I understand why the times are added for the Between operator on a Date field, but if a Date is really a Date then are those Times really needed in the filter criteria since the Date fields should not have a time associated with them?

Thanks.

[Isomorphic]

What you really want to do is throw away all your code for trying to deal specially with timezones. The framework already handles your use case - date and time information is conveyed unambiguously between client and server, and setDefaultDisplayTimezone allows you to configure all client-side UI components to use a specific timezone for display and input.

If you want to continue with your current (unnecessary) approach, the only way to influence JSON serialization to do what you want is to simply convert the embedded Date instances to your own desired String format *before* you ask for JSON serialization, in which case they will remain unchanged.

But we highly recommend getting rid of your unnecessary custom date/time handling instead.

[stonebranch1]

The only code we have for dealing with TimeZones is basically setting the default display time zone via setDefaultDisplayTimezone so we are in fact calling that method and then the only other thing we do on the client is parse/format DateTime fields to show the TimeZone.

So I am not sure what unnecessary approach you are referring to?

Look at the screen images and view the filter. That is just a FilterBuilder using a Date field where the setDefaultDisplayTimezone has been set to a TimeZone that is different from the browser TimeZone and viola different dates appear. The ONLY code of ours in play for the Date field is in fact the call we made to setDefaultDisplayTimezone.

I will create a simple code to reproduce it for you that wil show just calling setDefaultDisplayTimezone to a different TimeZone of the browser causes the FilterBuilder Between operator of a Date to show incorrectly on the UI.

[Isomorphic]

Your system for serializing date/datetime values as Strings in an attempt to avoid timezone shifts is what's unnecessary. Including whatever server-side code you've put in place to re-convert back to Dates. This is already handled properly by the framework.

That said, if you can show an issue with valid usage in a standalone test case, we'll of course correct it.

[stonebranch1]

Here is a sample which shows even more of a problem in that the restore of the filter doesn't even show the proper values at all. I don't have that problem which maybe due to the fact that I have parsing/formatting setup in my real application where as this sample is just using the defaults.

Anyway, select the between operator and select two explicit dates and then click the "Refresh" button. In my environment the Dates flip back (the problem I don't see in my application) and one of the dates (depending on your timezone) will show a different value in the input field from the display field (the problem I do see in my real application). See attached image.

public class Sandbox1 implements EntryPoint {

    @Override
    public void onModuleLoad() {
        // set user to Central TimeZone (make sure this timezone is different than browser).
        DateUtil.setDefaultDisplayTimezone("-05:00");

        final Layout layout = new VLayout(10);

        DataSource dataSource = new DataSource();
        dataSource.setID("xxxDS");
        DataSourceField dateField = new DataSourceField("adate", FieldType.DATE, "Date");
        dataSource.setFields(dateField);
        dataSource.setClientOnly(true);

        final FilterBuilder fb = new FilterBuilder();
        fb.setTopOperatorAppearance(TopOperatorAppearance.RADIO);
        fb.setTopOperator(LogicalOperator.AND);
        fb.setShowModeSwitcher(Boolean.TRUE);
        fb.setDataSource(dataSource);

        final Label label = new Label();

        IButton clear = new IButton("Clear");
        clear.addClickHandler(new ClickHandler() {
            @Override
            public void onClick(ClickEvent event) {
                fb.clearCriteria();
            }
        });

        IButton refresh = new IButton("Refresh");
        refresh.addClickHandler(new ClickHandler() {
            @Override
            public void onClick(ClickEvent event) {
                AdvancedCriteria ac = fb.getCriteria();
                if (ac == null) return;
                String criteria = ac.toJSON();
                label.setContents("Criteria: " + criteria);
                GWT.log("Criteria: " + criteria);
                fb.setCriteria(AdvancedCriteria.fromJSON(criteria));
            }
        });

        layout.addMember(label);
        layout.addMember(fb);
        HLayout buttonLayout = new HLayout(5);
        buttonLayout.addMember(clear);
        buttonLayout.addMember(refresh);
        layout.addMember(buttonLayout);
        layout.draw();

    }

}

[Isomorphic]

What's happening is that to/fromJSON cannot automatically round-trip date values, because the JSON format itself has no ability to represent date values.

Take a look at JSONEncoder as suggested in the main AdvancedCriteria docs - it supports a special not-quite-JSON format that can round-trip date values successfully.