Announcement

**Blama** · 3 Aug 2023, 16:43

Hi tece321,

Try adding a “notEqual randomValue”-criteria in the
FormItemCriteriaFunction of the FormItem1/2, that you want to always reload.

Also, for debugging purposes (and seeing all the requests in the Developer Console RPC Tab), try to disable client side filtering in all the relevant SelectItem/ComboBoxItem’s ResultSet properties.

Also, you could set the logging level for relevant categories in the Developer Console to info or debug. That might give more insights as well.

Best regards
Blama

**Isomorphic** · 3 Aug 2023, 19:27

We're puzzled by this question - if the criteria returned by your setPickListFilterCriteriaFunction differs, and you open the SelectItem, there should be a new fetch. You don't need to do anything manually, just the fact that the criteria differ is enough. You can see this working here:

https://smartclient.com/smartgwt/sho...bobox_category

The only purpose for which you might want a change/changed handler is to clear the value in the dependent select (as the sample above shows).

What may be happening here is that your SelectItem / DataSource is set up so that, at some point, it does a fetch with no criteria or with limited criteria, gets a result, and then when further criteria are added, filtering is then done against the local data, because the criteria is additive, so the system (correctly) assumes that further filtering can be performed locally.

Rather than bludgeon the caching system here, you just want to arrange things so that the criteria and responses match up.

For example, if you put in some code that makes it so that in the absence of criteria, the server returns "something reasonable" or perhaps knows the real criteria via some other means, that's going to fool SmartGWT's caching system, because the caching system is going to believe that whatever was returned in the absence of criteria must be the entire available data set.

What you definitely do not want to do is what Blama said, that is, turn off client-side filtering (sorry Blama - on the whole we really appreciate your help!). There's no apparent need to do this here, and our Adaptive Filtering system is one of the main reasons that SmartGWT is so much faster than anything else!

**Blama** · 4 Aug 2023, 01:40

Hi Isomorphic,

all good, I wrote "for debugging purposes". Sometimes it can be frustrating not to see what the framework is doing behind the scenes. Disabling client side filtering/sorting will make every criteria change show up in the RPC Tab for sure.

And what I meant with “notEqual randomValue” is described in this thread here. It's to force a fetch every time a SelectItem is opened. But I agree, this does not seem necessary from the requirements described in #1.

Best regards
Blama

**tece321** · 4 Aug 2023, 13:24

Originally posted by Isomorphic View Post

We're puzzled by this question - if the criteria returned by your setPickListFilterCriteriaFunction differs, and you open the SelectItem, there should be a new fetch. You don't need to do anything manually, just the fact that the criteria differ is enough. You can see this working here:

https://smartclient.com/smartgwt/sho...bobox_category

What you describe, the setPickListFilterCriteriaFunction changes when the Block_Name select Item changes, so I expected that it would 'just work', i.e. invalidate the cache as the criteria changed. But that's not happening. In short, just as the demo shows, I expected this to just work.

Could there be some form of the AdvancedCriteria that the FormItem cache is NOT detecting as changing? I will add the "notEqual randomValue" debug code and see if that works.
Is there some option ( don't think I'm using any ) of the Select/FormItem that could suppress the cache algorithm?

Won't get to this until next week.

**Isomorphic** · 4 Aug 2023, 13:47

No, there wouldn't be a form of criteria change that the framework would be unable to detect. It's basically a recursive diff. So pretty much the only explanation we have here is that there is a previous fetch with less restrictive criteria, causing the caching system to believe that it already has all the data required to fulfill new, narrower criteria.

Introducing an additional, randomly-generated criterion will indeed force a fetch every time, but that would be masking whatever is the actual problem.

The place we would start would be to look at the RPC tab in the Developer Console and see if there is an earlier fetch that has either no criteria or partial criteria (such that later fetches would seem to be a subset of those results).

Then, if you end up wondering how that fetch could occur, traceLogMessage() is a very powerful tool for figuring that out:

https://smartclient.com/smartclient-...raceLogMessage

**tece321** · 9 Aug 2023, 13:19

Am not making progress on resolving this issue.

1. If I change the code to always add a random criteria to every setPickListFilterCriteriaFunction() call, it does 'force' a fetch on every click of the select icon.
So in that code I am always - hacking - in a criteria OperatorId.NOT_IN_SET, new String[] { random_string} .

2. If I remove that 'random' hack and generate a request with this criteria:

Code:

dataSource:"BCG_STA_Runs",
    operationType:"fetch",
    operationId:"fetchSTACVT",
    componentId:"isc_PickListMenu_0",
    data:{
        operator:"and",
        criteria:[
            {
                fieldName:"PK_Project",
                operator:"equals",
                value:917
            },
            {
                fieldName:"masterGridID",
                operator:"equals",
                value:"projectTimingWaiverSum"
            },
            {
                operator:"inSet",
                fieldName:"Block_Name",
                value:[
                    "acebiu_ccb",
                    "mc2_68880_top_wrapper",
                    "pcie_ubus_x2x2x1x1g3_top_wrapper"
                ]
            },
            {
                fieldName:"FEorBE",
                operator:"equals",
                value:"Frontend"
            }
        ]

The first time I have selected with these 3 blocks in the SET, the fetch occurs and the data is returned.

But if I change the Block Name selections to say exclude the block named "acebiu_ccb" there is NO fetch of the select data and the drop down returns NULL :

In this failing case, by stepping through the code of the setPickListFilterCriteriaFunction() call I see it generate a criteria like this:

Code:

dataSource:"BCG_STA_Runs",
    operationType:"fetch",
    operationId:"fetchSTACVT",
    componentId:"isc_PickListMenu_0",
    data:{
        operator:"and",
        criteria:[
            {
                fieldName:"PK_Project",
                operator:"equals",
                value:917
            },
            {
                fieldName:"masterGridID",
                operator:"equals",
                value:"projectTimingWaiverSum"
            },
            {
                operator:"inSet",
                fieldName:"Block_Name",                        
                value:[ 
                    "mc2_68880_top_wrapper",                                       // note only difference is that    "acebiu_ccb" is not in the SET now
                    "pcie_ubus_x2x2x1x1g3_top_wrapper"
                ]
            },
            {
                fieldName:"FEorBE",
                operator:"equals",
                value:"Frontend"
            }
        ]

The above is what I expect the non issued RPCRequest would look like.

I could understand if the drop down showed values, the same perhaps as originally shown, but I can't understand why it is blank ??

It is possible in the UI to indicate from the Block Name selector that one wishes to generate a NOT_IN_SET operator. When that happens the RPC request IS issued and is generated like this:

Code:

dataSource:"BCG_STA_Runs",
    operationType:"fetch",
    operationId:"fetchSTACVT",
    componentId:"isc_PickListMenu_0",
    data:{
        operator:"and",
        criteria:[
            {
                fieldName:"PK_Project",
                operator:"equals",
                value:917
            },
            {
                fieldName:"masterGridID",
                operator:"equals",
                value:"projectTimingWaiverSum"
            },
            {
                operator:"NotinSet",
                fieldName:"Block_Name",                        
                value:[
                    "acebiu"                                  
                ]
            },
            {
                fieldName:"FEorBE",
                operator:"equals",
                value:"Frontend"
            }
        ]

But eventually I can generate a NotInSet criteria choice which will again cause the dropdowns to show "No items to show". So a criteria of

Code:

blah,blah......
            {
                operator:"NotinSet",
                fieldName:"Block_Name",
                value:[
                     "acebiu_ccb"             
                ]
            },
etc....

followed by attempting to generate a criteria like:

Code:

blah,blah......
            {
                operator:"NotinSet",
                fieldName:"Block_Name",
                value:[
                     "acebiu_ccb",
                    "mc2_68880_top_wrapper",               
                ]
            },
etc....

Will again cause the drop down to be NULL or show "No items to show".

**Isomorphic** · 9 Aug 2023, 14:46

The solution here is bound to cause a facepalm, we're afraid...

You have an operatorId of "NotinSet". The correct operatorId is "notInSet" (they all start with lowercase and are camelCaps).

We checked, and the SmartGWT constant for this operator (OperatorId.NOT_IN_SET) is correct. So this bad operatorId presumably had to do with the "hacking" of the criteria you referred to above - you presumably didn't use the constant.

If there was something about the AdvancedCriteria API that led to hacking something, please let us know, as we'd like to avoid anyone bypassing the constants in the future.

Some tips for going right to the solution in the future:

1. if this request had been allowed to go to the server, the server-side logs would tell you something like "invalid operatorId: NotinSet"

2. for the client-side, you can turn on the "ResultSet" log category to see all kinds of information about filtering.

Here also, you'd see complaints about invalid operatorIds (in fact, that looks to be a WARN-level log, so it's probably there by default, without enabling any special logging).

However, even better, the ResultSet logging will show you exactly what's going on with filtering. So you'll see messages like "criteria is more restrictive, proceeding with client-side filtering" and so forth.

If there's any way the logging can be improved here (client or server side), please let us know!

**tece321** · 10 Aug 2023, 06:28

A distraction. I copied and incorrectly modified the examples in my previous posting. It is using the correct OperatorId.NOT_IN_SET, and this correctly translates to the request criteria
of :

Code:


blah,blah......

      { operator:"notInSet",
         fieldName:"Block_Name",
         value: [
                      "acebiu_ccb",
                      "mc2_68880_top_wrapper"
                      ]
          },

    etc....

What I was trying to demonstrate was that if a fetch request was issued with this criteria:

Code:


blah,blah......

      { operator:"notInSet",
         fieldName:"Block_Name",
         value: [
                      "acebiu_ccb"
                      ]
          },

    etc....

and then changed to ADD a block name in the criteria:

Code:


blah,blah......

      { operator:"notInSet",
         fieldName:"Block_Name",
         value: [
                      "acebiu_ccb",
                      "mc2_68880_top_wrapper"
                      ]
          },

    etc....

That this SECOND criteria did NOT invalidate the SelectItem cache, and did NOT generate another fetch, and caused the pulldown to appear with "No items to show"

I will add ResultSet logging and see if it tells me more....

**tece321** · 10 Aug 2023, 06:47

Here is an example, with ResultSet debug information of using the OperatorId.IN_SET, and the SelectItem dropdown ending up with "No items to show'

First I generate a criteria where all 3 possible blocks are chosen, i.e. I select ALL of the Block Name SelectItem values:

Code:

dataSource:"BCG_STA_Runs",
    operationType:"fetch",
    operationId:"fetchSTACVT",
    componentId:"isc_PickListMenu_3",
    data:{
        operator:"and",
        criteria:[
            {
                fieldName:"PK_Project",
                operator:"equals",
                value:917
            },
            {
                fieldName:"masterGridID",
                operator:"equals",
                value:"projectTimingWaiverSum"
            },
            {
                operator:"inSet",
                fieldName:"Block_Name",
                value:[
                    "acebiu_ccb",
                    "pcie_ubus_x2x2x1x1g3_top_wrapper",
                    "mc2_68880_top_wrapper"
                ]
            },
            {
                fieldName:"FEorBE",
                operator:"equals",
                value:"Frontend"
            }
        ]

This results in DEBUG level logging of:

Code:

10:32:38.840:MDN5:DEBUG:ResultSet:isc_ResultSet_10 (dataSource: BCG_STA_Runs, created by: isc_PickListMenu_3):getRange: no scrolling direction detected
10:32:38.841:MDN5:INFO:ResultSet:isc_ResultSet_10 (dataSource: BCG_STA_Runs, created by: isc_PickListMenu_3):getRange(0, 125) will fetch from 0 to 125
10:32:38.841:MDN5:INFO:ResultSet:isc_ResultSet_10 (dataSource: BCG_STA_Runs, created by: isc_PickListMenu_3):fetching rows 0,125 from server
10:32:39.982:XRP2:INFO:ResultSet:isc_ResultSet_10 (dataSource: BCG_STA_Runs, created by: isc_PickListMenu_3):Received 14 records from server
10:32:39.983:XRP2:DEBUG:ResultSet:isc_ResultSet_10 (dataSource: BCG_STA_Runs, created by: isc_PickListMenu_3):full length set to: 14
10:32:39.983:XRP2:DEBUG:ResultSet:isc_ResultSet_10 (dataSource: BCG_STA_Runs, created by: isc_PickListMenu_3):integrating 14 rows into cache at position 0
10:32:39.983:XRP2:INFO:ResultSet:isc_ResultSet_10 (dataSource: BCG_STA_Runs, created by: isc_PickListMenu_3):Fetch request returned range 0,14 differs from requested range 0,125. Assuming client/server batch size mismatch and clearing loading markers greater than 14
10:32:39.984:XRP2:INFO:ResultSet:isc_ResultSet_10 (dataSource: BCG_STA_Runs, created by: isc_PickListMenu_3):cached 14 rows, from 0 to 14 (14 total rows, 14 cached)
10:32:39.984:XRP2:INFO:ResultSet:isc_ResultSet_10 (dataSource: BCG_STA_Runs, created by: isc_PickListMenu_3):Cache for current criteria complete

Now I DROP the "acebiu_ccb" from the criteria list. I EXPECT this to invalidate the cache, but it does NOT. I EXPECT this RPCRequest ( but it is NOT generated ):

Code:

dataSource:"BCG_STA_Runs",
    operationType:"fetch",
    operationId:"fetchSTACVT",
    componentId:"isc_PickListMenu_3",
    data:{
        operator:"and",
        criteria:[
            {
                fieldName:"PK_Project",
                operator:"equals",
                value:917
            },
            {
                fieldName:"masterGridID",
                operator:"equals",
                value:"projectTimingWaiverSum"
            },
            {
                operator:"inSet",
                fieldName:"Block_Name",
                value: 
                    "pcie_ubus_x2x2x1x1g3_top_wrapper",
                    "mc2_68880_top_wrapper"
                ]
            },
            {
                fieldName:"FEorBE",
                operator:"equals",
                value:"Frontend"
            }
        ]

This is what the console debugging reports:

Code:

10:33:19.529:DEBUG:ResultSet:isc_ResultSet_10 (dataSource: BCG_STA_Runs, created by: isc_PickListMenu_3):getRange(0, 14) satisfied from cache
10:33:19.562:INFO:ResultSet:isc_ResultSet_10 (dataSource: BCG_STA_Runs, created by: isc_PickListMenu_3):setCriteria: filter criteria changed, performing local filtering
10:33:19.563:DEBUG:ResultSet:isc_ResultSet_10 (dataSource: BCG_STA_Runs, created by: isc_PickListMenu_3):getRange(0, 0): returning empty list

So it detects that the filter criteria has changed, but it is "performing local filtering" and it returns an "empty list".

PLEASE NOTE THAT I HAVE CREATED THE RPCRequest above, as an example, but it is NOT generated by the framework.

If the criteria has changed what makes it believe that it can perform local filtering that returns an empty list?

**tece321** · 10 Aug 2023, 10:09

I got this to finally work.

The change I made was to stop using (1) OperatorId.InSet i.e. the include set, and (2) Operator.NOT_IN_SET i.e. the exclude set, and to replace these with

1: OR and Equals criteria: Example:

Code:

dataSource:"BCG_STA_Runs",
    operationType:"fetch",
    operationId:"fetchSTACVT",
    componentId:"isc_PickListMenu_3",
    data:{
        operator:"and",
        criteria:[
            {
                fieldName:"PK_Project",
                operator:"equals",
                value:917
            },
            {
                fieldName:"masterGridID",
                operator:"equals",
                value:"projectTimingWaiverSum"
            },
            {
                operator:"or",
                criteria:[
                    {
                        fieldName:"Block_Name",
                        operator:"equals",
                        value:"acebiu_ccb"
                    },
                    {
                        fieldName:"Block_Name",
                        operator:"equals",
                        value:"mc2_68880_top_wrapper"
                    },
                    {
                        fieldName:"Block_Name",
                        operator:"equals",
                        value:"pcie_ubus_x2x2x1x1g3_top_wrapper"
                    }
                ]
            },
            {
                fieldName:"FEorBE",
                operator:"equals",
                value:"Frontend"
            }
        ]

2: AND along with NOT_EQUAL criteria.

Code:

 dataSource:"BCG_STA_Runs",
    operationType:"fetch",
    operationId:"fetchSTACVT",
    componentId:"isc_PickListMenu_3",
    data:{
        operator:"and",
        criteria:[
            {
                fieldName:"PK_Project",
                operator:"equals",
                value:917
            },
            {
                fieldName:"masterGridID",
                operator:"equals",
                value:"projectTimingWaiverSum"
            },
            {
                fieldName:"Block_Name",
                operator:"notEqual",
                value:"acebiu_ccb"
            },
            {
                fieldName:"FEorBE",
                operator:"equals",
                value:"Frontend"
            }
        ]

So can I conclude that the use of the operators: OperatorId.NOT_IN_SET or OperatorId.IN_SET , causes the SelectItem to detect criteria changes but not attempt to refetch the data?

**Isomorphic** · 10 Aug 2023, 10:25

In post #10 above, you go from an IN_SET operator with 3 possible values to an IN_SET operator with only two possible values. So, criteria are now more restrictive, since there must be fewer records that match any of 3 records than the number of records that match only two possible values (which are a subset of the 3).

This is the basic premise of Adaptive Filtering, as explained in the ResultSet docs, samples etc (let us know if you were not already familiar): if we have a full set of matching records for a given criteria, and the criteria becomes more restrictive, then we can filter client-side.

So seeing client-side filtering in this case is exactly what is expected (and is extremely, extremely desirable!).

Now, the only question is: why did client-side filtering return 0 records? You seem to be saying there should be matches, and possibly that the same criteria on the server would cause matches (but we've never seen any sample data, so we don't know).

Generally speaking, the most common reason for the client and server filtering to disagree is that the data delivered to the client is incomplete, or not the same as the original data.

For example, values for the Block_Name field might be totally omitted for client data (since someone thought it might only be needed on the server), or, it might be formatted in some way, such as uppercasing it or adding a prefix. When this is done, clearly client filtering will not match server filtering. This is why we recommend that any formatting is done client-side, which both reduces server load, increasing performance, and also prevents problems when client-edited data is saved, for example.

Announcement

SelectItem, force cache invalidate on Criteria change

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment