Foreign key elements are fetched twice in my form

[mathias]

OK, so I've been meaning to ask about this for a long time, but you know what, there's only 24 hrs in a day.

I have a datasource with a foreign key item, like this:

<field name="reporterId" type="integer" title="Användare" foreignKey="user.id" required="false" combo="true"/>

I then have a DynamicForm with a bunch of items, including a comboboxitem for the reporter field.

I then have a list grid with all users, and when I click on the list grid, I populate the items.

Now, I notice that when I click on the first row in the grid and populate the Form for the first time, a fetch is triggered on my User datasource:

<operationConfig xsi:type="xsd:Object">
    <dataSource>user</dataSource><repo xsi:nil="true"/>
    <operationType>fetch</operationType>
    <textMatchStyle>exact</textMatchStyle>
</operationConfig>
<componentId>isc_DefaultForm_0</componentId><
<operation>user_fetch</operation>

I thought this was needed, because of the field described above. However, when I then go in and edit the form, and click on the ComboboxItem for the first time, another fetch is issued!

<operationConfig xsi:type="xsd:Object">
    <dataSource>user</dataSource>
    <repo xsi:nil="true"/>
    <operationType>fetch</operationType>
    <textMatchStyle>substring</textMatchStyle>
</operationConfig>
<componentId>isc_PickListMenu_0</componentId>
<appID>builtinApplication</appID>
<operation>user_fetch</operation>

As you can see, the textMatchStyle is the only thing that's different, but the first time the form component is requesting, and then the picklistmenu component requests again.

This is of course wasteful, is there anything I can do about it? I have the same issue for all forms with a databound foreign key FormItem in it.

[claudiobosticco]

Hi mathias, if I understand correctly, the first fetch is to show the reporter name (so a single record is returned), and the second to show the combobox picklist?
If yes, are you aware of the displayField/foreignDisplayField mechanism?

[mathias]

Hi Claudio, thanks for chiming in!
For reference see below how I create the form with the combobox (simplified).


I did some digging today, and from what I can gather, the two fetches are these:

1. The form fetches all the user rows the first time a record is clicked in the grid and it is a record is to be displayed in the form (by calling editRecord()), since their names are needed to show the text field of the combo box?

2. Then, the first time I edit the form, and click on the combobox field to show the rows to select from, a second user fetch is issued to display the combobox list to select a user from.


I don't really get why two fetches are needed for this? Is there a better way for me to define the combobox so that the data isn't fetched twice?

DataSource ds = DataSource.get(CSConstants.DS_TIMEBANK);

        DefaultForm form = new DefaultForm();

        DataSource userDS = DataSource.get(CSConstants.DS_USER);
        DataSourceField userField = ds.getField(CSConstants.FIELD_REPORTERID);
        reporter = new ComboBoxItem();
        reporter.setTitle(userDS.getTitle());
        reporter.setOptionDataSource(userDS);
        reporter.setValueField(CSConstants.FIELD_ID);
        FormItem note = FormUtils.createLargeTextArea(ds.getField(CSConstants.FIELD_NOTE), 2, 2);

        form.setDataSource(ds, reporter, balance, note);

[Isomorphic]

If you look at the docs for listGridField.optionDataSource, you'll see what's going on, why it's needed, and the pattern to make it more efficient.

In brief, in order to for the grid to display user names instead of just IDs, it needs to fetch the entire user table. You can avoid this by configuring a "displayField" on your FK which is populated by a join to users. Then the grid will be able to display user names without a separate fetch, and the ComboBox fetch will no longer seem "redundant" (it isn't, but we see why you thought it was).

This is all automatic / declarative with our server, however, if we recall correctly, you're using our server but doing your own persistence. So you'll have the extra work of getting a join set up.

[mathias]

In brief, in order to for the grid to display user names instead of just IDs, it needs to fetch the entire user table

Yes, of course, and that's one fetch. I am expecting this.

Of course it must be redundant for the purpose of this SelectItem. The underlying dataset is the same for displaying the current value of the selectitem, and it's possible values of said checkbox when the user is to select one of the items.

Note that no user fetch is used for the ListGrid. The two fetches are *both* for the DynamicForm. For the grid, I have a "username" field defined separately in the datasource, when I just need to display names (like in the grid) to avoid extra fetches. I then have a separate userid field when I need to have user select functions. Had I used the foreign key also in the list grid, it would have been three identical fetches to the same datasource.

--

So, in the grid I use the username returned in the timebank records, then in the form I use the reporterid field to show the select item.

This is why the users aren't fetched until the first time I click on an item in the grid, and the form is to be displayed for the first time, as described above.

This is why I don't get why a second fetch is done by the dynamic form again, when the select item is to pop up. It should already have all the records!

[Isomorphic]

This is now getting a bit vague as to which fetches you consider duplicates of others - in particular it's not clear if you are now talking about "fetchMissingValues" requests vs fetches for sets of records.

But, general guidance: the fact that listGridField.optionDataSource will fetch all records for a DataSource is more of a fallback for when a developer has set things up badly/incompletely. We honestly intend that when you see that fetch, you read the docs and realize the right way to approach things.

If, on the other hand, it really does make sense to fetch all records for a DataSource, you can just set cacheAllData on that DataSources, and that gives you a full-DataSource write-through cache with optional expiration, so you'll never see any component fetch records from it again for the lifetime of the page (until the cache expires, if so configured)

If you still think there are redundant fetches going on, and they aren't handled by the simple setting described above, then please describe what those fetches are (criteria? single record or multi?), and what initiates them.

[mathias]

Hey thanks for responding. As a general response: Oh I might surely have set up the form wrong, as I wrote in the first post, it's a question, not a bug report. My main question is why I see two fetches in my form when I would think I only need one.
I'll try to detail.

1. Setup:
I'm doing "timebank" management for users here. Two datasources, one that my tab is actually for, and a foreign key to the users. Both use Spring MVC bean calls.
Timebank:

<fields>
        <field name="id" type="long" hidden="true" primaryKey = "true"/>
        <field name="reporterId" type="integer" foreignKey="user.id" required="true"/>
        <field name="fullname" type="text" title="Användare" hidden="false" required="false" canSave="false"/>
...more stuff.

User:

<fields>
        <field name="id" type="long" hidden="true" primaryKey="true"/>
        <field name="fullname" type="text" length="64" hidden="false" title="" required="false" canEdit="false" canSave="false"/>
...more stuff

I have created the form as described above, with one item set up as a combobox with the optiondatasource pointing to the user datasource.

2. Grid on initial page load (autoFetchData)
It loads all records from the timebank. This is only the one call to the server since I use "full name" to show the names, no users are fetched


3. When a row is clicked first time, the form is shown for the first time., by calling editRecord on the form


At this point a complete fetch of the user datasource is performed. I suppose it has to do with the select item to be able to show the text field value part of the select item in the form. (even though the select item itself doesn't become visible until the user clicks the "new" button)
The header you see above the form is using "full name" which again is just a field in the datasource and triggers no fetch.

4. When a user clicks on the select item itself, all users are fetched again.
This is what I wonder about. The users have just been fetched for the select item component, and now they are fetched again.

So, yeah, I get the one time, but not two times.

[Blama]

Hi mathias,

you write SelectItem, but I can't see one in the screenshot. Is it only shown for "+" under "Time bank" when you create a new row or also for "Edit" on the right? And if so, can one edit the user?
Can you perhaps add more screenshots?

Best regards
Blama

[mathias]

Indeed:

When you press new, the select item is shown and can be clicked on, since it's only then you should select a user for one you're creating. For the same reason it's not shown on edit since you don't change an existing one (of course can be different in other parts of our system)

(here you can also see the thing where I'm filtering out users already in the grid :) )

Edit: with "edit the user" I assumed you meant changing the user for the time bank. That should not be possible here, each user has a timebank and you don't re-assign it, so we don't show the select item for the edit use case. You can of course edit the user itself, but that's in another tab.

[Blama]

Hi mathias,

OK, so to me it seems that the 1st fetch of all users is the unnecessary/wrong one. It's a form not showing the user-SelectItem and prefetching the data here seems unnecessary, especially as you can't edit the user anyway.
EDIT: This is not a read-only form, removed that word.

How is the "Mark Knopfler" above "Current balance: 0" in the screenshot in #7 generated? This is not a SelectItem, but a Label, correct?
Perhaps the SelectItem you can see in #9 is actually also present in #7, but hidden? This would not explain the re-fetch/2nd fetch, but perhaps why the fetch happens when just showing the form?

Also, can you give you version/build? I don't need it, but Isomorphic might.

Best regards
Blama

[mathias]

I have a "HeaderLayout" class that listens for form events, takes the selected record and populates the labels with specified attributes. For this, the "full name" attribute is also used, so the foreign key field is not touched.

I actually forgot to mention one thing: If I take out the select item from the form, no user fetch is performed at all, so that should prove that it is only the select item that's the source for the fetches.


Edit: 13.0-p20240228, although it's been like this for a very long time.

[claudiobosticco]

Hi mathias did you check in the dev console (RPC Tab) which are the components responsible for the double fetch? You may also track down to the line code responsible for the fetch in the "Call Stack" tab (in the lower left part)

[Blama]

Hi mathias,

what are you doing so that for ListGrid-selection -> editRecord() there is no SelectItem in the screenshot, but for the click on "+" below "Time bank" there is?
There must be some hiding/showing of the SelectItem going on, correct?

Best regards
Blama

[mathias]

Hey claudiobosticco thanks for pitching in. Yeah I did check that, thought I had written it here but apparently not.

As I write, the first time it's the dynamic form fetching, and the second time it's the pick list, as evidenced by the HTTP requests in the logs. The only differences between the 2 fetch requests are:

<componentId>isc_DefaultForm_0</componentId><textMatchStyle>exact</textMatchStyle>
<componentId>isc_PickListMenu_0</componentId><textMatchStyle>substring</textMatchStyle>

[mathias]

Blama Because when editing an existing timebank, you can't change the user, so there's no reason to show it. The user name is already in the header.

(We spend quite a lot of time reducing cognitive load, a lot of our users are not very computer savvy.)


If you wonder what we do with the select item, it's something like this:

1. on page load it's hidden.
2. when clicking around in the list grid, still hidden.
3. when editing/cancelling/saving, still hidden.

4. when clicking "new", we show it, then hide it again on save/cancel. (For other tabs, for example the scheduling module, we might show it both in edit mode and new mode)