Two questions about RestDataSource using a RubyOnRails back-end

[wallytax]

I'm refactoring code to communicate with RubyOnRails using the RestDataSource class. I have this already working using transformRequest, but I want to improve the code if possible. Therefore, I have two questions:

• By default, when there are two sort fields, the query string generated is &_sortBy=field1&_sortBy=-field2. In Rails only the last request parameter is returned. Ideally, I would like to have a comma separated argument (&_sortBy=field1,-field2) or an array, which can be achieved by suffixing the argument names with "[]" (&_sortBy[]=field1&_sortBy[]=-field2). I now use transformRequest() to achieve this, but was wondering if there's a setting that allows me to change the argument name.
• I've probably asked this before, but can't find the post(s) for it anymore. I would like an easy way to send the field names visible in a grid so that the server only needs to query those fields and return only that data (optimization). I now set a dataProperties.requestProperties.params.fields option in the grid's init() method, but I read something about an "outputs" option in DSRequest, but can't get my finger around it how to configure this exactly.
  
  The way I now fix this in the grid is like this:
  
  

  	this.dataProperties ||= {};
  	this.dataProperties.requestProperties ||= {};
  	this.dataProperties.requestProperties.params ||= {};
  	this.dataProperties.requestProperties.params.fields ||= this.fields.getProperty('name').join(',');

  Is this the easiest way, or can I do it simpler with isc.addProperties()? Nested or something. I tried this, but when I already have params in the request properties, those get wiped out.

[Isomorphic]

1. We're not experts on Ruby on Rails, but repeating params of the same name is a very core & very old part of the HTTP spec, so we'd be extremely surprised if there is no way to acquire the full parameter value in Rails. However, as far as your question, no there isn't a setting that reformats the sortBy field, transformRequest() is the right way to do this if your backend needs special parameter names.

Also note that we recommend in the docs to use JSON or XML posts to the server, because with multi-level sorts, AdvancedCriteria, etc, it is easy to exceed the max URL length and have requests just break. Not to mention that you can't use the Queuing feature set with URL params.


2. Fetching just the visible fields for a grid is not, generally, actually an optimization, since end users can choose which fields are visible (forcing a re-fetch) and since the framework provides various ways to quickly see the hidden fields without a refetch (see eg ExpansionMode, HoverMode) and/or to go immediately into editing data without a re-fetch (dynamicForm.editSelected()).

Because every new fetch has a lot of overhead (authentication & authorization, asking the DB to sort & join & page a large table just to deliver a handful of records, etc), eliminating just one fetch is far more of a savings than trimming off data from the original fetch.

So you almost certainly want to just abandon this attempted optimization, as you are likely to make your app more complicated and slower if you continue.

Note that there are uncommon cases where fetching limited fields makes sense, although it's usually a workaround for a problem elsewhere; for example, a DB table that has an enormous text log in one column, which should have been a separate table. In this case, you can use operationBinding.outputs to eliminate fields that shouldn't be delivered by default.

Likewise you can set dsRequest.outputs via the "requestProperties" parameter of fetchData, or via dataProperties.requestProperties (no "params" or "fields" as you had it above).

However, note that RestDataSource will not send dsRequest.outputs by default, so you would also need to add some transformRequest code to put the outputs into the request data.

Finally, note that for SelectItem/ComboBoxItem, where it is fairly common to see just 1-2 fixed fields and not be reusing the data in some other way, there's an option fetchVisibleFieldsOnly, which sets dsRequest.outputs.

[wallytax]

Thanks for the quick reply!

About question 1
How would I send a JSON post to the server? As operation binding? Will the request type change from GET to POST? And isn't this breaking the REST principle? But you are right that the URL params are not suitable for this, so I'm willing to change my current approach.

About question 2
I made a mistake in my question. I already request the data for all fields possibly visible in the grid. So even if a field is hidden, it gets requested. In some cases that invisible data might also be used to render cell data differently or change the styling of a row (eg. when a field "active" is false, use a strikethrough effect, of whatsoever).

But I did indeed notice that the outputs were not send by the RestDataSource. You probably have good reasons why this is done.

Conclusion
I think I need to use an approach where I post the request data so that I can have way more information in it.

[Isomorphic]

See the RestDataSource overview and in particular operationBindings and dataProtocol. You just want to replace the default operationBinding for "fetch" (and probably all the others) to use dataProtocol:"postMessage".

[wallytax]

I've implemented the transformRequest() method so that the "outputs" property on the request is still send using RestDataSource. But as soon as I enable the filterEditor and enter a search text, those outputs are no longer send.

[wallytax]

When I override filterData() like this:

filterData: function (criteria, callback, request) {
    isc.addProperties(request, this.dataProperties.requestProperties);
    this.Super('filterData', [criteria, callback, request]);
},

It seems to work. But I think that the requestProperties should already be sent (in every request). I set those in the init() like this:

this.dataProperties ||= {};
this.dataProperties.requestProperties ||= {};
this.dataProperties.requestProperties.outputs = this.fields.getProperty('name');

[Isomorphic]

We're not sure what's going on here - the filterEditor does not have any knowledge of DataSource internals, it cannot influence your transformRequest method. Also the FilterEditor does not call filterData(). Also if you have set requestProperties via dataProperties, they would be included on every request, there would be no need for a filterData() override anyway.

So something very odd is going on here as you seem to be reporting multiple impossible things - possibly you have calls to setData() which are wiping out the automatically generated ResultSet.

[wallytax]

This should be a stand-alone example (except for the server side JSON file) that reproduces the issue.

isc.ListGrid.create({
  allowFilterOperators: true,
  alwaysShowOperatorIcon: true,
  autoDraw: true,
  dataSource: isc.RestDataSource.create({
    dataFormat: 'json',
    dataURL: '/users.json',
    fields: [
      { hidden: true, name: 'id', primaryKey: true, type: 'integer' },
      { name: 'full_name', type: 'text' },
      { name: 'locale_code', optionDataSource: 'LOCALE_ENUM', type: 'text' }
    ],
    jsonPrefix: '',
    jsonSuffix: '',
    operationBindings: [
      { dataProtocol: 'postMessage', operationType: 'fetch' }
    ],
    transformRequest: function (request) {
      request.data ||= {};
      request.data.outputs = request.outputs;
      return this.Super('transformRequest', [request]);
    }
  }),
  fields: [
    { name: 'full_name', title: 'Full name' },
    { name: 'locale_code', displayField: 'name', title: 'Locale', valueField: 'code', width: 100 }
  ],
  height: '100%',
  init: function () {
    this.dataProperties ||= {};
    this.dataProperties.requestProperties ||= {};
    this.dataProperties.requestProperties.outputs = this.fields.getProperty('name');
    this.Super('init', arguments);
  },
  showFilterEditor: true,
  width: '100%'
});

The first request fires immediately, even when autoFetchData is unset, but this is not my main concern. When I check the payload in the developer tools, I see that there's an "outputs" property nested in the "data" property.

After selecting a different locale in the filter editor, a new request fires and the "outputs" is set to null.

As an extra check, I tested setting this.dataProperties.requestProperties.outputs to the hard coded array ['full_name', 'locale_code'] (in the init() method of the grid), but this doesn't matter either.

So why is the requestProperty lost?

[Isomorphic]

fetchData() / filterData() have a requestProperties argument, and that is applied to the auto-created ResultSet, overwriting dataProperties.requestProperties.

Although you can't count on the FilterEditor calling fetchData()/filterData() (because this is not documented), it does use most of the same code path, and has the same effect.

Although all of this is moot (since we provided a better approach in your other thread), it would probably be better that requestProperties provided to fetchData()/filterData() are merged with dataProperties.requestProperties (if present) rather than completely overriding it, so we'll make that change.

[wallytax]

Okay, great that you acknowledge the missing (or overwritten) request property and will fix this!

It might not be of use for me then, since you advice not to trust on filterData() being called. I will try to use the component ID in the data source.