Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
Clear All
new posts

    Behavior Change in DataSource.applyFilter() between SmartClient 12.1 and 14.0 with Dynamic Fields

    I’m seeing a behavior change between SmartClient 12.1 and 14.0 when using DataSource.applyFilter() with AdvancedCriteria referencing dynamic fields that are not declared in the DataSource’s original fields array.

    Behavior in SmartClient 12.1


    When calling applyFilter() on an array of records where the criteria references a field not declared in the DataSource, the framework still attempted to evaluate the criteria against the raw property values in the records.
    This worked fine as long as the property existed on the records themselves.

    Example scenario:
    • Records have a property customField.
    • The DataSource doesn’t declare customField.
    • applyFilter() returns filtered results as expected.

    Behavior in SmartClient 14.0


    In 14.0, the same call to applyFilter() returns all records unfiltered.
    What appears to be happening is that applyFilter() now performs early validation using isc.DS.getCriteriaValidator().validate().
    This calls getField() on each criterion’s fieldName, and if any field is unknown, validation fails and no filtering occurs—criteriaApplies() is never invoked.

    I confirmed this by patching applyFilter() and criteriaApplies() with logs; in 14.0, criteriaApplies() is never called if getField() returns undefined.
    How We Add Fields Dynamically


    In our application, we intentionally add fields to the DataSource dynamically at runtime as part of our design.
    Currently, the only way we have found to do this is by assigning to the fields array or object, e.g.:
    dataSource.fields["customField"] = { name: "customField", type: "text" };

    This worked well in 12.1 because filtering continued to function even if the field was not declared at DataSource creation time.

    If there is a better-supported or recommended approach to dynamically registering fields in a DataSource (so that applyFilter() will recognize them), we are very open to using it instead.

    Workaround


    To restore the previous behavior, I wrote a patch that extends getField() to also look in this.fields[fieldName] if the field is not found via the standard mechanism.
    After applying this patch, applyFilter() and criteriaApplies() work as they did in 12.1.

    Below is the minimal version of the patch without console logging and without restricting to a specific field name or DataSource:

    Patch:

    Code:
    if (isc.version.indexOf("v14.0") > -1) {
        var proto = isc.DataSource.getPrototype();
        var origGetField = proto.getField;
        proto.getField = function(fieldName) {
            var result = origGetField.call(this, fieldName);
            if (result) return result;
            // Fallback: if fields is an object, check there too
            if (this.fields && !isc.isAn.Array(this.fields)) {
                return this.fields[fieldName] || null;
            }
            return null;
        };
    }

    Questions:
    1. Was this stricter validation in 14.0 an intentional change in the framework?
    2. Is there an officially supported way to allow applyFilter() to handle dynamic fields without defining them statically in the DataSource configuration?
    3. Are there any recommended approaches for dynamically adding fields to a DataSource so they are recognized by getField() without requiring patches like the one above?

    Thanks in advance for any guidance on this topic.

    #2
    This kind of code:

    Code:
    dataSource.fields["customField"] = { name: "customField", type: "text" };
    ... is invalid, and always has been. Basically, you're modifying an immutable internal data structure after lots of things have been derived from that structure.

    So, filtering on fields that aren't declared has been invalid for a long time, you just got lucky that modifying internal structures in earlier versions happened to be enough for the fields to be considered "declared".

    By design, DataSources are considered immutable after construction, because there are all kinds of bizarre cases to consider if you allow them to be modified (what happens to components that are already bound? in-flight requests?).

    So the way to think of it is: if you need to change a DataSource on the fly, that is a new DataSource.

    You haven't mentioned whether you need these DataSources to be dynamic on the server, or only on the client.

    On the server, you use DynamicDSGenerator (see QuickStart Guide for a quick intro and links to deeper docs).

    On the client, just don't call DataSource.create() until you have the final fields.

    There's a lot more we could get into about dynamically changing DataSources while the app is running (e.g. the end user can add fields) and how to do that, and using DataSource inheritance and facade patterns, but first, just let us know if the above is enough.

    Comment


      #3
      Wow, very interesting. We've been running our app in production for 18 years now using this invalid method and it has worked for us. This datasource only needs to be dynamic on the client. The user can add a custom field using a different CustomField datasource. But, after the custom field is added, we dynamically add it to the primary datasource (FundAsset). This FundAsset datasource controls our client-server interactions as well so it can't just be defined dynamically on the client. I guess we could look at doing surgery to create a different datasource client-side. But, I'm not sure how it would work to create a new datasource every time the user adds or modifies their custom fields. For now, this simple patch to getField seems to have solved our problem. It seems like you are suggesting major surgery to the logic of our long-running application. I understand this has always been invalid but it has also always "just worked" for us. How do we proceed?

      Comment


        #4
        Unfortunately, the longevity of a hack has nothing to do with whether it's supported. It's not uncommon for a hack to work for 20 years and then stop working.

        We maximally maintain backcompat with documented APIs, but it's literally impossible to move forward if people expect every single internal structure to work in exactly the same way as in the last version - that is, to be "bug-compatible" as they say.

        So let's clarify the use case: is the end user adding a totally novel field on the fly? As in, you've never seen this field definition before - it's not from a library of fields - it's an organization-specific field that they just made up on the fly?

        Or are they just picking from a library of already-defined fields?

        If the latter, why wouldn't you put all the fields in the DataSource and just use dsRequest.outputs / operationBinding.outputs to control the amount of data and what's queried?

        What is being achieved by dynamically redefining the DataSource? If we know that, then we can suggest supported ways to do it instead - which might actually be simpler or more performant.





        Comment


          #5
          Yes, the end user is adding totally novel fields on the fly. I'm happy to work with you on a better way to do this long-term. For the moment, I am hoping to understand if the getField hack that I showed is workable for the moment since it seems to solve our immediate issue.

          Comment


            #6
            Well, anything that isn't a supported usage can break at any time. Your getField() override works now, but it's not a supported approach, so tomorrow you could download a new build and find that, because we fixed some unrelated bug or added a new feature, the override no longer works.

            There's really nothing we can do about this uncertainty because we can, at best, provide 100% coverage for supported APIs - we can't provide 100% coverage for unsupported behaviors, because then we couldn't change even a single line of code.

            So:

            1) in the immediate term, if you're not prepared to migrate to supported APIs, we could add a test to our test suite to make sure this unsupported override isn't broken, so we would at least notify you that it was about to be broken (this is a special service, beyond normal support, but available)

            2) longer term, you want to be on supported APIs. So two cases:

            a. if the end user can add a totally novel field on the fly, but it's OK to reload the page when they do this, then that's easy: just construct the DataSource before UI components are created, keeping the user's added fields in mind, and you're done

            b. if there's a requirement to not reload the page, then you can create a new DataSource on the fly with the added field, and just set up your UI creation logic so that it can run again with the modified DataSource, after destroying the old UI. Essentially the same thing as reloading the page, but may preserve caches or other application-level context that would otherwise be lost in a page reload

            The other question is: what kind of field is being added? You can of course add fields on the fly in the view (for example add a new ListGrid field which is a formula). It's just that if you add a field dynamically to the underlying data model then there are all kinds of troublesome cases (like in-flight requests).

            So, perhaps you could add the underlying field to the data model - but mimic it in the UI, with a formula field or something? A lot depends upon your requirements for whether you can reload in this situation, so you can rerun your code for DataSource creation.


            Comment


              #7
              Thank you. Yes, if you were able to add a test to your test suite for this case, that would be helpful. Let me know. We can't really reload the page each time a new custom field is added so I guess we'll have to look option B you mentioned.

              Comment

              Working...
              X