Decimals in JSON as strings

[wallytax]

I'm using SmartClient LGPL (version 13) in combination with a RubyOnRails 4.2 server. By default, RubyOnRails sends a BigDecimal instance (which is more precise than a float) as string in JSON responses. This doesn't work well with SmartClient. Fortunately, there is a Ruby gem (= plug-in) that sends a decimal not a string. I think it works well with a large number of decimal positions.

Right now I'm migrating that RubyOnRails server to the latest version (7) and finally want to get rid of the extra gem (which might not even work anymore) and have a decent solution where either RubyOnRails or SmartClient fixes it. I think the developers of RubyOnRails send those BigDecimals as strings with a good reason (I would think not to lose significant digits, but I'm not 100% sure), so I need SmartClient to understand those strings actually representing numbers. Is this possible?

[Isomorphic]

Our Java server does the same thing that your Ruby server is doing - delivering BigDecimals as Strings - see DataSourceField.stringInBrowser.

Basically JavaScript's single numeric type (Number) cannot represent BigDecimals, if you are actually using the full range of BigDecimal. Delivering these values as Strings means they can at least be displayed, and edited.

If you want to do math inside the browser using BigDecimals, you're going to need a library that handles this. If you Google "bigdecimal library JavaScript" you'll see that a few are available.

[wallytax]

Okay, I see the data in a grid but the decimal and grouping separators are incorrect. They are in American style because the string is like that, but I am from the Netherlands where we use a comma as decimal separator.

I don’t use more than 5 decimals so can I then use JavaScript’s Number class? And if so, what would be the place to do the conversion?

[Isomorphic]

That can definitely be represented as a JavaScript “Number”. DataSource.transformResponse() is probably the best override point to do the data transform.

[wallytax]

Thanks for the answer, but I'm still a little bit confused.

I would think this is common behavior in lots of applications: a server sending a large number data type as JSON-string. Apparently those number strings are not treated as real number in SmartClient, so things like localization do not work out-of-the-box. I would think this is so common that it should. Is there a good reason not to have it?

Just to make sure I'm not re-inventing the wheel I wrote the above. If I need to invent the wheel, then I would create the transformResponse() method in my own DataSource class (a subclass of RestDataSource) and check which fields are defined as "localeInt" or "localeFloat" (which is what I use now when defining such fields). Perhaps other types should be checked as well? Then I would use a conversion using the Number class in JavaScript (not sure yet how).

It seems this is quite heavy for each response, but I'll have to wait and see the behavior.

But does my approach make sense? If so, I will try to create that method and will post the result here so it might help others.

[Isomorphic]

When XML is parsed, everything is a String, so we indeed parse numeric values. With JSON, numbers can be directly expressed, so we don't parse.

The most correct solution here is to have your server deliver numbers, since its numbers that you actually want.

If you can't do that, SmartClient let's you work around this in transformResponse, with basically one line of code, roughly:

   for (record in dsResponse.data) record.[I]nameOfField[/I] = parseFloat(record.[I]nameOfField[/I])

[wallytax]

Thanks for this suggestion. Somehow, it doesn't feel right. First, the documentation of RubyOnRails states that BigDecimals are exported as strings on purpose to avoid losing any significant decimals. I will try to have it create a JSON response that does not contain the quotes. I think this would still mean it preserves all significant decimals. But then when SmartClient (or JavaScript) in general doesn't parse this large number as precise as it should be, what's the use then?

So, I don't want this thread to become long and I don't want to be rude or anything, but to me there seems to be a hiatus somewhere. Other developers should also have problems with these conversions, don't they? What if you need to make something for shares or crypto prices? Those are in 8 digits if I'm not mistaken. Will this work then?

Any suggestions, ideas would be really meaningful. Not only from SmartClient itself, but also from other companies. I'm very curious what others experience.

[wallytax]

Perhaps, helpful for others. I managed to get rid of the quotes in RubyOnRails (version 7) by creating the file config/big_decimal.rb with this code in it:

class BigDecimal
  def to_json(*args)
    as_json(*args)
  end
end

My application uses 5 decimals at most, so I hope this is precise enough when the data arrives in SmartClient.

[wallytax]

The problem seems to be way more complicated. I'm a bit too confused right now. Anyway, I tried the code above in the Rails console and it worked! But in my Rails controllers I return data via render(json: data) and then it doesn't work. So I will quit my attempts to fix this in RubyOnRails and try to implement the transformResponse() method as described above. I can't simply copy that suggestion, because not all fields not to be converted. So it means I have to check all fields for their data type.

Again, suggestions are really appreciated: I can't understand nobody has difficulties using (big) decimals in their SmartClient application retrieving JSON data.

[wallytax]

It's getting stranger and stranger: I've implemented a transformResponse() method in which I hopefully correctly convert decimal values. But I get all kinds of validation errors from the grid (already before I've implemented that method). I have now changed the field type from "localeFloat" to "float" and removed my transformResponse() method and it looks fine now. But I will probably get issues when creating forms and allowing my user to enter numbers in Dutch format.

[Isomorphic]

Wally, it seems like you've gotten somewhat confused here - this is actually pretty simple.

First of all, if you were really using the full range of BigDecimal, you would have to deliver the values to the browser as a String, otherwise, they would need to be rounded or truncated to become valid JavaScript Numbers. As we've covered, the SmartClient Java Server takes the same approach.

What you've tried to do with your Ruby override would break the correct behavior of BigDecimal being serialized as a String, needed for cases where you really need the full precision.

The thing is, you aren't using the full range of BigDecimal. You've said you're only using 5 decimals places.

So instead, the solution is simply to not declare these particular fields as BigDecimal. You just want ordinary floats, because that's the all precision you actually are using, and that's more efficient in every way, and will also solve your JSON serialization problem.

If you can't figure out how to do that, transformResponse() is one layer where you can fix this problem. We haven't seen any of your code involved here, so we can't tell what the problem is - but not that there's an equivalent per-field API you could be using, DataSourceField.getFieldValue().

[wallytax]

I'll give some feedback on how I got here. In production environment, I use SmartClient 12 and RubyOnRails 4.2. I need to upgrade RubyOnRails to version 7 because 4.2 is outdated. While doing the upgrade, I also want to use SmartClient 13 and clean up and fix old issues. My main goal is to stay as close as possible to the default SmartClient code without creating a lot of custom client side code.

I've created a server side framework in which I define models with their fields, similar to what SmartClient Server is doing. I want to avoid a discussion on why I use Ruby instead of Java. I once switched and still like its syntax and usage. I do believe I can make the two frameworks work together, but there are probably some issues and this is one of them. :-) My server side framework offers a way of creating the JavaScript code for a data source class (subclassing a subclass of RestDataSource). Because I want the framework to be more generic than just for this application, I decided to support either integers or (big) decimals, not floats. Reading all the above, I still want RubyOnRails to send strings for decimals and handle things in the RestDataSource subclass.

I've already created the transformResponse() method but I get all kinds of validation errors even before that method is called. Is that possible? Or is there something in the code running before the call to transformResponse()?

And should I use "localeFloat" and "localeInt" as DataSourceField type or is it better to use "float" and "integer"? Doing the latter seems to remove the validation errors and make the display of the numbers correct (by that, I mean a comma as decimal separator), but I'm not yet at the point of creating forms and I expect issues there. Users should be able to enter numbers in that same format as well.

Hopefully you can help me out a bit more. I feel bad asking you all this, but am indeed confused. ;-)

[Isomorphic]

This is where you're going wrong:

I decided to support either integers or (big) decimals, not floats. Reading all the above, I still want RubyOnRails to send strings for decimals and handle things in the RestDataSource subclass.

BigDecimals, which can include precision not representable in JavaScript, must be sent as Strings. And then, they cannot be manipulated by normal JavaScript code, because JavaScript cannot handle that precision.

You have introduced the special handling required for BigDecimals for ordinary floats, even though you have no apparent plans to actually use the range for BigDecimal.

And you have not increased the flexibility of your framework in any way. You could just represent ordinary floats as floats, and use BigDecimal only when you need it, such as when handling astronomy data.

Further, in working around this design problem on your server side via the client, you have created a further problem: now you are going to try to parse all float values on the client, so you can turn them back into the Number values you should have delivered in the first place.

So, when you receive a String that contains a true BigDecimal value, your code is going to try to parse that, and the result will be rounding or other approximation. Silently. Which means the built-in capability for actual BigDecimal values to be delivered as Strings is now broken. You can then perhaps install a special library on the client to try to figure out whether a given String contains a true BigDecimal value, and so should be left as a String.. but this is more and more effort, and more and more slowness, compounding the original design error on the server.

And, to make this mistake possible, there will be even more effort: you are correctly seeing validation errors for floats, this happens when you call Super() in transformResponse(), which does the initial response parsing. These errors are correct and cannot be suppressed without writing your own different protocol - a different protocol based on DataSource, not RestDataSource, losing all the automatic processing.

So again: deliver decimals as decimals in JSON as decimals, not as Strings. JSON was explicitly designed to allow transmission of floats in this way, and SmartClient is expecting you to use JSON as it was explicitly designed. To do something different - to use JSON incorrectly - will require a lot of effort and the final result will be worse in several ways from the result if you just use JSON as designed.

[wallytax]

Thanks for the extensive reply, I appreciate it very much!

I will - for now - use floats where I used decimals, so that SmartClient has no problems parsing them!