Announcement

**Isomorphic** · 16 Feb 2012, 12:44

Technically, we should probably quote that character when serializing to XML, however, as far as we can tell this would only come up in a synthetic test - trying to display that character would never work.

If you just want to be able to successfully round-trip a binary value from server to browser and back, but don't intend to display it, base64 encode it.

**mathias** · 16 Feb 2012, 13:35

Well, I have found the culprit.

In my table i have a "note" field that is sent in through our IOS or Android phone apps.

For some reason, no idea why, its entered in a textfield in an app by users, that text has, on occasion, contained 0x1a values. In the database, and in the web its not visible, the character is omitted.

So, i have to check why on earth it's possible that android has sent that in. I will also update my rest service to strip stuff out i guess.

And i agree that you really should escape characters like that out, full-fledged stack traces in the browser are never fun :)

EDIT: so my problem is that i have a "normal" text field that for some reason contains 0x1a-values sometimes. i have to strip them out somehow i guess

**godonnell_ip** · 3 Aug 2016, 08:07

My apologies for resurrecting an old thread, but it's very relevant to a problem we are experiencing.

Our system uses a DB2 database hosted on an IBM i. As you are probably aware, character set handling is interesting to say the least in such an environment. Long story short is that certain characters entered by a user in the UI may be translated by the DB and/or OS to a byte string that may contain 0x1A. We are subsequently able to read such data, but cannot update it because the 0x1A character is an invalid character in XML and therefore in the request.

I've looked into adding a requestTransformer to each of our datasources on the client side, but that seems like some effort to ensure we enforce the instantiation of datasources with such a transformer. Alternatively we could look at implementing something on the server side to transform responses so that we never receive 0x1A back to us (remember, this character is only introduced AFTER the request is processed by the server-side so the initial insert or update that introduced it would have worked - we simply want to make sure it doesn't get back down to the client before then being included a subsequent update).

Although those options are open to us, it would be far preferable if you would simply quote that character as you suggested above. Is that feasible?

Thanks,
Gary O'Donnell

**Isomorphic** · 3 Aug 2016, 08:28

We'd definitely recommend avoiding the 0x1A byte string ever being saved. Even if it could be round-tripped to the client and back, it's clearly data corruption. Whatever characters are being turned into 0x1A are being garbled or lost, and that's a UI issue, not a data layer issue.

As far as where to add a data transformation, we would recommend doing it server-side in a custom DataSource implementation. This should be easy to re-use across all of your DataSources, and it means that other possible clients of your system will also get the same behavior, as opposed to making it specific to just browsers.

**godonnell_ip** · 5 Aug 2016, 15:10

Thank you for the super-quick response.

Unfortunately, though, the transformation is performed by the database and/or OS (I'm not sure which) and we are not entirely in control either the system (owned by the client) or the database and table schema configurations (owned by legacy applications). In addtition, it is possible for third-party or client-written programs to further modify the data. I would dearly love to be able to configure the system to enable support for the UTF-8 character set, and I would also like to drive a Bugatti...

So, within the constraints in which we must live, here is our problem scenario:

A user enters a value into a FormItem.
1. The value contains a character as benign as the Euro symbol (�).
Our SmartGWT client code happily accepts the value and submits an update request to the server.
The server is similarly happy to handle the value and constructs an UPDATE statement to write the data to DB2.
At some point when the UPDATE statement is executed by DB2, the supplied values are mangled to conform to the code page configuration determined by a combination of the system, database and table (possibly even field) configurations (again, not sure if DB2 or the OS is in charge of this, but either way it is well past where we or even Isomorphic has control of it).
1. The mangling introduces various nasty byte strings, which may well include 0x1A. These values make perfect sense in the context of the iSeries and the values show up nicely in greenscreen - provided the codepage is appropriately configured.
At some future point, the same mangled values are requested by the SmartGWT client, and again the server is happy to oblige.
1. At this point, the same FormItem now does not contain the Euro symbol (�), but it does contain the byte string that will allow the iSeries to display the character correctly in greenscreen.
User maybe edits the value slightly, but the value still contains the mangled byte string which includes 0x1A.
Client attempts to submit an update request, but the 0x1A causes a parsing error in SAX and so the request cannot be submitted.

To your point, this whole round-trip experience is far from optimal, but to my point you can see now why it occurs and that we are currently helpless to prevent it unless we either:

Strip out the 0x1A characters from the response or
Properly escape the character in the update request

If we strip out the 0x1A character we really are corrupting the data. If the character were simply escaped in the request then we'd at least leave it in whatever silly form the iSeries preferred it to be. Unless you can help advise of a global mechanism we can use to implement this type of transformation on all requests, I think we're back to asking if you would mind please moving this from my "I would really like" list to your "No problem, it's in the next nightly build" list.

Thanks for taking the time to listen.

Regards,
Gary

**Isomorphic** · 8 Aug 2016, 12:35

This explanation of steps is what we assumed was going on, and our recommendation is still the same - we'll try rephrasing it.

Whatever DB2 is turning into 0x1A shouldn't be turned into 0x1A, because that mangles whatever the user was trying to save. When the data round-trips to the client, you want the Euro symbol, not 0x1A. The framework is not in a position to further escape the Euro symbol - it is already being provided to you in the correct form. Your code needs to work around the conversion of the Euro into 0x1A so that does not happen.

Yet another way to put it: while the framework could be modified to deliver 0x1A to the client, that would not solve your problem, since the Euro symbol would still have been lost (and presumably, a lot of other characters that users will expect to be preserved).

**Isomorphic** · 8 Aug 2016, 12:40

Yet more detail: you've said the transformation performed by DB2 switches normal UTF-8 encoding for something that appears correctly in greenscreen. So presumably, the Euro symbol maps to 0x1A, and other special characters map to other special codes.

If it is impossible to turn this transformation off, and it is impossible to ask DB2 to reverse the transformation when you select data, then your remaining option is to transform the data back to normal UTF8 encoding after receiving it from the database: in Java code, probably in a server-side Custom DataSource, you'd iterate through dsResponse.data and perform replacements on any string fields, then call dsResponse.setData() with the resulting repaired data and allow normal processing to continue from there.

**godonnell_ip** · 13 Aug 2016, 09:39

Thank you for the further replies.

It may not be impossible for us to turn off this transformation, but it will certainly be difficult to coordinate not just with our legacy code but our customer base who likely have custom modifications and/or external applications that can work with the data as it is currently being encoded, but would not necessarily work successfully with UTF-8 character encoding.

That said, we were hoping as a first step to simply retain the encoding (yes, understanding that it wouldn't display correctly in our browser-based UI) while still allowing the client to initiate update requests without the exception occurring during parsing of the XML.

As I mentioned earlier we had already considered transforming replies. We may, though, be better off just transforming the requests since they will generally be smaller. It sounds like you won't be considering this enhancement to be added to the libraries (even though the suggestion was originally yours earlier in this thread), so I guess we'll go one way or the other. Either way, I agree with you that the whole experience is sub-optimal and ideally there would be a better way to deal with it, but I'm afraid we all have our constraints. Who knows - maybe we'll bump into a better approach while we continue our work on this.

Thanks again for your responses.

Regards,
Gary

**Isomorphic** · 13 Aug 2016, 11:21

We didn't suggest changing the libraries, we meant to say only that a system for XML serialization might be expected to handle it somehow. But in this case, our XML-serialization system is purpose-specific, and there is no point in handling this character since it could never be correctly displayed, and actually represents a data corruption problem that the libraries cannot work around. In addition, scanning for and quoting such characters would make serialization slightly slower for everyone. So it definitely doesn't make sense.

Announcement

listgrid exportclientdata as csv causes top-level IDACall exception!

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment

Comment