Problem withjson-rpc for datasources and error handling

[schlonz]

Hi,

I am trying to get smartclient talk to a json-rpc 2.0 (draft) compatible server. Unfortunately I am running into troubles with getting back error messages.

Can you please:
a) Confirm that I am on the right track regarding a way to talk to jsonrpc and do not follow a stupid way since there are better ones?
b) Maybe tell me a way how to get errors presented to the user?
c) Maybe tell me a way how to solve the http status issue?


Also, if I am right, error handling of http errors is currently not in place. Think it would be great if it would be. :)

Thanks a lot in advance for your help!

The details:

First hurdle is that the preferred way to contact json-rpc is with a POST - also for fetching. Also the data structures are quite a bit different to what RestDataSource expects. Instead of overwriting nearly everything in RestDataSource I started to write my own data source JsonRPCDataSource based on RestDataSource.

So far fetch works fine but as soon as an error occures I run into troubles. JsonRPC over http maps errors also into the http status. And if smartclient gets a e.g. http status 500 the code does not even reach my transformResponse function but fails earlier in DataSource.js in the _handleJSONTextReply function at the line

if (evalText.match(/^\s*\{/)) {

with an error

evalText.match is not a function

This is what the developer console says to it:

20:10:00.343:XRP6:WARN:Log:Error:
	'Object doesn't support this property or method'
	in http://localhost:8001/
	at line 297
    DataSource.$379(rpcResponse=>Obj, jsonText=>Array[1], rpcRequest=>Obj)
    Class.fireCallback(_1=>Obj, _2=>"rpcResponse,data,rpcRequest", _3=>Array[3], _4=>[JsonRPCDataSource ID:processListDS], _5=>undef) on [Class Class]
    Class.fireCallback(_1=>Obj, _2=>"rpcResponse,data,rpcRequest", _3=>Array[3], _4=>undef)
        "return isc.Class.fireCallback(_1,_2,_3,this,_4)"
    RPCManager.fireReplyCallback(_1=>Obj, _2=>Obj, _3=>Obj, _4=>Array[1])
    RPCManager.fireReplyCallbacks(_1=>Obj, _2=>Obj)
    RPCManager.performOperationReply(_1=>Obj, _2=>Obj)
    RPCManager.$39d(_1=>2)
    RPCManager.performTransactionReply(_1=>2, _2=>Array[1], _3=>undef)
    callback(transactionNum=>2, results=>Obj, wd=>undef)
        "isc.RPCManager.performTransactionReply(transactionNum,results,wd)"
    ** recursed on Class.fireCallback

For the user all this is hidden. No feedback. Just the "Finding records that match your criteria..." window blocking everything else. The only thing a user can do is to hit the browser's refresh button.

Well, I came to the conclusion that currently without rewriting smartclient code there is no way how to deal with http error codes. Just for testing I changed the backend to always send http 200 status codes - also for errors.

My transformResponse deals with the different data structure (I think :)) and should map them to the dsResponse structure. But I never see an error message in the user interface. Just "No items to show." in the ListGrid.

To shorten things I tried to force an error by rewriting my transformResponse this way:

    transformResponse : function (dsResponse, dsRequest, data) {
        dsResponse.status = isc.DSResponse.STATUS_VALIDATION_ERROR;
        dsResponse.errors = {   pid:[
                      {errorMessage:"First error on field1"},
                      {errorMessage:"Second error on field1"}
                  ]
              };
        dsResponse.data = {   pid:[
                      {errorMessage:"First error on field1"},
                      {errorMessage:"Second error on field1"}
                  ]
              };
        return dsResponse;}

pid is a field in my ListGrid and data source. Still no difference.

The console is well aware of the error:

21:43:08.515:XRP0:WARN:RPCManager:Error performing rpcRequest: error: VALIDATION_ERROR, response: {data: Obj,
startRow: 0,
status: -4,
endRow: 0,
totalRows: 0,
httpResponseCode: 200,
transactionNum: 2,
clientContext: Obj,
context: Obj,
errors: Obj}

Regards,
Erich

PS: Just in case, the respective json-rpc specifications can be found under:
http://groups.google.com/group/json-...c-1-2-proposal
http://groups.google.com/group/json-...-rpc-over-http


edit: Maybe I should also mention that I am using:
dataFormat: "json"
and
dataProtocol:"postMessage"

[Isomorphic]

Hi schlonz,

Starting with DataSource rather than RestDataSource is the right approach.

The fact that you can't get to transformResponse() with a 500 server error using dataFormat:"json" is a bug. We'll correct it and add automated tests so it cannot recur - in the meantime, use dataFormat:"custom" and you'll get the plain text of your response as dsResponse.data in transformResponse(), which you can then eval() yourself.

Your error reporting code is close - take a look at this example to see this working properly. Both your dsResponse.error and dsResponse.data have an extra level of structure ("pid") which doesn't match the intended format of those dsResponse properties.

Also, a failure to fetch data shouldn't be reported as a validation error, which specifically means a failed attempt to save. For a total failure to fetch, use any negative value for dsResponse.status other than VALIDATION_ERROR, which has special meaning. Don't set dsResponse.errors (which again is specifically for validation errors), instead set dsResponse.data to an error message. This will trigger an error dialog by default, which you can override via overriding DataSource.handleError() or RPCManager.handleError()

[schlonz]

Hi Isomorphic,

First thanks a lot for your fast reply!

Hi schlonz,

Starting with DataSource rather than RestDataSource is the right approach.

Sorry, that is what I meant. My data source extends DataSource.

The fact that you can't get to transformResponse() with a 500 server error using dataFormat:"json" is a bug. We'll correct it and add automated tests so it cannot recur - in the meantime, use dataFormat:"custom" and you'll get the plain text of your response as dsResponse.data in transformResponse(), which you can then eval() yourself.

Regarding the bug: good to hear! dataFormat: custom does not show the problem as it does with json. That helps a lot!

Your error reporting code is close - take a look at this example to see this working properly. Both your dsResponse.error and dsResponse.data have an extra level of structure ("pid") which doesn't match the intended format of those dsResponse properties.

You are right. My code was indeed wrong.

Also, a failure to fetch data shouldn't be reported as a validation error, which specifically means a failed attempt to save. For a total failure to fetch, use any negative value for dsResponse.status other than VALIDATION_ERROR, which has special meaning. Don't set dsResponse.errors (which again is specifically for validation errors), instead set dsResponse.data to an error message. This will trigger an error dialog by default, which you can override via overriding DataSource.handleError() or RPCManager.handleError()

Hmmm. I fear I do not get it. If I do so I run into the next problem. You should be able to easily reproduce it: Just copy and paste the code below into the 'JS' tab in the URL you mentioned above, press 'Try it' and then on the View tab press 'Save'. The code is the same as the original one just, the status is set to another number, the error message is saved in dsResponse.data instead of .errror and it is using a ListGrid instead of the DynamicForm.

Result here with Firefox 2.0.0.14 as well as with Internet Explorer 7.0.5730.11 under Windows XP SP2: 'Finding records that match your criteria...' pops up, vanishes, ppops up, ... endless until I close the browser or refresh, etc.

Trying the same with a DynamicForm works btw. Does that mean a ListGrid expects a different type of error message/handling?

isc.DataSource.create({
    ID: "users",
    dataFormat: "json",
    dataURL: "/isomorphic/system/reference/inlineExamples/dataIntegration/json/serverValidationErrors/serverResponse.js",

    fields: [
        {name: "userName", title: "Username", type: "text", required: true, length: 50},
        {name: "firstName", title: "First Name", type: "text", required: true, length: 50},
        {name: "lastName", title: "Last Name", type: "text", required: true, length: 50},
        {name: "email", title: "Email", type: "text", required: true, length: 100},
        {name: "password", title: "Password", type: "password", required: true, length: 20}
    ],
    transformResponse : function (dsResponse, dsRequest, jsonData) {
        var status = isc.XMLTools.selectObjects(jsonData, "/response/status");
        if (status != "success") {
            dsResponse.status = -1234;
            var errors = isc.XMLTools.selectObjects(jsonData, "/response/errors");
            dsResponse.data = errors;
        }
    }
});


isc.ListGrid.create({
    ID: "boundForm",
    dataSource: "users",
    fields:[{name: "userName"}]
});


isc.VLayout.create({
    members: [
        boundForm,
        isc.IButton.create({
            title: "Save",
            click : "boundForm.fetchData();"
        })
    ]
});

Isomorphic, thank you for your patience!

Regards,
Erich

[Isomorphic]

Error handling is uniform between the ListGrid and DynamicForm (both go through a DataSource).

The first problem in your code is that you set dsResponse.data to the errors, which is an Object, not a String. So this will not cause the warning dialog, as indicated in the docs for RPCManager.handleError()

The second is more subtle. You have changed from an "add" operation type to a "fetch" operationType and SmartClient's databinding system is doing it's best to recognize this response structure as a valid respond to "fetch", which expects an Array of matching records. So the entire response structure is interpreted as one record, placed in dsResponse.data, and dsResponse.endRow and dsResponse.totalRows are defaulted accordingly. But then the data is swapped for a single String without updating totalRows or endRow, with the result that you are advertising that one record is available but there was a transient error fetching it - so SmartClient tries again.

Enabling the ResultSet log category gives you something pretty close to a play-by-play account of the problem.

This would be a correct way of updating a dsResponse for a "fetch" that encountered an unrecoverable error:

            dsResponse.status = -1;
            dsResponse.data = "Data not available";
            dsResponse.totalRows = dsResponse.endRow = 0;

[schlonz]

Hi Isomorphic,

I think I am starting to understand the mechanism. Already changed my code and it works great.

Thank you for your quick and competent answers. I am impressed. :)

Regards,
Erich

[ledifni]

I like what you're doing here. Any chance Isomorphic could add a JsonRpcDataSource to SmartClient? I put a post in the Wishlist forum.

[Isomorphic]

Yeah we'd be happy to - Schlonz do you think you'll end up with something general-purpose enough to post here, that Isomorphic could use as a starting point?

[schlonz]

Sure! That was my plan from the beginning on. I would also love to find other people to have a look at the code and maybe improve it.

Currently the code consists just of a few lines supporting only the most basic use case. Give me 1-2 weeks to improve it and I will post it.

But beware: I am a lousy Javascript developer (simply not much experience with this language) and my experience with smartclient consists of 4 evenings.
So please keep your expectations low! Given my experience someone more experienced with both technologies will have to review the code in any case before it can be used in production!

Currently I am trying to make the data source JSON-RPC 2.0 compliant. Version 2.0 is only a draft so far. Would certainly be good if it supported 1.0 as well.


Regards,
Erich

[ledifni]

I hesitate to offer too much time for this right now, since my schedule tends to be a little hairy these days and I can't always keep up non-paying commitments -- but what time I have I'd be happy to offer. Can you think of a convenient way for us to exchange code so we can collaborate on this? If you like, you can email me at jodakim@hotmail.com (my spam-magnet email address) and I'll reply with my GMail address so we can send attachments back and forth.

[schlonz]

Hi Isomorphic, ledifni,

ledifni, you got mail.

Both of you: You know the community here better. What do you think would be the best approach to share the code? Posting it into the forum as non-attachment is probably not really what we want.

I could put it on google code or similar. Any better suggestions?

For me putting it on google code would mean it's there open for improvements until eventually isomorphic integrates it into SmartClient.


The code grows and is getting into a state where it can be used for at least some purposes. Currently it handles json-rpc 2.0 dataProtocol:"postMessage" for http POST (tested by me) and dataProtocol:"getParams" for a http GET (not tested and probably failing - my backend does not support GET so far).

Furthermore dataFormat "json" works if your backend returns errors as http status code = 200. I guess we better wait for the fix from isomorphic here.

As a work around dataFormat "custom" is also supported if your backend uses status codes != 200.

I have not looked into security so far.

The support for dataFormat "custom" is a dirty hack: it reuses (copy & pasted & modified) functions from DataSource.js.

It uses the http://www.JSON.org/json2.js library (public domain) and a base64 library (BSD License) from - ouch - Tibco GI. Sorry for that. ;)
I was not in the mood to rewrite these things. But I guess these licenses should not give us any troubles with the LGPL license of SmartClient.

Currently I see only one hurdle to fully support JSON-RPC: If the back end uses http status codes != 200 AFAIK smarclient does not forward the body of such messages to transformResponse. A pitty since there we could find more meaningful information.


Regards,
Erich

[Isomorphic]

Hi Schlonz,

Because of the substantial dependent libraries, hosting with Google code and simply linking from here probably makes the most sense.

We'll try to get you a patch for the problem with 500 server errors so you don't have to keep working with the dataFormat:"custom" pathway. However, it looks like it would be pretty straightforward to create such a patch yourself, for example, override _handleJSONTextReply and call the original function only after ensuring that the jsonText argument is an empty String rather than null.

[schlonz]

Please see http://forums.smartclient.com/showthread.php?t=2051 -
Announcement: CustJsonRPCDataSource - a JsonRPC data source. :)

[Isomorphic]

We've fixed the problem with HTTP 500 server responses on JSON data types. The fix will be included in the upcoming 6.5.1 maintenance release.

Regards,
Isomorphic Software

[schlonz]

We've fixed the problem with HTTP 500 server responses on JSON data types. The fix will be included in the upcoming 6.5.1 maintenance release.

Hi Isomorphic,

Are you sure the problem has been solved?

I still run into the same problem when HTTP server responses are e.g. 500 or 404.

I might be completely wrong since I am not so familiar with the smartclient code, but:

You changed DataSource.js but simply introduced a check in _handleJSONTextReply:

if (rpcResponse.status >= 0) {

an in there you run

if (evalText.match(/^\s*\{/)) {

which fails as described above. I believe the reason for this behaviour is caused in RPCManager.js in performTransactionReply. At the end it does the following:

            // All HTTP 2xx codes indicate success.  Success codes other than 200 OK are
            // somewhat obscure, but are used by Amazon S3 and possibly other REST APIs
            if (status > 299 || status < 200) { //error
                var url = transaction.URL;
                if (transaction.isProxied) {
                    url = transaction.proxiedURL+" (via proxy: " + url +")";
                }
                results = this._makeErrorResults(transaction, {
                    data : "Transport error - HTTP code: "+ status
                           +" for URL: " + url
                           + (status == 302 ? " This error is likely the result"
                              + " of a redirect to a server other than the origin"
                              + " server or a redirect loop." : ""),
                    status: isc.RPCResponse.STATUS_TRANSPORT_ERROR
                });
                this.logDebug("RPC request to: " + url 
                             + " returned with http response code: " 
                             + status +". Response text:\n"
                             + xmlHttpRequest.responseText);
                             
            }

That means, rpcResponse.status will be 0 and therefore _handleJSONTextReply will perform the evalText.match check. But RPCManager.js/performTransactionReply has set jsonText to an Array:

[{data: "Transport error - HTTP code: 404 for URL: /rpc/custJsonRPCDataSource_jsonrpc"
status: -90}]

And of course the Array does not do well with .match.


I was about to check whether CustJsonRPCDataSource (http://forums.smartclient.com/showth...light=json-rpc) has to be changed for smartclient 6.5.1. It seems to work fine as long as you use dataFormat: "custom" for backends which answer with HTTP status codes > 299 for Json-RPC errors.

If you have a look at CustJsonRPCDataSource.js/_handleJSONTextReply - it handles such situations by taking into account that the returned result can be a string, array or whatever else.


Regards,
Erich

[Isomorphic]

Hi Erich,

This check:

if (rpcResponse.status >= 0) {

should be safe. At that point, only a negative status should indicate an error. From your description, you seem to say that you have a status of 0, but indicating an error condition like 500 or 404. Is that the case?

If so, are you accesing your server functions through a filesystem URL like "file://", or some other non-HTTP means?