DynamicForm URI is too large

[aderosso]

We're running into an issue with a DSRequest coming from an add operation from a DynamicForm. It look like the request payload is also appended in the request URL so we're getting a 414 (URI is too large) http response code.

Is this normal? The same behavior isn't present coming from a ListGrid. I've attached screenshots of the request from the browsers DevTools console.

[Isomorphic]

Does this form involve a binary field that you are uploading? If so, SmartClient will take the rest of the form's data (the non-binary fields) and append it to the URI, in order to allow transparent binary field handling (in other words, even though an upload is potentially occurring, you still just call DynamicForm.saveData() and the server side system is able to treat it as a normal "update" DSRequest.

If you can let us know whether a binary field is present and that's the problem, we can suggest approaches to get around the problem.

[aderosso]

Yes, there are binary fields in the form.

[Isomorphic]

Thought so.

OK, there are a few simple ways to reduce the request size:

1. you may be populating the form with fields that are not actually edited. If you just omit values for fields that are not edited, those won't be sent

2. the framework automatically sends the "original" values to the server as dsRequest.oldValues, for use in things like concurrent change detection (sometimes called "long transactions"). If you don't have server logic that depends on oldValues, you can strip this out (in DataSource.transformRequest, set it to null) and this will reduce the request size by a bit less than half

3. you can do the upload in a separate form, where none of the other values are edited

[aderosso]

I don't think any of those work. This is an add operation so all of the values are new. We have a rich text editor, allowing users to paste images and other content into it. The text field (including base64 encoded images), was over 1M characters long, not much way to trim this down.

Is there a way to force it only to be in the post body instead of the url?

[Isomorphic]

Values for binary fields are not encoded into the URL. So take a second look at strategy #1 above: if you look at dynamicForm.getValues() you'll probably find that you have something large in there that isn't the binary value. So just set that value to null, and you'll be done.

While that should fix it, also note, if you have a base64 String in memory in the browser, there is no particular reason to use a binary field. You could just treat it as a normal field value, and then 1M is fine because a normal POST will be issued.

[aderosso]

The form has both a binary field for file attachments, and a rich text editor field that the user can paste content into (like images). We actually encountered this from one of our customers that simply put too much text in the rich text editor field. The url limit is 8192, the rich text editor field value plus the rest of the request data was over this limit.

[Isomorphic]

Ah OK, it sounded before like the binary field was the rich text data.

In this case, #3 above is your best approach. You can just split the form so the binary field is separate, save one, wait for the callback, save the other. It can be made to look visually the same and appear the same to the end user.

If it's daunting to split the form (for whatever reason), another approach is:

1) clear the rich text value and hold onto it in a variable

2) save the form with just the binary

3) save the rich text value with a direct DataSource.updateData() call so it doesn't involve the form with its binary field.

Note by the way that it is technically possible to POST form data and a binary to the server, but it requires an HTML5 API ("FormData"), but unfortunately, that API is not available in all supported browsers, and has some pretty bad bugs in some browsers where it's "supported". We'll switch to that API for this use case once it's better supported - you could accelerate that via our Feature Sponsorship process if you want it now and don't care about older browsers.

[aderosso]

Awesome, thank you for the guidance. We will try the 3rd approach.

[aderosso]

So we actually encountered this in 2 places in our application. One place was from a form saved to a database so we can split it to multiple forms. The other however is using a datasource, but not backed by ant database table. We use that for sending email from our application. There is a form with to addresses, attachments, subject, body, etc. In this case it would be difficult to persist with 2 calls since the data sends an email all in one request. Is there a way to use a DMI call or something directly that can send both form data and a binary file in the same call?

[Isomorphic]

Unfortunately, all the options here boil down to the special kind of HTTP request that needs to be sent to deliver all the data (mixed binary and form values) in one single request body, and to browser support for the necessary APIs to form such a request.

However, there are a couple of approaches here:

1. the HTML5 FormData API, mentioned above

2. directly forming the HTTP request body (with multipart boundaries). To do this, you use the FileReader API to turn the binary value in the upload field into a base64 string. This has better browser support (IE10+) than FormData.

Unfortunately, there's a further wrinkle, which is that you need server-side processing for this HTTP format as well - it wouldn't "just work" with existing server processing. So you would need a special servlet that received this kind of request, but that servlet could be relatively simple: use off-the-shelf libraries to parse the request, then turn it into a standard SmartClient DSRequest, execute() that, and you're done (well, except some error handling..).

We can help with this, if you like. We can do a Feature Sponsorship to add automatic handling of this kind to new versions (covering all the browser support details, trying to maximize support, working around bugs, etc), then provide a "patch" for your app, consisting of a JavaScript file and replacement server .jar that can be used with your current version.

Going back, assuming we don't do any of that and you handle this in app code: your problem is basically how to store either the binaries or non-binaries on the server until a second request comes through. Although it's generally a scalability "anti-practice", if the volume of emails isn't too high, it could be OK to take the non-binary data and stick it in the servlet session, awaiting the second request that has the binary data, then send the email and delete everything.

You can also mix approaches: sponsor the feature so it's going to be ready next release, then use the short-term approach of temporary session storage and put in comments saying that the session storage approach can be deleted after upgrade.

Finally, note, if you look at how the large email providers handle this (like GMail), you can see that binary email attachments are saved first, while you continue to edit your email message. So they are clearly taking the attachments and sticking them into temporary storage, probably a NoSQL key-value DB. The separate save also has the advantage that the overall message can exceed both the maximum size and possible HTTP timeout limit for a very large email.

Obviously, you're not trying to compete with GMail, you just need an app-specific email function, so your requirements are different.

And a last note: we've never seen anybody hit this limit before, and we're sorry it's so complex to get around. We'd like to just make this work transparently, but it's only been in the past few months that browser support has become good enough to implement an approach that can handle binary upload forms with non-binary data over 8k, without multiple requests (or without having to say something like: we can handle over 8k non-binary values if all of your users are on Chrome 42+, which we don't like to do, for obvious reasons).

[aderosso]

Thank you for the suggestion. I think we will try saving the binary file first, I think that will be the easiest approach for us, and like you said, that matches other email programs. Thank you!