BatchUploader question

[ecruz]

I have a question re. custom validations and the BatchUploader. We are using the latest SmartGWT EE edition, and have a need to upload, validate, and import CSV data.

For the validation part, we have created a custom datasource and overridden the validate() method (calling the super version in that method as well, of course). This has worked nicely when the CSV file is first uploaded, but we have discovered a scenario where it is not working for us.

Basically, some of the custom validations require that two fields be present in combination, call them fields A & B. We then import a CSV for testing with two rows of where A & B are blank in both rows. The batchuploader correctly flags these fields in both rows with the (!).

The first problem is that when the A&B fields are entered manually on the first row, the (!) indicators actually go away on both that row and on the 2nd row as well, even though that 2nd row is still missing the A&B fields. This then enables the import of both rows when the Commit button is selected, which is not the desired result!

The second problem is that the validation of fields A & B on the 2nd blank row can only be triggered when a value is manually entered and then tabbed on one of the fields (onChange, basically). But if the user just keeps the fields blank and tabs through them (not entering a new value), then the validation is NOT triggered, which allows the row to be (erroneously) committed...

I've tried to find a workaround so that the custom validate() would be called as every field is tabbed through (onBlur, basically), but have had no luck so far.

Is there a workaround that you can suggest? Of maybe a fix needed?

Thanks.

[Isomorphic]

That's such a bizarre effect that we can't even speculate as to the cause. Make sure you have the latest nightly in case you hit some kind of overnight regression. Aside from that, we'll need a way to reproduce the problem in order to help.

[Isomorphic]

Actually, we'll speculate a little - this might happen if there are no primaryKeys in the data or the primaryKeys are not unique. There needs to be some way of uniquely identifying rows or the client and server basically cannot communicate about things at a per-row level, which is required in order to edit individual rows.

[ecruz]

Thanks very much for the suggestion... It inspired me to look under the hood some more, and I did make a discovery that I think indicates a bug (or possibly an omission):

The problem we were seeing with validations crossing rows actually had to do with what validations were being called (and by what method). On it's own, the BatchUploader seems to automatically invoke the validations in the ds.xml, which is good so far. But in our case, we also have more complex custom validations in a custom server-side class we created for the datasource.

The confusion occurred because we had only overwritten the validate() method in the custom datasource code -- with this in place, the custom validations were being invoked ONLY INITIALLY, when the file was first uploaded. Later, when fields were changed, only the ds.xml validations were being invoked because we had no similarly accommodated a validation RPC request, which was what was happening for subsequent validations. Once we added logic there to also call the custom validations, things got better (but not perfect).

The problem that I am seeing now using the javascript console is that not all navigation events in the uploaded data grid trigger an RPC validate event. Basically, this is evident when a field with a custom validation problem is clicked in, but no data is entered for it, and instead, the user navigates away from the (empty) field using the return key, or up/down/left/right arrows, or the tab key.

Without the RPC validate to trigger the custom validation code, the grid displays no validation markers (!) after this navigation and allows the problem row to be uploaded...

I guess my question is: shouldn't the RPC validation request be triggered consistently by any navigation event in the upload grid?

Thanks.

[Isomorphic]

Trying to follow this, but, what do you mean by an RPC validate request? There are DataSource requests of type "validate", is that what you mean?

If you've isolated the cause here, the clearest thing might be to just take one of the sample DataSources, add some kind of trivial serverOnly validator (like value != 1) and explain how to reproduce the problem with that.

[ecruz]

I was able to recreate the problem on the SmartGWT EE showcase.

I uploaded the sample CSV, which includes multiple validation errors that need to be corrected. But instead of correcting them, I just double clicked in the first field with a (!), Description in the first row, then tabbed through the remainder of the fields on that row. As I did this, all of the (!) indicators disappeared, even though I did not correct any of the problems. I did this for the remainder of the rows as well, leaving no validation indicators again in spite of making no corrections. I then clicked the commit button, and it displayed the "added records" popup.

I monitored the javascript console why doing this, and no RPC validation requests occurred as I navigated/tabbed through the fields.

I have several screenshots that show this behavior, but they are too large to attach to this posting. If you email me directly, I can send them to you.

Thanks!

[Isomorphic]

You can make the screenshots smaller to upload them here, but all we need is the CSV file and instructions.

[ecruz]

To recreate the problem, go to the SmartGWT EE showcase and load the BatchUploader page:

http://www.smartclient.com/smartgwtee/showcase/#batch_uploader

- Open the javascript console so that the RPC request for the upload process can be monitored.

Then follow the first four steps on that page to upload the csv file provided there.

Once the csv data has been displayed in the upload grid along with the validation issues (!):

- Double click on the first error, the Description field.
- Do NOT enter a value there, and instead tab to the the next field

Note that the validation indicator (!) goes away although the problem (i.e., no value) is still present

- Continue tabbing through the fields on this first row

Note that the remaining validation indicators are also removed although none of the validation issues are addressed.

Also note in the javascript console that no RPC validate request occur as you tab through the fields.

- Repeat these steps for the remaining rows.

Note that the (!) validation errors are completely removed for all rows although no corrections have been made.

- Click the commit button.

Note that the system indicates that the rows have been added. This should not be the case, since none the original validation issues have been addressed.

[ecruz]

See attached in chronological (filename) order... The batch uploader screenshot shows the validation indicators (!) missing, although no corrections have been made to the data. The console screenshots show that no validation requests have been triggered while navigating from field to field (omitting required data to correct the validation problems).

[Isomorphic]

Thanks for the additional detail - we should have said, it's been queued up to be looked at just based on your instructions for reproducing the problem, but more detail always helps.

[Isomorphic]

Thanks for the report - this has been fixed in mainline and will appear in nightlies shortly

[ecruz]

So the fix will appear here?

http://www.smartclient.com/builds/SmartGWT/3.0/PowerEdition

Thanks again,

- E.

[Isomorphic]

The fix had only been made in the current dev branch for the next release (/builds/SmartGWT/3.1d/PowerEdition).

It has now been back-ported to the 3.0 release/patch branch at /builds/SmartGWT/3.0p/PowerEdition. This is a stable branch, updated only with bug-fixes.

[ecruz]

I will check it out and retest things here shortly.

I do have one more question based on something mentioned earlier in this thread: if a ds.xml specificies a primary key field, then if a csv file for upload includes rows with duplicate keys (either against the existing db records, or in the yet-import-csv data), shouldn't the upload grid indicate with a validation prompt(!) where that duplication exist?

As it stands right now, the showcase EE example doesn't seem to handle this scenario... See the attached csv file that specifies two new records that have duplicate itemId fields (both have 1) but otherwise good import data. If you process it through the batch upload process, multiple system-level error popups appear as the system allows you to try to process this import. Shouldn't validation indicators (!) appear instead on the initial upload of the file, and import be prevented because of this?

(Note: csv file are supported for upload in the forum, so you will have to rename from .txt to .csv).

Thanks.

[Isomorphic]

This appears to be because it's a field of type="sequence": values aren't expected at all, and if included, would be ignored.