Best practice / HowTo: User triggered background processing?

[Blama]

Hi Isomorphic,

as written here, I have a problem with BatchUploader and the amount of data I can transfer.
It is clear that BatchUploader will hit a limit eventually, let it be 1000 or 10000 records, as the browser is blocked then and this leads to a bad user experience.
So the solution is to turn to Background processing. For me, this will consist of

1. Upload CSV to DB as a file
2. Process it and insert data into a temporary table (by "temporary table" I mean normal DB table, but used for import and not the production table. I don't mean something like this)
3. Display temporary data and errors (like BatchUploader after click on "Upload", but already persisted in the DB and not in-memory)
4. Possible error correction by the user
5. Transfer data from temporary table to production table

Now steps 2 and 5 can take some time. What is the best practice here?
I can think of two solutions:

1. "Normal" custom DSRequest and some showPrompt:false setting where the request returns eventually (perhaps without a Callback, perhaps with a Callback + Notification in 12.1), but the user continues his or her work.
2. "Normal" custom DSRequest that triggers a new thread and returns directly.

I don't know if 1) has a problem, but to me it seems more easy with less issues w.r.t. request lifecycle, type="creator" .ds.xml-fields etc.

I did not find related threads here (only this one) and also don't know about a docs topic on this.

What is your suggested solution for this use case?

Best regards
Blama

[Isomorphic]

The problem with #1 is that the browser will severe the connection after 4 minutes, and some proxies and firewalls will sever it sooner.

We generally recommend returning immediately, and using either Messaging or polling to find out that the processing is complete.

[Blama]

Hi Isomorphic,

thanks, I assumed that this might be a drawback of 1).
Also thanks for the possible solutions of keeping the user informed. As this is not that important, polling and / or a mail once finished will be enough for me.

W.r.t. to the triggering:
I could of course create a backlog entry and use a servlet and hit that via a wget cron job. I could also use Quartz.
Both would involve something like RequestContext.instance(this, servletRequest, servletResponse) somewhere, I assume.
Both would suffer from some delay, until the next run of the scheduler.

Is there some way and something to watch if I want to do this like this:

    public DSResponse startBackgroundProcessing(DSRequest dsRequest, HttpServletRequest servletRequest) throws Exception {
        // new Thread to start background processing
        doBackgroundProcess(dsRequest, servletRequest);
        return new DSResponse().setSuccess();
}

Am I allowed to use dsRequest / servletRequest in this situation?
Anything to watch in order not to leak objects, connections or similar?

Best regards
Blama

[Isomorphic]

If you are in a non-servlet thread you are in the same situation that you are in if you are running from the command line, and our Standalone DataSource Usage overview applies.