No announcement yet.
  • Filter
  • Time
Clear All
new posts

    Best practice / HowTo: User triggered background processing?

    Hi Isomorphic,

    as written here, I have a problem with BatchUploader and the amount of data I can transfer.
    It is clear that BatchUploader will hit a limit eventually, let it be 1000 or 10000 records, as the browser is blocked then and this leads to a bad user experience.
    So the solution is to turn to Background processing. For me, this will consist of
    1. Upload CSV to DB as a file
    2. Process it and insert data into a temporary table (by "temporary table" I mean normal DB table, but used for import and not the production table. I don't mean something like this)
    3. Display temporary data and errors (like BatchUploader after click on "Upload", but already persisted in the DB and not in-memory)
    4. Possible error correction by the user
    5. Transfer data from temporary table to production table
    Now steps 2 and 5 can take some time. What is the best practice here?
    I can think of two solutions:
    1. "Normal" custom DSRequest and some showPrompt:false setting where the request returns eventually (perhaps without a Callback, perhaps with a Callback + Notification in 12.1), but the user continues his or her work.
    2. "Normal" custom DSRequest that triggers a new thread and returns directly.
    I don't know if 1) has a problem, but to me it seems more easy with less issues w.r.t. request lifecycle, type="creator" .ds.xml-fields etc.

    I did not find related threads here (only this one) and also don't know about a docs topic on this.

    What is your suggested solution for this use case?

    Best regards

    The problem with #1 is that the browser will severe the connection after 4 minutes, and some proxies and firewalls will sever it sooner.

    We generally recommend returning immediately, and using either Messaging or polling to find out that the processing is complete.


      Hi Isomorphic,

      thanks, I assumed that this might be a drawback of 1).
      Also thanks for the possible solutions of keeping the user informed. As this is not that important, polling and / or a mail once finished will be enough for me.

      W.r.t. to the triggering:
      I could of course create a backlog entry and use a servlet and hit that via a wget cron job. I could also use Quartz.
      Both would involve something like RequestContext.instance(this, servletRequest, servletResponse) somewhere, I assume.
      Both would suffer from some delay, until the next run of the scheduler.

      Is there some way and something to watch if I want to do this like this:
          public DSResponse startBackgroundProcessing(DSRequest dsRequest, HttpServletRequest servletRequest) throws Exception {
              // new Thread to start background processing
              doBackgroundProcess(dsRequest, servletRequest);
              return new DSResponse().setSuccess();
      Am I allowed to use dsRequest / servletRequest in this situation?
      Anything to watch in order not to leak objects, connections or similar?

      Best regards


        If you are in a non-servlet thread you are in the same situation that you are in if you are running from the command line, and our Standalone DataSource Usage overview applies.