No announcement yet.
  • Filter
  • Time
Clear All
new posts

  • Best practice / HowTo: User triggered background processing?

    Hi Isomorphic,

    as written here, I have a problem with BatchUploader and the amount of data I can transfer.
    It is clear that BatchUploader will hit a limit eventually, let it be 1000 or 10000 records, as the browser is blocked then and this leads to a bad user experience.
    So the solution is to turn to Background processing. For me, this will consist of
    1. Upload CSV to DB as a file
    2. Process it and insert data into a temporary table (by "temporary table" I mean normal DB table, but used for import and not the production table. I don't mean something like this)
    3. Display temporary data and errors (like BatchUploader after click on "Upload", but already persisted in the DB and not in-memory)
    4. Possible error correction by the user
    5. Transfer data from temporary table to production table
    Now steps 2 and 5 can take some time. What is the best practice here?
    I can think of two solutions:
    1. "Normal" custom DSRequest and some showPrompt:false setting where the request returns eventually (perhaps without a Callback, perhaps with a Callback + Notification in 12.1), but the user continues his or her work.
    2. "Normal" custom DSRequest that triggers a new thread and returns directly.
    I don't know if 1) has a problem, but to me it seems more easy with less issues w.r.t. request lifecycle, type="creator" .ds.xml-fields etc.

    I did not find related threads here (only this one) and also don't know about a docs topic on this.

    What is your suggested solution for this use case?

    Best regards

  • #2
    The problem with #1 is that the browser will severe the connection after 4 minutes, and some proxies and firewalls will sever it sooner.

    We generally recommend returning immediately, and using either Messaging or polling to find out that the processing is complete.


    • #3
      Hi Isomorphic,

      thanks, I assumed that this might be a drawback of 1).
      Also thanks for the possible solutions of keeping the user informed. As this is not that important, polling and / or a mail once finished will be enough for me.

      W.r.t. to the triggering:
      I could of course create a backlog entry and use a servlet and hit that via a wget cron job. I could also use Quartz.
      Both would involve something like RequestContext.instance(this, servletRequest, servletResponse) somewhere, I assume.
      Both would suffer from some delay, until the next run of the scheduler.

      Is there some way and something to watch if I want to do this like this:
          public DSResponse startBackgroundProcessing(DSRequest dsRequest, HttpServletRequest servletRequest) throws Exception {
              // new Thread to start background processing
              doBackgroundProcess(dsRequest, servletRequest);
              return new DSResponse().setSuccess();
      Am I allowed to use dsRequest / servletRequest in this situation?
      Anything to watch in order not to leak objects, connections or similar?

      Best regards


      • #4
        If you are in a non-servlet thread you are in the same situation that you are in if you are running from the command line, and our Standalone DataSource Usage overview applies.