Run an asynchronous task from a DMI

[streckerd]

We need a DMI to kick off a long running task. This task can take anywhere from 5 to 60 minutes (interfacing with external devices). Our first work-around was to increase the RPC timeout value to 60 minutes. Not a good solution. A better solution is have the DMI start an asynchronous task and return. The asynchronous task will need to perform some database updates which must behave as transactions (all or nothing) as well as database reads. However, since the DMI kicks off the task and returns a success to the client indicating the asynchronous task was started, the task no longer has access to the RPCManager for interacting with database.

We have been looking into following options but have not yet attempted to implement any of them yet.
* Standalone DataSources - It's not clear if this would work since the task is running inside the server code and not a true stand-alone application
* Writing embedded SQL or stored procedures to perform the database operations. Defeats the beauty of DataSources.
* Researching how SQLDataSource.getSQLClause() and SQLConnectionManager.getConnection() could be used. with a DataSource?

Does anyone have any suggestions on how to easily (via a DataSource ideally) call a DMI which launches an asynchronous task which perform database actions (read/write) and have the DMI return before the task is complete. I guess the real question is how to perform database reads and transactions from task (java runable) launched from a DMI? Any ideas would be greatly appreciated.

Thanks.

[Blama]

Hi streckerd,

I'm doing such a thing with my Mailimport, where I get the mails in the background via a servlet that is hit by cron job (your way 1).
In order for the servlet to act as if a specific user called it, some steps are needed:

These links help:
Thread on standalone usage
RequestContext-docs
Convenience constructor thread
Also related

My Convenience method overloads (you might only need some):

public static void prepareRequest(DSRequest dsRequest, RPCManager rpcManager, DSTransaction dsTransaction, Long userID,
            RequestContext requestContext) throws Exception {
        if (rpcManager != null) {
            dsRequest.setRPCManager(rpcManager);
        } else {
            prepareRequest(dsRequest, requestContext);
            if (dsTransaction != null) {
                dsRequest.setDSTransaction(dsTransaction);
                if (userID != null && dsTransaction.getUserId() == null) {
                    dsTransaction.setUserId(userID.toString());
                    dsTransaction.setClientRequest(false);
                }
            } else if (dsTransaction == null) {
                if (userID != null) {
                    dsRequest.setUserId(userID.toString());
                    dsRequest.setClientRequest(false);
                }
            }
        }
    }

    public static void prepareRequest(DSRequest dsRequest, DSTransaction dsTransaction, Long userID, RequestContext requestContext)
            throws Exception {
        prepareRequest(dsRequest, null, dsTransaction, userID, requestContext);
    }

    public static void prepareRequest(DSRequest dsRequest, Long userID, RequestContext requestContext) throws Exception {
        prepareRequest(dsRequest, null, null, userID, requestContext);
    }

    public static void prepareRequest(DSRequest dsRequest, RPCManager rpcManager) throws Exception {
        prepareRequest(dsRequest, rpcManager, null, null, null);
    }

    public static void prepareRequest(DSRequest dsRequest, RPCManager rpcManager, DSTransaction dsTransaction) throws Exception {
        prepareRequest(dsRequest, rpcManager, dsTransaction, null, null);
    }

    public static void prepareRequest(DSRequest dsRequest, RPCManager rpcManager, DSTransaction dsTransaction, RequestContext requestContext)
            throws Exception {
        prepareRequest(dsRequest, rpcManager, dsTransaction, null, requestContext);
    }

    public static void prepareRequest(DSRequest dsRequest, DSRequest parentDSRequest) throws Exception {
        prepareRequest(dsRequest, parentDSRequest.getRPCManager(), parentDSRequest.getDsTransaction(),
                Long.getLong(parentDSRequest.getUserId()), parentDSRequest.getContext());
    }

    public static void prepareRequest(DSRequest dsRequest, RequestContext requestContext) {
        dsRequest.setContext(requestContext);
    }

Most important:

dsTransaction.setUserId(userID.toString());
dsTransaction.setClientRequest(false);

That way, creator fields will be filled correctly. setClientRequest(false) disablesDeclarative Security you most likely don't want in this case.

A request looks like this:

                            DSRequest uploadReq = new DSRequest("MYDS", DataSource.OP_ADD);
                            Helper.prepareRequest(uploadReq, userID, receiveMailRequestContext);
                            uploadReq.setValues(mailData);
                            final DSResponse uploadResp = uploadReq.execute();

Now in the GUI you can see the status of the background task, if you maintain the status accordingly.
By having requests that explicitly do not take part in the Transaction you could also implement a progress bar for the GUI.

Please let me know how that works for you.
@Isomorphic: Any feedback to this is appreciated.

Best regards
Blama

[Isomorphic]

Standalone DataSources is the right way to do this, and you'll see in the docs that we mention your use case explicitly.

We'd also recommend that you read the standalone DataSource Usage overview before trying to understand Blama's answer, as he's talking about finer points such as how to do transactions and how to enforce security rules when in standalone mode.

[streckerd]

Blama, Thank you for the quick and detailed response. As we work through this, I'll let you know how it works out.

Isomorphic Administrator, Thank you for confirming that Standalone DataSources is the Right path.

Again, that's for the quick response.

[streckerd]

We are still working on this, but just out of curiosity, what if the RPCManager from the client is passed to the asynchronous task? Is the RPCManager invalidated once the initial response is returned to the client even though the task continues to run?

[Isomorphic]

Yes, it would be invalid to hold onto the RPCManager instance past the end of the HttpRequest lifecycle. If you need something like the user's authenticated roles, extract those and hold onto them as a separate variable.

[streckerd]

Update on how things worked out - We have successfully used standalone data source to perform database transaction updates from an asynchronous task created from a DMI and also from Restful services our application support. The standalone datasource make this very easy. Well, it was easy because of the great post provided by Blama, Anyone who needs to perform transaction-based database updates where they do not have access to a RPC Manager should review this thread as it contains all the information needed to be successful.

Thank you Blama and Isomorphic Administrator.

[tece321]

Am interested in this subject.

We have a case where we need to run a DB SQL task that can take from 1 minute to 60 minutes to complete, so the RPC client request frequently times out.

You mention in a reply "you read the standalone DataSource Usage overview before trying to understand Blama's answer". I'm not clear on what this reference is.

I assume it is "https://www.smartclient.com/smartgwt...ataSource.html" , but that doesn't make sense as a reference.

Finally you state: "We have successfully used standalone data source to perform database transaction updates from an asynchronous task created from a DMI and also from Restful services our application support. The standalone datasource make this very easy."

Where can I find this implementation example?

Thanks

[Blama]

Hi tece321,

the relevant docs are StandaloneDataSourceUsage and possibly the concept described here, if you need user data for modified_by fields etc and user roles.

Best regards
Blama

[tece321]

@Blama,

Thanks for the prompt reply! I can see I'm going to be busy with this....

[tece321]

I have read the references. I'm still unclear on some parts of the concepts. My intention is that a Client action will start this process. It will issue a DMI request to the server, and get a response - almost immediate - that the request has been queued by the server. So I'm expecting standard SmartGWT DMI processing in this portion of the overall process.

Meanwhile the server will pass this request to the Standalone DataSource, which may take up to 60 minutes to process.

Is the Standalone DataSource running inside a servlet? Or is it a self running process on the server with it's own Main() entry point? And if so how does the server communicate with that standalone process?

Excuse my inexperience.

[Isomorphic]

In your DMI, you can start a thread or run a stand-alone process. In that thread or standalone process, you use DataSources in the style shown in the Standalone DataSource Usage docs.

For inter-thread or inter-process communication, you can use any standard Java approach you like.

[tece321]

Understand. Thanks

[tece321]

I think the process of starting up a new thread, to communicate with the the stand-alone DataSource thread, may be best suited to the Messaging jar/app. Allow the client to message with the Messaging.jar on the server, and use its asynchronous nature to exchange data with the client that is monitoring the overall status.

Rather than reinventing a thread handler or communications with a stand-alone process. Does this seem like a reasonable approach?

[Isomorphic]

HTTPRequest threads are meant to be short-running, so we did not mean to suggest using Semaphores or anything like that..

Yes, typical approaches here include sending a notification that the result is ready via Messaging, or having the client poll for it, or some combination (to handle cases where the long-running process hangs or fails).

You might also look at Quartz instead of starting a thread per se, as it essentially manages a thread pool for you.

Finally, note that EJB environments do not allow servlets starting threads as they are trying to implement throttling and other features that rogue threads would ruin.