Processing DSRequest queue

[matus_b]

Hi Isomorphic,

please, can you advice me how to approach processing of the queue of the dsRequests?

1. queue of validation requests

We send 2 validate DSRequests in queue to DS.
DS has DMI with validate server method.
In the validate server method we have to call remote validate service.
Remove validate service validates records in bulk - we must send all records from the queue in one call.

Our first approach was processing on last.

• In the validate server method acquire all dsRequests.
• If the currently processed dsRequest is not last in the queue then return dummy success dsResponse.
• If the currently processed dsRequest is last in the queue, then call remote service with records from the whole queue.
  • Then get dummy success response from the rpcManager and set it up with proper response data and status.

This approach has several fallbacks.

Therefore I strive to do all the processing on first:

• In the validate server method acquire all dsRequests.
• If the currently processed dsRequest is first in the queue, then call remote service with records from the whole queue.
  • Then for call rpcManager.send() for every dsRequest from queue and provide proper dsResponse built from the remote service answer.
• If the currently processed dsRequest is not first in the queue then return already set dsResponse in the rpcManager.

Unfortunately this approach produces some kind of weird dsResponse in which the array of responses is repeated 2 more times (please see next comment with the code and request/response).

2. queue of update requests
(for this one I don't have showcase example prepared, but if you would require it, let me know)

We send 2 update DSRequests in queue to DS.
DS has DMI with update server method.
In the update server method we first call validate.
We can update only when the records is marked as valid in the response from the remote service.

When we are using processing on last approach, then I don't how during the processing of last update request from queue, to see the validation responses of the previous requests.
If I call rpcManager.getResponse(dsRequest), then I will get the dummy update request and not the validate response.

Therefore I think that processing on last approach is not possible to use at all and the only option will be to process on the first.

I suspect that the rpcManager.send() doesn't produce proper queue response and that maybe you will find out that you need to fix something.
Or maybe I don't really understand how to properly handle queue and you can give some hint what to try.

Regards,
Matus

PS: code and requests/response example in next post

[matus_b]

Example for queue of validation requests while processing on first:

Reproduced on: SmartClient Version: v10.0p_2017-05-14/PowerEdition Deployment (built 2017-05-14)
Browser version: GC 58

CLIENT:

final DataSource queuingUserDS = DataSource.get("queuing_userHB");
final Record record1 = new Record();
record1.setAttribute("firstName", "Jack");
record1.setAttribute("surname", "Sparrow");
record1.setAttribute("email", "jack.sparrow@email.com");
final Record record2 = new Record();
record2.setAttribute("firstName", "Alfred");
record2.setAttribute("surname", "Hitchcock");
record2.setAttribute("email", "alfred.hitchcock@email.com");

final Button button = new Button("send validate queue");
button.addClickHandler(new ClickHandler() {
    @Override
    public void onClick(ClickEvent event) {
        RPCManager.startQueue();

        queuingUserDS.validateData(record1);
        queuingUserDS.validateData(record2);

        RPCManager.sendQueue(new RPCQueueCallback() {
            @Override
            public void execute(RPCResponse... response) {
                SC.say("queue is finished");
            }
        });
    }
});

DS:

<DataSource
    ID="queuing_userHB"
    serverType="hibernate"
    beanClassName="com.smartgwt.sample.showcase.server.User"
    testFileName="/ds/test_data/queuing_user.data.xml"
>
    <fields>
        <field name="userID" type="sequence" primaryKey="true" canEdit="false" />
        <field name="firstName" type="text" required="true"/>
        <field name="surname" type="text" required="true"/>
        <field name="department" type="text" />
        <field name="email" type="text" required="true">
          <validators>
            <validator type="isUnique"/>
          </validators>
        </field>
    </fields>

    <serverObject lookupStyle="new" className="com.smartgwt.sample.showcase.server.CustomUserDataDMI" />

    <operationBindings>
        <operationBinding operationType="validate" serverMethod="validate" requires="true"/>
    </operationBindings>
    
</DataSource>

DMI:

package com.smartgwt.sample.showcase.server;

import com.isomorphic.datasource.DSRequest;
import com.isomorphic.datasource.DSResponse;
import com.isomorphic.rpc.RPCManager;

import java.util.List;

public class CustomUserDataDMI {

    public DSResponse validate(DSRequest dsRequest) throws Exception {
        System.out.println("validate method - qwerty");
        final RPCManager rpcManager = dsRequest.getRPCManager();
        final List<DSRequest> queueRequests = rpcManager.getRequests();

        if (isFirst(dsRequest, queueRequests)) {
            for (DSRequest queueRequest : queueRequests) {

                // calling remote service with bulk of data
                final DSResponse queueResponse = new DSResponse(queueRequest);
                // first one valid other not
                queueResponse.setStatus(dsRequest == queueRequest ? 0 : -4);
                queueResponse.setData(queueRequest.getValues());

                // setting all responses
                rpcManager.send(queueRequest, queueResponse);
            }
        }

        // returning previously set response via rpcManager.send()
        return rpcManager.getResponse(dsRequest);
    }

    private boolean isFirst(DSRequest dsRequest, List<DSRequest> requests) {
        return dsRequest != null && requests != null && !requests.isEmpty() && requests.get(0) == dsRequest;
    }

}

Queue request:

<?xml version="1.0"?>
<transaction xmlns:xsi="http://www.w3.org/2000/10/XMLSchema-instance" xsi:type="xsd:Object">
  <transactionNum xsi:type="xsd:long">1</transactionNum>
  <operations xsi:type="xsd:List">
    <elem xsi:type="xsd:Object">
      <values xsi:type="xsd:Object">
        <firstName>Jack</firstName>
        <surname>Sparrow</surname>
        <email>jack.sparrow@email.com</email>
      </values>
      <operationConfig xsi:type="xsd:Object">
        <dataSource>queuing_userHB</dataSource>
        <repo xsi:nil="true"/>
        <operationType>validate</operationType>
        <textMatchStyle>exact</textMatchStyle>
      </operationConfig>
      <validationMode>full</validationMode>
      <appID>builtinApplication</appID>
      <operation>queuing_userHB_validate</operation>
      <oldValues xsi:type="xsd:Object">
        <firstName>Jack</firstName>
        <surname>Sparrow</surname>
        <email>jack.sparrow@email.com</email>
      </oldValues>
    </elem>
    <elem xsi:type="xsd:Object">
      <values xsi:type="xsd:Object">
        <firstName>Alfred</firstName>
        <surname>Hitchcock</surname>
        <email>alfred.hitchcock@email.com</email>
      </values>
      <operationConfig xsi:type="xsd:Object">
        <dataSource>queuing_userHB</dataSource>
        <repo xsi:nil="true"/>
        <operationType>validate</operationType>
        <textMatchStyle>exact</textMatchStyle>
      </operationConfig>
      <validationMode>full</validationMode>
      <appID>builtinApplication</appID>
      <operation>queuing_userHB_validate</operation>
      <oldValues xsi:type="xsd:Object">
        <firstName>Alfred</firstName>
        <surname>Hitchcock</surname>
        <email>alfred.hitchcock@email.com</email>
      </oldValues>
    </elem>
  </operations>
</transaction>

Queue response (seems to be wrong):

//isc_RPCResponseStart-->
[{
    affectedRows: 0,
    data: {
        firstName: "Jack",
        surname: "Sparrow",
        email: "jack.sparrow@email.com"
    },
    invalidateCache: false,
    isDSResponse: true,
    queueStatus: -1,
    status: 0
}, {
    affectedRows: 0,
    data: {
        firstName: "Alfred",
        surname: "Hitchcock",
        email: "alfred.hitchcock@email.com"
    },
    invalidateCache: false,
    isDSResponse: true,
    queueStatus: -1,
    status: -4
}]
//isc_RPCResponseEnd
//isc_RPCResponseStart-->
[{
    affectedRows: 0,
    data: {
        firstName: "Jack",
        surname: "Sparrow",
        email: "jack.sparrow@email.com"
    },
    invalidateCache: false,
    isDSResponse: true,
    operationType: "validate",
    queueStatus: -1,
    status: 0
}, {
    affectedRows: 0,
    data: {
        firstName: "Alfred",
        surname: "Hitchcock",
        email: "alfred.hitchcock@email.com"
    },
    invalidateCache: false,
    isDSResponse: true,
    queueStatus: -1,
    status: -4
}]
//isc_RPCResponseEnd
//isc_RPCResponseStart-->
[{
    affectedRows: 0,
    data: {
        firstName: "Jack",
        surname: "Sparrow",
        email: "jack.sparrow@email.com"
    },
    invalidateCache: false,
    isDSResponse: true,
    operationType: "validate",
    queueStatus: -1,
    status: 0
}, {
    affectedRows: 0,
    data: {
        firstName: "Alfred",
        surname: "Hitchcock",
        email: "alfred.hitchcock@email.com"
    },
    invalidateCache: false,
    isDSResponse: true,
    operationType: "validate",
    queueStatus: -1,
    status: -4
}]
//isc_RPCResponseEnd

[Isomorphic]

Your verbal description for the "processing on first" approach is right, but your code for doing it is wrong. From a DMI you can't start providing responses for DSRequests that haven't started processing yet; you're causing the weird doubled responses because you provide a DSResponse early but then the normal processing flow continues and obtains a second DSResponse.

Instead, call your service as the first request is being handled, then keep all the data that you need to fill in subsequent responses, but only provide the DSResponse in the normal way (returning it from a DMI, for example) rather than trying to call rpcManager.send() in the middle of the queue (which is invalid).

[matus_b]

I think, I didn't understand the rpcManager.send() documentation properly.

Anyway, we were able to implement processing on first by using rpcManager attribute as carrier of DSResponses.
During the processing of the first DSRequest from queue we add attribute where the key is combination of dataSource name, operation type and operation id; and the value is the list of DSResponses we created from the response from the remote service. Then during the processing of later DSRequests we simply pick from the attribute the DSResponse we need to return by index.

Thanks