Creating Dynamic grid without a specific ds

[dawityifter]

We have a requirement to generate dynamic grid that is determined by a search criteria user inputs. Just as an example, a user enters customerId, StartMonth and EndMonth to generate a specific result grid. We pass these three fields to the service layer which determines the exact sql to be executed and returns records with arbitrary fields. We can not define the ds as we don't know every column that the server returns and I am wondering if you have any suggestion (good practice to handle such cases.)
we have smartgwt 2.5 version with gwt 2.3 and running spring based DMI . I have looked at another suggestion to use DataSource.addDynamicDSGenerator() but I have not seen any example on how it is used and solves the issue I'm facing.
Thank you,

[Isomorphic]

You can either:

1. not declare fields in the .ds.xml definition, and only put field definitions on the ListGrid itself

or

2. use addDynamicDSGenerator() to create a fully dynamic DataSource

If this is just a read-only grid, there may not be much of a point of having the dynamically generated fields as part of the DataSource definition (approach #2), since server-side field definitions mostly control validation and other things that do not apply for read-only data.

One reason you might need to do #2 would be Excel export (if you need it) since in this case the server needs the field definitions in order to do correct export of typed data such as dates.

[dawityifter]

Yes we want read only access for now and I used the first approach you suggested. I am having hard time getting it to work and am wondering if you can help me. Even though I am sending records with three fields from the server, response.getAttributes().length on the client side returns 21 and I am not able to redraw the grid with the dynamic fields returned by the fetch operation.
DS....

<DataSource 
	ID="executedLeaseReport"
	serverType="generic" >
   <fields></fields>
  <serverObject lookupStyle="spring" bean="executedLeaseQuery" />
    <operationBindings>
        <binding operationType="fetch" serverMethod="makeExecutedLeaseQuery"/>
    </operationBindings>
</DataSource>

DMI ....

...
public DSResponse makeExecutedLeaseQuery(DSRequest dsRequest, HttpServletRequest request) {
      //  Some dummy data instead of actual dao call     
      ArrayList<HashMap<String, String>> result = new ArrayList<HashMap<String,String>>();
      HashMap<String, String> ttt = new HashMap<String, String>();
      ttt.put("name", "Elsa");
      ttt.put("age", "12");
      ttt.put("gender", "male");
       HashMap<String, String> ttt1 = new HashMap<String, String>();
      ttt1.put("name", "David");
      ttt1.put("age", "24");
      ttt1.put("gender", "female");
      result.add(ttt);
      result.add(ttt1);
       DSResponse dsResponse = new DSResponse();
       dsResponse.setDropExtraFields(false);
       dsResponse.setData(result);
       return dsResponse;
    }

And finally the client side code ....

submitButtonItem.setTitle("Generate");
        submitButtonItem.addClickHandler(new ClickHandler() {

            @Override
            public void onClick(ClickEvent event) {
                dynamicForm.validate();
                executedLeaseListGrid.fetchData(null, new DSCallback() {

                    @Override
                    public void execute(com.smartgwt.client.data.DSResponse response,
                            Object rawData, DSRequest request) {
                       // SC.say("attribute count :" + response.getAttributes().length);  // this is putting 21 instead of 3 
                        executedLeaseDS.addField(createField("test", "TEXT", "some test"));
                        for (Record record : response.getData()) {
                            for (String attribute : record.getAttributes()) {
                                Record field = record.getAttributeAsRecord(attribute);
                                executedLeaseDS.addField(createField(attribute,
                                        "TEXT",
                                        attribute));

                            }
                        }
                        executedLeaseListGrid.redraw();
                    }
                });
            }
        });

The entire file for the last snippet is attached. Thanks,

[Isomorphic]

Look in the RPC tab of the Developer Console to see what response you're actually sending to the browser.

[dawityifter]

DSRequest:

{
    "dataSource":"executedLeaseReport", 
    "operationType":"fetch", 
    "componentId":"isc_ListGrid_0", 
    "data":{
    }, 
    "startRow":962, 
    "endRow":1000, 
    "textMatchStyle":"exact", 
    "resultSet":[ResultSet ID:isc_ResultSet_0 (created by: isc_ListGrid_0)], 
    "callback":{
        "caller":[ResultSet ID:isc_ResultSet_0 (created by: isc_ListGrid_0)], 
        "methodName":"fetchRemoteDataReply"
    }, 
    "willHandleError":true, 
    "showPrompt":true, 
    "prompt":"Finding Records that match your criteria...", 
    "oldValues":{
    }, 
    "clientContext":{
        "requestIndex":2
    }, 
    "requestId":"executedLeaseReport$5441"
}

DSResponse ...

[
    {
        queueStatus:0, 
        isDSResponse:true, 
        invalidateCache:false, 
        status:0, 
        data:[
            {
                age:"12", 
                name:"Elsa", 
                gender:"female"
            }, 
            {
                age:"14", 
                name:"David", 
                gender:"male"
            }
        ]
    }
]

[Isomorphic]

You don't want to try to addField() on the DataSource. Create ListGridFields and provide them to the ListGrid.

[dawityifter]

Thank you guys for your support. I have a couple more question on this issue.
1. Now the grid is calling the fetch request twice even though the number of records is very small. Here is what I see on the RPC DSRequest tab:

{
    "dataSource":"executedLeaseReport", 
    "operationType":"fetch", 
    "componentId":"isc_ListGrid_0", 
    "data":{
        "startDate":"2011-12-29", 
        "endDate":"2012-03-29", 
        "property":"2332"
    }, 
    "startRow":0, 
    "endRow":999, 
    "textMatchStyle":"exact", 
    "resultSet":[ResultSet ID:isc_ResultSet_0 (created by: isc_ListGrid_0)], 
    "callback":{
        "caller":[ResultSet ID:isc_ResultSet_0 (created by: isc_ListGrid_0)], 
        "methodName":"fetchRemoteDataReply"
    }, 
    "willHandleError":true, 
    "showPrompt":true, 
    "prompt":"Finding Records that match your criteria...", 
    "oldValues":{
        "startDate":"2011-12-29", 
        "endDate":"2012-03-29", 
        "property":"2332"
    }, 
    "clientContext":{
        "requestIndex":1
    }, 
    "requestId":"executedLeaseReport$5440"
}

followed by :

{
    "dataSource":"executedLeaseReport", 
    "operationType":"fetch", 
    "componentId":"isc_ListGrid_0", 
    "data":{
        "startDate":"2011-12-29", 
        "endDate":"2012-03-29", 
        "property":"2332"
    }, 
    "startRow":962, 
    "endRow":1000, 
    "textMatchStyle":"exact", 
    "resultSet":[ResultSet ID:isc_ResultSet_0 (created by: isc_ListGrid_0)], 
    "callback":{
        "caller":[ResultSet ID:isc_ResultSet_0 (created by: isc_ListGrid_0)], 
        "methodName":"fetchRemoteDataReply"
    }, 
    "willHandleError":true, 
    "showPrompt":true, 
    "prompt":"Finding Records that match your criteria...", 
    "oldValues":{
        "startDate":"2011-12-29", 
        "endDate":"2012-03-29", 
        "property":"2332"
    }, 
    "clientContext":{
        "requestIndex":2
    }, 
    "requestId":"executedLeaseReport$5441"
}

And here is the specific java code rendering the grid :

...
submitButtonItem.setTitle("Generate");
        submitButtonItem.addClickHandler(new ClickHandler() {

            @Override
            public void onClick(ClickEvent event) {
                dynamicForm.validate();
                final List<ListGridField> fields = new ArrayList<ListGridField>();
                Criteria criteria = dynamicForm.getValuesAsCriteria();
                executedLeaseListGrid.fetchData(criteria, new DSCallback() {

                    @Override
                    public void execute(com.smartgwt.client.data.DSResponse response,
                            Object rawData, DSRequest request) {
                       Record[] record = response.getData();
                       if(record.length > 0){
                            for (String attribute : record[0].getAttributes()) {
                                Record field = record[0].getAttributeAsRecord(attribute);
                                fields.add(createField(attribute,
                                        "TEXT",
                                        attribute));

                            }
                        }
                        
                        executedLeaseListGrid.setFields(fields.toArray(new ListGridField[0]));
                        executedLeaseListGrid.markForRedraw();
                    }
                });
            }
        });
...

Where as the DMI side of the fetch is :

...
 ArrayList<LinkedHashMap<String, String>> calibratedResult = calibrateResult(results,startDate, endDate);
       if(calibratedResult != null){
                totalRows = calibratedResult.size();
            }
       DSResponse dsResponse = new DSResponse();
       dsResponse.setDropExtraFields(false);
       dsResponse.setTotalRows(totalRows);
       dsResponse.setStartRow(startRow);
       endRow = Math.min(endRow, totalRows);
       dsResponse.setEndRow(endRow);
       dsResponse.setData(calibratedResult);
        return dsResponse;

2. Eventhough I am using LinkedHashMap to keep the order of fields, some how dsResponse mangles the order of fields. Is there a way to keep the order?
Thanks,

[dawityifter]

Was my question not very clear? I would really appreciate your expert advice.
Thanks,

[Isomorphic]

1. we need a way to reproduce the problem and it clearly can't be related to just the code you've shown, which is just an ordinary fetchData() call.

2. the field order in the data is generally based on the order that the fields are declared in the DataSource

[dawityifter]

For the first issue, I will try to get you a test case but for the second issue, I am not declaring the fields in the datasource as I don't know it beforehand. As you can see from the original issue on this thread, I took your first advice

1. not declare fields in the .ds.xml definition, and only put field definitions on the ListGrid itself

I am populating a list of linked hashmaps and setting them to the dsResponse on my DMI. Looking at the history of this thread will help on where I am coming from.
Thank you,

[Isomorphic]

We wouldn't expect them to be reordered in this simple case, and we've made a note to look into how it's happening. However, we don't actually try to preserve order (or document that it will be preserved) and in some cases, preserving order would have serious performance consequences, so the best approach is to explicitly provide the order.

One way to do so is to include the order you want as an Array of Strings stored in the first record. You will probably end up needing to do something like this in the future anyway, as it's likely you will want to provide type information or other per-field metadata to your grid as you continue to implement this screen.

[dawityifter]

This is in regard to fetch being called twice (Issue #1 above). I have attached the client code. And this is the RPCRequest as shown on the console. I am not sure where/how the starRow and endRow values were set on both queries.
Request #1.

{
    "actionURL":"http://localhost:8080/ysconfig-1.0-SNAPSHOT/isomorphic/IDACall", 
    "showPrompt":true, 
    "prompt":"Finding Records that match your criteria...", 
    "transport":"xmlHttpRequest", 
    "promptStyle":"cursor", 
    "bypassCache":true, 
    "data":{
        "criteria":{
            "property":"2332", 
            "startDate":"2012-01-30", 
            "endDate":"2012-03-30"
        }, 
        "operationConfig":{
            "dataSource":"executedLeaseReport", 
            "repo":null, 
            "operationType":"fetch", 
            "textMatchStyle":"exact"
        }, 
        "startRow":0, 
        "endRow":999, 
        "componentId":"isc_ListGrid_0", 
        "appID":"builtinApplication", 
        "operation":"executedLeaseReport_fetch", 
        "oldValues":{
            "property":"2332", 
            "startDate":"2012-01-30", 
            "endDate":"2012-03-30"
        }
    }
}

Request #2

{
    "dataSource":"executedLeaseReport", 
    "operationType":"fetch", 
    "componentId":"isc_ListGrid_0", 
    "data":{
        "property":"2332", 
        "startDate":"2012-01-30", 
        "endDate":"2012-03-30"
    }, 
    "startRow":962, 
    "endRow":1000, 
    "textMatchStyle":"exact", 
    "resultSet":[ResultSet ID:isc_ResultSet_0 (created by: isc_ListGrid_0)], 
    "callback":{
        "caller":[ResultSet ID:isc_ResultSet_0 (created by: isc_ListGrid_0)], 
        "methodName":"fetchRemoteDataReply"
    }, 
    "willHandleError":true, 
    "showPrompt":true, 
    "prompt":"Finding Records that match your criteria...", 
    "oldValues":{
        "property":"2332", 
        "startDate":"2012-01-30", 
        "endDate":"2012-03-30"
    }, 
    "clientContext":{
        "requestIndex":2
    }, 
    "requestId":"executedLeaseReport$5441"
}

My DMI snippet is again:

public DSResponse makeExecutedLeaseQuery(DSRequest dsRequest, HttpServletRequest request) {
     // Send dummy data for test;
     //Query nativeQuery = null;
      //List<Object[]> results = null;
     
      Map modifiedFieldsOnRecord = dsRequest.getValues();
      Date startDate = null;
       Date endDate = null;
       long startRow = dsRequest.getStartRow();
       long endRow = dsRequest.getEndRow();
       long totalRows = 0;

      ArrayList<LinkedHashMap<String, String>> result = new ArrayList<LinkedHashMap<String,String>>();
     for(int i=1; i<=80; i++){
      LinkedHashMap<String, String> ttt = new LinkedHashMap<String, String>();
      ttt.put("name", "Dawit"+i);
      ttt.put("age", "100"+i);
      ttt.put("ccc", "male");
       result.add(ttt);

    }
            }
       DSResponse dsResponse = new DSResponse();
       dsResponse.setDropExtraFields(false);
       dsResponse.setTotalRows(totalRows);
       dsResponse.setStartRow(startRow);
       endRow = Math.min(endRow, totalRows);
       dsResponse.setEndRow(endRow);
    dsResponse.setData(result);
       return dsResponse;
}

And Finally the ds info:

<DataSource 
	ID="executedLeaseReport"
	serverType="generic" >
   <fields></fields>
  <serverObject lookupStyle="spring" bean="executedLeaseQuery" />
    <operationBindings>
        <binding operationType="fetch" serverMethod="makeExecutedLeaseQuery"/>
    </operationBindings>
</DataSource>

[Isomorphic]

Note that the server code isn't important here, just the client code and the server response (as shown in the RPC tab).

It seems like you've got off by one issues - you can cause an extra fetch by advertising that there are still rows to be fetched. totalRows should be the total count of rows, and endRow the index of the last row returned and your code is not doing the right thing. In this case, where you're returning all rows, you can just call setData() and all the row count numbers will be properly