New bug with declarative aggregation since 5.1p 2016-04-28

[Blama]

Hi Isomorphic,

there is a bug with 5.1p build newer than 2016-04-28.
It seems that my custom aggregation in .ds.xml code is no longer used. Instead a SELECT * is done.

This is my .ds.xml file:

<DataSource xmlns="lmscompany/ds" xmlns:fmt="lmscompany/fmt" dbName="Oracle" tableName="V_LEAD_RES_MATCH_ALL_RD_GUI" ID="V_LEAD_RES_MATCH_ALL_RD_GUI"
serverType="sql" serverConstructor="com.lmscompany.lms.server.LMSSQLDataSource">
<fmt:bundle basename="com.lmscompany.lms.server.i18n.DSXMLResources-utf8" encoding="utf-8" />
<fields>
<field name="TENANT_ID" type="integer" hidden="true" />
<field name="LEAD_ID" primaryKey="true" type="integer" hidden="true" />
<field name="RESELLER_ID" primaryKey="true" type="integer" displayField="RESELLER_NAME" foreignKey="T_RESELLER.ID" joinType="outer">
<title><fmt:message key="reseller" /></title>
</field>
<field name="RESELLER_NAME" includeFrom="T_RESELLER.NAME" type="text" length="70" escapeHTML="true" hidden="true" />
<field name="DISTANCE" type="float">
<title><fmt:message key="distanceToCustomer" /></title>
</field>
<field name="CUST_NAME" length="80" type="text" escapeHTML="true">
<title><fmt:message key="company" /></title>
</field>
<field name="RESELLER_AVAILABLE" type="boolean" sqlStorageStrategy="singleCharYN">
<title><fmt:message key="allowReseller" /></title>
</field>
<field name="LEAD_NOTRESELLER_ID" type="integer" />
<field name="RESELLER_COUNT" type="integer" customSQL="true" nativeName="RESELLER_ID" />

<!-- TENANT_IDs for WHERE-clause generation -->
<field name="RESELLER_TENANT_ID" includeFrom="T_RESELLER.TENANT_ID" hidden="true" />
</fields>

<serverObject lookupStyle="new" className="com.lmscompany.lms.server.worker.V_LEAD_RES_MATCH_ALL_RD_GUI" />
<operationBindings>
<operationBinding operationType="update" />
<operationBinding operationType="fetch" />
<operationBinding operationType="fetch" operationId="countPossibleResellers">
<customFields>RESELLER_COUNT</customFields>
<summaryFunctions>
<RESELLER_COUNT>count</RESELLER_COUNT>
</summaryFunctions>
<whereClause>RESELLER_AVAILABLE = 'Y' AND ($defaultWhereClause)</whereClause>
</operationBinding>

<operationBinding operationType="fetch" operationId="countPossibleResellersWithCustName">
<customFields>RESELLER_COUNT, LEAD_ID, CUST_NAME</customFields>
<summaryFunctions>
<RESELLER_COUNT>count</RESELLER_COUNT>
</summaryFunctions>
<whereClause>RESELLER_AVAILABLE = 'Y' AND ($defaultWhereClause)</whereClause>
<groupBy>LEAD_ID, CUST_NAME</groupBy>
</operationBinding>
</operationBindings>
</DataSource>

The client request goes to operationId "countPossibleResellers".

This is the server log from 5.1p 2016-05-07:

 INFO  RequestContext - URL: '/lms/lms/sc/IDACall', User-Agent: 'Mozilla/5.0 (Windows NT 6.3; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0': Moz (Gecko) with Accept-Encoding header
DEBUG LMSIDACall - Header Name:Value pair: host:localhost:8080
DEBUG LMSIDACall - Header Name:Value pair: user-agent:Mozilla/5.0 (Windows NT 6.3; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0
DEBUG LMSIDACall - Header Name:Value pair: accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
DEBUG LMSIDACall - Header Name:Value pair: accept-language:de-de,de;q=0.8,en-us;q=0.5,en;q=0.3
DEBUG LMSIDACall - Header Name:Value pair: accept-encoding:gzip, deflate
DEBUG LMSIDACall - Header Name:Value pair: content-type:application/x-www-form-urlencoded; charset=UTF-8
DEBUG LMSIDACall - Header Name:Value pair: referer:http://localhost:8080/lms/Lms.jsp?gwt.codesvr=127.0.0.1:9997
DEBUG LMSIDACall - Header Name:Value pair: content-length:1285
DEBUG LMSIDACall - Header Name:Value pair: cookie:JSESSIONID=E4A93F776D1CF16B5BD96B51B1BA9F8E; GLog=%22%5Bobject%20Object%5D%22; _ga=GA1.1.761878784.1443531764; isc_cState=ready
DEBUG LMSIDACall - Header Name:Value pair: connection:keep-alive
DEBUG LMSIDACall - Header Name:Value pair: pragma:no-cache
DEBUG LMSIDACall - Header Name:Value pair: cache-control:no-cache
DEBUG LMSIDACall - session exists: E4A93F776D1CF16B5BD96B51B1BA9F8E
DEBUG LMSIDACall - remote user: csuperuser
DEBUG RPCManager - Processing 1 requests.
DEBUG RPCManager - Request #1 (DSRequest) payload: {
    criteria:{
        operator:"equals",
        fieldName:"LEAD_ID",
        value:3606,
        _constructor:"AdvancedCriteria"
    },
    operationConfig:{
        dataSource:"V_LEAD_RES_MATCH_ALL_RD_GUI",
        repo:null,
        operationType:"fetch",
        textMatchStyle:"exact"
    },
    appID:"builtinApplication",
    operation:"countPossibleResellers",
    oldValues:{
        operator:"equals",
        fieldName:"LEAD_ID",
        value:3606,
        _constructor:"AdvancedCriteria"
    }
}
INFO  LMSIDACall - Performing 1 operation(s)
DEBUG ServerObject - Couldn't find a public method named: fetch on class: com.lmscompany.lms.server.worker.V_LEAD_RES_MATCH_ALL_RD_GUI
DEBUG DataSourceDMI - DataSourceDMI: no public method name: fetch available on class: com.lmscompany.lms.server.worker.V_LEAD_RES_MATCH_ALL_RD_GUI - defaulting to builtin operations.
DEBUG AppBase - [builtinApplication.countPossibleResellers] No userTypes defined, allowing anyone access to all operations for this application
DEBUG AppBase - [builtinApplication.countPossibleResellers] No public zero-argument method named '_countPossibleResellers' found, performing generic datasource operation
INFO  SQLDataSource - [builtinApplication.countPossibleResellers] Performing fetch operation with
    criteria: {criteria:[{value:1,fieldName:"TENANT_ID",operator:"equals"},{value:3606,fieldName:"LEAD_ID",operator:"equals"},{criteria:[{value:1,fieldName:"RESELLER_TENANT_ID",operator:"equals"},{fieldName:"RESELLER_TENANT_ID",operator:"isNull"}],operator:"or"}],operator:"and",_constructor:"AdvancedCriteria"}    values: {criteria:[{value:1,fieldName:"TENANT_ID",operator:"equals"},{value:3606,fieldName:"LEAD_ID",operator:"equals"},{criteria:[{value:1,fieldName:"RESELLER_TENANT_ID",operator:"equals"},{fieldName:"RESELLER_TENANT_ID",operator:"isNull"}],operator:"or"}],operator:"and",_constructor:"AdvancedCriteria"}
WARN  SQLSelectClause - [builtinApplication.countPossibleResellers] SQLSelectClause is null or zero size, remap and conversions won't work!
INFO  SQLDataSource - [builtinApplication.countPossibleResellers] derived query: SELECT $defaultSelectClause FROM $defaultTableClause$defaultAnsiJoinClause WHERE RESELLER_AVAILABLE = 'Y' AND ($defaultWhereClause)
INFO  SQLDataSource - [builtinApplication.countPossibleResellers] 594: Executing SQL query on 'Oracle': SELECT * FROM V_LEAD_RES_MATCH_ALL_RD_GUI LEFT OUTER JOIN T_RESELLER ON V_LEAD_RES_MATCH_ALL_RD_GUI.RESELLER_ID = T_RESELLER.ID WHERE RESELLER_AVAILABLE = 'Y' AND (((V_LEAD_RES_MATCH_ALL_RD_GUI.TENANT_ID = 1 AND V_LEAD_RES_MATCH_ALL_RD_GUI.TENANT_ID IS NOT NULL) AND (V_LEAD_RES_MATCH_ALL_RD_GUI.LEAD_ID = 3606 AND V_LEAD_RES_MATCH_ALL_RD_GUI.LEAD_ID IS NOT NULL) AND ((T_RESELLER.TENANT_ID = 1 AND T_RESELLER.TENANT_ID IS NOT NULL) OR (T_RESELLER.TENANT_ID IS NULL))))
DEBUG SQLTransaction - [builtinApplication.countPossibleResellers] Started new Oracle transaction "1984994984"
DEBUG SQLDataSource - [builtinApplication.countPossibleResellers] Setting DSRequest as being part of a transaction
INFO  SQLDriver - [builtinApplication.countPossibleResellers] Executing SQL query on 'Oracle' using connection '1984994984': SELECT * FROM V_LEAD_RES_MATCH_ALL_RD_GUI LEFT OUTER JOIN T_RESELLER ON V_LEAD_RES_MATCH_ALL_RD_GUI.RESELLER_ID = T_RESELLER.ID WHERE RESELLER_AVAILABLE = 'Y' AND (((V_LEAD_RES_MATCH_ALL_RD_GUI.TENANT_ID = 1 AND V_LEAD_RES_MATCH_ALL_RD_GUI.TENANT_ID IS NOT NULL) AND (V_LEAD_RES_MATCH_ALL_RD_GUI.LEAD_ID = 3606 AND V_LEAD_RES_MATCH_ALL_RD_GUI.LEAD_ID IS NOT NULL) AND ((T_RESELLER.TENANT_ID = 1 AND T_RESELLER.TENANT_ID IS NOT NULL) OR (T_RESELLER.TENANT_ID IS NULL))))
INFO  DSResponse - DSResponse: List with 1 items
DEBUG RPCManager - Content type for RPC transaction: text/plain; charset=UTF-8
DEBUG SQLTransaction - Committing Oracle transaction "1984994984"
DEBUG RPCManager - non-DMI response, dropExtraFields: false
DEBUG SQLTransaction - getConnection() looked for transactional connection for Oracle:  hashcode "1984994984"
DEBUG SQLTransaction - Ending Oracle transaction "1984994984"
DEBUG SQLDataSource - About to clear SQLDriver state for DS instance 594
DEBUG SQLDataSource - About to clear SQLDriver state for DS instance 679
DEBUG SQLDataSource - About to clear SQLDriver state for DS instance 679
DEBUG SQLDataSource - About to clear SQLDriver state for DS instance 594
DEBUG SQLDataSource - About to clear SQLDriver state for DS instance 594
INFO  Compression - /lms/lms/sc/IDACall: 534 -> 388 bytes

In consequence, in the Developer Consoles Result you can that data for all columns *but* the custom field RESELLER_COUNT is returned, where only data for RESELLER_COUNT would be expected.
Are there changes in this area in builds newer than 5.1p 2016-04-28?

I checked that if I include a serverMethod with the operationBinding with just return request.execute() and put a breakpoint in it in the java file, I can see that it is hit, so the operationBinding is being used, it is just not interpreted correctly.

This one stops me from releasing product updates that include other fixes you made since 2016-04-28, so it is very important to me.

Best regards
Blama

[Isomorphic]

This is caused by recent changes preventing having multiple fields with the same nativeName in a datasource. You already have one field with name="RESELLER_ID", so you aggregation field is skipped, this is reported during datasource initialization.
To fix this you may use customSelectExpression approach instead of nativeName in RESELLER_COUNT field config, something like this:

<field name="RESELLER_COUNT" type="integer" customSQL="true" [b]customSelectExpression[/b]="RESELLER_ID" />

[Blama]

Hi Isomorphic,

OK thanks, I'll try the change here and let you know how this works for me.
Do you mean this warning?

=== 2016-05-11 11:31:43,722 [c-16] WARN  SQLSelectClause - [builtinApplication.countPossibleResellers] SQLSelectClause is null or zero size, remap and conversions won't work!

That's the only one I spotted.

Best regards
Blama

[Blama]

Hi Isomorphic,

OK, I'm seeing the warning you are mentioning:

=== 2016-05-11 11:53:38,845 [c-18] WARN  BasicDataSource - Datasource 'T_CATEGORY1' has multiple fields with the same nativeName='POSITION. This is not allowed.

It is only shown at 1st access to the DataSource, is this correct?

IMHO the change/error will not be noticed at 1st access always, but perhaps only when a subsequent write fails. Could you output the log message every time the DS is accessed?

Also the docs still mention a case where a duplicate nativeName is OK (but with a different tableName). I don't know if this is correct.
Could you include the mention of "every nativeName only once" and the workaround/correct solution with customSelectExpression / customUpdateExpression you gave here in the docs?

This might be a delicate change in the framework as the duplicate nativeName wasn't explicitly disallowed before and a developer had no reason to assume that it is this way (at least not for readOnly operations, besides me at least one other user (godonnell_ip) used it like this).
This is no problem for me now that I know about it, but I assume many people will hit the problem.

Best regards
Blama

[Isomorphic]

We've added a note to the docs telling that same nativeName is disallowed for fields bound to the same table, so sample with different tableName is still correct.

Note that warnings are shown only in versions since 5.1. Older versions (5.0 and below) do not show bad usage warnings, but the usage is still disallowed there.