Received null criteria for remove operation

[eliasbalasis]

On behalf of "Nigel Stirzaker", while license seat is being assigned,

Hi we’ve upgraded to Smart GWT 13. On some, but not all of our ListGrids using the delete option via the red icon created by setCanRemoveRecords(true); causes the following error

“Received null criteria for remove operation – would remove all records – ignoring.”

This is what we send

{
dataSource:"fieldDS",
operationType:"remove",
componentId:"grid_FieldGrid",
data:{
fieldFormat:"ddMMyyyy",
fieldName:"PUBPD1",
maxRepeats:10,
dataType:"BOOLEAN",
tagPattern:"PUBLISHER_PUBLICATION_DATE1",
fieldDescription:"PUBLISHER_PUBLICATION_DATE1",
isFieldDeprecated:true,
isOrgField:true,
isXml:true,
isRepeatable:true,
lastChangedDatetime:null,
lastChangeUser:null,
fieldId:"a58001e2-e191-11ec-86b6-8038fbf30607"
},
textMatchStyle:"exact",
callback:{
target:[ListGrid ID:grid_FieldGrid],
methodName:"removeDataComplete"
},
willHandleError:true,
showPrompt:true,
oldValues:{
fieldFormat:"ddMMyyyy",
fieldName:"PUBPD1",
maxRepeats:10,
dataType:"BOOLEAN",
tagPattern:"PUBLISHER_PUBLICATION_DATE1",
fieldDescription:"PUBLISHER_PUBLICATION_DATE1",
isFieldDeprecated:true,
isOrgField:true,
isXml:true,
isRepeatable:true,
lastChangedDatetime:null,
lastChangeUser:null,
fieldId:"a58001e2-e191-11ec-86b6-8038fbf30607"
},
requestId:"fieldDS$62722",
internalClientContext:{
removeDataCallback:{
},
editInfo:{
rowNum:11,
values:Obj,
editCompletionEvent:"programmatic"
}
},
fallbackToEval:false,
lastClientEventThreadCode:"TMR1",
bypassCache:true,
dataProtocol:"getParams"
}
“

And below is what the underlying datasource sends back. I hasten to add that the message appears to come from the DataSource.class. Our code overrides the DataSource.class’s executeRemove method but any breakpoint we add to this is not reached

{
affectedRows:0,
data:"Received null criteria for remove operation - would remove all records - ignoring.",
invalidateCache:false,
isDSResponse:true,
status:-1
}

[Blama]

Hi eliasbalasis,

this happens when your remove request does not include a value for the primaryKey-field in data.
Does your DataSource define a PK and is the data for the PK included in the request?

Best regards
Blama

[nrstirzaker]

Hi Blama , that was my thinking but this is the contents of the relevant DS.xml file

<DataSource
ID="fieldDS"
serverConstructor="com.nielsen.book.quicksilver.ui.server.datasource.field.FieldDataSource"
schemaBean="com.nielsen.book.quicksilver.ui.shared.product.FieldDTO"
>

<fields>
<field name="fieldId" primaryKey="true" />
</fields>



</DataSource>

In the request you can see that we are sending a value for fieldId, hence my and eliasbalasis confusion. Also, we upgraded from version 6 where all this code worked fine

[Isomorphic]

Hi nrstirzaker et al

From what you've shared with us, it's not obvious what's wrong. The ListGrid remove icon will use the record values from the grid as the remove request data [essentially criteria for removal]. If your ListGrid data was missing a value for the primary key field in the record you clicked, you could see an error like the one you've shared, but the request snippet you shared does appear to have a value for fieldId [which is your primary key].

We do notice that the DataSource has a custom serverConstructor, so you're presumably running through your own execute() or executeRemove() implementation - so our best guess at this stage is that there's something in that code which ultimately puts together a server side DSRequest / executes a remove operation and is missing the necessary field value.

The server logs should show a stack trace indicating where things went wrong. This may give you a hit on what happened.
If not, we'd recommend debugging this on the server - add a breakpoint to your executeRemove operation and step through, checking what ultimately leads to the error.
If this points to a SmartClient server bug we're happy to look further but we'll need enough information to fully understand what's going wrong and ideally reproduce it on our end.

The fact that this was introduced with a version update (and worked perfectly with older SmartGwt versions) is definitely interesting. This doesn't necessarily indicate a regression, you may have been relying on a pattern that was technically invalid but used to not have any ill effects or similar. But we'll certainly take a look if you can get us more information.

Regards
Isomorphic Software

[Blama]

Hi nrstirzaker,

I had another (unlikely) idea. Does the DataSource have only one PK field? If it has more, sending a value for fieldId might not be enough.

Best regards
Blama

[nrstirzaker]

Thank for the reply

So there are three extra bits of information that make things more "intriguing"

1) Yes we override the executeRemove() but not execute(). We have placed an initial breakpoint in our executeRemove but the error occurs before we reach the breakpoint
2) For our "SGWT6.1" implementation we had our own delete icon and called grid.removeData(record) ourselves. invoking"removeData" gives us the error. But we've also bypassed this old code by setting setCanRemoveRecords(true) but we still get the same error
3) We have another grid on the same page for which the delete works perfectly using but our original manual invocation of grid.removeData and the setCanRemoveRecords one

When we do any debugging we soon leave our code and end up in files like ISC_core.js etc, the error appears to be coming from DataSource.class's execute() method or from some validation it's running. As you can see the field called fieldId is defined as our PK and a value for this is sent is the DSRequest. The error message implies that the whole criteria is null and not just missing the primary key value. But that could be chicken and egg. My working assumption is that the criteria object is created by something in the underlying Datasource.class. I'm very much open to suggestions and pointers. Thanks again Nigel

[nrstirzaker]

Addendum
I've added an execute() method to our GeneralDataSource class which inherits from BasicDataSource, I've added a breakpoint but again it is never reached

public abstract class GeneralDataSource extends BasicDataSource {

private static final Logger LOG = LoggerFactory.getLogger(GeneralDataSource.class);

..............................................

@Override
public DSResponse execute(DSRequest req) throws Exception {
return super.execute(req); //breakpoint here but code fails before it's reached
}

[nrstirzaker]

Blama good question, but this particular datasource does only have 1 PK , it almost feels to me like something in our datastructure (DTO) cannot be converted via the Record to a criteria

[nrstirzaker]

Solution:

We've found the problem. From the JSON "output" from our DSRequest you can see the data we are sending. This includes the key "fieldName", If I rename this to fildName or remove it altogether from the DTO then everything is fine and the executeRemove operation competes successfully. I think this should enable you to reproduce the error. Regards Nigel

[eliasbalasis]

It seems "fieldName" is a reserved element name for DTOs in SmartGWT 13.0 but not in SmartGWT 6.1

Is our observation correct ?

Keep in mind that renaming our DTO attributes is not necessarily a desirable course of action for us.

Is there a better way to transmit DTOs ?

[Isomorphic]

The discovery that field named "fieldName" is responsible for the issue is helpful and it is definitely related to the problem, although we don't see the root of the problem and we could not reproduce it. The "fieldName" is not restricted to be used as field.name and should work as a normal field.

Considering you can make the issue either occur or not depending on whether "fieldName" field is present, could you please provide more details? Or, in a sense, less details. Ideally this should be as minimal as possible. If you simplify the DataSource to just couple of fields all defined locally in the "fieldDS" DataSource (including the "fieldName" field) and perform builtin remove operation, does that reproduce the issue in your environment?

[eliasbalasis]

Due to the nature of the problem and the complexities involved we cannot provide anything minimal wouldn't be enough to accurately reproduce and study the issue.
Yet, we do have a Maven project which we use to share our SmartGWT experiments.
We will extend it to accurately to reproduce the problem on both versions of SmartGWT and send it to you for deeper study soon.

[firebet]

I have put together a small standalone test case that demonstrates this issue (relevant files attached).

The error generates the following stacktrace on the server-side

=== 2022-06-24 15:39:41,173 [0-56] WARN RequestContext - dsRequest.execute() failed:
java.lang.Exception: Received null criteria for remove operation - would remove all records - ignoring.
    at com.isomorphic.application.AppBase.executeDefaultDSOperation(AppBase.java:594)
    at com.isomorphic.application.AppBase.executeAppOperation(AppBase.java:555)
    at com.isomorphic.application.AppBase.execute(AppBase.java:498)
    at com.isomorphic.datasource.DSRequest.execute(DSRequest.java:3182)
    at com.isomorphic.servlet.IDACall.handleDSRequest(IDACall.java:226)
    at com.isomorphic.servlet.IDACall.processRPCTransaction(IDACall.java:183)
    at com.isomorphic.servlet.IDACall.processRequest(IDACall.java:148)
    at com.isomorphic.servlet.IDACall._processRequest(IDACall.java:119)
    at com.isomorphic.servlet.IDACall.doPost(IDACall.java:79)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:648)
    at com.isomorphic.servlet.BaseServlet.service(BaseServlet.java:178)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:95)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org.eclipse.jetty.server.Server.handle(Server.java:499)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
    at java.lang.Thread.run(Thread.java:748)

I confirm that if you change the FieldTestPojo fieldName field to testFieldName and the get/setFieldName() methods to getTestFieldName and setTestFieldName and update the fieldName ListGridField in FieldTestListGrid to testFieldName, then the executeRemove method in the DataSource runs as expected when the Delete? column icon is clicked.

[eliasbalasis]

As promised, you can find the Maven project we use to share our SmartGWT experiments, extended to accurately to reproduce the problem on both versions of SmartGWT (6.1-p20210902 and 13.0-p20220616), at https://github.com/eliasbalasis/poc-...t-smartgwt.zip

I will be deleting the file as soon as you download it though.

The file contains a Maven hierarchy which you can import into Eclipse IDE (with GWT Eclipse Plugin installed) and run the "com.nielsen.book.poc_shared.gwt.smartgwt.BasicDataSource.version.test.GwtTestBasicDataSourceFieldName" classes as "GWT JUnit Test", which reproduce the problem quite accurately.

[Isomorphic]

Thank you for the details. We can now reproduce the issue and see what's happening. The problem is that we are not recognizing "fieldName" being a dataSource field, since datasource definition misses the autoDeriveSchema="true" flag, so it does not inherit fields from the schemaBean. Side effect of this is that the criteria is stripped more aggressively resulting in an empty criteria.
This unintentionally worked without the autoDeriveSchema flag, which is fixed in recent versions to match the docs. So, in order to make things work as expected you need to add autoDeriveSchema="true" to the datasource (like below). Let us know please how it worked for you.

<DataSource
    ID="fieldDS"
    serverConstructor="com.nielsen.book.quicksilver.ui.server.datasource.field.FieldDataSource"
    schemaBean="com.nielsen.book.quicksilver.ui.shared.product.FieldDTO"
    autoDeriveSchema="true"
>
    <fields>
        <field name="fieldId" primaryKey="true" />
    </fields>
</DataSource>

We are also considering to default the autoDeriveSchema more intelligently in the future, so that your current code would work as well, but for now adding the autoDeriveSchema="true" is the correct fix.