obfuscateKeys for Advanced Criteria is not working

[ksenia_korenkova]

Hi Isomorphic,

We have requirement to mask some sensitive data (e.g, credit card number) in all logs. Under logs I mean also logging which is written inside isomorphic classes. This data can come from client as advanced critreia for example.
The only post which I found out related to this issue is http://forums.smartclient.com/forum/...m-submitted-to
I tested this solution but unfortunately it doesn't work for AdvancedCriteria.Is it a bug or not?
In ideal case we would like to configure fields to mask on datasourse level, but as I understand you provide global solution.
Is it possible to extend you solution to work on Advanced Criteria as well?
Or may be there is another way for configuration to achive what we want?
Currently we use version 5.0p.2016-12-08.
This is what I have in server.properties file:
map.obfuscateKeys.orderNumber: <obfuscated>
map.obfuscateKeys.id: <obfuscated>
and as result what I see in logs (fist is with advanced criteria not masked, the second one is with usual criteria - masked):

=== 2016-12-28 14:51:30,481 [80-2] DEBUG RPCManager - Request #1 (DSRequest) payload: {
criteria:{
_constructor:"AdvancedCriteria",
operator:"and",
criteria:[
{
fieldName:"orderNumber",
operator:"iContains",
value:"1111111111111"
}
]
},
operationConfig:{
dataSource:"blablaDataSource",
repo:null,
operationType:"fetch",
textMatchStyle:"exact"
},
startRow:0,
endRow:75,
componentId:"isc_Result_grid_0",
appID:"builtinApplication",
operation:"blablaDataSourse_fetch",
oldValues:{
_constructor:"AdvancedCriteria",
operator:"and",
criteria:[
{
fieldName:"orderNumber",
operator:"iContains",
value:"1111111111111"
}
]
}
}

=== 2016-12-28 16:12:13,138 [80-2] DEBUG RPCManager - Request #1 (DSRequest) payload: {
criteria:{
id:"<obfuscated>"
},
operationConfig:{
dataSource:"blablaDatasourse",
repo:null,
operationType:"fetch",
textMatchStyle:"exact"
},
appID:"builtinApplication",
operation:"fetchId",
oldValues:{
id:"<obfuscated>"
}
}

[Isomorphic]

This is fixed and will be available for download in nightly builds since Jan 3 (tomorrow). Field values will be obfuscated in advancedCriteria as well.

[ksenia_korenkova]

Hi Isomorphic,

When we applied your fix we noticed such strange behavior: we can see how the field value is masked in advanced criteria and all other logs too,
but we also see that this field (which we specify as obfuscated in server.properties) is masked the same way on UI (grid cell, form item, etc..) when we are trying to show the user the data which contains such field.
Basically when we do fetch.
But this is not what we want to achieve. Our goal is just to mask sensitive data in all logs, but allow user to see the original value on UI.
Is this expected behavior for such fields which are specified as map.obfuscateKeys. in server.properties? I can't find any documentation about this specific properties.
If yes, is there another way to achieve what we want?

Currently we use 5.0p.2017-08-15

[ksenia_korenkova]

Hi Isomorphic,
Do you have any news about my last post?
Did you notice it?

[Isomorphic]

This has been assigned and is being worked on. It's most likely a relatively simple bug where the code is intended to modify the structure used for logging, and actually modifies the original structure instead.

[ksenia_korenkova]

Hi Isomorphic,

Do you have any updates or bug-fixes?

[Isomorphic]

Apologies for the delay. This is fixed and will be available for download in nightly builds since Nov 8 (today).

[ksenia_korenkova]

Hi Isomorphic,

As I already mentioned above currently we use 5.0p.2017-08-15
I cant find nightly build for 5.0 version dated by Nov 8, only since 5.1
Could you please add this fix for 5.0 Power version also

Thank you in advance.

[Isomorphic]

The fix is included in all versions from 5.0 since Nov 8, try downloading 5.0p 2017-11-10 nightly and please let us know how it worked for you.

[ksenia_korenkova]

Hi Isomorphic,
I was able to download 5.0p 2017-11-10 version.
But during testing I found out that in some places in logs values are still not obfuscated.
For example for SQLDataSource class :
I see this in logs:

2017-11-12 17:59:14,890 [ec-9] INFO SQLDataSource - [builtinApplication.customGridFetch] Performing fetch operation with
criteria: {strictSQLFiltering:"true",_constructor:"AdvancedCriteria",criteria:[{fieldName:"fieldName_obfuscated",operator:"iContains",value:"OBFUSCATED"},{fieldName:"canEdit",operator:"equals",value:true}],operator:"and"} values: {strictSQLFiltering:"true",_constructor:"AdvancedCriteria",criteria:[{fieldName:"fieldName_obfuscated",operator:"iContains",value:"OBFUSCATED"},{fieldName:"canEdit",operator:"equals",value:true}],operator:"and"}

But then :
2017-11-12 17:59:14,920 [ec-9] DEBUG SQLDataSource - [builtinApplication.customGridFetch] SQL windowed select rows 0->75, result size 75. Query: SELECT * FROM (SELECT a.*, rownum myrownum FROM (SELECT
TableName.fieldName_obfuscated
FROM TableName
WHERE
(((LOWER(TableName.fieldName_obfuscated) LIKE LOWER('%BLA-BLA-BLA%') ESCAPE'\' )))
..........

In server.properties files I have this:
map.obfuscateKeys.fieldName_obfuscated: OBFUSCATED

As you can see the first line of logs shows the correct (obfuscated) value for the field "fieldName_obfuscated", but the second one from the same class shows the real value.
It looks like when you logs the query itself you didn't mask this special fields.

UI part looks good.

Best Regards, Ksenia

[Isomorphic]

We don't currently plan to try to also obfuscate SQL queries, and doing so would be an entirely new feature, not related to how obfuscateKeys works. So we would recommend either turning off this particular log category for production deployments, or directing such logs to a separate, security controlled file.