TreeGrid where primaryKey changes after update

[JonFields]

I am attempting to implement a TreeGrid that for files on a file system. I've implemented a simple server DataSource that implements the data source operations for files. As for the data source fields, the file's pathname is the primaryKey, and a "parent" field specifies "pathname" as the foreignKey. Common file operations (move, rename) are implemented in the "update" data source operation.

The problem is that the pathname is not an immutable primaryKey. If you move (reparent) the file, or rename the file, the pathname changes, so the cached value in the ResultSet is no longer valid. Indeed according to the FAQ: "The primary key value returned on an update must exactly match the value in the cached record. "

I can of course call invalidateCache() on the datasource, but this redraws the entire tree.

My question is: In the absence of an immutable primaryKey, as you would normally have for a database record, is there any way to properly implement move (reparent) and rename operations? I was thinking perhaps some related updates attached to the DSResponse. I thought I would ask first if there is a known/supported way of doing this before attempting that.

I have also thought of trying to use the UNIX I-nodes and NTFS file identifiers, but this gets into native code that I'd rather avoid.

Any tips would be greatly appreciated!

1. SmartClient Version: v8.3p_2012-12-31/Pro Deployment (built 2012-12-31)
2. Firefox 17.0.9

[Isomorphic]

If you don't wan't to generate a stable primaryKey, you can represent an update to the primaryKey as a remove following by an add, and this will update the client-side cache as expected.

[JonFields]

Thanks. To be clear, what should be passed as the data to the DSResponse() upon return from executeUpdate() when the pathname has changed:

DSResponse  response = new DSResponse(???);
DSResponse remove =  new DSResponse(oldValues);
DSResponse add = new DSResponse(newValues);
response.addRelatedUpdate(remove);
response.addRelated(update(add);
return response;

Should that just be left null?

Also, I discovered NIO BasicFileAttributes.fileKey() as a possible portable way to generate a unique primaryKey.

[Isomorphic]

There are a lot of ways to do this, and the simpler ways would involve two separate requests (a remove and an add) being issued from the client, such that executeUpdate is never called server-side.

But if you were already in executeUpdate(), you could return a response indicating the rest of the update succeeded but the PK was not changed, then use addRelatedUpdates() to send additional remove and add DSResponses.

[JonFields]

Got it, thanks for the quick response. To be honest, now that I have discovered NIO fileKey(), I will use that if it turns out to be portable/reliable. But if not I now have this approach as a backup.

[JonFields]

I spoke too soon... Using a stable primaryKey like inode number does not work because there is no way to find a file given that value for operations like fetch and remove. So that approach does not work, and I have to go back to using the file pathnames as the primary key. (BTW, I will need to get this working with Real Time Messaging as well, which I just purchased along with Power so I can reflect updates to the files made by other users.)

I attempted the use of related updates in the data source update operation, as suggested, and it partly works - the new re-parented/re-named node is displayed as a result of the related "add", but the old node is not removed from the TreeGrid as a result of the "remove". Following are two examples with the DSRequest and DSResponse from the console.

In the data source, the "path" field is the primaryKey, and the "parent" field has foreignKey="path". Omitting the rest of the fields for brevity, there is a "name" field with the file's name, and a boolean "isFolder" field.

Here is the DSRequest for a "rename". It is initiated as the result of a TreeGrid.startEditing() on the selected row. The user enters the new file name and presses return. In this case it was being renamed from "test3.txt" to "test4.txt":

{
    dataSource:"file",
    operationType:"update",
    componentId:"isc_FileTreeGrid_0",
    data:{
        path:"/test3.txt",
        name:"test4.txt"
    },
    callback:"isc_FileTreeGrid_0.$337(dsResponse, dsRequest)",
    willHandleError:true,
    showPrompt:false,
    oldValues:{
        path:"/test3.txt",
        name:"test3.txt",
        parent:"/",
        isFolder:false
    },
    requestId:"file$6274",
    clientContext:{
        saveCallback:{
        },
        newValues:{
            path:"/test3.txt",
            name:"test4.txt"
        },
        editInfo:{
            editValuesID:"_1",
            rowNum:0,
            colNum:0,
            values:Obj{name:test4.txt},
            oldValues:Obj{name:test3.txt},
            editCompletionEvent:"enter"
        }
    },
    fallbackToEval:false,
    bypassCache:true
}

Here is the resulting DSResponse. I have returned the old values as the data of the DSResponse() as suggested, and added a "remove" and an "add" related response to handle the change in the primary key ("path") field:

[
    {
        data:{
            path:"/test3.txt",
            name:"test3.txt",
            parent:"/",
            isFolder:false
        },
        invalidateCache:false,
        isDSResponse:true,
        operationType:"update",
        queueStatus:0,
        relatedUpdates:[
            {
                dataSource:"file",
                isDSResponse:true,
                invalidateCache:false,
                status:0,
                operationType:"remove",
                data:{
                    path:"/test3.txt",
                    name:"test3.txt",
                    parent:"/",
                    isFolder:false
                }
            },
            {
                dataSource:"file",
                isDSResponse:true,
                invalidateCache:false,
                status:0,
                operationType:"add",
                data:{
                    path:"/test4.txt",
                    parent:"/",
                    name:"test4.txt",
                    isFolder:false
                }
            }
        ],
        status:0
    }
]

The new "/test4.txt" node is displayed, but the old "/test3.txt" node is not removed from the TreeGrid.

Here is the DSRequest for re-parenting, which results as a drag and drop within the TreeGrid. Here I have dragged "/test4.txt" to "/folder2/test4.txt".

{
    dataSource:"file",
    operationType:"update",
    data:{
        path:"/test4.txt",  
        name:"test4.txt", 
        parent:"/folder2",  
        isFolder:false
    }, 
    showPrompt:true, 
    oldValues:{
        path:"/test4.txt",  
        name:"test4.txt", 
        parent:"/",  
        isFolder:false
    }, 
    requestId:"file$62719", 
    fallbackToEval:false, 
    dragTree:[ResultTree ID:isc_ResultTree_0 (created by: isc_FileTreeGrid_0)],
    dropIndex:0,
    bypassCache:true
}

The resulting DSResponse is:

[
    {
        data:{
            path:"/test4.txt",  
            name:"test4.txt",
            parent:"/"
        },
        invalidateCache:false,
        isDSResponse:true,
        operationType:"update",
        queueStatus:0,
        relatedUpdates:[
            {
                dataSource:"file",
                isDSResponse:true,
                invalidateCache:false,
                status:0,
                operationType:"remove",
                data:{
                    path:"/test4.txt",  
                    name:"test4.txt", 
                    parent:"/",  
                    isFolder:false
                }
            },
            {
                dataSource:"file",
                isDSResponse:true,
                invalidateCache:false,
                status:0,
                operationType:"add",
                data:{
                    path:"/folder2/test4.txt",  
                    name:"test4.txt",
                    parent:"/folder2",  
                    isFolder:false
                }
            }
        ],
        status:0
    }
]
~

In the TreeGrid the new "/folder1/test4.txt" node is displayed, but the old "/test4.txt" node is not removed.

In both cases, a call to invalidateCache() properly refreshes the data.

AFAIK I am doing what was suggested in the data source as the DSResponses indicate, so why is the related "remove" operation having no effect?

1. SmartClient Version: v8.3p_2012-12-31/Pro Deployment (built 2012-12-31)
2. Firefox 17.0.9

Thanks.

[Isomorphic]

This could either be because:

1. you are signalling success, but sending back a new record that doesn't match the user's edits - this may throw off the grid. Try accepting the "name" change without changing the path (so send back name:"test4.txt").

2. an order-of-operations issue - the "remove" doesn't work because the grid hasn't fully processed the "update" and removed the editValues it was tracking. If this is the issue, you should see that doing an equivalent DataSource.updateData() separately from the grid causes the expected result

[JonFields]

Using suggestion (1) I was able to get rename to work, where "name" changes but "parent" remains the same. However, re-parent, where "name" is the same, and "parent" changes does not work. I end up with two nodes with the same "name", and "parent" values, one with the old "path" and one with the new "path".

I cannot spend more time on this so for now I will punt and set invalidate cache on re-parent which provides an absolutely awful user experience. Isomorphic, if you can provide a working solution that would be great. Otherwise I will conclude that TreeGrid only works with immutable primaryKey values, and will need to design a solution that works around that (e.g. generating ids and tracking an ID to path mapping in a DB).

[Isomorphic]

Hopping back to our first comment:

There are a lot of ways to do this, and the simpler ways would involve two separate requests (a remove and an add) being issued from the client, such that executeUpdate is never called server-side.

But if you were already in executeUpdate(), ...

You headed down the not-so-simple path, we weren't sure why..

Sending "add" and "remove" requests from the client instead of allowing the grid to auto-save via an "update" is sure-fire. You will need to set autoSaveEdits:false on the grid, set up an event handler for saving yourself (probably RowEditorExit), call discardEdits() to have the grid stop tracking the edits rid of the edits, then use DataSource.removeData()/addData() to do your save.

[Isomorphic]

Another note on this - SmartGWT's need for a stable primaryKey is just for the lifetime of the page, it doesn't actually need to correspond to a permanent, server-side primaryKey.

You could provide a random primaryKey value (just an increasing number) and never change it. Even though SmartGWT will use this random number to identify the record in "update" requests, your server code will always know what file is actually being referred to because of the availability of dsRequest.oldVaues.

[JonFields]

Hopping back to our first comment:



You headed down the not-so-simple path, we weren't sure why..

Sending "add" and "remove" requests from the client instead of allowing the grid to auto-save via an "update" is sure-fire. You will need to set autoSaveEdits:false on the grid, set up an event handler for saving yourself (probably RowEditorExit), call discardEdits() to have the grid stop tracking the edits rid of the edits, then use DataSource.removeData()/addData() to do your save.

I was "already in executeUpdate()" because I need to move/rename the file on the server file system, and any associated metadata in a DB. I was hoping to leverage data binding for that purpose. On the server side, this is an update (rename/move file) not an add/delete...

I will see if I can get your approach to work. Basically I need to update the TreeGrid on the client side only with an add/remove, and then separately invoke server side logic to actually rename/move the file (and possibly metadata in the DB).

[JonFields]

Another note on this - SmartGWT's need for a stable primaryKey is just for the lifetime of the page, it doesn't actually need to correspond to a permanent, server-side primaryKey.

You could provide a random primaryKey value (just an increasing number) and never change it. Even though SmartGWT will use this random number to identify the record in "update" requests, your server code will always know what file is actually being referred to because of the availability of dsRequest.oldVaues.

I am working on this approach, as I am more comfortable with it. It is using TreeGrid as it was intended, and allows the use of data binding.

However, I believe there are still issues mapping the ID back to the file path. For example when the TreeGrid is expanding a node, it invokes a fetch with the "parent" (ID) as the sole criteria. Also, when it does a remove, it only sends the ID value. So unless I'm missing something, you still need to maintain some kind of server side mapping from ID to path name in these two cases.

As I see it there are two potential approaches. First is to maintain that ID to path mapping in the user's session. Second is to somehow cause the SmartGWT client to send additional fields, so that the path is always sent along with the ID. I'm hoping to look at transformers as a way of doing this (the path is always available on the client side). Any hints how to do this would be appreciated....

(BTW, the reason that we just don't put all of the files into a DB that gives us a stable primaryKey is that we have huge numbers of files, and only a small percentage of them have additional metadata. We don't want to have to insert millions of rows into a DB just to provide a stable primaryKey)

Thanks,Jon.

[Isomorphic]

However, I believe there are still issues mapping the ID back to the file path. For example when the TreeGrid is expanding a node, it invokes a fetch with the "parent" (ID) as the sole criteria.

This can be solved by implementing DataSource.transformRequest and grabbing additional values from dsRequest.parentNode.

Also, when it does a remove, it only sends the ID value.

You should be seeing oldValues in this case, if the remove is triggered by canRemoveRecords.

Also note: we tried to replicate your issue with addRelatedUpdates(), but our test worked fine. We'll post the code soon and perhaps you can point out something that differs.

[JonFields]

This can be solved by implementing DataSource.transformRequest and grabbing additional values from dsRequest.parentNode.



You should be seeing oldValues in this case, if the remove is triggered by canRemoveRecords.

Also note: we tried to replicate your issue with addRelatedUpdates(), but our test worked fine. We'll post the code soon and perhaps you can point out something that differs.

Thanks I think if I can get this to work, I will switch to using (generated) unique IDs. I'm more comfortable with that approach as it allows straightforward use of (Tree) Grids with data binding.

[JonFields]

Got it working with request and response transformer. The response transformer maintains a map from a stable file ID (NIO fileKey, Windows file ID) to pathname. The request transformer adds additional path fields from the map so that the server data source receives the path name in addition to the file ID. All the nice data binding features work (multiple grids stay in synch, etc) with rename and reparent.

Suggestion for a possible enhancement? Since it is sometimes necessary to synthesize stable keys get best results, but since there is not always a mapping from the synthesized key back to the actual keys (e.g. pathname -> inode, but not inverse), then perhaps a feature that allows specifying certain non-primary-key fields that must be provided along with primary-key values in data source requests. Not a compound primary key, but a field that is somehow "tied" to the primary key whenever it is sent in a request.....

BTW I was not able to use DSRequest.parentNode. It appeared to be null in most cases.

Thanks,
Jon