Paged TreeGrid stops loading children of rootNode

[SimonF]

Hi there,

I'm working again on the paged tree and have detected a bug.

Trying to use all features of the paged TreeGrid I have a case, where the tree does not load any additional children of the root.
At the serverside I have a big tree. There are mutliple children of root and multiple children of these children (root -> root-x -> root-x-y).
Every root-X noder should be open. The dataPageSize is set to 50, and the server does know this number. Because every root-x node is open, there is no need to respond all 50 root-x nodes. Instead I return the first 50 nodes, which could be root-x and root-x-y nodes.

With this I only have to transfer the nodes, which the user will see, if he begins scrolling.
This does work really well, so after scrolling a bit, there is a ds-request for root-x (to get the last 2 children).
Then the next children of the root are requested, by scrolling down there is another request for a root-x node to get the rest of the children to display.
After that there should be a request to get the next children of the root-node. But this request is not called. The tree does know, that there are some nodes missing, because the alternateRecordStyles is true, and there are some empty rows.
This is the network-log:


You can either scroll down (multiple times) or you can just close the folders one after the other:


It get's really hard to create a standalone example with the datasource to reproduce the same error. So I hope you can understand, that the example might look not so nice.

I have tested this with the latest nighty (SmartClient_v100p_2015-05-29_Pro) with chrome and firefox with the same result as shown above.

This is the code for reproduction:

isc.VLayout.create({
	"ID" : "rootLayout_5",
	"width" : "100%",
	"height" : "100%",
	"autoDraw" : true,
	"hideUsingDisplayNone" : false,
	"leaveScrollbarGap" : false,
	"members" :
	[
		isc.Button.create({
			title: "reload node 'root'",
			width: 200,
			click:function(){
				theTreeGrid.data.reloadChildren(theTreeGrid.data.findById("root"));
			}}
		),
		isc.TreeGrid.create({
			"ID" : "theTreeGrid",
			"width" : "100%",
			"height" : "100%",
			"selectionType" : "multiple",
			"canEdit" : false,
			"showFilterEditor" : false,
			"dataArrived":function (node){
				var theChilds = node.children; 
				for (var i = 0; i < theChilds.length; i++){
					eachChild = theChilds[i];
					if(eachChild._isOpen === true){
						delete eachChild._isOpen;
						theTreeGrid.data.openFolder(eachChild);
					}
				}
			},
			"alternateRecordStyles" : true,
			dataSource : isc.DataSource.create({
				"fields" :
				[{
						"name" : "treeGridGeneratedIndex",
						"primaryKey" : true,
						"hidden" : true,
						"canView" : false
					}, {
						"name" : "nameField",
						"title" : "Reason",
						"type" : "text"
					}, {
						"name" : "parentId",
						"rootValue" : "root",
						"foreignKey" : "treeGridGeneratedIndex",
						"hidden" : true
					}
				],
				"dataFormat" : "json",
				"dataURL" : "http://devset.de/treegrid3.php",
				"transformRequest" : requestTransformer,
				"transformResponse" : responseTransformer,
				"recordXPath" : "\/resultData",
				useHttpProxy : false
			}),
			dataProperties : {
				openProperty : "isOpen",
				childrenProperty : "children",
				canReturnOpenFolders: true,
				progressiveLoading : false
			},
			"autoFetchData" : true,
			"dataPageSize" : 50,
			"dataFetchMode" : "paged",
			"selectionProperty" : "isSelected",
			"fields" :
			[{
					"name" : "nameField",
					"title" : "Reason",
					"type" : "text",
					"canEdit" : true,
					"canSort" : true
				}, {
					"name" : "lastField",
					"title" : "Name",
					"type" : "text",
					"canEdit" : true,
					"canSort" : true
				}, {
					"name" : "randomField",
					"title" : "random",
					"type" : "text",
					"canEdit" : true,
					"canSort" : true
				}
			],
			"selectionProperty" : "isSelected",
			"members" :
			[]
		})
	]
});
function requestTransformer(dataSourceRequest) {
	var operationType = {
		operationType : dataSourceRequest.operationType
	};
	if (dataSourceRequest.operationType == "fetch") {
		var params = {
			delay: 300,
			sortBy : dataSourceRequest.sortBy,
			start : dataSourceRequest.startRow,
			end : dataSourceRequest.endRow
		};
	}
	return isc.addProperties({}, operationType, dataSourceRequest.data, params);
}
function responseTransformer(dataSourceResponse, dataSourceRequest, jsonData) {
	if (dataSourceRequest.operationType == "fetch") {
		dataSourceResponse.totalRows = jsonData.totalRows;
		dataSourceResponse.endRow = jsonData.endRow;
		dataSourceResponse.startRow = jsonData.startRow;
	}
	if (dataSourceRequest.operationType == "remove") {		
		console.log(dataSourceRequest.componentId);
		console.log(this.componentId);
		dataSourceResponse.nodeId = dataSourceRequest.data.id;
	}
}

Best regards

[Isomorphic]

It's required that your server provides a complete response to the request. So, if startRow:10, endRow: 20 is requested, you must provide all 10 rows if you have them. (If you only have row numbers 0-14, then obviously you can only supply rows 10-14 in answer to that request.)

If you're interested in some kind of optimization for a corner case, we could come up with some new property or API that allows control of level-by-level fetch-ahead behavior, as a Feature Sponsorship.

[SimonF]

Thanks for the explanation.

Like stated in the docs, i thought it may be also possible to ignore the start/endRow and only submit some of the children.

As with all paged DSRequests, the server is free to ignore startRow/endRow and simply return all children of the node. This allows the server to make on-the-fly folder-by-folder choices as to whether to use paging or just return all children. However, whenever the server returns only some children, the server must provide an accurate value for DSResponse.totalRows.

I'm referencing to http://forums.smartclient.com/showthread.php?t=32808.
Like you stated out, it's not supported to return folders with "isOpen: true" and not giving any children. Therefore i have set the dataArrived in the previous example. This works fine and after the treegrid is initialized the tree gets shown, but everytime the server responds with all children of the root, there is a request for every node. Even if the node is not yet visible. There seems to be no possibility or api to only open the node when the user scrolls near the position of the node.
Because of this, I have tried to only return the current needed children of the root.

It seems the only solution is, that for every children of the root i have to return one child, so i can use the normal "isOpen:" and the data will get loaded if the user scrolls by.

Is there any other recommended solution to this?

Best Regards

[Isomorphic]

The documentation you quote says that you may return more nodes than requested if you want.

It does not say that you may return less. That is not allowed in general, because the server doesn't have the information to know whether this is safe.

Similarly, your described strategy - returning an isOpen folder and providing a childCount and single node only - is also not allowed *in general* because on the server, you don't actually have enough information to know whether it is safe to return just a single node, or whether that will cause the tree to immediately request additional data (never allowed).

For the specific case of first-time root-level loading, if you arrange to send some additional context to the server so that the server is aware of how many rows are visible in the TreeGrid and that this the initial data load, your server code could detect that nodes you are returning as open are in fact scrolled offscreen. Returning such nodes with just one child should work for typical end user TreeGrid scrolling given the current client-side code, but is technically not supported, and could break in the future.

As previously stated, a safe approach would be a Feature Sponsorship to add enough context to requests to make the server able to do this optimization safely in *all* circumstances, and make it a supported and tested part of the protocol.

[Isomorphic]

Also, just to re-establish a bit of perspective here: if you have some nodes that are initially offscreen, and it is so expensive to load a page of children from these nodes that you are taking a deep dive into exactly what the tree paging algorithm allows, you should consider:

1. maybe those nodes shouldn't be returned as automatically open in the first place

2. scrolling through a gigantic tree with many large, already-open folders is usually not the best UI. Consider providing the user with search capability, or in some other way organizing the UI so that the end user is not scrolling around a gigantic tree trying to find relevant nodes.

[SimonF]

Thanks for the quick repsonses.

Similarly, your described strategy - returning an isOpen folder and providing a childCount and single node only - is also not allowed *in general* because on the server, you don't actually have enough information to know whether it is safe to return just a single node, or whether that will cause the tree to immediately request additional data (never allowed).

Your're right is should not be the situation that one request results into another request, but by definition in the doc it's allowed to return an the firt node with two children and a childCount of 20. This would lead to a immediately request for the child-nodes 3-20. At this point this makes no sense, but like described below, in rare use-cases this mages sense.

In my case I have about 30 nodes in first depth (children of root). I know that if the user enters the view, there can be only viewed at maximum 50 nodes. So if I would work with the dataArrived-event in the TreeGrid, by viewing the tree there would be one request for the children of the root and 30 single requests (one for each node). This can be reduced if the server already responds with the children for the first visible treenodes. For every treenode after the first 50 nodes (in first-level or second-level) there is no need for returning the complete child-set. So one child has to be set, that I don't have to set "_isOpen:true" and have to work with the dataArrived.

The worst case of this approach (deliver 1 child for each first-level node that is below the x nodes limit) is that there are some additional requests for displaying the data. But this is also better than a request for each first-level node. (Case as described at the beginning)

to 1. I'm aware that working with automatic opened folders in a large tree is a not the best practice and is even for us a rare case.

to 2. We have worked with a client-side-only tree and the user haven't had any problem working with this. Our users are used to it, that the first-layer nodes are open, and in fact it makes a lot of sense in this usecase. Most of the time this tree is quite small, but there are multiple cases, where the tree gets huge so it's not possible to deliver it as a client-side-only tree.
At this point we need to set to the paged treegrid, which behaves similar to the previous client-side-only tree.
Because we are working for 2 month on the implementation of the paged TreeGrid, with dozen of issues reported, we already have implemented search so the user can see only a smaller sub-tree and can interact with the tree as usual.

A Feature Sponsorship might be the clean solution, but also the more expensive solution for us.

Thanks for the effort trying to push me to the "good side", but in this rare case we have no other chance.

Best Regards

[Isomorphic]

To clarify again: no, returning a node with two children and a child count of 20, such that the tree must immediately request data again, is not allowed and not supported.

The documentation does not say you can do this.

We might attempt yet another clarification on the docs to make it more explicit that this is not allowed, but for anyone else reading this - no it's not allowed and not supported.

SimonF, as far as your strange edge case involving backwards compatibility with an older interface, consider: your users may be used to having the folders already open, but if these folders are so slow to load that you don't want to load ~50 children from each, maybe you are best off leaving them closed. In the previous version of your UI, with *different performance characteristics*, it made sense to just have the folders open. It may not make sense now; you may end up slowing down your end users by trying to replicate the old UI in circumstances where it's not the most efficient UI anymore.

[SimonF]

I really do understand why responses that triggers another requests are bad.

I dont' think I have understood the difference with this:

In addition, if any folder is returned already open, it must include children via the childrenProperty or there will be an immediate, new fetch to retrieve the children. When returning children, a partial list of children may be returned, but if so, the childCountProperty must be set to the total number of children.

This means if there is a request for the root I can add children for a first-layer node root-3. Assume the root-3 has 200 children. In the first response I can return each child of root and at the root-3 node I can add additionally add the first 100 children. Because root-3 has 200 children the childCount has to be set.

If I add 50 nodes to each root-1 (first child of root) and root-2, which also will be transfered completely in the first request, the chances are very high, that the user does only see(visible) this:

+root-1
  | root-1-1
  | root-1-2
  | ...
  | root-1-50
+root-2
  | root-2-1
  | root-2-2
  | ...
  | root-2-50
+root-3
  | root-3-1
  | root-3-2
  | ...
  | root-3-50

150 rows is a very high value (the screen has to be very high), and node root-3-51 to root-3-100 are not visible yet. So this wouldn't result into another request loading root-3-101 to root-3-200.
The rest of the children of root-3 (root-3-101 to root-3-200) would be loaded if the user scrolls down.
This is a normal use case and the reason for working with childCount, isn't it?

What I don't understand is this:
Now adding a root-4 node, with also 200 nodes, but only adding one child and childCount with 200. It seems that it's the same situation as in root-3, a subset of children is given, childCount is set and the not returned children will be loaded if the users scrolls near this sub-tree. Why is this case not supported? It seems to be a similar situation as with root-3.

In this case the load of the children are normal, but I don't want to render all children of all nodes in the first datasource-request of the tree.

Sorry for my lack of understanding.

Best Regards

[Isomorphic]

Once again, it's very simple: if it's expensive to load children of a node that may be offscreen, just return it closed. If it's expensive to load, the user won't want to wait for it to load up front *and* the user won't want to wait for nodes to load when they happen to scroll the folder into view while looking for something else.