Databound ListGrid + grouping + setShowRecordComponents creates extra whitespace

[beckyo]

SmartGWT 4.1p - 2014-06-19
FireFox 26, Chrome 35, IE 11

A ListGrid backed by a DataSource that combines grouping with show record components = true produces extra whitespace at the bottom of the list when the list is longer than its container.

The example below creates four list grids.

The first grid does not use a DS and combines both grouping and setShowRecordComponents = true. It renders fine (scroll to the bottom).

The second is a DS backed grid with just grouping - it looks fine.

The third is a DS backed grid with just show record components set to true - it's also OK.

The fourth is a DB backed grid with both grouping and show record components - scroll down and you'll see the extra whitespace.

Also note that if setShowAllRecords is not set to true only the second grid (the DS bound grid with grouping) renders without the extra whitespace.

Screenshots attached.

Standalone test case:

package standalone.client;

import com.google.gwt.core.client.EntryPoint;
import com.smartgwt.client.widgets.grid.HeaderSpan;
import com.smartgwt.client.widgets.grid.ListGrid;
import com.smartgwt.client.widgets.grid.ListGridField;
import com.smartgwt.client.widgets.grid.ListGridRecord;
import com.smartgwt.client.widgets.layout.HLayout;
import com.smartgwt.client.data.DataSource;
import com.smartgwt.client.data.fields.DataSourceTextField;
import com.smartgwt.client.types.GroupStartOpen;


public class TestCase implements EntryPoint 
{
	private static final String DSFIELD_GROUP = "group";
	private static final String DSFIELD_NAME = "name";
	
	public void onModuleLoad() 
	{
		ListGrid gridNonDSBoth = getNonDSGrid();
		ListGrid gridDSGroup = getDSGridGroupingOnly();
		ListGrid gridDSRecordComp = getDSGridRecordCompOnly();
		ListGrid gridDSBoth = getDSGridBoth();
		HLayout container = new HLayout();
		
		container.setMargin(20);
		container.setWidth(1200);
		container.setHeight(400);
		container.setMembersMargin(40);
		
		container.setMembers(gridNonDSBoth, gridDSGroup, gridDSRecordComp, gridDSBoth);
		
		container.draw();
	}
	
	
	private ListGrid getNonDSGrid()
	{
	    ListGrid grid = new ListGrid();
		
	    grid.setHeaderHeight(40);
	    grid.setShowAllRecords(true);
	    grid.setGroupByField(DSFIELD_GROUP);
	    grid.setGroupStartOpen(GroupStartOpen.ALL);
	    grid.setShowRecordComponents(true);
	    grid.setHeaderSpans(new HeaderSpan("Non DS + Group + Record Comp", new String[] { DSFIELD_GROUP, DSFIELD_NAME } ));
	    
	    grid.setFields(new ListGridField(DSFIELD_GROUP), new ListGridField(DSFIELD_NAME));
	    grid.setRecords(getRecords());
		
	     return grid;
	  }
	
	
	private ListGrid getDSGridGroupingOnly()
	{
	    ListGrid grid = new ListGrid();
		
	    grid.setHeaderHeight(40);
	    grid.setShowAllRecords(true);
	    grid.setGroupByField(DSFIELD_GROUP);
	    grid.setGroupStartOpen(GroupStartOpen.ALL);
	     grid.setHeaderSpans(new HeaderSpan("DS + Group", new String[] { DSFIELD_GROUP, DSFIELD_NAME } ));
		
	    grid.setAutoFetchData(true);
	    grid.setDataSource(TestDS.getInstance());
	    	
	    return grid;
	}
	
	
	private ListGrid getDSGridRecordCompOnly()
	{
	    ListGrid grid = new ListGrid();
	    	
	    grid.setHeaderHeight(40);
	    grid.setShowAllRecords(true);
	    grid.setShowRecordComponents(true);
	    grid.setHeaderSpans(new HeaderSpan("DS + Record Comp", new String[] { DSFIELD_GROUP, DSFIELD_NAME } ));
	    	
	    grid.setAutoFetchData(true);
	    grid.setDataSource(TestDS.getInstance());
	    	
	    return grid;
	}
	
	
	private ListGrid getDSGridBoth()
	{
		 ListGrid grid = new ListGrid();
		
		 grid.setHeaderHeight(40);
		grid.setShowAllRecords(true);
		 grid.setGroupByField(DSFIELD_GROUP);
		grid.setGroupStartOpen(GroupStartOpen.ALL);
		grid.setShowRecordComponents(true);
	        grid.setHeaderSpans(new HeaderSpan("DS + Group + Record Comp", new String[] { DSFIELD_GROUP, DSFIELD_NAME } ));
		
		grid.setAutoFetchData(true);
		grid.setDataSource(TestDS.getInstance());
		
	        return grid;
	}
	
	
	
	 protected static ListGridRecord[] getRecords()
	{
		ListGridRecord[] records = new ListGridRecord[44];
		 int recordCounter = 0;
		
		for (int groupCounter = 0; groupCounter < 4; groupCounter++)
		{
			 for (int nameCounter = 0; nameCounter < 11; nameCounter++)
			{
			    ListGridRecord record = new ListGridRecord();  
			     record.setAttribute(DSFIELD_GROUP, "Group " + groupCounter);
			     record.setAttribute(DSFIELD_NAME, "Name " + nameCounter);
			    
			     records[recordCounter] = record;
			     recordCounter++;
			 }
		 }
		
		return records;
 	}
	
	
	
 	private static class TestDS extends DataSource 
	{
            private static TestDS instance = null;  
        
            public static TestDS getInstance() 
            {  
	          if (instance == null) 
	          {  
	              instance = new TestDS("localTestDataSource");  
	          }  
	          return instance;  
            }  
  
            private TestDS(String id) 
            {
                setID(id);
                setClientOnly(true);
            
                DataSourceTextField group = new DataSourceTextField(DSFIELD_GROUP);
                DataSourceTextField name = new DataSourceTextField(DSFIELD_NAME);
          
                group.setPrimaryKey(true);
                name.setPrimaryKey(true);
          
                group.setCanEdit(false);
                name.setCanEdit(false);
            
                setFields(group, name);
            
                setTestData(getRecords());
            }
      }	
}

[beckyo]

Any luck with this one?

[Blama]

Hello,

I can confirm that behaviour as well in my app (v9.1p_2014-07-23, using modified Simplicity skin), but only in FF26 (dev and compiled mode). Chrome 36 and IE10 work as expected.

From the DS Console I know the framework knows it has all data:

[
    {
        affectedRows:0, 
        data:[
            {
               ...
            }
       ], 
        endRow:23, 
        invalidateCache:false, 
        isDSResponse:true, 
        operationType:"fetch", 
        queueStatus:0, 
        startRow:0, 
        status:0, 
        totalRows:23
    }
]

In FF, the scrollbar starts at the same size as Chrome's, when scrolling down, at some point (when it "sees" that end of the list??) it becomes smaller (=framework thinks it has more data to show).
When you scroll more then you can scroll to exactly to far that the last pixelrow of the ListGrid's data becomes hidden.
When you scroll up again, at some point (when you start seeing the topmost row??), the scrollbar goes back to the size it was in the beginning.

Important: This does only happen when you scroll with the mouse. Scrolling with the keyboard does not trigger the effect. Once I reached the end of the ListGrid with the keyboard, I can also use the mouse without triggering the error.
Calling invalidateCache() resets the ListGrid, so that I can trigger the error again with the mouse.

Does this help?

Best regards,
Blama

[beckyo]

Thank you so much for the additional info. I use FF almost exclusively and it's good to know this doesn't happen with other browsers.

[Blama]

Well, you wrote it happens with Chrome and IE for you as well. Could you retest? Could you also try if there is a difference between mouse- and keyboard-scrolling for you?
That might valuable information for Isomorphic.

Best regards,
Blama

[beckyo]

Most definitely. I'll follow up by the end of the day.

[Isomorphic]

This is actually a standard behavior which happens whenever listGrid.virtualScrolling is enabled.

For performance reasons, ListGrids support rendering only a subset of the rows actually loaded in the grid (see listGrid.showAllRecords and listGrid.drawAheadRatio / listGrid.quickDrawAheadRatio).
ListGrids also support having variable height rows -- rows which render at different heights to fit their content.

When both of these features are active at the same time, we have to go into 'virtual scrolling' mode.

What this means is that we don't actually know exactly what the scroll-height of the listGrid would be if all rows were rendered (since we don't know the rendered height of undrawn rows), so we have logic which estimates the required space to render out every row based on the number of rows and the average row heights and uses this to size the scrollbar thumb, and determine what "slice" of data to draw in the viewport.
As the user scrolls, they can bring more rows into view so we have to re-render. At this point there's some tricky logic to account for the fact that rows may actually render taller or shorter than we guessed *but* the user expects to see the subsequent rows - we can't have the content of the grid suddenly appear to jump around. Without going into details of exactly how this is achieved the "page of whitespace" below the last row is expected - it's required to allow us to handle things consistently.

Having recordComponents visible, plus incremental rendering (showAllRecords:false) gets us into this state as the recordComponents mean that row heights are unpredictable.

Regards
Isomorphic Software

[beckyo]

But in the first set of examples (first image attachment) I have showAllRecords(true) for each grid, so it shouldn't have to do that guesswork - the record count is known before rendering. Is there something I can do to turn off virtual scrolling? In this case I know ahead of time exactly how many rows there are.

[Blama]

Hello Isomorphic,

thanks for the explanation.
In my case I use setShowRecordComponents(true). But I do not set setFixedRecordHeights, which then should default to true. According to getVirtualScrolling() VirtualScrolling is disabled then, isn't it?

Also, this does not explain differences between Firefox and Chrome/IE (or scrolling with the mouse vs scrolling with the keyboard).

If you think I might be correct here, I can try to build a sample for my configuration as well.

Best regards,
Blama

[Isomorphic]

It's enabled when fixedRecordHeights is false or recordComponents are enabled (snip from the JSDoc you linked to):

...virtualScrolling is switched on automatically when fixedRecordHeights is false and when using the recordComponents subsystem, as recordComponents expand the rows that contain them.

In our testing, using the test case attached at the top of this thread, we didn't see a difference between browsers, so we're not sure what the explanation for that could be.
The difference between mouse scrolling and keyboard scrolling is to do with how the draw-ahead is calculated. When drag-scrolling the scrollbar thumb, you are "quick scrolling", so we suppress redrawing the rows you skate over and use the "quick-draw-ahead-ratio".
When keyboard scrolling the normal drawAheadRatio is used as you're essentially stepping through every record, so it makes sense for us to draw a larger number of records below the viewport when we render out the slice of records. This is all to minimize user-impact of incremental rendering, but could cause minor differences in the appearance of things like the scrollbar thumb size.
However you should still be seeing the whitespace below the last record, regardless of how you scrolled there.

If you think we're missing something which you're seeing, please show us a way to reproduce your particular case and we'll take a look

Regards
Isomorphic Software

[Blama]

Hi Isomorphic,

I read the "and when using..." as logical and - too much time in front of a computer I guess.

I'm using Simple Buttons as my RecordComponent's, which always have the same height. So far I did not use setRecordComponentHeight().
Two questions:

• Is it generally speaking good if I can provide a situation where VirtualScrolling is not needed (use setRecordComponentHeight() in my case)?
• So far I'm only using createRecordComponent(). From the docs I get that I definitely should use updateRecordComponent() and RecordComponentPoolingMode=recycle as well. Correct?

Besides this, from my side only the different behaviour between the browsers is open. I'll try to create a standalone testcase for this.


Thanks for the insight in the inner workings of ListGrid-scrolling and the explanation on fast mouse- vs. slow keyboard-scrolling. I'm happy to have a framework that encapsulates me from all this.

Best regards,
Blama

[beckyo]

This dialog is really great, thanks guys. I'm still tripped up on that first test case, though. The scrolling works beautifully with only record components, and with only grouping, but when the two are combined that's when I see the extra whitespace. Is virtual scrolling behaving differently for that specific combination?

[Isomorphic]

Beckyo: we'll double check why the grouping should have an impact here and respond shortly on that.

Blama: We've tweaked the docs (in mainline) to read "and also" rather than just "and" to make this non-ambiguous.

Reusing recordComponents via the recycle/update flags is definitely a good idea. This allows the grid to create only as many components as is actually needed, and simply reuse them for cells as they render.
It'll have a significant performance impact if you're talking about a grid with a large number of rows where otherwise you'd be creating and drawing new components for every row (or every cell) the user ever sees as they scroll through a grid.

If you have fixed height recordComponents - yes we'd recommend you setRecordComponentHeight() so the grid knows how tall rows will render.

We'll take a look at the standalone test case when you have it

Regards
Isomorphic Software

[Isomorphic]

It turned out there actually was a logic bug in the case where a grid was grouped and databound which was causing 'showAllRecords' to be essentially ignored.
We've fixed this issue in both 4.1p (SC 9.1p) and 5.0d (SC 10.0d) branches - you can pick up the change in the next nightly build.

Regards
Isomorphic Software

[beckyo]

That's great, thank you.