Suggestions for porting image manipulation from GWT to Smartgwt?

[jimwaschura]

My shipping application is written in GWT and uses HTML5 Canvas object and getContext2d() method to use context2d image manipulation methods to calculate RGB values and display them using context.drawImage() method on the screen. I can easily accomplish many refreshes per second this way. Here's an example of some of the GWT objects I use:

ImageData m_Image;
Canvas m_OffscreenCanvas;
CanvasElement m_OffscreenCanvasElement;

void ManipulateAndDrawImage( Context2d context, int columns, int rows )
{
m_Image = context.createImageData(columns,rows) ;

CanvasPixelArray pixels = m_Image.getData();
for( int r=0; r<rows; r++ )
for( int c=0; c<columns; c++ )
{
RGB rgb = calculatePixelRGB( r, c );
int linearIndex = r*columns+c;
pixels.set( linearIndex+0, rgb.red );
pixels.set( linearIndex+1, rgb.green );
pixels.set( linearIndex+2, rgb.blue );
pixels.set( linearIndex+3,255 ); /* alpha at full opacity */
}

m_OffscreenCanvas = Canvas.createIfSupported();
m_OffscreenCanvasElement = m_OffscreenCanvas.getCanvasElement();
m_OffscreenCanvasElement.setWidth(columns);
m_OffscreenCanvasElement.setHeight(rows);

m_OffscreenCanvasElement.getContext2d().putImageData( m_Image, 0, 0 );

context.drawImage( m_OffscreenCanvasElement, 0,0,columns, rows );
}

Now I'm porting to Smartgwt and looking for what to use to accomplish same functionality there. So far, much of my other requirements have been satisfied by using DrawPane and associated DrawItems. I expected to be able to accomplish the above with DrawImage object, however now that I've gotten there, I see it only allows setting an image URL and no means to access contents of image matrix of pixels.

Any suggestions or pointers towards other ways to accomplish this will be greatly appreciated, and if nothing comes to mind, then any comments about how I might find a way to integrate the GWT implementation together with the Smartgwt everything-else would also be greatly appreciated. I understand not to mix the two, generally, but technically, is it possible?

Thank you.

Jim Waschura

[Isomorphic]

The FAQ explains the issues and drawbacks for GWT integration - this is one of the cases where it could make sense, since there's no inter-nesting of GWT and SmartGWT controls, and presumably no components that require keyboard accessibility.

If you're existing GWT code basically adds a <canvas> element to the DOM and starts manipulating it, this is pretty straightforward to do - see the DOM Integration overview.

Note, if you work with some internal, JS-only APIs like DrawPane.getBitmapContext() (see Drawing.js), you could also add your custom rendering to DrawPane. Or, via Feature Sponsorship, you could get some APIs added that would allow you to integrate your code without using internals - we're thinking something like a drawItem.registerNativeRenderer() API that would allow you to register to be called, with the native bitmapContext, after any given DrawItem finishes rendering.

[jimwaschura]

Thank you. I sent an email to your sales/development group regarding Feature Sponsorship, and asked for features to be added to the DrawImage draw item, so that not only may they display a source image specified by a URL address, but also to permit a means to define dimensions of a bitmap that can be the source, and to provide a means to fill/access the contents of the bitmap directly. I hope this strategy is fruitful.

I am thinking of another implementation that I'd like your feedback on, please. My back-end server already triggers all browsers that a new image of the oscilloscope view is available, which triggers the browsers to request the matrix of data from the server. My server is technically two pieces. First is a C++ program that supports simultaneous automation via multiple sockets, and the second is the Java servlet built in the eclipse smartgwt project that acts as a full-duplex byte-stream gateway between the C++ server and each browser instance, so that each browser can synchronously request and receive responses from the C++ server via the Java servlet. Enough on the architecture. The point is, there is C++ server running, inside of which, I could create a file-based socket in a known location in the Linux file system, maybe even in the war/images directory. This pseudo-file, when open and read, would trigger the C++ server to calculate and produce a byte-stream equivalent to a .BMP or .MPG file. Then, I would attempt to specify this file name in the URL of the DrawImage control. I would need a means to trigger refreshing the URL contents inside the browser when a new oscilloscope view was ready, and to enforce not using a cached image. Any feedback would be greatly welcome. Thank you.

[Isomorphic]

What's the actual goal here? Are you trying to switch from streaming the image through Java to instead having the Java call return immediately and then later get an asynchronous notification that the file is ready? As far as avoiding caching, just adding a timestamp as a URL parameter would do that. As far as notifying the browser that a server-side process has completed, you would use our Real-Time Messaging module.

[jimwaschura]

I have an image that is created by a server about every 200 ms. It happens to be a picture of an electronic signal a'la an oscilloscope. Next, I have potentially multiple instances of browsers running smartgwt applications that are served by a tomcat server running on the same ARM embedded linux system that is producing the oscilloscope images. These raw images are received by each browser and integrated into a custom charting control that offers features like titles, axes, labels, cursors, cursor measurements, graticules, sub-graticules, pan, zoom, etc.. The charting control can be configured with data visualization layers including layers for multiple XY plot traces, and histograms, and strip charts. One important visualization layer is an "image" layer that is used to display the oscilloscope view with charting features like graticules and cursors coming from the browser control, and the raw images coming from the ARM server and inserted into the appropriate layer of the chart as it is drawn.

Currently (in my GWT implementation), the browser polls one property "ev?" which responds with change-notifications for any/all properties in the server that have changed since the last time that specific browser performed an "ev?" poll. Among the properties that may change, a "SequenceNumber" property gets incremented whenever a new oscilloscope frame is ready. When each browser receive a change notification for this property, they can synchronously request the "ImageMatrix" property that is an array of bytes representing a monochrome image. The browsers receive the array and transform it into RGB values based on a lookup-table which colorizes the image. Then, an RGB bitmap of the image is created and inserted it into the appropriate charting control visualization layer for display.

For the smartgwt implementation, I'm using DrawPane and different DrawItems to render the different features of the chart control. DrawRect, DrawLabel, DrawLine, DrawPolygon get most of the job done, but I can't get the matrix of colorized RGB values to work with DrawImage like I had expected to, so I'm looking for alternatives. Yes, it is "streaming," but no it isn't streaming in the sense of any video image streaming protocol. Its really just a series of images I want inserted into a stack of drawn objects, and if I can't do it directly, I'm working on an alternative strategy to fake the DrawImage control into working by controlling it to refresh from a URL on the backend server that I construct to produce sequential images using the file-based socket technique.

So I think as long as I can tell the DrawImage to refresh itself and that will cause it to go back to the Apache or Tomcat server to GET a new file at that time instead of using a cached version, then I think this will work. Wish me luck. Of course, if there were API's for creating and manipulating a bitmap associated with the DrawImage, that would be easier. ;-)

Thanks again, and sorry for the long-winded explanation. I am a big fan of smartgwt. - Jim W

[Isomorphic]

OK, this is still basically what you've said before.. is there a new question here?

It sounds like you've already implemented polling, and if you don't need to reduce latency down to be sub-millisecond (which Real-Time Messaging can do, on a fast connection), then there's no need to switch approaches.

As far as caching, again just changing the URL should work - is this still an issue for you?

[jimwaschura]

No, I have a few approaches to try. Thank you.