Building the LGPL version from source

Hello,

This question relates to the LGPL version.

I’ve been hacking away with SmartClient over the past few weeks and believe that sometimes having access to the non obfuscated version of the source code at runtime would be helpful in debugging some tricky problems.

As part of my due diligence in evaluating the software, I’d also like to be sure that the source code matches the runtime (obfuscated) version.

I am aware from http://forums.smartclient.com/showthread.php?t=2037 that you use the FileAssembler internally to build the runtime (obfuscated and compressed) from the source. I understand that loading up the source directly may end up loading extremely slowly, however I’m prepared to accept this occasional limitation in order to allow me to debug some hard to track down bugs in my software.

I’ve noted that the source code is in the “source/” tree and that in the source/system/modules tree are files with conspicuously similar names to those in the “isomorphic/system/modules” tree. I spent a few minutes looking at the module files in the source tree and noted that they merely load up all of their included files using a simple array and document.write loop.

I’ve also noted that they load from the “system” directory which doesn’t actually exist, so as a quick hack I copied the client directory tree to an equivalent “system” directory tree. After then loading the source module files into my browser I was greeted by innumerable javascript syntax errors.

I believe that you are using the (non-LGPLed) FileAssembler to do this construction when you release the product, which may also be addressing the syntax errors. However, I have been unable to find any build documentation on how you have exactly configured the FileAssembler to do this. I’m not really keen on reverse engineering the module files into the XML syntax required run the FileAssembler.

I believe that to be truly “LGPL” it must be possible to build the runtime version from the supplied source code (even if this requires non-LGPL tools), or it must be possible to run directly from the source code (which I have been unable to do).

Can you provide further information on how I can independently verify that the obfuscated code was indeed constructed from the supplied source code? Can you provide details on how to run the source code directly, performance caveats noted?

On a related topic, the patch files are only supplied in their obfuscated form. Again, to be LGPL compliant, these should also be supplied in source code form.

Hopefully more and more developers will discover SmartClient and a community may build up to help debug issues and contribute new functionality. However, I don’t see the tools currently being provided making this a possibility for the time being.

And by the way, SmartClient is a great product with some great ideas! Keep up the good work and I look forward to seeing version 7.

To run directly from the source (again, ill-advised because of the speed), the module files (ISC_Core.js et al) under the source tree are the right starting point.

To run from the FileAssembler, it’s pretty straightforward to generate FileAssembler XML definitions from the list of files in each module file, no reverse engineering involved.

The syntax errors you see are all of the same kind - either a missing comma between two method definitions or an extra comma at the end of the block of methods. There are, I think, 5-6 of them. These are auto-corrected by the FileAssembler and because we use the FileAssembler exclusively in-house, we were not even aware of them.

If you really want to work this way, Firebug should make quick work of them. Unfortunately, a number of people working from source had to slog through these, but these commas were fixed shortly after 6.5.1.

Since you’ve actually called into question whether either approach works, we’d appreciate it if you post confirmation once you’re up and running.

Thanks for the reply and confirming my understanding. I’d still like you to address a couple more issues for me please.

You have obviously detected the missing or extra commas in the source code. You also mentioned that a number of people “had to slog through these”. I’d prefer not to do that if it has already been done by someone else previously. It would be helpful if you could release a diff of the changes you made post 6.5.1 to fix these specific errors. I didn’t bother to look at the errors in any detail, but there were more like dozens of errors, although I’m happy to accept that only 5-6 changes may be all that is necessary to make them all go away. I really didn’t want to waste any time going down this rabbit hole if my initial assumptions had been incorrect.

Do you have any intention of releasing the FileAssembler xml config that you use internally to build the GPL version? I’ll reconstruct it myself if I have to, but again as this has already been done it would be much simpler from my POV if you could just supply it. I’m sure I’m not the only who would find this useful as part of their research into the product.

You haven’t addressed my question regarding non ofuscated patches.

See below for the necessary changes. I shouldn’t have said “slog”, really, having this list is going to make it a 3 minute instead of 5 minute job, since Firebug takes you right to the file and line number of syntax errors.

Our internal FileAssembler configs are rather different from what you would want in the SDK, as they deal with producing multiple different builds of the libraries with additional settings that have to do with our development environment and project structure. Whether we do it or you do, the fastest approach is to generate XML from the module files.

No one has ever requested an unobfuscated patch before. It’s straightforward for anyone to translate patches to unobfuscated source if necessary; this is so rarely needed that if we bothered to do this with every patch I really think it would be a disservice to people waiting for us to get releases out.

ScreenReader.js

isc.TreeGrid.addMethods({
    waiRole:"tree",
    rowRole:"treeitem";  //change semi colon to comma


DynamicForm.js

setAction : function (action) {
    this.action = action;
    var form = this.getForm();
    if (form) form.action = action;
},  //comma was missing here


ListGrid.js

doubleClick : function () {
    return isc.EH.STOP_BUBBLING;
},  //comma was missing here

Window.js

destroyModalMask : function () {
    if (this.modalMask) {
        this.modalMask.destroy();
        this.modalMask = null;
    } 
}, // comma was missing here

SectionStack.js

    removeItem : function (section, item) {
        var sectionHeader = this.getSection(section);
        sectionHeader.items.remove(item);
        this.removeMember(item);
    }, //comma was missing here


Hover.js

hoverCanvasHidden : function () {
    var lhc = this.lastHoverCanvas;
    delete this.lastHoverCanvas;
    if (lhc != null) {
        lhc.hoverHidden();
    }
}, // comma was missing here 


Date.js

setLocaleStringFormatter : function (functionName) {
    if (isc.isA.Function(Date.prototype[functionName]) || isc.isA.Function(functionName)) 
        Date.prototype.localeStringFormatter = functionName;
}, //comma missing here

Array.js

_isChangingData : function () {
    return (this._dataChangeFlag != null && this._dataChangeFlag > 0); 
}, //comma missing here

Ok, thanks for the information and saving me the time to address the comma and semicolon issues.

I got all of this up and running. There’s a few more things needed to be done to make this work that aren’t contained in this thread, which if anyone is interested I’m happy to explain. But given what I’ve discovered to far, I’m not sure there is much point at the moment.

Here are my initial observations:

1: No changes were required in DynamicForm.js

2: This wasn’t as slow as I had expected. The initial load is slow, but after that, performance is comparable enough to the obfuscated version for it to be perfectly workable for debugging purposes. I do have a very fast machine and am working with local services.

3: There’s a bug in line 70 of ISC_DataBinding.js, just need to remove “.js” from this line. Related to this, I cheated and renamed all of the schema/…ds.xml files to schema/…ds.xml.js.

4: Some of my client code worked correctly, however there were a few surprises. I had to change line 4327 of DataSource.js to

    var qualify = this.mustQualify || (flags && flags.qualifyAll);

5: I was also very surprised that in some cases a whole bunch of extra XML was being posted to my service, starting with a deep and long “clientContext” tag. I haven’t used clientContext, but according to the documentation “The clientContext is never sent to the server.”!

Just to be sure, I switched my includes back to the obfuscated version and everything started working again as I expected.

I haven’t spent much more time looking into any other potential differences. One (point 5 above) is enough for me to be concerned. My expectations were that the source version should function identically to the obfuscated version and this does not appear to be the case.

My conclusion at this point is either the source does not match the “runtime” or “obfuscated” version or there is some something else that I’ve missed.

I’m happy to email my version to you if this would be of use.

I find your comment on not providing open source versions of patches misleading. It isn’t necessarily straightforward to translate patches and as it has already been done, you are just requesting rework like you suggested with the FileAssembler. Just because no one has specifically asked for the source doesn’t mean it isn’t necessary. Yes, we are all keen for new releases to come out, but you are providing patches for an LGPL version so it is conventional (and I believe required by the license) that these include the source code to the changes.

If you are not isomorphic and you are concerned about any of this, please post a comment here to let them know…

Thanks for catching the “.js” in ISC_DataBinding, that’s been corrected now.

Your clientContext difference is almost certainly due to using this patch against the obfuscated version. To match this patch, just remove the line in RestDataSource transformRequest that includes clientContext in the params.

You’ll note that line in DataSource.js (“this.mustQualify”) is identical in the obfuscated source. You might have introduced this issue by omitting or incorrectly translating the schema files (.ds.xml) loaded by ISC_DataBinding, it’s not clear.

As far as source matching binaries: as we speak, the SmartGWT Early Adopters group is developing against a correct set of binaries derived from the LGPL source. While we haven’t made working against the source as straightforward and convenient as we plan to, there is absolutely no question that the source we provide matches the binaries.

While we have noticed that you are stopping just barely short of a claiming we’re providing bogus source, please consider how it affects people at Isomorphic to see you raising such possibilities at every hurdle you encounter when you are still a relative novice with the system.

Investigate the LGPL offering all you want and you will find that it is a real open source offering and we will immediately correct any legitimate user concerns. However while doing so, realize that we do deserve the benefit of the doubt.

Ok, so the patch worked. As I had been unable to use the obfuscated patch and this was indeed a difference between running with the obfuscated and source versions. I should have commented out the patch in my obfuscated version as I would have realized that this was the problem. I’ve done this and now confirmed identical behavior.

I’ve also removed my change to line 4327 of DataSource.js and confirmed that all is working correctly.

I take your point about about raising these issues. In my defense, if your policy was different on providing source code patches then my results would have been different. You have had a long time to provide better LGPL support so perhaps fostering the open source community sooner than later could really provide everyone with some benefit?

Anyway, thanks for your help. At this stage I’m convinced I know what I’m working with and still think you have created an excellent piece of software.

==================================================================

So if anyone has a need to do this, here are the steps I followed:

1: Start with a clean version of the smartclient_651_LGPL version.

2: Copy smartclientSDK/source/client to smartclientSDK/source/system

3: Copy smartclientSDK/isomorphic/skins to smartclientSDK/source/skins (note: only to be used in accordance with license!)

4: Modify the following source files in the smartclientSDK/source/system tree as per Isomorphic’s post on 15th Oct @ 15:46.

ScreenReader.js
DynamicForm.js (this did not require updating in my case)
ListGrid.js
Window.js
SectionStack.js
Hover.js
Date.js
Array.js

5: Modify line 70 of ISC_DataBinding.js, removing “.js” from this line

6: Add “.js” to all of the files in the smartclientSDK/source/system/schema directory. This is a bit of a hack.

7: To apply the patch at http://forums.smartclient.com/showthread.php?t=2562, remove or comment line 419 of RestDataSource.js, “clientContext:dsRequest.clientContext”, ensuring you do not leave a dangling comma on the previous line.

8: Change your include files so that they read something similar to the following:

<script>
var isomorphicDir = "<%= application.getContextPath() %>/source/"
</script>

<script src="<%= application.getContextPath() %>/source/system/modules/ISC_Core.js"></script>
<script src="<%= application.getContextPath() %>/source/system/modules/ISC_Foundation.js"></script>
<script src="<%= application.getContextPath() %>/source/system/modules/ISC_Containers.js"></script>
<script src="<%= application.getContextPath() %>/source/system/modules/ISC_Grids.js"></script>
<script src="<%= application.getContextPath() %>/source/system/modules/ISC_Forms.js"></script>
<script src="<%= application.getContextPath() %>/source/system/modules/ISC_DataBinding.js"></script>
<script src="<%= application.getContextPath() %>/source/system/modules/ISC_History.js"></script>
<script src="<%= application.getContextPath() %>/source/skins/TreeFrog/load_skin.js"></script>

Load time is much slower, but ok once your browser has cached. Runtime performance is slower, but not so much as you might expect.

You can still use the developer console: javascript:isc.showConsole(). In this case its still using the smartclientSDK/isomorphic tree. Using the developer console and Firebug at the same time is a really bad idea unless you are really patient!

Now when you get a JavaScript error and you are using Firebug, you will land in some useful source code with a useful stack trace, allowing you to examine variables and figure out what you have done wrong. Good luck!

No worries Magnus. Thanks for posting those instructions for getting completely up and running. And we do likewise take your point that patches that work against both the obfuscated and unobfuscated source are useful.

One point about your post: the license to the skins is also LGPL, with the exception of a handful of 16x16 icons we purchased from Icon Experience - these icons are free for use in SmartClient-based applications and products but are not LGPL. We clarified the public copy of the license to reflect this and you will see this updated license in 7.0 packages.

Thanks to Magnus & Isomorphic

I have followed the same steps & now my source code is also working. Thanks again.

Isomorphic Admin,

Do we have any seperate documentation to understand the code & its flow. I understand that the code itself is having lots of comments. But please let us know if there is any more documentation.

Thanks,

Yes, SmartClient’s documentation is available online here.