Announcement

Collapse
No announcement yet.
X
  • Filter
  • Time
Clear All
new posts

    Automated Application Testing (selenium or canoo)

    We are currently using WebTest Canoo for basic testing of our application every time we release. It worked great when we only had simple elements, but once we began to get fancy, the javascript elements were not accessible. Instead of rewriting everything, we attempted other tools like Selenium, but quickly ran into the same problems.

    Buttons are rendered as div with a table/tbody/tr/td element (yuck), the onclick event is buried so deep I believed it warranted a post to the forums. I can't believe this question hasn't been asked before (these two tools work beautifully with other popular Ajax frameworks like the Dojo toolkit, OpenLaszlo, MooTools, etc...)

    To summarize, 2 questions:

    1) Do you have any free tools you would recommend for automated testing of smartClient apps? Or do you know how to hack Canoo or Selenium to work correctly?

    2) If not #1, then at least tell me how I can force your "click" event from the DOM or the JavaScript console...

    Thanks in advance.

    #2
    Hi sbarthelmess,

    You must not have been through a framework upgrade cycle with those tools (eg Dojo 0.4 -> 0.9). As frameworks evolve they change the DOM structures they use, breaking test cases that rely on those structures. Unfortunately this means that when you most need the tests - framework upgrades are scary - they become unreliable.

    The more sophisticated a framework becomes, the more inappropriate it is to record DOM structures. For example, in SmartClient, DOM structures vary by browser, skin, and version of SmartClient - the structure you mentioned for a button is true only in some skins. These differences are necessary for performance, pixel-perfect cross-browser rendering, and features such as auto-sizing to content.

    When working with an Ajax platform, DOM structures are rightly regarded as internals. That's the only way an Ajax platform can avoid you having to do cross-browser testing and debugging.

    The automated testing tools are slowly, every so slowly waking up to this reality, and you can see some tools vendors working on approaches that coordinate explicitly with the Ajax platform to create a reliable recording stream.

    We have partnered with a company called SOASTA to provide an integrated solution, which is already in use at mutual customers. Contact us here to get into the beta program.

    On your specific questions:

    1. We can't fix Canoo or Selenium. As I mentioned, the approach these tools take is fundamentally flawed. SOASTA is what we recommend - not free but very reasonable, and with much greater capabilities (eg, load testing, cloud testing, etc).

    2. "click" is not reliably fired by browsers when content is changed during the mouseDown/mouseUp events, so SmartClient components react instead to a mouseDown/mouseUp sequence. If you go down this path (it's a serious effort) just be sure to get the page coordinates correct in your generated events.

    Comment


      #3
      Good afternoon Isomorphic Gurus...

      I totally see your point, but we will not be upgrading our framework at the drop of a hat - we have standardized on 6.5.1 (and of course we chose the best framework out there - and you guys would never make HUGE framework updates ;)

      Don't throw the baby out with the bath water though...

      Those two testing frameworks are excellent for smoke-tests of an application directly following a build (we use them for Continuous Integration). So the value for us is that after every code commit a developer can see if he broke something seemingly completely unrelated to his code. And re-recording the tests if something big changes is very simple. So the off-chance that a framework update breaks things, it's not the end of the world (isn't that what QA is for anyway)?

      Some SmartClient buttons work, others do not because - as stated - you use different click/mousedown handlers for even different skins. I would argue that approach is fundamentally flawed instead, you should have a uniform click handler - even if it's overkill for "simpler" themes.

      Regardless, if I wanted to track a button click w/ the TreeFrog skin, as you stated earlier I could track down the MouseDown event. How would I access the IsoMorphic variable for a single button on a page?

      Something like:
      Code:
      document.getElementById('PLEASE_TELL_ME_THE_ISO_BUTTON_ID').clickHandler()
      (obviously this one doesn't work)...

      Comment


        #4
        We don't break backwards compatibility for documented APIs. I think we are the only platform that has never done so, actually. However internals change frequently, for example, frozen fields (a 6.0 -> 6.5 feature upgrade) necessitate a different DOM structure in grids.

        So again let me emphasize that the DOM structure is considered internals. We have a consistent button click API across all versions of SmartClient, all skins, and all browsers: isc.Button.click(). Critiquing the DOM structure is like saying it is hard to work with decompiled C++ :) Of course it is, because that's not how you're intended to work with the framework.

        The DOM we generate is complicated for very good reasons. If it were simpler it would not work. Claiming this is fundamentally flawed is tantamount to saying that it's a flaw that SmartClient works as advertised.

        I am surprised to hear you talk about losing your tests on a framework upgrade, because larger organizations frequently view their tests as very valuable. Even if you are only using record & playback, and never hand-writing tests, a test suite for a large product can easily represent 1000 man hours. We have designed around preserving that investment.

        Continuous Integration is great and SOASTA supports it very well (it's a web-based testing solution). We highly recommend you look into it and get a hold of the beta of SOASTA/SmartClient integration.

        As far as jerry-rigging something with Selenium or other tools that record DOM internals, some tools can successfully record and playback test runs if you stick to keyboard navigation exclusively. If you want to write test scripts by hand, don't try to simulate a DOM click, call the click handler you have installed on the button directly instead.

        Comment


          #5
          Selenium, most used open source functional test framework

          Hi,
          I agree with "sbarthelmess" when he says "Don't throw the baby out with the bath water though..." concerning compatibility of selenium framework and Smartclient object.
          The most important thing with Selenium is not the record/replay of the test, but to write manually a test that works with SmartClient.
          Selenium Recording tools does'nt work even with other ajax frameworks.
          But we can write functional selenium test for those frameworks.
          Don't ask us to buy a solution when we have a best one and a free one !
          Thanks.

          Comment


            #6
            Selenium, most used open source functional test framework

            Hi,
            I agree with "sbarthelmess" when he says "Don't throw the baby out with the bath water though..." concerning compatibility of selenium framework and Smartclient objects.
            The most important thing with Selenium is not the record/replay of the test, but to write manually a test that works with SmartClient.
            Selenium Recording tools does'nt work even with other ajax frameworks.
            But we can write functional selenium test for those frameworks.
            Don't ask us to buy a solution when we have a best one and a free one !
            Thanks.

            Comment


              #7
              Originally posted by Isomorphic
              1. We can't fix Canoo or Selenium. As I mentioned, the approach these tools take is fundamentally flawed. SOASTA is what we recommend - not free but very reasonable, and with much greater capabilities (eg, load testing, cloud testing, etc).
              Is $1,000 / mo. "very reasonable"?
              maybe for some project yes but it's not acceptable for ours, an internal editing tool.
              Unless I'm misunderstood the SOASTA site we need a different solution.
              Have anybody tried with greasemonkey/imacros tools?

              Second question, the problem with Selenium is only on clicks (vs MouseUp/Down) or are there other incompatibilities?

              Comment


                #8
                Hmm, there used to be a lower-end offering for smaller shops. We'll look into it.

                We are also working with some of the core Selenium guys to see if we can get support added for Ajax RIA systems like SmartClient, ideally in a standard way that multiple systems can use.

                Comment


                  #9
                  Any updates on this?

                  Has Selenium responded yet? Their last release was in June of 2008. We are still without testing in our LARGE shop because of security risks using your "approved partner SOASTA". I am impressed with their offering, but there will likely be no resolve b/c they don't open their source and our corporate security policies are VERY strict.

                  Please advise on your progress and give an ETA.

                  Comment


                    #10
                    Hi sbarthelmess,

                    Selenium is on the verge of a major release, and we hope that support for testing advanced Ajax framework will be incorporated into the release after that.

                    We are have proposed (along with a potential partner) a detailed architecture that would allow any Ajax framework that treats the DOM as internal information to work with Selenium, and it has been well received, and we have also offered to co-sponsor the necessary development work. But we don't have an ETA. It's not our product so we can't directly fix it.

                    In the meantime we'd recommend simply writing tests in JavaScript that do not try to directly simulate mouse and keyboard events. If you, eg, have logic in a button click handler, have the test code call that logic rather than trying to simulate click().

                    Comment


                      #11
                      Up

                      Is Selenium still not compatible ?
                      Thanks.

                      Comment


                        #12
                        Selenium

                        Selenium is very impressive and offer so much functional testing tools (cubitest, selenium grid, ...) whitch make it UNIQUE in the open source world !
                        Please take a look at it.

                        Comment


                          #13
                          What about QA ?

                          What about QA with automatic funtional testing ?
                          This doesnt seem to interest many people in this area.
                          Even Extjs/extgwt is compatible with Selenium and it's a serious concurrent of SmartGWT.
                          I thing that, We, community of SmartGWT, must pay attention to the fact that Selenium is going to be widely used if it's still not. And Quality is being a leitmotiv for many company.
                          So If extjs/extgwt has less functionality than SmartGwt but allows people to be more compliant with QA, they will use it instead of using SmartGWT.
                          So let's pay attention to that.

                          http://forums.smartclient.com/showthread.php?t=3264
                          Last edited by taxaw; 16 Apr 2009, 02:28.

                          Comment


                            #14
                            bump: any updates?

                            Comment


                              #15
                              Hi,
                              As this is an open question for me to, bumping it up also....

                              gr. Martin

                              Comment

                              Working...
                              X