Selenium tests failing after skin upgrade

[acarur01]

After upgrading our Enterprise skin from 8.2-3-26-2013 to any SC 8.3 build, we noticed that some of our Selenium testcases started failing.

For example, when trying to get the slider width using the code below, if we use the latest skin, we get 148px. If we revert back to the old skin, we get 150px which is what we had recorded originally.

Other code such as the click() have also become unresponsive using the following code:

selenium.click("scLocator=//DynamicForm[ID=\"page$content$$content$SliderElement$gridWrap\"]/item[name=page%24content%24%24content%24SliderElement||index=0||Class=SliderItem]/canvas/track[Class=StretchImg||index=0||length=1||classIndex=0||classLength=1]/stretch");

Also, we noticed that the SectionStack section header locator have been modified:

Before:  var expanded = selenium.getEval("selenium.browserbot.getCurrentWindow()['isc_ImgSectionHeader_0'].expanded");
After: var expanded = selenium.getEval("selenium.browserbot.getCurrentWindow()['isc_SectionHeader_0'].expanded");

Could you please let us know if this is a bug or if it's one of those changes that we have to deal with. If so, how do we deal with them?

Thanks

[Isomorphic]

It looks like you might have intended to paste something related to the 150/148px issue but forgotten to. In general, our Selenium support will not record sizes (because skins do indeed cause on-screen elements to have different sizes) and we would advise against manually coding tests so that they rely on sizes.

On the second one, did you change the "name" attribute of the FormItem? That would be expected to cause an issue. If not, can you explain what structure this locator is involved in?

On the last one, there's an auto-generated ID in the locator (isc_ImgSectionHeader_0), which should not happen. You may have recorded such a locator before a bug was fixed that avoided this. If you obtain a locator for the same element now and it doesn't have this auto-generated ID, use the new locator. Otherwise please let us know what UI structure results in a locator like that.

[acarur01]

Here's the missing code:

1.

var width = selenium.getElementWidth("scLocator=//DynamicForm[ID=\"page$content$$content$SliderElement$gridWrap\"]/item[name=page%24content%24%24content%24SliderElement||title=%3Cnobr%3Eleft%26nbsp%3B%3C%24fs%24nobr%3E||value=0||index=0||Class=SliderItem]/canvas/track[Class=StretchImg||index=0||length=1||classIndex=0||classLength=1]/stretch/");

We have A LOT of testcases checking sizing. Because the skin we use for our automation never changes, we expect that our screens look the same each time. I don't think modifying width/height of elements should be taken lightly. Some of our customers are picky to the last pixel.

2. No. This is the same test case that we have been running since last year. It is a simple dynamic form with a slider element.

3. If we record it now then it shows isc_SectionHeader_0 as stated in the "After" code. I can try to figure out how many testcases are failing because of this. If it's a huge amount, then this is really bad since different automated applications were created at different times. Obviously we do not update nightly but this would be such a waste of time for us especially when we have more pressing issues to look into. Perhaps it is a just a quick find-replace (I hope).

In general, we don't want to have to deal with changing our automated scripts every time we upgrade the library. I know I've complained about this before and I hope there aren't too many more. Is this just a Selenium weakness? Are there other tools you'd recommend over Selenium?

[Isomorphic]

1. If you are checking the element width you are getting the native width of an HTML element. This can differ even with identical on-screen appearance because of how the CSS box model works and is something we definitely don't (and cannot) guarantee will be stable across skins, browsers or releases.

If you want a stable value to test that actually does reflect on-screen appearance, use the Canvas.getVisibleWidth() API.

2. We'll look into this one.

3. The second locator may happen to work, but the question is, what locator is returned if you try to get a locator in Selenium IDE or manually? You should find that no longer includes the ID of the particular header.

As far as the overarching question: Selenium is absolutely the tool to use, #1 is clearly a usage issue, #2 is probably a bug caused by big performance enhancements in 8.3 and should be fixable if so without changes to your scripts, #3 remains to be seen.

[acarur01]

Ok thanks will try getVisibleWidth() but is this available for formitems as well?

[Isomorphic]

Yes it is. FormItem.getVisibleWidth().

[Isomorphic]

What specific build of 8.3p are you using?
What browser version and OS are you using?

[acarur01]

We are using 8.3 06-11-2013 but I also tried yesterday's build (07-07-2013). I am using Windows 7, Firefox 12

[Isomorphic]

On #2, we're unable to reproduce a problem with a locator like that. Can you be more specific about the problem you're experiencing:

a. if you try to retrieve an element using that locator, do you get an element or do you get null?
b. if you're getting an element, what was this click expected to do in the application - move the slider by the track click amount?

#3 is still waiting on a response from you, in case you didn't realize.

[acarur01]

The testcase is supposed to test that the change function is fired when the the value is modified.

Here's the code when we automate it now:

NEW:
selenium.click("scLocator=//DynamicForm[ID=\"page$content$$content$SliderElement$gridWrap\"]/item[name=page%24content%24%24content%24SliderElement||title=%3Cnobr%3ERun%20Trigger%26nbsp%3B%3C%24fs%24nobr%3E||value=1||index=0||Class=SliderItem]/canvas/");


OLD:
selenium.click("scLocator=//DynamicForm[ID=\"page$content$$content$SliderElement$gridWrap\"]/item[name=page%24content%24%24content%24SliderElement||index=0||Class=SliderItem]/canvas/track[Class=StretchImg||index=0||length=1||classIndex=0||classLength=1]/stretch");

#3 Oops sorry didn't realize.

Here's the issue. Went I use the developer console, I get the code below. No it does not include the header name (and it probably never did).

//SectionStack[ID="page$content$$content$SectionStack"]/section[Class=SectionHeader||index=0||length=1||classIndex=0||classLength=1||roleIndex=0||roleLength=1||title=Expand%20True%26nbsp%3B||scRole=heading||name=page%24content%24%24content%24Section]/

But I cannot use this to determine if the first section is expanded. I think that is why the original testcase was using the SectionHeader name. If you do not recommend using this auto generated name, how should we figure out what is expanded? I can do it the longer way by generating this eval:

selenium.getEval("(selenium.browserbot.getCurrentWindow()['page$content$$content$SectionStack']).sectionIsExpanded((selenium.browserbot.getCurrentWindow()['page$content$$content$SectionStack']).sections[0])")

[Isomorphic]

Could you clarify on #3 - what was the older strategy for checking whether the section was expanded, that you liked better?

If you want the SectionHeader to have a stable global ID, you can give it one via SectionStackSection.ID.

Probably a better overall strategy would be to check for whether one of the items within the section is visible.

[acarur01]

It really isn't a question of whether I like what we currently have or not. The problem is that the test cases used to work and now they are failing.

What we CURRENTLY have is this code that I gave in my first post where we retrieve the section object and evaluate its expanded state. The strategy still works but instead of passing in isc_ImgSectioHeader_0, we would have to pass in isc_SectionHeader_0.

selenium.getEval("selenium.browserbot.getCurrentWindow()['isc_ImgSectionHeader_0'].expanded");

[Isomorphic]

When usage is incorrect, tests are expected to fail with new versions.

This is an instance of relying on an auto-generated ID, which you should never do. Please use one of the alternative strategies we recommended.

So that covers #1 and #3 as usage issues where you'll need to adjust your tests. We're still looking at #2, which we believe we'll able to resolve without changes to your tests.

[Isomorphic]

We've made a change which should address the remaining issue.
Please try the next nightly build (dated July 23 2013)

Regards
Isomorphic Software