> The end user interacts with the application visually. That's why we test visually.
Not all users interact visually. Selecting interactive elements through accessible labels, not based on visual appearance, is a better practice imo. I want important parts of the DOM that are critical to building a correct accessibility tree to be a part of the test, and to fail the test if we change something that makes the accessibility tree incorrect. Because that is the API for assistive technology and it communicates the user interface for lots of people. "Correct behavior" of an app or website includes "the accessibility tree accurately reflects the structure and nature of the content" and "form fields are labelled correctly". I might be in the minority in thinking that, but I believe it 100%.
Nothing I've seen so far (including the post you linked) suggests that the OCR-like approach can tell us anything about the accessibility tree.
The post does make a similar point to mine:
> If your app changes visually, your test will fail. This is of course a tradeoff and a conscious decision to couple your tests to your UI instead of your code, but the upsides bring much more value than that the downsides take away.
I disagree on the tradeoff. I can run screenshot diffs etc to catch unexpected UI changes, they are a little noisy, but I'm ok with that tradeoff because I care about more than just the visual appearance of the app.
A "visually correct" app with click handlers on divs and no semantic HTML is a liability (legally, maintenance wise, etc).. I'd like the E2E testing tool to assert that the app is working correctly, which does mean some assertions about the DOM are appropriate to me. I agree with the author of the linked post that "we want a solution that is not brittle in unwanted ways." We can be selective about what DOM things are important.
In the linked post the author says "In particular, DOM tests are tightly coupled to the structure of your code." and gives an example about a Selenium test that uses a brittle xpath that depends on a specific DOM structure.
Maybe I have not been exposed to enough of the industry to know that there are thousands of setups relying on flaky xpaths to target elements for testing. To me, it is not true that DOM tests are tightly couple to the structure of your code by default. It's a false statement made for marketing purposes and it is gross.
DOM tests "can be flaky", "are sometimes coupled to DOM structure" or whatever, is a fair assertion, but flakiness in DOM-driven testing is not a fact, it's a sign of badly written DOM-based tests. This is often the first thing I address in a code review of new tests written by somebody who does not write lots of FE test code, and they easily learn how to avoid it.
Maybe I'm wrong but it seems really really basic stuff to not create brittle selectors that fail tests for reasons we don't care about.
I like the OS-level interaction and agree that provides some advantages. I totally disagree that these advantages mean your solution wins at the "best way" to test, but it does clearly cover a different surface area than other automated solutions for E2E testing, and it seems like tests are pretty quick to knock out.
This solution could be a complement to other automated E2E tests, and I would see no reason that a PM or other party couldn't spin up and maintain some tests this way as quick-to-create set of tests to run against various builds, knowing that design changes will break them but that this is OK cause in theory it is quick to rewrite them.
But I couldn't see using this tool as the only E2E testing solution as though it is a superset of what Cypress/Selenium/Whatever tests are capable of. It is actually not a competitor with those tools. It's addresses different concerns with a little bit of overlap.
I'm happy to check out the free trial and see if I'm missing something, and eat crow if I'm being unfair here.
Hi author, I found parts of this downright wrong, especially on the Cypress side, which is where I have the most familiarity:
>Compare this to code-based automation solutions like Selenium or Cypress, where a coder would need to manually update the code across all the relevant tests any time the product changes.
... Well, no. Those tools can have reusable chunks and references that work just like your steps and are organized in the same manner, so that things used multiple places are stored once and reused. That's a basic programming good practice. When a change is needed to, say, a login flow, you don't change 100 UI tests, you change the login function that those tests are all calling.
> Tests created in code-based automation solutions like Selenium and Cypress evaluate what’s going on in the code behind the scenes of your product’s UI. That is, these solutions test what the computer sees, not what your users see.
and
> For example, if the HTML ID attribute of the signup button in the example above changes, a Selenium or Cypress coded test that identifies that button by its ID would likely break.
This is all very disingenuous in my book. In these tools the tests interact with the rendered web page and _can_ be made to use IDs, classes, and other brittle stuff as selectors, but that's a naive approach that a professional wouldn't typically use, because the brittleness surfaces early and there are better patterns that avoid it. One of those is to use selectors based on something that itself should be present - eg, click a button based on its label being "Log In". If that test fails because the button label was changed or removed, that's a reasonable failure, because labels are important in the UI. In the case where we are selecting/making assertions about DOM elements that don't have meaningful user-facing labels, dedicated test-id attributes solve the brittleness problem. But by and large, if something is interactive to the user, it should have a label, so the fallback to test attrs for selectors is rare, and used for the case where we want to measure some specific effect took place that is not itself an interactive element.
It's unfair to suggest that these tools inherently create brittle tests.
> Rainforest QA tests don’t break when there are minor, behind-the-scenes code changes that don’t change the UI.
but
> It works by pixel matching and using AI to find and interact with (click, fill, select, etc.) the correct elements.
Look, maybe your tool is GREAT, I haven't tried it. But a functional UI test based on label contents may be more robust than something that breaks when the visual structure of a page changes, which happens itself pretty frequently. Maybe the AI is good enough to know that a bunch of stuff moved around but the actual flow is the same and can find its way through. With tests based on labels and dedicated selectors, devs can make massive structural changes to the DOM and not break any tests if all the workflows are the same.
I would sure love if this post contained more info on how the visual AI system of selecting elements is better than using labels, test attributes etc as is conventional in automated testing done by professionals. Likewise it would be good to compare with an actual competitor like Ghost Inspector, where non-programmers have been able to generate step-based tests like this for years. The main gripe I had with Ghost Inspector was that it creates brittle selectors and so tests need to be changed a lot in response to DOM changes, unless a developer gets in there and manually reviews/picks better selectors.
If what you have a is a tool that _makes more robust tests than Ghost Inspector_ but is _as easy for a PM to use_, then that is interesting.
I actually support this underlying idea completely - I love systems where non-developers can create and maintain tests that reflect what they care about in the UI. I even love it when tools like this create low-quality tests that are still expressive enough that a dev can quickly come in and fix up selectors to make them more suitable. Cypress Studio is a currently-experimental feature that looks promising for this too, allowing folks to edit and create tests by clicking in the UI, not editing code files. It's a good direction for automated test frameworks to explore.
I'm just really uncomfortable after reading this post. It strays beyond typical marketing hyperbole to be actually deceptive about the nature of these other tools and the practices of teams using them. Instead of highlighting the actual uniqueness of your product, you exaggerate the benefits of this approach vs other solution. Come on, you can do tests in parallel with many tools, and you can avoid captcha for automated testing in various ways. Some of the process points you make are fair but also, again, kind of weak, because sure "QA at the end with no consultation during planning and development" is bad, but that's inherently bad for well known reasons.
What you've said is basically "our tool is better than every other idea about QA, if those ideas are deliberately implemented in the worst possible way, and our tool is implemented in the best way". Well sure. But also, of course it is.
Sorry if this seems harsh. To give you the benefit of the doubt: Marketing copy walks a fine line when trying to be persuasive and you might not have expected to create this impression. It's also possible that you did some research into weaknesses of automated test frameworks and just don't understand that those things are compensated for quite easily and routinely, because maybe you don't have the background. I don't know, but I hope future materials are a little more grounded in reality.
Huge difference between "feeling depressed" and "being unable to function on a daily basis due to a medical condition called depression".
Sick leave is appropriate if a person is sick. Depression can take over your life and make you unable to work. If sick leave does not exist for a depressed person they either become dependent on family or just get fired, risking homelessness, credit problems, increased chances of suicide etc.
In some countries and jobs ... well, that's what happens. But some places treat sick leave differently and if a doctor says you are sick you can get a lot of paid leave in order to recover.
I'm not fond of big nested menus in general either, but they are good for quick exploration and can be pretty forgiving of poor categorization.
I do wonder if sometimes the nested nav is too in-between. Maybe the solution is not to try to squeeze it all into that structure but make "finding the page I want" to be a more top-level experience. EG don't take me to a Category Page, but do let me select a category and start drilling down, whilst keeping it easy to switch between top level categories.
I agree I was wrong, there's another reply to me that's a lot more coherent about what I was going for. I do think it's important to still understand and confront the problems of whiteness as an identity head on.
> Being in denial about how divisive these racially-connoted messages are is just not productive.
I don't think it's denial, I think it's strategic choice made in the full knowledge of the offense and discomfort it might provoke. There's definitely part of messages like "whiteness is racist" or "be less white" that ii ... intended to generate offense & reflection and is a valid way of getting the work done. Being polite and working within the status quo can be seen to upload the status quo, and hasn't seemed to work at actually making chaneg. Sometimes to cut through people choose to say very challenging things. I'm personally fine with that, and even if I'm not I don't think it's up to me to tell the people affected by white supremacist ideas how to go about their business.
Sure, but the history of the term matters in this context & maybe should be taught a little more. Like when we grow up these days, we sorta get the idea that White or Black are these inherent traits that people naturally have and have always had.
We rarely get the chance to see that the idea of whiteness comes from a need of protecting an in-group from exploitation, while excusing and justifying the exploitation of outsiders to that group.
The census is an interesting example because for the first 70 years or so, starting in 1790 the US census had just 3 "race/ethnicity" categories. You could be "Free white male/Free white female", "All other free person" or "Slave". Which is part of the reason for creating the category of whiteness. There's also some really gross stuff right through the 1800s where the census was tracking people with various percentages of "black blood".
I suppose I'm just saying that the census, as part of the machinery used to implement and maintain white supremacy, is not necessarily the best authority on race.
"White" isn't a racial group. "White" as an idea to describe a group of (roughly) light skinned European people, came about in order to justify enslaving and subjugating other groups. Various groups worked their way into being considered white over time, in part by contributing to the subjugation of others. And various groups once considered white, later were not considered so.
"White" as what you call a "social group" was created in service of this power dynamic. Before, say, about 400 years ago, whiteness was not an idea used to identify a "race" of people.
So yeah, I'm all for being less white. I'm fine with just being like ... Irish.
You can research the history on this pretty easily, but a good place to start is a podcast series from Scene On Radio called Seeing White. My memory is a little fuzzy but I think the broad outline is correct.
The problem is partly that page speed measurement tools care a lot about that first load. Even if everything after that is super snappy due to cached resources, if you are only using 5% of the total CSS file in the first page the user hits, that other 95% is dragging down the experience on first load, and especially dragging down lighthouse scores.
It doesn't mean though, that you can't have a combined approach where a small amount of genuinely global CSS is loaded from an external file that is cached, while component-specific CSS lives in the component and renders inline with it. But then it's like ... why do that extra request if the file is that small, and you end up with inline everything.
Definitely the trend from a business point of view is to optimize for good perf scores on lighthouse, even if that's a little illogical at times.
Another thing though is that often the JS for whole chunks of a page (which contains the CSS) can be code-split and cached as its own file, so whole components made up of HTML CSS and JS can be lazy loaded and if that same chunk of stuff is needed again, it could be a cached JS file.
Especially if these are "below the fold" and pulled in as the user scrolls or something, you get a small initial payload + small JS files being called in to run as needed.
I don't know how many implementations there are of this in the wild, but it's possible.
"Authoritative reference documents of MDN are going to become cluttered with comments and disagreements"
Which suggests previously authoritative docs are becoming less so due to being editable. I'm just pointing out they are and always have been subject to edits from the public. If anything the PR process will likely increase the reliability of the info because there would be a chance to do some vetting of changes.
I totally agree that MDN is as close to authoritative as you can get, but it's one (important) step away.
MDN has never been "authoritative reference documents". It's a wiki. Anybody can edit and republish pages, change examples, do whatever. I noticed somewhere about a year ago that it was a wiki, and have made minor changes to a couple of pages since then, when something was missing an example or something.
I change, I publish, it's done. And if my change was wrong, or opinionated, or bad, it's just like that until somebody else catches it and fixes it back.
Something that hasn't been pointed out yet as far as I can tell is that there are accessibility and quality errors on your website. If I am looking to hire somebody it's one of the things I look at when I want to see if they know what they are doing. Not everybody is like me, but these problems are worth fixing anyway:
- The "Explore" button is a span with a click handler, the click handler updates the URL hash. It's not keyboard focusable, and since it's just a link to a different location in the page, it could just be a link, but if it must be JS, it needs to be a button element.
- The form does not have accessible labels. Placeholder text is not a label. Also "fullname" looks like an error.
Lighthouse reports some other low hanging fruit.
Overall vibe of that website is that somebody chose some readily-available bootstrap landing page template and didn't check it for basic errors. That combined with the scant information about the company & the fact that your website is really just a single page does not inspire confidence. Even in the small amount of copy that is on the site, there are three errors in this one sentence:
"Healthcare providers will need an infracstructure what is open, interoperable and standards-compliant, which ensures the security, confidentiality and privacy of presonal data."
If you are not spellchecking your copy & shipping valid HTML for a single web page you make for yourselves, it would be wrong of me to expect higher quality in something you are hired to build. FHIR and HIPAA are tough and I have a ton of respect for you for being able to work in that context. The quality of your website should reflect the high standard you bring to that work.
I've had a rough time with GrandPad. Tried to set it up for a family to use during covid, the mother is in her 60s and pretty competent with tech - they use iPads, Kindles etc just fine. We tried to change the registration of the GrandPad from her billing and they said they needed to ship us a whole new device. Then they said they could just ship a new sim. So anyway she's gotten the new sim in the mail and has had such a frustrating time with it & can't get it to do video calls. She's wanting me to take it back.
The whole idea was that it would be simple enough that her son with Down Syndrome could use it to keep in touch with friends. He's smart, he can use gmail, but he can't use this thing.
Honestly my perspective on them has been that it's a cheap thing, UX is a real mixed bag, customer service not great, & I deeply regret losing the time we've spent trying to work with it vs exploring other solutions. Ugh. I even don't want to go pick it up to try and deal with it from my side.
Glad you have had a better experience, but let's just say it's not universal!
This tool definitely seems very close in spirit to mavo, but paid. I didn't last long enough after the autoplaying video to learn if there's anything it can do that mavo doesn't do (maybe hosting services?) that's worth paying for.
Not all users interact visually. Selecting interactive elements through accessible labels, not based on visual appearance, is a better practice imo. I want important parts of the DOM that are critical to building a correct accessibility tree to be a part of the test, and to fail the test if we change something that makes the accessibility tree incorrect. Because that is the API for assistive technology and it communicates the user interface for lots of people. "Correct behavior" of an app or website includes "the accessibility tree accurately reflects the structure and nature of the content" and "form fields are labelled correctly". I might be in the minority in thinking that, but I believe it 100%.
Nothing I've seen so far (including the post you linked) suggests that the OCR-like approach can tell us anything about the accessibility tree.
The post does make a similar point to mine:
> If your app changes visually, your test will fail. This is of course a tradeoff and a conscious decision to couple your tests to your UI instead of your code, but the upsides bring much more value than that the downsides take away.
I disagree on the tradeoff. I can run screenshot diffs etc to catch unexpected UI changes, they are a little noisy, but I'm ok with that tradeoff because I care about more than just the visual appearance of the app.
A "visually correct" app with click handlers on divs and no semantic HTML is a liability (legally, maintenance wise, etc).. I'd like the E2E testing tool to assert that the app is working correctly, which does mean some assertions about the DOM are appropriate to me. I agree with the author of the linked post that "we want a solution that is not brittle in unwanted ways." We can be selective about what DOM things are important.
In the linked post the author says "In particular, DOM tests are tightly coupled to the structure of your code." and gives an example about a Selenium test that uses a brittle xpath that depends on a specific DOM structure.
Maybe I have not been exposed to enough of the industry to know that there are thousands of setups relying on flaky xpaths to target elements for testing. To me, it is not true that DOM tests are tightly couple to the structure of your code by default. It's a false statement made for marketing purposes and it is gross.
DOM tests "can be flaky", "are sometimes coupled to DOM structure" or whatever, is a fair assertion, but flakiness in DOM-driven testing is not a fact, it's a sign of badly written DOM-based tests. This is often the first thing I address in a code review of new tests written by somebody who does not write lots of FE test code, and they easily learn how to avoid it.
Maybe I'm wrong but it seems really really basic stuff to not create brittle selectors that fail tests for reasons we don't care about.
I like the OS-level interaction and agree that provides some advantages. I totally disagree that these advantages mean your solution wins at the "best way" to test, but it does clearly cover a different surface area than other automated solutions for E2E testing, and it seems like tests are pretty quick to knock out.
This solution could be a complement to other automated E2E tests, and I would see no reason that a PM or other party couldn't spin up and maintain some tests this way as quick-to-create set of tests to run against various builds, knowing that design changes will break them but that this is OK cause in theory it is quick to rewrite them.
But I couldn't see using this tool as the only E2E testing solution as though it is a superset of what Cypress/Selenium/Whatever tests are capable of. It is actually not a competitor with those tools. It's addresses different concerns with a little bit of overlap.
I'm happy to check out the free trial and see if I'm missing something, and eat crow if I'm being unfair here.