> If anything, unit tests are the most brittle and susceptible to low-level changes.
I don't find this to be the case if the unit tests are precise (which they should be).
That is, if you are writing non-flaky unit tests which do all the "right" unit-testy things (using fakes/dependency injecting well and so isolating and testing only the unit under test), you should end up with a set of tests that
- Fails only when you change the file/component the test relates to
- Isn't flaky (can be run ~10000 times without failing)
- Is quick (you can do the 10000 run loop above approximately interactively, in a few minutes, by running in parallel saturating a beefy workstation)
This compares to integration/e2e tests which inherently break due to other systems and unrelated assumptions changing (sometimes legitimate, sometimes not), and can have rates of flakyness of 1-10% due to the inherent nature of "real" systems failing to start occasionally and the inherently longer test-debug cycle that makes fixing issues more painful (root causing bug that causes a test to fail 1% of the time is much easier when the test takes .3 CPU-seconds than when it takes 30 or 300 CPU-seconds).
Very few tests I see are actually unit tests in the above sense, many people only write integration tests because the code under test is structured in inherently un- or difficult- to test ways.
That's true. I probably used brittle in the wrong sense there.
What I mean is that after any code change that isn't a strict refactoring you will inevitably have to touch one or more unit tests. If you're adding new functionality, you need to test different code paths; if you change existing functionality, you need to update the related test(s); if you're fixing a bug, you need to add a unit test that reproduces it, and so on. All this means is that unit tests take the most effort to maintain, so I'm not sure why the article claims they have "high" maintainability, or that that's a good thing. In contrast, higher-level tests usually require less maintenance, assuming they're stable and not flaky, since you're not touching them as often.
> Very few tests I see are actually unit tests in the above sense, many people only write integration tests because the code under test is structured in inherently un- or difficult- to test ways.
Very true. I think the recent popularization of alternative guidelines like The Testing Trophy is precisely because developers are too lazy to properly structure their code to make pure unit testing possible, and see the work of maintaining unit tests as too much of a chore, so they make up arguments that there's no value in them. This couldn't be farther from the truth, and is IMO an irresponsible way to approach software development.
I don't find this to be the case if the unit tests are precise (which they should be).
That is, if you are writing non-flaky unit tests which do all the "right" unit-testy things (using fakes/dependency injecting well and so isolating and testing only the unit under test), you should end up with a set of tests that
- Fails only when you change the file/component the test relates to
- Isn't flaky (can be run ~10000 times without failing)
- Is quick (you can do the 10000 run loop above approximately interactively, in a few minutes, by running in parallel saturating a beefy workstation)
This compares to integration/e2e tests which inherently break due to other systems and unrelated assumptions changing (sometimes legitimate, sometimes not), and can have rates of flakyness of 1-10% due to the inherent nature of "real" systems failing to start occasionally and the inherently longer test-debug cycle that makes fixing issues more painful (root causing bug that causes a test to fail 1% of the time is much easier when the test takes .3 CPU-seconds than when it takes 30 or 300 CPU-seconds).
Very few tests I see are actually unit tests in the above sense, many people only write integration tests because the code under test is structured in inherently un- or difficult- to test ways.