I had the opposite reaction: I wonder if tests make it easier for you to fix code without forcing you to develop a mental model of it, assuming you're working in an unfamiliar codebase. That seems like something of a hidden drawback.
That may be possible, but it isn't inevitable. I use tests to validate that my mental model is correct. When I'm doing something greenfield, you'll see my tests are full of rather stupid-looking assertions of really basic stuff, and the reason for that is that about 5% of the time, my really basic so-simple-it-couldn't-be-wrong is wrong.
If you're building something that's going to be used as a foundation by lots of other things, those 5% errors add up really fast.