Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

A ‘full’ test suite? I’m not sure that one of those has ever existed for any large program, let alone one under active development. We have to work in the real world here!


Yes, a full test suite. 100% code coverage is a thing, even in compiler validation.

Also, every test that was ever written for a compiler is still out there, being run, somewhere.

And, for the record, the real world is pretty big.


But 100% code coverage isn't necessarily a useful thing in this context, as different code (and different options) mean different interactions between passes. And you're never going to cover all the different paths through the compiler.

Back when I worked on a commercial compiler, we had a smoke test which we ran before committing. It would take ~30 minutes to run. We had a longer, roughly eight hour, set of tests which we ran daily. And we had machines running tests 24/7, which managed to clock up roughly 20 million distinct test cases (combinations of code and options) over the course of a year.

Every single compiler bug we'd ever seen was present (in a reduced form) in the conformance tests we'd run before release. That took about a week.


>And you're never going to cover all the different paths through the compiler.

Depends who you work for and how willing they are to let such justifications stand in a court of law.


100% code coverage does not mean a full test suite. You can still find huge numbers of bugs in code that has 100% coverage in a test suite. The two are not the same!


I'm not offering 100% coverage as anything but a level playing field, that things are tested. Also, code-path analysis is pretty much more advanced than you might think, especially when you get auto/generated testing rigs configured for particular tasks - and especially if you have a compiler to test, as well as all the (reproducible) applications it may produce ..




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: