I think the venn diagram of “people who find primes interesting” and “people who understand password security” is pretty close to a circle.

3,5,7 line up on the diagonal, leaving 2 as the only other prime. So people using physical patters are likely to choose them. Not to mention any mathematically inclined person may also choose all primes.