Time symmetry applies to the physical laws governing the universe, not the distribution of mass in it.

For a simpler example look at Newton's 3 laws. They have time symmetry but still allow for things to move around!

It also doesn't apply to the structure of space-time. Which is not a situation that Newton's laws can deal with.