Yes you do have to keep up. Because one day you'll go install your 2.3 app on a new server, and you'll have an incompatible version of some library because Rails 2.3 didn't specify version number dependencies.
Like what happened to Rails 3.0.7 and Rake 2 weeks ago.
That's why you use Bundler. In my experience Bundler solves this problem 100%. Lock your gems and get Ruby from RVM and the chances of nasty surprises are next to nil.
I wish the Rubygems guys would put down the crack pipe though. They're the one problem in this setup and, admittedly, they're a massive problem :(
Like what happened to Rails 3.0.7 and Rake 2 weeks ago.