I know hard-core Java people that use Vim. It integrates with Eclipse (by talking to a headless instance over a socket) for the refactoring stuff. Vim for the editing, Eclipse for the Java munging. Seems to work very well.

Never use an "IDE" unless there is absolutely no other way.