From my experience using Sphinx, it would be a non-trivial task to make it work with Haystack. There's the django-sphinx project which is what I currently use, but I remember getting started with it was kind of a pain. It's speed is really nice though.
I started coding up a site using this (yesterday, actually) and it's very easy. Whoosh support is cool, although you have to use a forked version that reverts a couple changes. One thing that bothers me is that you have to label the template files for search results as *.txt for some reason.
It currently support Solr really well, and Whoosh (Pure Python) decently.