Unfortunately there is a very, very, large set of possible situations a car may find itself in that requires 'user intervention'. Especially without much context-aware systems, or AI, the best error handler a driver-less car has is "stop, inform driver". Is there a curb to pull over to? What if the system is unable to determine if the path is clear because something like a branch is in the way?
Well, my line of argument is that any driver in a self-driving car is going to be completely inattentive (think available-to-public, not a test-phase car). For any self-driving car to be considered safe enough for public use, the car will need to be proven absolutely safe in "emergency" situations (i.e., auto-shutoff, etc.).
So, if the car is proven to be absolutely safe, then why do we need a driver? To pull it off the road after it shuts off or stops?
No licensed driver, car encounters situation it's not programmed for.
Now what?