I personally don't like crashing my RC car or quadcopters, either- it isn't something I like to do. I'm also a new driver, and I've had several near misses. One of the possibilities with tele-op is that someone entirely different can take over if someone else falls asleep, messes up, or does something belligerent. Technology is a double-edged sword.
That doesn't solve the "hand-over" problem --- which I already mentioned in another comment here --- if you were suddenly given control of a moving car headed in a dangerous direction, how long would it take to assess the situation enough to make a decision on what to do? You have to be already paying attention to react in reasonable amount of time.
Of course there is the "workaround" of having multiple (alert) drivers per vehicle, but that seems more trouble than it's worth just to avoid putting a single driver in a vehicle. (In addition, as all the stories about self-driving-car accidents have shown, even when people are in the vehicle itself, they don't tend to pay as much attention if they already have the "someone/thing else is driving" notion anyway.)
