No. It really is that simple. There are x units and y people looking for units in an area. If we had enough housing it wouldn't be seen as investment because the returns would be terrible since landlords would be competing for renters instead of the other way around. Leaving housing vacant only makes sense if you've cornered the market or are a land speculator. Most American cities do not have a cornered rental market so we just need to stop land speculation.
I think you missed my point entirely. I'm saying that supply-and-demand is too simple to be useful, not that it is untrue. It does not point directly at the dominant effect that is causing housing to be priced the way it is, although it is part of the overall equation. I still believe that building lots of housing is a great idea (and finding a way to make more cheaper non-luxury-priced housing get built would be even better). But it's not a good-enough theory of housing prices.