> Asimov predicted that moon colonies would be common by now, for example. The overoptimism may have been due to the rate of technological growth leading up to 1964.
I think it is more likely a sad indication of how we have moved away from spending money on space exploration, and instead spending it elsewhere. We certainly could have moon bases if we had continued spending and researching into space.
We could have moonbases but what use would they be? In general any futurist has been ridiculously optimistic about energy generation/storage, robots, and transportation, and hopelessly pessimistic about (or even oblivious to) computers, communication, and anything biological (except perhaps for sanguine predictions about curing cancer, ageing, or the common cold). Arthur C. Clarke, whose predictions are linked from the article, completely missed miniaturization. Asimov was one of the few who foresaw miniaturization but oddly, as a biologist, seemed completely oblivious to genetic engineering etc.
In general, futurists tends to overestimate trends from technologies that are at the middle of an s-curve and underestimate technologies at the beginning of the s-curve. I expect future predictions to shift along the same trends, overestimating trends that are currently skyrocketing.
And of course, which technologies the futurist is personally familiar with can vary widely. Asimov, for example, had a blind spot for robots.
I'm not sure as a general observation the "s-curve" turns out to work that well. E.g. futurists have (with Kurzweil and his crowd a notable exception) consistently underestimated progress in computer capabilities even in the 80s and 90s when progress was insanely rapid, while over-estimating advances in AI (which is, optimistically, at the beginning of its s-curve). You could almost include Asimov in that. Similarly, Arthur C. Clarke was oblivious to miniaturization while living in the thick of it.
I think one of the factors is romance -- SF writers want certain things to happen and just wish it into being, regardless of how plausible it really is (faster than light travel, time travel, anti-gravity, artificial gravity, laser pistols).
On the reverse side they tend to ignore technologies that are either unromantic or make stories more complicated or seem to fight against cliched dramatic opportunities. (Fast travel adds to romance and drama, but fast communications detract from romance and drama.)
I think over-estimation of AI comes less from the understanding the position of the technology on the s-curve, but more from easily conceivable vision of machine behaving just like human. Nobody wants to listen to people who predict AI in future being pattern-scanners, optimizers or programing languages, because that, as you said, isn't romantic or dramatic at all.
And I would dare to argue that the AI is at the beginning of the s-curve. The field of study have been around for many decades already, many applicable results found and it is used in the industry. It's kind of like saying that current transportation technology is at the beginning of the s-curve, because I am waiting for near light-speed transportation, and current speeds are very small compared to my arbitrary expectation.
Alternately, though this may be stretching the point, you could say that '60s AI-as-human was at the _end_ of its curve, shortly to be replaced by the more practical actual AI. While specific writers have different biases--and some end up way ahead of the curve--in the aggregate I think futurists tend to overestimate the technologies of their era.
Though you're definitely right that literary biases play into it. While not not all SF writers are futurists, most futurists are trying to tell stories. Even some who aren't primarily SF writers: Kurzweil has a narrative arc to his ideas.
I am sure that's true, but the problem is that the costs of space exploration haven't gone down the way that e.g. the costs of air travel have gone down over the same period. That, combined with the fact that none of the expected advantages of manufacturing and research in space have panned out means that it turns out there's not much for people on a Moon base to usefully do and it's a lot more expensive than expected for them to do it.
If SpaceX actually does manage to crack efficient rocket re-useability that will certainly help with the cost side of things. The potential there is really exciting, but since there are plenty of uninhabited bare rocks on Earth for people to establish colonies on if they want to, it's still hard to see why they'd choose to do so on the Moon.
I think it is more likely a sad indication of how we have moved away from spending money on space exploration, and instead spending it elsewhere. We certainly could have moon bases if we had continued spending and researching into space.