You’ve hit on a giant thing that bothers me about this discourse: endless rationalistic discourse about systems and phenomena that we have no experience with at all, not even analogous experience.
This is not like the atomic bomb. We had tons of experience with big bombs. We just knew atom bombs if they worked could make orders of magnitude larger booms. The implications if real big bombs could be reasoned about with some basis in reality.
It wasn’t reasoning about wholly unknown types of things that no human being has ever encountered or interacted with.
This is like a panel on protocols for extraterrestrial contact. It’d be fine to do that kind of exercise academically but these people are talking about passing actual laws and regulations on the basis of reasoning in a vacuum.
We are going to end up with laws and regulations that will be simultaneously too restrictive to human endeavor and ineffective at preventing negative outcomes if this stuff ever manifests for real.
> This is not like the atomic bomb. We had tons of experience with big bombs. We just knew atom bombs if they worked could make orders of magnitude larger booms. The implications if real big bombs could be reasoned about with some basis in reality.
Well, we thought we did.
We really didn't fully appreciate the impact of the fallout until we saw it; and Castle Bravo was much bigger than expected because we didn't know what we were doing; and the demon core; and the cold war arms race…
But yeah, my mental framing for this is a rerun of the first stage of the industrial revolution, and it took quite a lot of harm for what is now basic workplace health and safety such as "don't use children to remove things from heavy machinery while it's running", and we're likely to have something that's equally dumb happen even in the relatively good possible futures that don't have paperclip maximisers or malicious humans using AI for evil.
There are so many differences here vs organisms shaped by evolution and involved in a food web with each other. This is much closer to space aliens or beings from another dimension.
If there are huge risks here they are probably not the ones we are worried about.
Personally one of my biggest worries with both sentient AI and aliens is how humans might react and what we might do to each other or ourselves out of fear and paranoia.
This is not like the atomic bomb. We had tons of experience with big bombs. We just knew atom bombs if they worked could make orders of magnitude larger booms. The implications if real big bombs could be reasoned about with some basis in reality.
It wasn’t reasoning about wholly unknown types of things that no human being has ever encountered or interacted with.
This is like a panel on protocols for extraterrestrial contact. It’d be fine to do that kind of exercise academically but these people are talking about passing actual laws and regulations on the basis of reasoning in a vacuum.
We are going to end up with laws and regulations that will be simultaneously too restrictive to human endeavor and ineffective at preventing negative outcomes if this stuff ever manifests for real.