>But not everything can be automated, because some situations are more complex
This is where you lost me. You are very right that brains do all kinds of things to regulate the body that we don't usually think of as influencing thought. But we can't model those because they're too complex? I don't think environmental variables are too complex.
But (1) if those are important, we can model them too and (2) it may be that we don't need to model computer 'thinking' after the structure of human brains anyway to solve problems intelligently. And (3) the totality of things an AGI might be 'aware of', even without simulating biology, could very well mean that the 'intelligence' of a system is nestled in a complex web of variables that give it the ability to have the equivalent of our tacit knowledge. That's probably an informational question rather than a question of needing to simulate biology.
If you are in the jungle and a lion comes at you, you will not have time to sit down and think what to do, and your brain is prepared to act in those situations as well.
I'd say that's an extremely crude understanding of what computers can actually model. There are ways that humans stream through thoughts that depend on tacit knowledge and unconscious connections, and there's no reason why those can't be modeled by a computer. And the equilavence betweeen "rational" (in some informal, human sense of the term, as in talking out loud like spock), and "computable" is just a misunderstanding. Computable contains much more than this naiive conception of what is rational.
This is where you lost me. You are very right that brains do all kinds of things to regulate the body that we don't usually think of as influencing thought. But we can't model those because they're too complex? I don't think environmental variables are too complex.
But (1) if those are important, we can model them too and (2) it may be that we don't need to model computer 'thinking' after the structure of human brains anyway to solve problems intelligently. And (3) the totality of things an AGI might be 'aware of', even without simulating biology, could very well mean that the 'intelligence' of a system is nestled in a complex web of variables that give it the ability to have the equivalent of our tacit knowledge. That's probably an informational question rather than a question of needing to simulate biology.