Maybe kexec-ing into the new kernel would be feasible?
I investigated this many years ago to see if it was, but I found scant information on compatibility requirements when using kexec to hand over execution to an arbitrary kernel.
The problem seems really hard though. One issue that stands out to me is that even if you properly shutdown the old kernel, will all system devices be in a 'good enough' state to be reinitialized properly by the new one? Or do some devices require a reboot for some reason?
When I've built custom kernels, I'm pretty sure that the new kernel wasn't active until I rebooted. But rebooting after even dist-upgrade has just become automatic.
For unattended upgrades, you can disable automatic reboot. But then, I think there's risk that some upgrades won't take effect.
Userspace sure, but what about underlying hardware? How will device drivers react if they come up and encounter hardware that is not coming out of a ACPI induced reboot? Will some devices and their corresponding drivers be OK? Or will the drivers panic when they encounter a device in a weird state?
I'm genuinely curious is all. At the time I was pursing this I decided it was going to get too complicated and that I had to live with a reboot.
This is a concern but it usually works for two reasons:
1. Most firmware is sufficiently broken that Linux drivers are already hardened against devices being brought up in arbitrary states.
2. kexec walks the device tree to shut down all devices before starting the new kernel. This usually gets devices closer to a startup state, or at least a smaller number of known shutdown states.
I investigated this many years ago to see if it was, but I found scant information on compatibility requirements when using kexec to hand over execution to an arbitrary kernel.
The problem seems really hard though. One issue that stands out to me is that even if you properly shutdown the old kernel, will all system devices be in a 'good enough' state to be reinitialized properly by the new one? Or do some devices require a reboot for some reason?