IIRC the KeyKOS/EROS/CapROS tradition used capabilities for everything including the scheduler. Of course, pervasive persistence makes those systems somewhat esoteric (barring fresh builds, they never shut down or boot up, only go to sleep and wake up in new bodies; compare Smalltalk, etc.).
Amoeba was my favorite, as it was a homogeneous, decentralized operating system. Different CPU architectures spread across different data centers, and it was all homogenized together into a single system image. You had a shell prompt where you typed commands and the OS could decide to spawn your process on your local device, in the server room rack, or in some connected datacenter in Amsterdam, it didn't make a difference. From the perspective of you, your program, or the shell, it's just a giant many-core machine with weird memory and peripheral access latencies that the OS manages.
Oh, and anytime as needed the OS could serialize out your process, pipe it across the network to another machine, and resume. Useful for load balancing, or relocating a program to be near the data it is accessing. Unless your program pays special attention to the clock, it wouldn't notice.
I still think about Amoeba from time to time, and imagine what could have been if we had gone down that route instead.
There's no free lunch of course, so you would have circumstances where a network partition at a bad time would result in a clone instead of a copy. I don't know what, if anything, Amoeba did about this.
In practice it might not be an issue. The reason you'd typically do something like move processes across a WAN is because you want it to operate next to data it is making heavy use of. The copy that booted up local to the data would continue operating, while the copy at the point of origin would suddenly see the data source go offline.
Now of course more complex schemes can be devised, like if the data source is replicated and so both copies continue operating. Maybe a metric could be devised for detecting these instances when the partition is healed, and one or both processes are suspended for manual resolution? Or maybe programs just have to be written with the expectation that their capabilities might suddenly become invalid at any time, because the capability sides with the partition that includes the resource? Or maybe go down the route of making the entire system transactional, so that partition healing can occur, and only throw away transaction deltas once receipts are received for all nodes ratcheting state forward?
Guess I'm too ignorant. I need to read up on these. I did know about the persistence feature. I think it's not terrible but also not great, and systems should be designed for being shut down and apps being closed.
> I think it's not terrible but also not great, and systems should be designed for being shut down and apps being closed.
The problem with shutdowns and restarts is the secure bootstrapping problem. The boot process must be within the trusted computing base, so how do you minimize the chance of introducing vulnerabilities? With checkpointing, if you start in a secure state, you're guaranteed to have a secure state after a reboot. This is not the case with other any other form of reboot, particularly ones that are highly configurable and so easy for the user to introduce an insecure configuration.
In any case, many apps are now designed to restore their state on restart, so they are effectively checkpointing themselves, so there's clearly value to checkpointing. In systems with OS-provided checkpointing it's a central shared service and doesn't have to be replicated in every program. That's a significant reduction in overall system code that can go wrong.
It's fallacious to assume that the persistence model of the system can't enter an invalid state and thus cause issues similar to bootstrapping. The threat model also doesn't make sense to me: if an attacker can manipulate the boot process, I feel like they would be able to attack the overall system just fine. Also, there's the bandwidth usage, latency, and whatnot. I think persistence is a strictly less powerful, although certainly convenient, design for an OS.
> The threat model also doesn't make sense to me: if an attacker can manipulate the boot process, I feel like they would be able to attack the overall system just fine.
That's not true actually. These capability systems have the principle of least privilege right down to their core. The checkpointing code is in the kernel which only calls out to the disk driver in user space. The checkpointing code itself is basically just "flush these cached pages to their corresponding locations on disk, then update a boot sector pointer to the new checkpoint", and booting a system is "read these pages pointed to by this disk pointer sequentially into memory and resume".
The attack surface in this system is incomparably small compared to the boot process of a typical OS, which run user-defined scripts and scripts written by completely unknown people from software you downloaded from the internet, often with root or other broad sets of privileges.
I really don't think you can appreciate how this system works without digging into it a little. EROS was built from the design of KeyKOS that ran transactional bank systems back in the 80s. KeyKOS pioneered this kind of checkpointing system, so it saw real industry use in secure systems for years. I recommend at least reading an overview:
EROS is kind of like what you'd get it if you took Smalltalk and tried to push it into the hardware as an operating system, while removing all sources of ambient authority. It lives on as CapROS:
I don't deny that bootstrapping in current systems is ridiculous, but I don't see why it can't be improved. It's not like EROS is a typical OS either. In any case, I'll read up on those OSes.
