what version of the kernel do you use (the github page says 5.10 but isn't that quite old?)
what (extremely minimal, I imagine?) kernel configuration
What do you use to build the 'micro' images (I'm guessing many won't even have a classic pid-1 such as systemd, but put their software as pid-1?).
How do you keep timesync of you're not using a timesync daemon? Can you make one of these daemons work on af_vsock (I know firecracker gives a virtio-backed Ethernet device but what if you only want af_vsocks?).
Handle kernel and app logs without adding an log daemon, and same through vsocks, etc?
> what version of the kernel do you use (the github page says 5.10 but isn't that quite old?)
Right, they have tested with 5.10, but it also works with higher kernel versions. Our host currently runs 5.19 and we're planning to upgrade to 6.1 soon. The guest runs 5.15.63, we use a config very similar to the recommended config by FC team (it's in the FC repo). It's important to mention that we had to disable async pagefaulting (a KVM feature) with more modern kernel versions, as VMs could get stuck waiting for an PF resolve.
> What do you use to build the 'micro' images
We created a CLI that creates a rootfs from a Docker image. It pulls the image, creates a container and then extracts the fs from it to an ext4 disk. For the init, we forked the open sourced init from the Fly team (https://github.com/superfly/init-snapshot) and changed/added some functionality.
> How do you keep timesync of you're not using a timesync daemon?
IIRC we expose the time as a PTP device (handled by kvm) and run phc2sys to sync the time in an interval. Firecracker has some documentation on this, where it recommends chrony. It can also be done with vsock, but it would be more manual.
> Handle kernel and app logs without adding an log daemon, and same through vsocks, etc?
The init forwards stdout/stderr of the command it runs to its own stdout, which Firecracker then logs out by itself. A supervisor reads these and writes the logs to files.
> We created a CLI that creates a rootfs from a Docker image. It pulls the image, creates a container and then extracts the fs from it to an ext4 disk.
You guys don't happen to have a public writeup about how this works, do you? Maybe it's as simple as it sounds, but Fly and CodeSandbox both have some magic to turn Docker images into VM disks that I'd like to know how to build :)
Fly is doing fancy stuff to avoid using docker entirely, but with docker you can just run "docker export" to dump an image to a .tar file that contains the whole filesystem. Built-in feature. I use this as a convenient way to grab a foreign platform sysroot for clang cross-compilation; just pick a Docker image and rip the filesystem out.
Oh thanks a lot. PTP! I need to try this out. And thanks for the init from fly.io too! And... I think I know now why my vms would get stuck thanks for the tip!
6.2 kernels work fine with Firecracker, and you can compile out a lot compared to a normal x86-64 defconfig: PCI, USB, SCSI, ATA, md, non-virtio ethernet, etc. You probably still want CONFIG_SERIAL_8250 and CONFIG_SERIAL_8250_CONSOLE but everything else can be virtio net and block. (Firecracker uses memory-mapped virtio not PCI, which enabled you to compile out PCI altogether, unlike qemu.) 100ms boot time to exec()ing init is definitely realistic.
Ah yes thanks, I tried going down the absolute slimming down of the kernel rabbit hole, but couldn't find a write-up on the actual best practice on this for firecracker and micro/lightweight VMs, I found some things in the embedded/microcontroller side of things, but it still has lots of devices and features. Is there a minimal, canonical conf somewhere for this?
That is actually a very nice post (and far more advanced than the usual Julian Evans post I see - not judging here, I really enjoy reading everything she publishes) and you posted the HN link with a comment by Thomas Ptacek too, with interesting links about performance and a discussion of actual isolation.
what version of the kernel do you use (the github page says 5.10 but isn't that quite old?)
what (extremely minimal, I imagine?) kernel configuration
What do you use to build the 'micro' images (I'm guessing many won't even have a classic pid-1 such as systemd, but put their software as pid-1?).
How do you keep timesync of you're not using a timesync daemon? Can you make one of these daemons work on af_vsock (I know firecracker gives a virtio-backed Ethernet device but what if you only want af_vsocks?).
Handle kernel and app logs without adding an log daemon, and same through vsocks, etc?