Linux binaries, not recompiled.
A different kernel underneath.

Rinux is a new kernel, written in Rust and designed for servers. It runs already compiled Linux binaries, as they are, inside domains: each one with its own memory, CPU, root and faults. When something goes down inside a domain, the rest of the machine does not notice.

rinux-1 Google Cloud n2-standard-2kernel up 0 s


          
faults contained 0

Press a domain to trigger a fault in it: the kernel restarts it with the next generation and the others keep running. The session is real, copied from the VM's serial port and from SSH.

This same Rinux node also serves hullcondorstilapollo11bakeroxidestore x3
PostgreSQL 17, GCE200/ 200

committed transactions still there after a power cut under load

boot, GCE129ms

from kernel entry until it is ready to work

Linux 6.12, x86_64252/ 375

system calls implemented, each with its status written down

Rinux is the operating system, not something that runs on top of one. It boots directly on the server or the cloud VM, in place of Linux.

Debian 13, from an OCI image verified layer by layer with SHA-256, boots with systemd in a Rinux domain. Over SSH, apt installs packages and curl speaks HTTPS. Everything they do underneath is handled by Rinux.

Stronger than a container, lighter than a virtual machine.

A domain is not a namespace trick on top of a shared kernel. It is an object of the kernel itself, which owns its resources and answers for its faults.

docker dockerd shared Linux rinux Rinux kernel a kernel faultreaches everyone each domain ownsits resources, faults

It is not Docker

Docker needs a Linux underneath, and its containers share that kernel. Here there is no Linux: Rinux takes Docker images as they are and runs each one in a domain isolated by the kernel itself.

VMs linuxlinuxlinux hypervisor rinux Rinux kernel one guest kernelper workload one kernel isolatesall the domains

It is not a virtual machine

There is no guest Linux per workload. A single kernel isolates all the domains, and it boots in 129 ms.

apt, sshd, postgres openat() Linux 6.12 personality Rinux vfs, rfs, network the program sees no difference

It is Linux compatible

Debian, systemd, apt and sshd do not know they are not on Linux, because Rinux speaks the same system call language.

This page is served by Rinux.

A node in production, with no Linux in any layer

This page and other sites of ours run on a Rinux node in Google Cloud. The same machine runs the Mentat control plane, which decides which workload goes where, and Cóndor, the proxy that terminates TLS and obtains the Let's Encrypt certificates. Each site lives in its own domain.

15to 21 sof downtime when the whole VM is restarted
463msto fetch this page from Santiago, full TLS
13workloads and the control plane on 2 CPUs and 8 GiB

Hard reset of the VM from Google Cloud, with a probe to each domain every half second: six came back at 15 s and the heaviest, at 60 MB, at 21 s. The 463 ms is the median of 20 new connections, with a 148 ms round trip to the cloud.

Loader with two slots

A new kernel boots once on trial and, if the node does not confirm it, the loader goes back to the previous one. It only accepts signed images.

Network closed by default

Each domain opens only the ports and the egress it is granted. A workload with internet egress cannot reach the metadata server.

Egress by list, warning and quarantine

A workload reaches only the destinations it is given by name; the kernel also filters its DNS queries. Every attempt to go elsewhere is refused and reaches a phone as a warning within seconds. In production, a test workload insisted six times in a minute: it lost its egress and kept serving. No workload on the node has open egress to the internet.

What real traffic found

Three bugs, each with its test: a TCP segment held back, a lock that made a thread spin, and a control plane that was slow to bring workloads back.

We tried to trap an attacker.

We deployed a test workload of our own on the production node and made it act like an attacker that had already taken over an application. Nobody broke in from outside: the workload is ours, and the point was to see what it can do from its domain. It tried to reach the internet, the metadata server —where the node's keys live—, the other workloads, even just to see them. It could do none of it.

0destinations reached beyond the single one it was granted
0other workloads seen, reached, or keys read
secto refuse, warn a phone, and cut its egress on its own

The workload was granted egress to a single destination. It tried the internet by address and by name, the metadata server, the gateway, the ports of the other workloads and of the control plane, and reading their data: the kernel refused it all and counted every attempt; on the sixth in a minute, Mentat took its egress away while it kept serving. What we proved is that a workload already taken over cannot move or take anything out. How an application gets taken over in the first place depends on the application, not the kernel — that is a different test, not this one.

A 7.8 GB store, moved to this node.

OxideStore, our S3-compatible store, runs here as three domains with Raft between them. We copied 1,147 objects into it from three Linux virtual machines. The copy found what the tests had not: kernel limits that only show up with real data, each one fixed and covered by a test.

28min

to copy the 7.8 GB with four parallel uploads, with the same objects and sizes on all three nodes

15.9to 4.6 s

to write, read back and assemble 100 MiB on a network disk, before and after the fix

789to 1,856 MiB

the memory limit of one store node, raised and lowered back by Mentat without restarting it

The disk, a run at a time

The file system asked the disk for 4 KiB per request, and on a network disk each request costs half a millisecond. It now reads the neighboring blocks of a file in a single request. Completing a 92 MiB upload went from 13 s to 3 s.

Memory charged when it is used

Reserving memory no longer costs anything: each page is charged to the domain when it is touched, and what a program gives back stops counting. At rest, each store node shows 43 MB instead of 92.

Elastic memory

The kernel allows the limit of a running domain to be changed, and counts every time it denies the domain memory. Mentat looks at this every 5 seconds: it raises the limit of a workload that gets close to it and lowers it back when usage drops.

A stopped domain holds no memory

When it stops it releases everything: its in-memory file system and, as of this kernel, also the System V shared memory and semaphores that nobody removed. After the fuzzer, a domain with no processes was still charged 55 MiB; now all stopped domains together add up to less than half a MiB.

A table that only grew

Finished processes stayed recorded until someone listed their domain, and that cleanup stalled the rest of the node. Now each process leaves the table when it exits: after 2,000 that start and die, 2 entries remain, not 2,002.

Before production, the real machine

Each fix went through the 40 tests on aarch64 and through an x86_64 Rinux node in Google Cloud before getting here. The disk fixes were measured with disks identical to the ones on this node.

PostgreSQL, with real power cuts.

A database is the hardest test for a kernel: every committed transaction waits for an fsync, and that data has to be there after any power loss. PostgreSQL 17 from Debian runs unmodified in a Rinux domain, and we cut its power under full load.

200/ 200

committed transactions that survive the power cut under load

368

transactions per second with 50 clients in Google Cloud, without a single failed one

7.5ms

per transaction with one client: almost all of it is the disk fsync

The power cut in the test

With pgbench writing, 200 marked transactions are committed and the VM is powered off without warning. When it comes back, all 200 marks are there, the balances add up and pg_amcheck finds no corruption. On x86_64 and aarch64.

Hung on purpose

The main process is frozen: the port stays open but nobody answers. The kernel's liveness check detects it and restarts the domain. It comes back with all 200 marks in under a minute.

The power cut in the cloud

Hard reset of the VM from the Google Cloud console: 200 of 200 marks, balances that add up over almost 80,000 transactions and a clean pg_amcheck.

What PostgreSQL asked for

Seven kernel bugs, each with its test: System V shared memory, vectored I/O, stacked mounts, per-domain local networking and a disk read that failed under load.

Measured in the cloud

n2-standard-2 with a pd-balanced disk, pgbench scale 10: 133 tps with 1 client, about 314 with 8 and 368 with 50.

Oracle Database, with nothing changed.

Oracle is among the most demanding things you can ask of a kernel that is not Linux: a 434 MB executable, dozens of processes that share memory, threads that notify each other with signals and a listener that inspects the node's network before accepting anyone. Oracle Free 26ai runs in a Rinux domain from its container image, with its own user, and opens the database.

1,000/ 1,000

rows written and read back through the listener, over TCP

191s

from domain start to the database opened, used and shut down cleanly

5.5GiB

of memory at peak usage, with a limit of 6.5

What the test does

It boots the machine, waits for Oracle to open the database, creates a table, inserts 1,000 rows through the listener and reads them. If startup fails, it leaves the Oracle alert log on the console. Today it runs on aarch64.

An executable that is not copied

Each process had its private copy of the program, and one larger than 256 MiB would not start. Now its pages are taken from the file system cache and only the ones the process writes are copied, as Linux does.

The thread that never started

Each new Oracle thread reads its own entry in /proc. Rinux answered per process, not per thread: the thread gave up and Oracle reported that it could not create it. It was the hardest bug to see.

What else it asked for

The image's user for the first process, a /dev/shm that anyone can write to, signals that carry a value, the node's addresses over netlink and over ioctl, and sending several messages in a single call.

Where it stands today

In the test, not in production. Of the 5.5 GiB, 2.9 are the image layers, which are still kept in memory: moving them to disk is what comes next.

The kernel checks itself.

At the end of each test, the kernel checks its own accounting. If something does not add up, the test fails even if the program inside finished correctly. This way every real workload that runs on Rinux also tests the kernel.

43

boot tests on aarch64 that end with a clean kernel verdict

55MiB

that a stopped domain was still charged: the first thing it found

100/ 100

identical runs of a three-node cluster on the deterministic kernel

What it checks

It walks each whole volume and compares it with its block map: no block lost, none used twice, every name with its file. That what is charged to the domains does not exceed the node's memory. That every process is in the domain it belongs to.

The first thing it found

After the fuzzer, a domain with no processes kept the System V shared memory that nobody had removed. It now returns it on exit. The fix already runs on the node that serves this page.

The wall clock that leaked in

On the deterministic kernel, the same seed has to give the same execution. A cluster with systemd gave one trace before the top of the hour and another one after it: the first process of each domain was reading the machine's clock without going through the kernel.

The same thing a hundred times

With that closed, 100 runs with one seed gave the same trace of 844,211 events. Half of them were done with the machine's clock set to dates between 2024 and 2032. Since then every deterministic test is repeated with a different clock.

Also on real x86_64

Before production, seven tests and the deterministic fuzzer ran on a Rinux node in Google Cloud: all with a clean verdict, and two boots of the same disk with the same trace.

Built so that nobody has to go and fix it.

Rinux targets nodes that nobody can go to: the network starts closed and everything goes through the control plane. A node like that has to come back on its own from whatever takes it out of service.

129ms

from kernel entry until it is ready to work

40s

until SSH on Debian, with PostgreSQL and the encrypted disk

18s

from a failure to SSH again: the Debian domain restarts completely, on its own

A disk that survives power loss

Copy-on-write and an intent log: a power cut leaves the last committed state, never a mix.

Per-domain encryption

Each volume has its own key, sealed in the TPM. Deleting a domain destroys its key.

Updates with rollback

If the new kernel fails to boot properly twice, or crashes three times in 10 minutes, the loader goes back to the previous one on its own.

Lockup detector

A CPU stuck in the kernel for 20 seconds is detected, the reason is left in the log and the node is restarted.

Per-domain liveness check

Every 10 seconds the kernel verifies that the service accepts connections. A hung one is restarted. One stopped on purpose is not.

Watchdog at two levels

The hardware one restarts the VM if the kernel freezes. Each domain has its own /dev/watchdog for systemd.

A separate control channel

A second SSH in an independent domain answers when Debian does not. fail2ban blocks in the kernel itself.

Scans that do not shut the door

Thousands of half-open connections do not leave the node without SSH: they expire, and the oldest one gives up its place.

Memory with a clear limit

Each domain pays for the memory it uses, not the memory it reserves. If it goes over its limit, only the process that touched the extra page dies. The rest of the node does not notice.

Every claim has a test that boots the machine.

Nothing counts as done until a test boots Rinux in QEMU, on x86_64 and on aarch64, runs real Linux programs inside a domain and verifies the result from the inside.

252/ 375

Linux 6.12 system calls on x86_64, each with its status written down

220/ 318

on aarch64, with that architecture's own syscall number table

42

full boot tests, on both architectures

The tests, in the order they were added
TestWhat it verifiesChecksx86_64aarch64
bootUEFI, signed envelope, rinuxd in user mode, shutdown1passespasses
mmframes, heap, paging, faults, scheduler, timer8passespasses
elf-static64 static processes on 4 CPUs1passespasses
domainsfault isolation, capabilities, memory and CPU limits8passespasses
vfstmpfs, devfs, procfs, file syscalls10passespasses
procfork, execve, wait4, pipes, posix_spawn, /proc/self/exe17passespasses
signalshandlers, masks, Linux signal frames11passespasses
ttyterminals, pseudoterminals, job control11passespasses
dynamicdynamically linked BusyBox from Alpine as init15passespasses
alpineOCI image from a virtio disk, tampered digest rejected21passespasses
threadspthread, futex, copy-on-write, lazy pages, TLB, mremap, growing stack15passespasses
ipcshared mmap, FIFOs, permissions, System V shared memory and semaphores, vectored I/O, native AIO18passespasses
eventsepoll, eventfd, timerfd, signalfd, POSIX timers15passespasses
socketsAF_UNIX, descriptor passing, credentials, netlink12passespasses
systemstacked mounts, tmpfs options, sysfs, cgroup2, inotify15passespasses
systemdDebian 13 with systemd as init, journal, login, node name, key-only SSH, memory according to the node, process start time, watchdog, license notices15passespasses
netvirtio-net, DHCP, ARP, ICMP, TCP3passespasses
inetTCP and UDP sockets, network closed by default, DNS, netlink, inbound connections, each domain's own local network, egress to metadata only17passespasses
gceGoogle Cloud image: PCI without MCFG, legacy virtio, login, SSH and a real update4passesn/a
blkwdisk writes over virtio-blk, virtio-scsi and NVMe, verified from the host3passespasses
rfsour own on-disk file system, checked from the host at shutdown9passespasses
crypt-persistencrypted volumes that survive restarts and are destroyed with their key13passespasses
tpmkeys sealed in the TPM, across three boots11passespasses
loaderour own loader with A/B slots1passespasses
updatelive update, signature, a kernel that does not boot and automatic rollback18passespasses
load128 processes writing at once to an encrypted volume2passespasses
persistDebian installed on disk: changes survive a restart27passespasses
pgPostgreSQL 17 with a power cut under full load, and hung on purpose15passespasses
lockupa stuck CPU is detected and the node restarts on its own2passespasses
unstablea kernel that crashes three times goes back to the previous one on its own13passespasses
healtha hung service is restarted; a healthy or stopped one is not; per-domain watchdog6passespasses
watchdogthe whole kernel frozen: the hardware watchdog restarts the machine1passespasses
syna scan of half-open connections does not leave the port unserved1passespasses
keysSSH keys from the metadata server, users, live changes12passespasses
hvimage for Proxmox and KVM: without a TPM it explains why; with a TPM it installs encrypted and boots again3passespasses
sysfuzz144,000 system calls with random number and arguments from a seed: the kernel does not panic or lock up, and keeps serving4passespasses
sysfuzz-detthe same fuzzer on the deterministic kernel: the same seed gives the same kernel trace, call by call2passespasses
oracleOracle Free 26ai from its container image: opens the database, and 1,000 rows written and read through the listener1untestedpasses
cluster-detthree nodes with systemd and a load generator on the deterministic kernel: the same seed gives the same trace, with the machine's clock set to another date2untestedpasses

All the unsafe code lives in one place.

Rust does not stop you from touching hardware. It makes you declare it. In Rinux, everything that needs unsafe (page tables, device registers, DMA, context switching) is in a single crate with its invariants written down. The rest of the kernel is forbidden from using it.

Domains

Unmodified Linux programs, with musl or glibc, and rinuxd, the node controller. They run in user mode.

ring 3 on x86_64, EL0 on aarch64

rinux-kernel

Virtual memory, multiprocessor scheduler, domains and capabilities, file system, sockets, TCP/IP networking, signals and the Linux personality that translates each syscall.

ring 0 / EL1, no unsafe

rinux-frame

Paging, interrupts, CPU context, MMIO and DMA, exposed to the kernel as safe types.

the only crate with unsafe

Firmware

UEFI boots a single image, RINUX.EFI. The kernel discovers the machine through ACPI and verifies the Ed25519 signature of the boot envelope before executing anything.

x86_64 and aarch64 from the same source

Capabilities with a generation

When a domain restarts, its generation advances and every old permission stops working.

Limits that are enforced

CPU and memory quotas. Going over is an error of the domain, never of the node.

Syscall profile

What is not allowed is denied with the error Linux would give, and is recorded in the trace.

Its own local network

Each domain has its own localhost. None can reach the local network of another.

Where it runs.

Rinux boots as the operating system of the virtual machine. It is not published yet: the code and the images come out with the first release, under the MPL-2.0 license.

Google Cloud

Today on n2-standard-2 VMs, with a signed boot disk and updates with rollback. C3, C4 and N4 need the gVNIC driver, which is coming.

Proxmox, QEMU/KVM, libvirt

Image ready: three qcow2 disks and a virtual TPM, tested on x86_64 and aarch64.

AWS and Azure

On the way. AWS needs the ENA driver. Azure needs the Hyper-V devices.

Keys from the cloud

The image ships with no keys or passwords: it takes them from the metadata server.

The base of our stack.

Each piece runs in its own domain, with its own authority, on the same node.

  • Hullorchestrates workloads on Rinux domains.
  • Cóndorreverse proxy, with direct network queues.
  • OxideStoreobject storage and image registry.
  • Bakerdeterministic testing of distributed systems on Rinux.
  • Melangereceives the telemetry of every node and every domain.
  • LoaPostgreSQL proxy in front of databases in domains.
Rinux mascot: a rhinoceros in a tuxedo