Booting 5000 Erlangs on Ampere One: The Ultimate IoT Virtualization Madness

Hello everyone. Today’s flavour of fun is a perfect storm of raw silicon muscle, obscure ARM CPU privilege levels, and a bootloader written because – well, apparently wrestling u-boot is about as productive as trying to teach a goldfish algebra. All this is wrapped up in a bit of showmanship about running thousands of virtual IoT devices… because why use a server for something sensible when you can push it to the brink of madness?

The Premise: Because We Can

So, we’ve got an Ampere One beast – 192 cores, 1 TB of RAM – and the mission was clear: cram as many virtual IoT devices into it as possible using Nerves under Elixir on the BEAM VM. Last year? 500 VMs. Maybe 1000 if you squinted. But now, with KVM acceleration in the toolbox, we’re aiming for hilariously absurd territory. Spoiler: they did it.

Nerves, for the uninitiated, basically treats Linux like a glorified kernel and driver delivery mechanism while the BEAM runtime is the real operating system. It has the stability of an unshakable monk and the memory safety of a paranoid surgeon. Perfect for embedded IoT – your JSON-mangling devs can’t segfault the device any more than they can install malware on a Swiss watch.

The Ridiculous Bootloader Saga

Frank Hunleth, clear co-conspirator in this madness, decided the path to glory was writing his own bootloader. Not debugging u-boot. Not tweaking configs. He just wrote one. Because why not rebuild the wheel if you can make it shinier? Supposedly, ChatGPT threw some ARM64 crumbs his way and Frank MacGyvered them into a tiny, functional boot mechanism. It’s cute, it works, and it restores the A/B firmware upgrades that Nerves thrives on. Frankly (pun intended), it’s tech wizardry with a side of masochism.

KVM: The Nitro Button

KVM on ARM64 turned this from a wheezy little VM farm into an industrial-grade server horde. Instead of emulating a meek Cortex-A53, we’re letting our VMs ride shotgun directly on the Ampere cores. Result? Boot-to-IEx in single-digit seconds, 500MB less memory overhead per guest, and the smug sense that “this is now stupidly fast” while sipping coffee over SSH.

qemu-system-aarch64 \
-machine virt,accel=kvm \
-cpu host \
-smp 1 \
-m 150M \
-kernel ../little_loader.elf \
-netdev user,id=eth0 \
-device virtio-net-device,netdev=eth0 \
-drive if=none,file=special.img,format=raw,id=vdisk \
-device virtio-blk-device,drive=vdisk \
-nographic

That’s the heart of it – one core per VM, skinny memory allowances, attached storage, no GUI fluff, just headless terminals of doom.

From 3000 to 5100 Devices

The progression was satisfyingly ludicrous. First the server hit 3,389 connected devices before the OOM killer paid a visit. Then through some good old memory tuning – zram, swappiness tweaks, vfs_cache_pressure magic – they bullied it all the way up to 5,100 stable guests. Each running Linux, Erlang, and Nerves base features. And yes, the CPU? Chilling at <20% usage like it’s bored in a tutorial level.

• Each VM: ~110MB internal RAM at start, ~160MB steady state.
• Stable over thousands of instances.
• Updates pushed concurrently to hundreds at a time without meltdown.

Doctor’s Orders (Memory Tuning)

Like a doctor prescribing a diet to an overfed patient, Frank’s advice slimmed the system down. Tweak BEAM’s allocator appetite, swap to zram, adjust kernel greed – voila, leaner, meaner guest instances. Yes, performance “might” dip in theory, but the goal wasn’t Olympic sprints, it was a VM clown car stunt act – how many can we stuff in the chassis before the wheels come off?

Why This Matters (Sort Of)

If you squint past the showboating, there’s utility here: consistent, hardware-free testing environments for Nerves projects. Apple Silicon, ARM64 Linux, even x86 – if you’ve got qemu and some patience, you can replicate embedded IoT behaviors without touching a physical board. The bootloader hack is definitely not just a vanity project… probably.

Final Prescription

This was nerd fireworks: unnecessary, vaguely impractical, but executed with style. The Ampere One didn’t break a sweat, KVM turned sluggish emulation into warp speed, and memory tuning unlocked record VM densities. Do you “need” to run 5,100 Erlang-powered IoT devices on one box? Absolutely not. Was it worth it as tech theatre and tooling evolution? Absolutely.

Verdict: Highly entertaining, technically impressive, completely over the top – just the way I like it.

And that, ladies and gentlemen, is entirely my opinion.

Source: Booting 5000 Erlangs on Ampere One 192-core, https://underjord.io/booting-5000-erlangs-on-ampere-one.html