#atsiekratsiepersonal tech blog

From a dead Proxmox box to a 3-node ZimaBlade cluster

· updated 12 October 2026
VirtualisationHardwareNAS

It was fine. Until it wasn’t.

My Proxmox server died on me. One box, all my VMs, and suddenly the house got very quiet: no Hermes, no Sonarr, and no Uptime Kuma to even tell me things were down. It ran on an old single-board computer that simply stopped working one day. Old hardware with a thick layer of dust on it, so my money is on the power supply.

The good news: the disks of my guests were on an NFS share on my Synology. The better news: I already had six ZimaBlade 7700 boards waiting for a project. So instead of rebuilding the same single point of failure, I built a 3-node Proxmox cluster on three of them. The other three are saved for another project (that’s a future post).

This post is mostly for future-me, but if your homelab just went dark too, it might save you an evening.

The hardware

The ZimaBlade 7700 is a tiny, passively cooled x86 board. Per board you get a quad-core Intel Celeron, one DDR3L SO-DIMM slot (max 16 GB), 32 GB eMMC, two SATA ports, one network port and a PCIe 2.0 x4 slot.

Unboxing the ZimaBlade 7700: the bare board, a 16 GB DDR3L SO-DIMM, the SATA data and power cable, and a board already in its clear case

Per node I added 16 GB of RAM, the maximum. And that’s it: no SSD. Proxmox runs on the onboard eMMC, and every guest disk lives on the Synology. The nodes only need to boot and run guests.

Why three nodes and not two? I started with two plus a QDevice on the Synology as tie-breaker. It works, but it’s one more moving part. With three nodes you get three votes out of the box, one node can die, and the cluster keeps going. Simpler is better.

One thing to know up front: the CPU has no AVX. If a guest needs it, it won’t run on these boards, no matter what CPU type you pick.

Installing Proxmox VE 9 on eMMC

Here’s the first surprise. The Proxmox installer doesn’t do eMMC. Pick /dev/mmcblk0 as target and it bails out at the very end:

Unable to get device for partition 1 on device /dev/mmcblk0

Proxmox staff will tell you it’s not supported. The installer just doesn’t know how eMMC partitions are named (mmcblk0p1 instead of sda1). iBug figured out the fix, and it’s two lines:

  1. Boot the ISO and choose Install Proxmox VE (Debug mode).

  2. At the first shell, type exit. Too early to do anything.

  3. At the second shell, open the installer’s disk module:

    nano /usr/share/perl5/Proxmox/Sys/Block.pm

  4. Search for unable to get device and add the mmcblk branch right before the final else:

    } elsif ($dev =~ m|^/dev/nvme\d+n\d+$|) {
    return "${dev}p$partnum";
} elsif ($dev =~ m|^/dev/mmcblk\d+$|) {
    return "${dev}p$partnum";
} else {
    die "unable to get device for partition $partnum on device $dev\n";
}

  5. Save, exit, and the normal graphical installer starts.

The patch only lives in the installer’s RAM, so you redo it on every node. Three times, coffee in between.

My installer choices:

  • Target: /dev/mmcblk0. Not mmcblk0boot0 or boot1, those are tiny hardware boot areas.
  • Filesystem: ext4. ZFS likes RAM, and 16 GB is not a lot.
  • swapsize = 0, because swap eats eMMC for breakfast.
  • maxvz = 0, because there’s no point in local VM storage when everything lives on NFS.
  • Fixed IPs 192.168.1.21, .22 and .23 for zb1, zb2 and zb3, outside my DHCP pool.

After the install I switched to the free no-subscription repo. On Proxmox 9 the GUI does this best: Updates → Repositories, disable the enterprise ones, add No-Subscription. Then:

apt update && apt full-upgrade -y

I also put all three nodes in /etc/hosts on every node, so the cluster doesn’t depend on DNS. And check the clock: chronyc sources -v should show one line starting with ^*. Corosync really doesn’t like nodes that disagree about the time.

Keeping the eMMC alive

Proxmox writes to its OS disk all day: logs, the cluster database, graph data. On an SSD you don’t care. On eMMC, that’s what kills the board. So before building the cluster, I cut the avoidable writes on every node.

Keep the journal in RAM, in /etc/systemd/journald.conf.d/emmc.conf:

[Journal]
Storage=volatile
RuntimeMaxUse=64M

The trade-off: logs are gone after a reboot. I can live with that.

Add noatime to the root line in /etc/fstab, so reading a file doesn’t count as a write:

/dev/pve/root / ext4 errors=remount-ro,noatime 0 1

Double-check there’s no swap (swapon --show should print nothing), and don’t store ISOs or backups locally. That all goes to the Synology.

The nice thing about eMMC: it reports its own wear. Worth a look once a month:

cat /sys/block/mmcblk0/device/life_time      # 0x01 = 0-10% of rated life used
cat /sys/block/mmcblk0/device/pre_eol_info   # 0x01 = normal

And if one does die? Nothing on it matters. Remove the node with pvecm delnode, reinstall, join again. The guests are safe on NFS.

Building the cluster

This part is almost suspiciously easy. On the first node:

pvecm create homelab

Then on zb2, and once that one shows green, on zb3:

pvecm add 192.168.1.21

Join each node while it’s still empty: a node with guests on it can’t join. Then check:

pvecm status

Three expected votes, quorum of two. Lose any one node and the other two carry on. No QDevice, no tie-breaker, no extra container.

Hooking up the Synology

Before a new node touched the old share, I took a Btrfs snapshot of it (Snapshot Replication). If a recovery step goes wrong, that’s a one-click undo. Do this. Seriously.

Then the NFS rules: one per node IP instead of the whole LAN, read/write, Squash set to No mapping (Proxmox wants to write as root), and Allow connections from non-privileged ports. I also removed the rule for the dead server.

Because storage config is cluster-wide, you only add it once. The trick is to use the same storage ID the old server used, so every path in an old config still points to the right place:

pvesm add nfs synology --server 192.168.1.10 --export /volume1/proxmox \
  --content images,rootdir,backup,iso,vztmpl --options vers=4.1

All three nodes now see the old share under /mnt/pve/synology/, with the guest disks still sitting in images/<vmid>/.

Recovering the guests

Three guests to bring back: Uptime Kuma, Hermes and Sonarr. The disks on the share hold everything right up to the crash. The only thing missing is the configs: those lived in /etc/pve on the dead box. So the job is putting a config back next to each disk. No copying, no importing.

First, figure out what you’re holding:

ls -l /mnt/pve/synology/images/*/
file vm-101-disk-0.raw     # partition table = VM, bare ext4 = container
fdisk -l vm-101-disk-0.raw # EFI System partition = UEFI boot

If you have a vzdump backup, steal the config from it. It’s right there in the archive:

# container
tar --zstd -xOf vzdump-lxc-102-<date>.tar.zst ./etc/vzdump/pct.conf > /root/102.conf
# VM: decompress on the NAS, not on the eMMC
zstd -d -o /mnt/pve/synology/tmp.vma vzdump-qemu-101-<date>.vma.zst
vma config /mnt/pve/synology/tmp.vma > /root/101.conf && rm /mnt/pve/synology/tmp.vma

Drop it in /etc/pve/qemu-server/ or /etc/pve/lxc/, and qm rescan picks up any disk the config doesn’t know about yet.

If you only have the disk, which was my situation, you rebuild the guest around it. For a VM, create an empty shell that points straight at the existing disk. I reused the old MAC from my DHCP lease list, so the guest came back on its old IP:

qm create 101 --name hermes --memory 4096 --cores 2 --cpu host \
  --net0 virtio=<old-mac>,bridge=vmbr0 --scsihw virtio-scsi-single --ostype l26
qm set 101 --scsi0 synology:101/vm-101-disk-0.raw --boot order=scsi0

For a container, peek inside the disk first to see what you’re dealing with:

mount -o loop,ro vm-102-disk-0.raw /mnt/peek
cat /mnt/peek/etc/hostname
ls -ln /mnt/peek/etc/hostname   # owner 100000 = unprivileged, 0 = privileged
umount /mnt/peek

Then write /etc/pve/lxc/102.conf by hand: rootfs: synology:102/vm-102-disk-0.raw, net0 with the old MAC, memory, and unprivileged: 1 if the owner said so.

Before the first boot, check the disks. They were open when the box died:

qemu-img check vm-101-disk-0.qcow2   # qcow2 VMs only
pct fsck 102                          # containers

Then qm start / pct start, and watch the console. All three came back with their data from the day of the crash.

The biggest catch for me: the old server was dead, so I had no access to its setup. No VM configs, no container settings to copy. Every guest was a matter of rebuilding around the raw file. Annoying, but the data was all there, and that’s what counts.

HA and backups

With shared storage, live migration only copies RAM, so moving a guest between nodes takes seconds. And with three nodes, HA actually makes sense: under Datacenter → HA I added every guest, with a soft preference for a node so they’re spread out. Uptime Kuma prefers a different node than the rest, so it actually notices when one goes down.

One rule to keep in mind: when a node dies, the other two have to carry its guests. So keep the total RAM of your HA guests around 24 GB, not 48.

Backups now run nightly to the Synology:

pvesh create /cluster/backup --schedule "02:00" --storage synology --all 1 \
  --mode snapshot --compress zstd --prune-backups keep-daily=7,keep-weekly=4

And this time I also back up the configs. A tiny script in /etc/cron.daily/ tars /etc/pve, the network config and /etc/hosts to the NAS every night. That’s the stuff I was missing.

Then I tested it for real: pulled the power on one node. The cluster stayed quorate and its guests came back on the other nodes in about two to three minutes.

Lessons learned

  1. Back up your configs, not just your disks. The data survived; rebuilding every guest by hand is what cost me the evening.
  2. Snapshot before you recover. One Btrfs snapshot turns every mistake into an undo.
  3. Three nodes beat two plus a tie-breaker. Real quorum, no QDevice, room for HA.
  4. eMMC is fine if you’re nice to it. No swap, logs in RAM, and check the wear once in a while.
  5. Reuse the old storage ID. Old paths keep working, and recovery is putting configs back instead of moving data.
  6. Test a restore. A backup you never restored is a hope, not a backup.

Net effect: one dead box turned into three small, SSD-less nodes that boot from eMMC and run everything from the Synology. Losing a node now means a few minutes of HA restarts instead of a silent house.

Next up: the 10 inch rack and the 3D printed fronts that hold these boards.