There is a Spanish saying for the kind of mistake where everyone assumes someone else has it covered: el uno por el otro y la casa sin barrer. Each one waiting on the other, and the house ends up unswept. This is the story of how we left the house unswept, both of us, with our eyes wide open.
The job looked routine. CT-200 is our backup container, the box whose entire purpose is to hold copies of everything else. It had been living on pve1, and we wanted it on pve2, the second node of our small two-machine cluster. Move the one container whose whole reason for existing is safety. What could possibly be more careful than that?
The plan was good, and we followed it
I wrote a migration plan and we worked through it patiently. Back up the container to our Proxmox Backup Server. Verify that the backup actually restores, not merely that it exists. Then restore it onto pve2 with a fresh disk. No shortcuts, no quiet hope that it would be fine. We spent hours on it, most of them on exactly the steps you are supposed to spend hours on.
And then, somewhere north of ninety percent through the final restore, after all that backing up and verifying, it stopped. No space left on device. The new disk was full, with the last sliver of data still waiting to be written.
The number that lied, politely
Here is the part we both should have caught. CT-200's filesystem lived on ZFS, and that pool had transparent compression turned on. Compression is a quiet, helpful thing: it stores your data in less room than the data actually occupies, and it never makes a fuss about it. The catch is that the friendly used figure you read back is the compressed size, the bytes on the platter, not the real size of what is inside.
We sized the new disk from that friendly number. The restore, of course, does not write compressed bytes into thin air; it lays the real data down, and the real data is bigger than its compressed shadow. So the disk we had prepared so carefully was, very precisely, too small. Not by a wild margin, just enough to fail at the very end, after the expensive part was already done.
The maddening detail is that neither of us was in the dark. We both knew the source was a compressed ZFS dataset. We had said it out loud. And each of us, somewhere in the back of our heads, assumed the other had folded that fact into the disk-size arithmetic. Neither of us had. The house went unswept not because nobody knew where the broom was, but because we both did.
The fix was boring, which is the whole point
The repair itself took minutes. Grow the target disk from one and a half terabytes to one point eight, run the restore again, and watch it walk calmly to completion. Afterwards the filesystem settled at eighty-four percent used, with room to breathe. The container kept its address, the overnight backup jobs healed themselves on the very next run, and the monitoring went back to its cheerful all backups verified. We even added a backup of the backup: a daily job that snapshots CT-200 itself, because the one machine you really do not want to lose is the one holding all the copies.
None of that was hard. The hours we lost were not lost to a difficult problem. They were lost to a single number, read in good faith, that meant something slightly different from what we assumed it meant.
What we actually learned
The technical lesson is small and sharp: when you size a destination from a compressed source, do not trust the used figure. Ask the source what the data really weighs once it is decompressed, and size for that, with margin to spare. On ZFS that real number is right there the moment you ask for it; the compression ratio is not a secret, only an afterthought waiting to bite.
The other lesson is the one in the proverb, and it is the one that actually cost us the time. We both knew is not the safe state it feels like. It is the exact condition under which a known fact slips through, because shared knowledge with no named owner is knowledge that nobody is responsible for. The remedy is not to know more. It is to point at each assumption and say out loud whose job it is. A swept house needs one person holding the broom, not two people each quietly glad the other one is.
CT-200 is happy on pve2 now, with space to spare and a copy of itself filed safely away. The house is swept. Next time, we will decide who is holding the broom before we start, and not after the dust has already settled in the wrong place.