What I changed after my home server's disk filled up
The little server in the hallway stopped answering on a Tuesday. No dramatic failure: the disk was at 100 percent and everything that wanted to write a temp file had quietly given up.
The cause took ten minutes to find and was entirely mine. In spring I had switched one service to debug logging to chase a bug, fixed the bug, and never switched it back. Six months of very detailed logs later, here we were.
du -xh / --max-depth=2 | sort -h | tail -n 15What I changed
- Log rotation with a size cap, not only a daily schedule.
- A weekly email with disk use, sent whether or not anything is wrong.
- A note in my work log whenever I raise a log level, with a date to lower it again.
- Twenty percent of the disk left unallocated, so there is room to recover.
The weekly email is the one I would recommend to anyone. An alert that only fires on trouble is easy to break without noticing. A boring message every Sunday tells me the reporting itself still works.