Old but still great: http://scholar.harvard.edu/files/mickens/files/thenightwatch.pdf

However I take issue with one quote: "Indeed, I would do that if I hadn’t broken every component that a logging system needs to log data. I have a network file system, and I have broken the network, and I have broken the file system, and my machines crash when I make eye contact with them. I HAVE NO TOOLS BECAUSE I’VE DESTROYED MY TOOLS WITH MY TOOLS." There's a person that really needs printk_persist.