Weekly Retro 2024-W26

I bought a 3D printer, I reformatted my Framework to add disk encryption, and I discovered some drive failures that I had no warnings for and try to save the data.

3D Printing

I’ve wanted a 3D printer for as long as I can remember. I remember in the late 2000s, my friend and I were in a CADD class together, using Solidworks to design parts. I was… okay, at it. But my friend was very good at it, so much so that he went on to become an engineer with a focus on human-centered mechanical engineering (I think).

During our time in that class together, he also designed a lot of custom twisty puzzles (which is apparently the generic name for Rubik’s cube type puzzles), and was able to make several sales of his designs to other enthusiasts, who would typically have them printed from Shapeways. 3D printers hadn’t really saturated the mainstream at this point, but I remember some of the early prototypes he got back from Shapeways being distinctly 3D printed in that coarse granularity that came with the early technology. It was so cool to have an idea, design it on the computer, and then have it delivered to your doorstep.

Fast forward several years and I had moved to California and several folks I knew at work were into the Maker sphere, attending hackathons, spending all their spare time at Hacker Lab in Sacramento, etc. It was very clear that 3D printers were getting significantly better, and significantly more affordable. I even went to a Hak5 Hack Across America event in 2013 at the Printrbot headquarters, where I got to meet Darren (who, little did I know at the time, would later become a friend and mentor). But I didn’t have the disposable funds, nor the mechanical inclination to deal with putting together my own 3D printers and fighting with them to get prints to work.

Well, now I’ve solved the first part of that, at least – I have the disposable funds. Still no mechanical inclination, but we’ll get there. 3D printers have also come a long way in the past 10 years, so I feel like I’ll spend less time fighting the printer and more time just printing. The 3D printing community has also come a long way, so I’m not worried about being able to get help should I struggle.

All this to say, I bought a Bamboo A1 mini combo (which is currently on sale), and it’s sitting in the box on the floor behind me as I write this. I plan to get it setup this week and hopefully have some pictures to share next week (or perhaps in a standalone post about my experience, not sure yet). I picked this printer because it seems fairly beginner friendly, the price point is affordable for a random hobby I’ll probably neglect (hello ADHD), and it has a small enough footprint that it should fit on the shelf behind my desk.

Declarative Disk Encryption w/ NixOS

Those of you following my RSS feed have probably seen my Framework laptop posts, in which I have been documenting my experience getting NixOS setup on it. I’ve been in pursuit of my ideal desktop operating environment – one I can easily repair, one I can make configuration changes on without (as much) fear of bricking it, and one I can wipe and get back to an ideal state as easy as possible.

I’m not going to dive too deep into the whole series of posts, but you’re welcome to check out the #nix tag to see them all. But my latest update is that I now have my disk partitions and filesystems declaratively managed with disko, using a lukscrypt volume with btrfs volumes. My end goal is to subscribe to the Erase Your Darlings philosophy, using something like Impermanence.

Accidentally Discovering Disk Failures

I’ve been a bit of a data hoarder for as long as I’ve had a computer, I think. I love having data on my device. It all began with my first brand new desktop computer. It had a 500GB hard drive in it, and no real room for expansion. After working through the summer for minimum wage, I had managed to buy a new, bigger case for it, as well as a new 1TB hard drive. This was around summer of 2011. (Note: I had several computers before this, but most were scavenged from hospital or school dumpsters, or bought secondhand).

This was my only computer, so it was also my gaming computer, and the computer I did my college homework on. It ran Windows. Over the years, I would buy another hard drive so I could continue to expand my gold pile (In this analogy, I am like a dragon, except instead of hoarding gold coins and treasures, I am hoarding installers of software that stopped working 10 years ago).

Back in 2019, I think, I replaced that computer with a new one meant purely for gaming, and the old one became purely used for running services and media. But at this point, it had 6 4TB drives in it (in pairs of 2 mirrored NTFS volumes, so I was throwing away a lot of space). It was still running windows, and I didn’t have the free hard drive space to move the data around and reinstall linux on it and use linux filesystems. So I just left it running windows. It ran qbittorrent to download linux ISOs¹Plex to serve those linux ISOs¹ to my friends, sonarr and radarr to automatically find new linux ISOs¹and occasionally it was also a Minecraft or Valheim server.

A couple years ago, I bought a Synology NAS, which currently has 4 18TB drives in it, in a RAID 5 array. Just one of these drives was more than enough to migrate the whole collection of data I had on the old machine. The old machine’s drives became purely for seeding linux ISOs¹. Some 1500 of them, spanning 8TB or so, and all the long term storage was moved to the NAS.

But I kept running into issues where I’d fill up a volume and then future linux ISOs¹ wouldn’t download correctly. So I finally had enough of it this week and decided I would move files around, break the mirrored volumes, and get into a parity configuration. When I went into the Windows Disk Management tool, I discovered that one of the drives in one array was entirely offline, and another was in an error state.

Purely by happenstance, I managed to find this issue. I had no monitoring in place, got no notifications, saw no warnings, nothing. So I started to move quickly to take the remaining drives that were in good condition, and put them into a Windows Storage Space drive pool. I know what you’re thinking, why am I doubling down on Windows at this point, just hear me out.

It turns out Windows Storage Spaces is the only way to get a parity disk configuration – simple disk management only supports striped and mirrored volumes. But that’s okay, I can use the storage space, add three of the 4TB drives (this required breaking the mirrored volumes of all my remaining healthy drives, unfortunately), and start to copy data over from the standalone drives. At time of writing, I’ve found a couple corrupted files, but nothing too major – I still have something like 3TB of data moving off of one drive, then 4TB more from another drive to move, so we’ll see.

I don’t expect this solution to last forever, and at a certain point, I have to put a cap to how many linux ISOs¹ I’m going to seed at a time. But today is not that day. I will run these drives into the ground.

I’m not replacing the whole thing with linux right now, because my longer term plan is to replace the whole system. The CPU in this system is a i7-4790K, which I think just crossed it’s 10 year birthday. The SSD boot drive is a drive that I was given when I worked in the SSD division at Intel – it had something like 800TB written to it when I got it, and I cleared the SMART data before using it. Since then, it’s crossed well over 100TB written to it, over 90,000 hours powered on. It says it’s still running fine, which I don’t doubt, but it’s also a SATA SSD, and I want that new NVMe hotness². It’s time to retire this system and replace it in it’s entirety. I’m just trying to save the data til I make that happen.

Fun fact: This drive officially launched in Q1’14 – I received this post-reliability test drive around that time. It has averaged 330.47 hours between reboots since then. If we assumed that this drive was released on January 1, 2014 (which it wasn’t), that would be 91,992 hours ago since time of writing. This SSD has been powered on for 98.79% of it’s possible lifetime. Probably more than that, since I received the drive sometime in February I think – Feb 1 was 91,248 hours ago, which would put the drive’s uptime at 99.59%.

Not bad for surviving 2 cross-country moves, 4 intra-California moves, and 9 years of PG&E power service.

What I’m Reading

Interesting Links

• Authenticated Boot and Disk Encryption on Linux – Approaching 3 years old, this post helped me to better understand how secureboot and encrypted disks work. It’s a good dive into the topic.
• What can we remove? – Another concise essay from Steph Ango (the CEO of Obsidian) about the importance of removing things and fighting against our ever-present urge to add, rathe than subtract.
• Verizon’s New Logo – While this article talks about some interesting elements of brand design, the new Verizon logo looks too much like Valvoline to me, which probably isn’t the association it was trying to make
• Using a TPM for SSH authentication – While working on secureboot for my NixOS/Framework project, I found a lot of interesting posts about manual key generation and whatnot. This one uses TPM-backed keys for SSH authentication, which is pretty cool.
• I will fucking piledrive you if you mention AI again – A wonderful rant about the hype cycle around AI. “I started working as a data scientist in 2019, and by 2021 I had realized that while the field was largeit was also _large_ly fraudulent.”
• Why I Attack – A great explanation of why offensive security is valuable and not, as some might have you believe, anti-social behavior.
• Keyoxide – I used to love Keybase. But then Keybase became a chat app. And then it became a Git server. I don’t even know what it is these days. But it’s not just the “Cryptographically verifiable social identity” that I liked it for. I have wanted a way to do the same concept, signing proofs of my identity for each of my social networks, websites, whateevr, but in a self-hosted / decentralized / federated sort of way. Keyoxide seems to have similar goals, though I’ve not yet tried it out. Also do not tell me to just use a PGP keyserver. I will find you.
• Local Energy Monitoring using the Emporia Vue 2 – I was talking with coworkers about home energy monitoring this week, and remembered that my friend wrote up his experience setting up home energy monitoring with Emporia Vue 2, but keeping the data local (which I value tremendously). If you’re interested, I recommend checking it out. One day I’ll figure out why our apartment uses 2.5x more than “similar homes”. And it better not be my server rack in the closet.

Upcoming Projects

• BSides Las Vegas Talk – Accepted! – I will be presenting “Free Your Mind: Battling Our Biases” at BSides Las Vegas 2024. This will be my first return to a public stage in like 6 years, and my first time speaking in Vegas. Stay tuned.
• Defcon 32 Call for Artists – Submitted, not selected. I won’t be rocking the artist badge this year, unfortunately. You can still find me at the Hak5 booth at Defcon this year!
• Defcon 32 Call For Soundtrack – I've submitted my new song “Oh Dade”, produced by Mikal kHill. If it’s accepted, it will debut on the Defcon soundtrack. If it’s not accepted, I will release it the same day I find out it’s not accepted. (Due: N/A – Done)
• PyBay 2024 Talk – Submitted – I have wanted to get out of the security conference space and talk about security-related things at other types of conferences for a while, and I had an idea for a talk that I think fits perfectly with PyBay. Bonus, PyBay is local, so I don’t have to travel. (Due: N/A – Done)