Remember back last year when I added a USB stick to my Pi to increase reliability <link>? Serious mistake, it died. Not only did it die, I reacted wrongly to every single thing and messed myself up and had to rebuild the Pi from almost the ground up.
The story goes like this, but I don't have nice pretty examples of the output of the various efforts to fix and recover from this. I wasn't thinking about blogging it, I was trying to get my Pi house controller back up.
I tried to open the garage doors with a remote command it didn't work. I walked out to the garage, pushed the button and the door opened. OK, the door opener was working, so I took a quick look at the garage controller, it looked fine, all the little blinking lights were working. I took a look at the various house control logs and they were weird. I was getting a message that said that the USB stick was a read-only file system. It's important to note that all the devices were still controlling things perfectly. Over the years I made sure that their default operation didn't depend on the house controller, The house controller is a convenience and logging device, nothing totally depends on it. This is when I made my most serious mistake, I tried rebooting the device.
Naturally, it wouldn't boot. I took the whole mess of wires and components to the kitchen table and started trying to understand what was going on. I couldn't read the file system on a laptop since it was a Linux file system, so I went looking for something that could help, but instead I found a number of tools that wouldn't work with a USB stick. I didn't want to spend hours trying things out, so I just edited my boot SD card back the way it was before I added the USB stick. The little Pi booted first try with software a year old. Of course this messed up my data logging because things have changed a lot in the last months.
I searched the web for solutions to the read-only-USB-stick and found lots of people that had also encountered the problem, but nothing that would actually solve it. Time to look at my backups...I didn't have any! Yep, I relied on the stick to keep working and my source repositories on GitHub as my backup plan.
Fine, I'll just rebuild it. The problem was that I no longer had my source repository on the Pi, it was gone with the USB stick. If I had been coherent while the Pi was running read only, I could have just copied it off and gotten on with life. The source was there in GitHub, all I had to do was get it. So, I wiped my really old source directory from the SD card and downloaded a zip file from GitHub and installed it.
Now, I had the source, but the various libraries and packages that I've accumulated over the months weren't there. It became a chore of try something, get the missing stuff, try it again - repeat for each and every process I'd put together and modified over time.
My notes here on the blog were extremely valuable. I kept going back to see what I did in a particular instance which gave me clues to what I needed to add and modify. As I stepped through things, I even noticed a couple of bugs that hadn't given me problems before. Basically, I recreated my house controller from notes on this blog. But, my configuration file 'houserc' was a total loss and I built it up from scratch one line at a time, rediscovering the various entries I needed as processes printed error messages. I can't put that one on GitHub, it has the keys and passwords used for all the cloud servers. Eventually though, the pieces came together and everything started working again.
I managed to recreate my source repository so I could get back in sync with GitHub and update the minor changes I made to the code. That turned out to relatively easy. My objections to GitHub from the past seem really silly now. You folk should consider using it.
The USB stick is a total loss. I can't format it because it's read only, and I can't read it on a laptop because I can't format it back to DOS. Guess I'll take it apart and see what's in there.
So, did I learn anything? I learned not to trust USB sticks, they just aren't meant to be the primary device on a system. How about making backups more often? Well, I'm backed up now, but when life starts interfering, I'm sure I'll get lax about it. Automate the backup process? There's a possibility, but it costs money in the form of somewhere to back it up to, and time rotating the backups so they stay relevant. There's no really good solution, just a bunch of compromises.
I guess I'll start saving for a network appliance, but I still think the 3D printer is more important.
The story goes like this, but I don't have nice pretty examples of the output of the various efforts to fix and recover from this. I wasn't thinking about blogging it, I was trying to get my Pi house controller back up.
I tried to open the garage doors with a remote command it didn't work. I walked out to the garage, pushed the button and the door opened. OK, the door opener was working, so I took a quick look at the garage controller, it looked fine, all the little blinking lights were working. I took a look at the various house control logs and they were weird. I was getting a message that said that the USB stick was a read-only file system. It's important to note that all the devices were still controlling things perfectly. Over the years I made sure that their default operation didn't depend on the house controller, The house controller is a convenience and logging device, nothing totally depends on it. This is when I made my most serious mistake, I tried rebooting the device.
Naturally, it wouldn't boot. I took the whole mess of wires and components to the kitchen table and started trying to understand what was going on. I couldn't read the file system on a laptop since it was a Linux file system, so I went looking for something that could help, but instead I found a number of tools that wouldn't work with a USB stick. I didn't want to spend hours trying things out, so I just edited my boot SD card back the way it was before I added the USB stick. The little Pi booted first try with software a year old. Of course this messed up my data logging because things have changed a lot in the last months.
I searched the web for solutions to the read-only-USB-stick and found lots of people that had also encountered the problem, but nothing that would actually solve it. Time to look at my backups...I didn't have any! Yep, I relied on the stick to keep working and my source repositories on GitHub as my backup plan.
Fine, I'll just rebuild it. The problem was that I no longer had my source repository on the Pi, it was gone with the USB stick. If I had been coherent while the Pi was running read only, I could have just copied it off and gotten on with life. The source was there in GitHub, all I had to do was get it. So, I wiped my really old source directory from the SD card and downloaded a zip file from GitHub and installed it.
Now, I had the source, but the various libraries and packages that I've accumulated over the months weren't there. It became a chore of try something, get the missing stuff, try it again - repeat for each and every process I'd put together and modified over time.
My notes here on the blog were extremely valuable. I kept going back to see what I did in a particular instance which gave me clues to what I needed to add and modify. As I stepped through things, I even noticed a couple of bugs that hadn't given me problems before. Basically, I recreated my house controller from notes on this blog. But, my configuration file 'houserc' was a total loss and I built it up from scratch one line at a time, rediscovering the various entries I needed as processes printed error messages. I can't put that one on GitHub, it has the keys and passwords used for all the cloud servers. Eventually though, the pieces came together and everything started working again.
I managed to recreate my source repository so I could get back in sync with GitHub and update the minor changes I made to the code. That turned out to relatively easy. My objections to GitHub from the past seem really silly now. You folk should consider using it.
The USB stick is a total loss. I can't format it because it's read only, and I can't read it on a laptop because I can't format it back to DOS. Guess I'll take it apart and see what's in there.
So, did I learn anything? I learned not to trust USB sticks, they just aren't meant to be the primary device on a system. How about making backups more often? Well, I'm backed up now, but when life starts interfering, I'm sure I'll get lax about it. Automate the backup process? There's a possibility, but it costs money in the form of somewhere to back it up to, and time rotating the backups so they stay relevant. There's no really good solution, just a bunch of compromises.
I guess I'll start saving for a network appliance, but I still think the 3D printer is more important.