The Frivolity of Software and Software Updates

I had a few moments when I bricked my system because of some settings and when I was playing around with various things.
At one point decided to play a forensic analysis hero, Without out knowing it I broke my whole partition table due to a fat finger /sda instead of file:///sdd.[/sdd.]
It was at such a moment in life that you realize you’re just one week away before the monthly disaster backup is due.
Command Line Fu to the rescue, i rebuilt it based on metadata from each disk plus a little bit of luck.

Remember kids, forensic analysis should be done in virtual machines

Then a few months later I dediced to do a upgrade to version 9.
Everything worked fine, except that my setup had 3 disks. 1 SSD which had the / partition and another fast partition. Then 2 disks where in RAID 0 that used MDADM, I had also manually setup encryption for my home folder.
After the upgrade i got a rare message "You need to wait for 1 min and 30 seconds.." After the waiting time was done I could hit CTRL+D to go in EMERGENCY MODE.
Oh crap! What happened?

Well, looking at the logs it seems that something related to /etc/fstab couldn’t boot to the /dev/md0 device.
But after I typed CTRL+D I recieved the login screen.
After looking into the logs and figuring out what went wrong I still to this day can’t understand WTF!?

Anyway, I fixed the 90 seconds wait with by editing /etc/fstab and adding x-systemd.device-timeout=2
The error still existed, but everything worked fine. Guess it’s a minor bug.

Fast forward 1 year later 2018.

I was working on an Elixir project. I updated Fossil from 1.37 to 2.5 by downloading 1 self contained executable form the Fossil website.
Everything seemed to work fine, except the clone and push commands gave a Segmentation Fault
I started digging in and found out that some library on my debian crashed it.
I tried it on another debian 9 and the binary worked flawlessly. Great. it’s my debian that has issues.

I decided it was time to check for updates. I installed them all. Rebooted. Same error.
CTRL+D didn’t seem to react. I could however access another TTy. But couldn’t start the Xorg server.
Looking in the logs I found some weird errors. I decided hey, let’s do a dist upgrade.
Reboot.. This time whenever I hit CTRL+D or tried to go to a TTY it began flickering my screen.

Great God NO! Anyway, I had multiple errors and tried to fix multiple things, but after each step I figured got a new error.

Reinstalling Linux is a breeze. I’ve always had my / and /home partitions kept separate and I encourage you to do the same.
I usually reinstall Linux only of something goes wrong with my fiddling or I decide to use another distro as my main one BEFORE i’ve tested it out in a virtual machine.
I decided to give Linux Mint another try.

During the installation process I saw I couldn’t select my RAID disks.
I then began to look into gparted. It was still there.
Then I decided it was time to do it with mdadm.
But mdadm wasn’t installed. Another WTF! Why wouldn’t they ship it?

The mdadm --examine --scan looks at all disks and scans for arrays.. If it finds something it will output it. So whenever you brick your system, your data is still there, don’t start formatting everything!

Great, it still exists, before I started the installation I decided to backup everything.
I had some backup external HDD’s and considering the fact that i had RAID 0 anything that could happen I still had my data there.
On the safe side, let me backup everything again.

I went through a little hassle in remounting everything with ecryptfs.
Then I hit anothe rroadblock when trying to backup, the sheer size of data and the lovely NTFS.
This was the last time I’ll ever use NTFS on external backup HDD’s for "interoperability" with windows machines.
Ext4 all the way baby (or something else Linux can work with easily)

Without a proper recent backup (2 months of changes) I set out to reinstall it and I said If the installation bricks my encrypted mdadm raid then it’s a sign I should become a lumberjack, painter, priest or anything else except a Programmer/Linux enthousiast.

I reinstalled Linux Mint, tried to login.
POOF Linux Mint was endlessly starting up. It’s a sign, I’ll have to let my beard grow.
Then i thought the distro probably didn’t have my settings nor mdadm. Couldn’t login to a TTy either since there was no home directory.
Eventually got a recovery shell, yep, no mdadm.

Restarted with the Linux Mint USB again.
Viewed the /etc/fstab and it had everything there. Uncommented the /home directory form fstab.

Rebooted. Created /home/myuser as root, logged in.
And then I installed mdadm and generated the mdadm config again
sudo su -c "mdadm --examine --scan > /etc/mdadm/mdadm.conf"

It was time to find out if it will work or fail.
After the reboot, I logged in with my user and voila, ecryptfs worked automatically.
I ignored the fact that cinnamon crashed and went to fallback mode endlessly, probably due to my old cinnamon settings.
This was later fixed with
dconf reset -f /org/cinnamon/

Yes, my data is still there! Thank you Super Robust Linux features!
Now for the fun part. I need to backup everything again. This means copying things from 3 exernal hdd’s to my disk and vice versa.. Format each one of them to Ext4.