Recently, one of my backup servers started complaining - during its latest reboot, I ended up with an emergency boot prompt loop!
The server was a Fedora (but the workaround should also apply to almost every CentOS, Ubuntu and maybe to other distros too!).
Why no regular boot up?
Here's a 'shot of what I got:
I'll paste the error for the search engines:
*** An error occurred during the file system check.
*** Dropping you to a shell; the system will reboot
*** when you leave the shell.
Give root password for maintenance
(or type control-D to continue):
By interpreting the above error, it seems there's an issue while checking the filesystems.
The md arrays looks alright ("
clean"), although there's a problem with /dev/sdc1 ("
No such file or directory while opening /dev/sdc1").
IMHO, it's a good practice to keep an external HDD plugged (and maybe also rsync'ed).
As I discovered with my work-mates, that was the case, 'though (for reasons unknown), it seems the external HDD had been removed.
I also discovered the external HDD automount was (manually) hardcoded in fstab.
In conclusion, to fix the issue, I had to patch fstab.
'Thoug I couldn't edit fstab since the "slash" was mounted Read-Only.
After searching through endless posts and forums (OK, I may have used the wrong terms...), I just found what I needed:
mount -n -o remount,rw /
The above command mounted the ROOT (aka slash), as a READ-WRITE filesystem.
After mounting slash in RW-mode, I was able to manually patch fstab, namely, by commenting out the offending drive.
The "-n" parameter prevents the mount command to populate the mtab lines (while mounting devices).
The "-o" parameter is a switch to specify additional, comma-separated options (similar to the options used inside fstab - ie. 'defaults', 'sync', 'noatime', etc.).
The "remount" option tries to remount the fs as-is (since it is already mounted RO).
Finally, the "rw" option enables read & write mode for the desired filesystem.
Once done, I rebooted the backup box and ta-daaa! (Windows 3.1 welcome sound): onsite backup server back online.