[Solved] Fedora/CentOS emergency prompt after boot.

October 11th, 2016 by Andrea Matesi

Recently, one of my backup servers started complaining - during its latest reboot, I ended up with an emergency boot prompt loop!

The server was a Fedora (but the workaround should also apply to almost every CentOS, Ubuntu and maybe to other distros too!).

Why no regular boot up?

Here's a 'shot of what I got:

I'll paste the error for the search engines:

*** An error occurred during the file system check.
*** Dropping you to a shell; the system will reboot
*** when you leave the shell.
Give root password for maintenance
(or type control-D to continue):

By interpreting the above error, it seems there's an issue while checking the filesystems.

The md arrays looks alright ("clean"), although there's a problem with /dev/sdc1 ("No such file or directory while opening /dev/sdc1").

IMHO, it's a good practice to keep an external HDD plugged (and maybe also rsync'ed).

As I discovered with my work-mates, that was the case, 'though (for reasons unknown), it seems the external HDD had been removed.

I also discovered the external HDD automount was (manually) hardcoded in fstab.

In conclusion, to fix the issue, I had to patch fstab.

'Thoug I couldn't edit fstab since the "slash" was mounted Read-Only.

After searching through endless posts and forums (OK, I may have used the wrong terms...), I just found what I needed:

mount -n -o remount,rw /

The above command mounted the ROOT (aka slash), as a READ-WRITE filesystem.

After mounting slash in RW-mode, I was able to manually patch fstab, namely, by commenting out the offending drive.

MOUNT-command details.

The "-n" parameter prevents the mount command to populate the mtab lines (while mounting devices).

The "-o" parameter is a switch to specify additional, comma-separated options (similar to the options used inside fstab - ie. 'defaults', 'sync', 'noatime', etc.).

The "remount" option tries to remount the fs as-is (since it is already mounted RO).

Finally, the "rw" option enables read & write mode for the desired filesystem.

Once done, I rebooted the backup box and ta-daaa! (Windows 3.1 welcome sound): onsite backup server back online.

Posted in LINUX, System Administration | No Comments »

11 exim cpanel golden checks for quick mail troubleshooting.

March 19th, 2016 by Andrea Matesi


The following is just some random advice derived from my experience on dealing with email-related issues.

More specifically, here I'll be referring to exim (a very popular mail daemon), Cpanel/WHM & CentOS.

  1. Check if the user & password combination is correct.
  2. Check if the SMTP Authentication is enabled.
  3. Check if the User's mailbox is full.
  4. Perform an nslookup of the domain thru a public DNS Server.
  5. Perform an nslookup of the MX RRs thru a public DNS Server.
  6. Verify that the SPF RR is applied to the domain.
  7. Telnet (or putty with the telnet option enabled) to the destination server address to see if it answers.
  8. Check if the domain name is present inside /etc/localdomains.
  9. Check the logs with exigrep /var/log/exim_mainlog.
  10. Check with vi /etc/userdomains
    Look for some blank spaces or broken lines near the domain that is having incorrect authentication data issue error 535
  11. Check whether the folder "etc" within /home/"cpanel-username"/etc/ is owned by cpanel-username:mail
    If it is not then change it with:
    chown username:mail /home/username/etc/ -R

Hope you might find those useful & feel free to share your own special/unique checks on the comments section.


Posted in LINUX, System Administration, Tips and Tricks. | No Comments »

4 useful lsof commands explained

July 12th, 2015 by Andrea Matesi

This short post introduces you 4 useful lsof commands by examples.

Due to their usefulness, I'd like to "remember to use" those commands more often.


lsof -u "username".

Example running lsof -u root

lsof-u root

The command above will show you all "root's user" open files.


lsof -a -p "PID".

lsof -a -p 1

lsof -a -p 1

-a is a simple AND operator. Used this way is the equivalent of "lsof -p 1".

-p 1 limits the output to PID 1 (usually that is the kernel...). You get PIDs by running the ps command.

When you specify more than 1 lsof -X -Y command switches (ie. "lsof -p 1 -u johndoe"), by default lsof will perform an OR operation (ie. EITHER "PID = 1" OR "User = johndoe").

IF you type, say, "lsof -p 1 -a -u johndoe", lsof will filter your output by "PID = 1" AND "User = johndoe".


lsof "/var/log/filename.log".

lsof /var/log/messages

lsof /var/log/messages

lsof with a file parameter will show you who & what daemon is using the file (ie. the "messages"-log file).

On the above screenshot, /var/log/messages is opened by root thru rsyslogd (which has a PID of 1078).


lsof -i :TCP|UDP-PortRange.

[root@host:~]#-> lsof -i :1-100
sshd     1216 root    3u  IPv4  11823      0t0  TCP *:ssh (LISTEN)
sshd     1216 root    4u  IPv6  11827      0t0  TCP *:ssh (LISTEN)
sendmail 1240 root    4u  IPv4  11922      0t0  TCP localhost:smtp (LISTEN)
sshd     1446 root    3r  IPv4  22798      0t0  TCP> (ESTABLISHED)

lsof -i :1-100

The above command (with a space-char after "-i"), queries your system about “what services are running on the first 100 ports”?

If you want to know only what TCP ports are in use, then type:

lsof -i tcp

That'll show you all the open TCP ports.

My short examples are only the tip of the iceberg of what lsof can do.

lsof is extremely useful and has an extensive (and sometimes arcane) list of options and switches - check for yourself at the lsof man page: http://linux.die.net/man/8/lsof

Posted in LINUX, System Administration | No Comments »

Fast-enable vncserver on CentOS.

January 3rd, 2015 by Andrea Matesi

...Assuming it is already installed (if not then "yum install vnc").

Launch the server by typing the following on a terminal:

vncserver :1

Then edit ~/.vnc/xstartup as follows:

# Uncomment the following two lines for the normal desktop:
exec /etc/X11/xinit/xinitrc
[ -x /etc/vnc/xstartup ] && exec /etc/vnc/xstartup
[ -r $HOME/.Xresources ] && xrdb $HOME/.Xresources
xsetroot -solid grey
vncconfig -iconic &
xterm -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" &
gnome-session &

And done!

Posted in LINUX, System Administration | No Comments »

4 most useful vmrun commands.

August 23rd, 2014 by Andrea Matesi

Did you pay the vTax?!

If you "vmware", then vmrun should be your screwdriver.

If you only "Hyper-V" then know that vmrun is like "Set-VM"-equivalent (in Powershell-speech).

Among other things, vmrun allows you to boot & reboot your VMs by skipping the screamingly fast (!) WEB-UI.

vmrun relies on webservices to send commands to the host but is invoked from the CLI.

For it to work, you need a client (say a linux distro of your liking), and a vSphere server with a couple of VMs.

In my examples below, I'll be referring to vSphere 4.1 Server ( managed from an Ubuntu box.


vmrun requirements.

To use vmrun, you'll need the VIX Standalone API Libraries.

You can Download the VIX Standalone API Libraries from the vmware website ('though you'll have to register first).

Register with vmware (it's free), then download and install the VIX standalone API libraries.

  • Follow the enclosed instructions on how to install the VIX API libs to your system.

I downloaded and installed the VIX Standalone API Libraries on my occasional Ubuntu client of choice.


vmrun -h https://vSphere-IPAddr:8333/sdk -u root -p pwd list.

This command will provide you a list of available VMs.

It relies on the list parameter.

Once you installed VIX, from your Linux client type:

[root@localhost ubudevsrv1]# vmrun -h -u root -p p@ssw0rd list


Total running VMs: 7
[standard] ubuntu_server_1/Ubuntu.vmx
[standard] ubuntu_server_2/Ubuntu.vmx
[standard] centos_server_1/centos-5.3-i386-server.vmx
[standard] ubuntu_server_3/Ubuntu 9.04 Server.vmx
[standard] ubuntu_server_4/Ubuntu.vmx
[standard] centos_server_2/Centos-5.5.vmx
[standard] centos_server_3/CentOS-5.2.vmx

You can interact with those VMs remotely.


vmrun -h https://vSphere-IPAddr:8333/sdk -u root -p pwd stop "[type] vm-name.vmx".

This second command will show you how to shutdown a VM.


[root@localhost ubudevsrv1]# vmrun -h -u root -p p@ssw0rd stop "[standard] centos_server_1/centos-5.3-i386-server.vmx"

The above will gracefully Shutdown (provided your VM does have the "VMWARE Tools" installed), the VM named "[standard] centos_server_1/centos-5.3-i386-server.vmx".

For the above command to work, please note you'll have to refer to the VM by enclosing any spaces and brackets within double quotes ("...").


vmrun -h https://vSphere-IPAddr:8333/sdk -u root -p pwd start "[type] vm-name.vmx".

This command will simply Boot your VM of choice.


[root@localhost ubudevsrv1]# vmrun -h -u root -p p@ssw0rd start "[standard] ubuntu_server_1/Ubuntu.vmx"

Will Boot the VM named [standard] ubuntu_server_1/Ubuntu.vmx.


vmrun -h https://vSphere-IPAddr:8333/sdk -u root -p pwd reset "[type] vm-name.vmx".

Finally, if you wish to Restart your VM, use the reset parameter.


[root@localhost ubudevsrv1]# vmrun -h -u root -p p@ssw0rd reset "[standard] centos_server_2/Centos-5.5.vmx"

The above command will Reboot the VM named "[standard] centos_server_2/Centos-5.5.vmx".

Posted in LINUX, System Administration, Virtualization | Comments Off on 4 most useful vmrun commands.

Samba share on Ubuntu Server Host.

December 29th, 2013 by Andrea Matesi

I experimented with the manual creation of a samba share on an instance of Ubuntu Server. Here's the commands I used...

sudo su
useradd amatesi
smbpasswd -a amatesi
vi /etc/samba/smbusers <- insert the following: amatesi = "network username"

smbpasswd -a amatesi
addgroup samba-users
adduser amatesi samba-users

chown -R root:samba-users /var/samba-share
chmod -R 771 /var/samba-share

/etc/samba/smb.conf content:

comment = samba-share directory
path = /var/samba-share
public = yes
writable = yes
valid users = amatesi
create mask = 0771
directory mask = 0771
force user = amatesi
force group = samba-users

/etc/init.d/samba restart

One synonym for guest access is "public = yes".

To access the shared folder from Windows, "Start" -> "Run...":


And replace 192.168.x.x with your actual Ubuntu Server IP address sharing the folder.

To access the shared folder from a Linux GUI, type the following in Konqueror or Nautilus:


To access the shared folder from a Linux terminal or console:

smbclient -L SERVER-NAME

If the command above doesn't work, try the following command:

smbclient -L SERVER-NAME -I 192.168.x.x

...where SERVER-NAME = netbios name from /etc/samba/smb.conf and 192.168.x.x is the Ubuntu server hosting the files.

To delete a network user:

sudo smbpasswd -x system_username

To Enable SWAT.

edit the /etc/xinetd.d/swat file and make it similar to the following:

# SWAT is the Samba Web Administration Tool.
service swat
socket_type = stream
protocol = tcp
port = 901
wait = no
user = root
server = /usr/sbin/swat
#    only_from =
log_on_failure += USERID
disable = no

/etc/init.d/xinitd restart

Then restart the xinitd daemon


Posted in LINUX, System Administration | Comments Off on Samba share on Ubuntu Server Host.

My Fav bash less commands.

September 13th, 2013 by Andrea Matesi

Just kidding...

Everybody knows less (it doesn't mean people are becoming more stupid!), if you don't, you don't know what you're missing.

Let me show you less:

Knowing less means also knowing less is better than more :D

less I-know-less.log

It's especially useful when used with the log files from /var/log.

Once you know less, you should also know less commands!

less commands to do more than more!

This is a list of my favourite less commands.

I evidenced the most useful one (in my experience).

command description
My favourite less commands.
<spacebar> Scroll forward 1 page.
CTRL+B Scroll backward 1 page.
Finds the "txt2srch" string inside your open file.
Next occurrence of searched string (ie.with a /).
Search backwards.
Go to beginning of open file.
Go to EOF.
v Opens current file with vi (but then you'll have to quit twice...).
Opens next file (ie. less file1 file2).
Opens previous file.
Quit less.















Posted in LINUX, System Administration | Comments Off on My Fav bash less commands.

nmap basic and fast scan.

August 3rd, 2013 by Andrea Matesi

There are way too many different nmap scanning options to talk about, but a typical first scan for this kind of test would be the w/t following command:

nmap -A -v -T 3

nmap basic and fast scan options

That'll do:

  • OS Detection.
  • Services Version Detection.
  • TCP SYN Connect, with a timing of 3.

What you’re looking for is open ports, and to see if the scan returns the name of the Operating System and/or the name and version number of the service(s) that are running on any open port(s). If you get nothing of the sort back, then you’re OK. This scan really only affects people who are running services on their computer, such as the SSH Daemon, sendmail, apache httpd, and so forth.

Other useful scans.

nmap -sU -sV -v

UDP+UDP Services and verbose.

nmap -sU -P0 -p161 -v -v IP

Port 161 UDP.

nmap -sU -P0 -p161 -sV -v -v

Port 161UDP + Service.

sudo nmap -vv -O -P0

Fast OS Check (fast).

nmap -vv -O -P0 -sSU

OS Check (+ slower).

Posted in LINUX, Tips and Tricks. | Comments Off on nmap basic and fast scan.

netstat -puntalo.

July 13th, 2013 by Andrea Matesi

Every netstat article usually starts with a description and some tiny code examples, but since this tool is all about OPTIONS, so this time I'll use some visual aids to do you a favor.

- netstat -nr shows the route without resolving any name (ie."link-local" would look like "").

- netstat -i shows interfaces statistics (ie. TX-OK, RX-OK, MTU, etc.).

- netstat -ta or -ua shows all TCP or UDP connections.

- netstat -lotanpu shows you all you need to know, including the PID/Program name: just enlarge your terminal to at least 124!

That's should keep you going for the most part.
Keep reading for more details.

netstat -nr

netstat -nr shows the route without resolving names.

MSS (Maximum Segment Size) is the size of the largest L3 datagram the kernel will construct for transmission via this route.

Window is the maximum amount of data the system will accept on a single burst from a remote host.

irtt (initial round-trip time) is the value that the TCP protocol will use when a connection is first established.
TCP protocol keeps a running count of how long it takes for a datagram to be delivered AND an acknowledgement to be received (ie. the total time, for a packet, to go and come back), so that it knows how long to wait before assuming a datagram needs to retransmitted: round−trip time!


G = Is a Gateway.

U = Interface is up.


netstat -i

netstat -i shows interfaces statistics.

The MTU and Met fields show the current MTU and metric values for that interface.

The RX−OK TX−OK are obvious: received or transmitted error−free packets.

RX−ERR and TX−ERR are how many packets were damaged.

RX−DRP and TX−DRP how many were dropped.

RX−OVR/TX−OVR) how many packets were lost because of an overrun.


B = a broadcast address has been set.

M = All packets are received (aka promiscuous mode).

O = ARP is turned off for this interface.

P = This is a point−to−point connection.

R = Interface is running.

U = Interface is up.

L = this interface is a loopback device.

Posted in LINUX, System Administration | Comments Off on netstat -puntalo.

Linux Kernel Logging howto.

June 23rd, 2013 by Andrea Matesi

If Linux crashes, by default, there are not so many places you can go look for a complete log.

If you have a crashy Linux system you need to debug, you can do so by enabling kernel logging.

Enable kern.log.

(From http://serverfault.com/questions/308503/how-do-i-view-enable-kernel-logs-on-an-ec2-instance-amazon-linux): dmesg give you the kernel logs but it doesn't include the timestamp by default (it can be enable by recompiling kernel with CONFIG_PRINTK_TIME=y).

With [r]syslog, you can log all kernel messages to a file (with timestamp) by inserting a below line into /etc/[r]syslog.conf:

kern.* /var/log/kern.log

Don't forget to restart [r]syslog daemon.

For CentOS:

/etc/init.d/syslog restart

Will restart the syslog.

Logrotate kern.log.

So, basically, by enabling kernel logging, you'll get a timestamped log without the need to recompile the kernel. But since the file logged by the kernel may easily become fat, it's more indicated to put it on logrotate.d.

So, create a new file inside /etc/logrotate.d/ and name it kern.log (ie. vi /etc/logrotate.d/kern.log) , then fill it with the following:

# Logs kernel messages into kern.log.
/var/log/kern.log {
        rotate 6
                        /sbin/killall -HUP syslogd

Now it's also a good idea to cat /var/log/kern.log. If it exists, you’re done.

Hopefully the next crash won’t be so clueless!

Posted in LINUX, System Administration | Comments Off on Linux Kernel Logging howto.

« Previous Entries