pwrusr.com

Intro.

• Say you have an old crufty CentOS host that you cannot absolutely touch because it is working^tm (apart from the occasional apache-OOM, that is).
• Say you are required to migrate this old system to a new hardware (perhaps to a VM), and, for the occasion, to a more recent CentOS version.

Fortunately with Linux, you have plenty of options!

One of them could be by manually cloning the old CentOS configuation to the new one (essentially by trying to replicate the existing setup on the new system).

But how to clone the installed programs list to the new one? It can be done!

On the Old CentOS host:

Launch (on the old system):

rpm -qa --queryformat “%{NAME}\n” > yum.packages.list

This command will export the list of the applications installed on the old CentOS without the architecture extension, version number, revision, etc. (ie: "gnome-utils-libs" vs "gnome-utils-libs-2.28.3-1.fc11.i586").

Much better, but the real advantage of the previous command is that it will create a list of packages compatible to yum (ie.: this packages list could be directly supplied to the yum command)!

On the New CentOS host.

This way you could:

yum -y install < yum-packages.list

And yum will simply fetch and install the latest versions of the packages specified on the list file directly from its repos. Please note that, before launching the previous command, I'd recommend you to at least check the repos (on both systems).

Now you can focus on the serious stuff: the /etc folder and the configuration files. Good luck!

SRC:

http://www.cyberciti.biz/tips/linux-get-list-installed-software-reinstallation-restore.html

Intro.

The Apache Bandwidth module allows you to limit the allocated bandwidth on a per-VirtualHost basis, which is very useful on hosted scenarios (many sites on a single IP) or if you're simply hosting your own site at home.

First check if you have it installed already:

apachectl -M |grep mod_bw

Install mod_bw.

Then, on CentOS/Ubuntu, install it by doing:

yum install mod_bw; apachectl -k graceful
sudo apt-get install libapache2-mod-bw; sudo a2enmod bw

Here's an example I successfully used to limit the available bandwidth on a named virtual host site:

<VirtualHost server-ip:80>
    BandwidthModule On
    ForceBandWidthModule On
    Bandwidth all 20480
    MinBandwidth all 500
    LargeFileLimit * 1024 500
    ServerName www.site.it
    ServerAlias site.com.au
[...]
</VirtualHost>

mod_bw vhost config.

Bandwidth all 20480 allocated 20Kb/s (20480/1024) to every user (it was a very fascist limit, but hey, I was in Italy!). Instead of "all", you may specify IP Addresses or subnets with an /XX notation and you can also declare multiple Bandwidth-entries (ie. BW localhost XX; BW 192.168.0.0/24 YY).

ForceBandWidthModule On forces the module to apply to every request (the man says mod_bw by default doesn't apply to every request), but I suppose this may increase the server load -- just speculating here, so bear with me.

MinBandwidth all 500 allocated a minimum bandwidth of 500 bytes (0.48k!) to every user.

Both the previous commands, when applied, meant the 1st user would have a total of 20Kb/s, from the 2nd client on, the total Bandwidth would've been splitted (10k each), with a MinBandwidth for every user of 0.48k.

LargeFileLimit * 1024 500 is a limit on a per-file basis. Here I did limit ALL the files with an asterisk, but I coulda specified different specific extensions (i.e.: by specifying multiple LargeFileLimit-entries -- ".avi", ".tbz2", ".mp3", etc.). 1024 specifies the single file size limit in Kbytes, the 500(!) specifies the maximum bandwidth available to filetypes in bytes/s.

A good alternative way to use mod_bw is to declare the "MinBandwidth -1" parameter, which means each user will have a top Bandwidth specified by the "Bandwidth" directive (ie.: everybody will have the same top bandwidth) like so:

    Bandwidth all 20480
    MinBandwidth all -1
    MaxConnection all 8

In this example all woulda had a top limit of 20Kb/s, which sounds decisively less fascist!

I've also added a statement to limit the maximum number of connections. Please note this can be used only if you also declared a "Bandwidth" statement AND it must also be applied only to the same Bandwidth "XXX"-users (i.e.: don't use MaxConnection if you don't use Bandwidth). BTW, this should limit the maximum number of connections per IP to 8.

Some considerations.

That said, mod_bw is a beautiful toy that allows you to set your bandwidth limits on a per customer-basis, and that's the best way (absolutely IMHO) to use it.

You need to first identify your "resource-hog" sites, study how much bandwidth per-user they need and, depending on how much bandwidth you have available, verify how it works under different scenarios, but this is also risky since how a sites develops is (generally) not easy to predict (and good values for today usage are not always good values for tomorrow scenarios).

That could represent a good intro, for more info, look at the man page: http://bwmod.sourceforge.net/files/mod_bw-0.7.txt

This short guide explains how to redirect the "Home page" of your site to a temporary web page.

I'm assuming your website is PHP-based (ie. Joomla, WordPress or other) and it is hosted on a LAMP Server.

Those assumptions also imply your home page (usually) is named “index.php”.

But for additional clues, please check your apache loading order defaults (on the apache configuration file).

2 Ways to do Redirect your Home Page to a Temporary Page:

1) Your redirection could be as simple an an “Under Construction” basic HTML page named "index.html", which you will simply upload to your site's root.

By default, "index.html" has loading precedence over "index.php" and will simply be loaded first.

That's simple redirection at work!

 

Alternatively, if your website is hosted on Cpanel/WHM, you may:

2) Create a manual redirection through the "Redirects" function:

• From type select "Permanent (301).
• Select your domain from the dropdown and leave the following field empty.
• Put the checkmark on "Redirect with or without www".
• Do Not click enable the Wildcard Redirect otherwise "your.site.sucks.domain.com" will be considered as valid and will take you to your website!
• Click on Add and this is it, Cpanel redirection in place!

To successfully understand what am I talking about, you should read my previous article:

http://www.pwrusr.com/system-administration/relocating-a-raid1-mdadm-array-to-bigger-hdds-on-centos-pt-1

Remove the faulty hdd and disable the swap.

If you've followed me, here's some hands-on.

The first step involves removing sdb (the faulty hdd in my case), so:

mdadm /dev/md0 --fail /dev/sdb1 --remove /dev/sdb1
mdadm /dev/md1 --fail /dev/sdb2 --remove /dev/sdb2
swapoff /dev/sdb3
mdadm /dev/md2 --fail /dev/sdb4 --remove /dev/sdb4

• The previous commands will remove /dev/sdb from the active array. Also, have a look at fstab and comment the swap line (I had 2 different swaps on two different partitions, which were outside the mdadm RAID 1 array).

screenshot:

Add your new hdd.

Now you can shutdown your system, physically install the new hdd and pwr the system up.

At this point I'm assuming you've already taken care of the physical layer (i.e.: by physically adding the new, additional hdd to your server), so let's proceed.

On the left, how my original mdadm array was (before adding the 1TB hdd), on the right (followed by the "->"-simbol), how my new mdadm array will look like (ie. after following the whole guide).

(set the partition type to 0xDA -- Non-fs data or 0xFD, Linux raid autodetect if you are still using the deprecated autodetect check other disks and decide!)

Fdisk fun hour!

Now you should fire fdisk and create a new partition table and the new partitions on the new additional 1TB hdd.

Here's a summary of the possible fdisk commands:

fdisk /dev/sdb
   
    o for a New Partition TABLE.
    p to VIEW (print) the Partition table.
    n for a New PARTITION (1, default starting sector, +end_sizeM).
    a to make the (/boot) Partition ACTIVE.
    t to change/specify the Partition TYPE.

• Use fdisk to recreate your partition table and your partitions scheme (this is fun, the best part of my guide)!

Some adjustments.

After you manually created the partition table and your desired partitions, create and enable the swap partition like so:

mkswap /dev/sdb3
swapon /dev/sdb3

• Then open fstab and uncomment the previously commented swap string.

Once the new partitions are created and the swap is enabled, synch the new partitions to the existing mdadm array:

mdadm --add /dev/md0 /dev/sdb1
mdadm --add /dev/md1 /dev/sdb2
mdadm --add /dev/md2 /dev/sdb4

The previous commands will add the new partitions to your pre-existing array.

A background synchronization process will (automatically) start.

The synch process will simply replicate the content From the original (pre-existing) 80GB disk To the new 1TB disk.

Tweaking the mdadm Synchronization process.

To SPEEDUP the mdadm resync process:

echo 400000 >/proc/sys/dev/raid/speed_limit_max
echo 4000 >/proc/sys/dev/raid/speed_limit_min
ps fax | grep sync
renice -20 mdX_resyncPID (to add priority to the sync process).

TO WAIT for the RESYNC process to finish:
mdadm -W or --wait /dev/mdX

(AFAIR, wait will take your control of your prompt and won't be able to mess with your system in the meantime, but that's subjective -- just be patient!).

GRUB setup.

It would also be a good time to check the mdadm.conf, so have a look at /etc/mdadm/mdadm.conf and verify everything is alright on your new array.

Once the synch process has finished (believe me, it takes a lot even after the tweaking it), you will need to "grub-ify" the new hdd.

The commands I typed are as follows:

grub
root (hd0,0)
setup
root (hd1,0)
setup

After grub was setup on your new hdd, you'll need to check if it works, so remove the old original (booting) disk, and leave the new (just synched) 1TB hdd alone (i.e.: /dev/sdb), then verify If CentOS boots up from the new hdd (it will be recognized as /dev/sda).

Fantastic, if you made it successfully 'till here, you're on your way to succeed!

[ADD THE SECOND 1TB DISK - repeat the first disk steps!]

Now…it is time to add the second 1TB hdd to the array. I'm not going again on a step-by-step process (like a boring wallpaper which periodically keeps popping up), simply because the procedure is more or less the same as I explained above.

And I am pretty sure your setup differs from mine, so you'll have to adjust accordingly to your specific requirements.

Refreshed RAID 1 mdadm array booting successfully from BOTH NEW Hard Disk Drives.

At this point, you've already tested your system and it successfully boots from BOTH hard drives, because you were diligent and you DID test it by booting first from the 1st disk then from the 2nd disk.

Well done! You're almost finished.

The last step consists on the baptism of fire growing the new hdds for them to avail of the whole space.

GROW and RESIZE GOOD RAID 1 ARRAYS.

Before growing the array, check the partitions are clean with fsck, then grow the array and resize the filesystems to their maximum available size.

mdadm --grow /dev/md0 --size=max

The previous command will grow your array to the maximum available space, then:

resize2fs -p /dev/md1

To resize the filesystems online.

Please note for it to work, you'll need at least linux-2.6.x, which supports the ext3 "on-line resizing" feature.

Now run mdadm -W or --wait /dev/mdX!

Did I make your day?! Shout a Coopers for me (Paypal under my pic!).

Intro.

This is not for the fainthearted, but since you're here, you're probably looking for what my title says…

This procedure is especially important when one of your Hard Disk Drives breaks and you urgently need to swap it with another (and unfortunately you are not able to buy another new 80GB hdd, but you must cope with adding a new 1TB-hdds instead).

So, basically, you could simply add a new 1TB hdd to your existing 80GB RAID1 Array and live with it, but considering the cost of new hard drives, what's the point?

It's a lot better if you just swap BOTH the old AND the broken original 80GB Hard Disk Drives with fresh new 1TB hdds.

This way you:

1. Refresh your existing RAID 1 array.
2. Circumvent dealing with a potential problem that is going to happen (on your old but working 80GB hdd).

This article is split in 2 parts for organizational purposes. Part 1 involves the "information gathering" part. Part 2 will deal with hands-on operations.

Preliminary checks: Check IF there is a Bitmap on your /dev/mdX array.

mdadm /dev/mdX --examine-bitmap
cat /proc/mdstat | grep bitmap

The previous commands will check if your array has a bitmap.

Example n.1: Array with a bitmap.

cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md_d0 : active raid5 sde1[0] sdf1[4] sdb1[5] sdd1[2] sdc1[1]
      1250241792 blocks super 1.2 level 5, 64k chunk, algorithm 2 [5/5] [UUUUU]
      bitmap: 0/10 pages [0KB], 16384KB chunk

unused devices: <none>

Example n.2: Normal array example (without bitmap line):

cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdb1[1] sda1[0]
      264960 blocks [2/2] [UU]

md2 : active raid1 sdb4[1] sda4[0]
      65312192 blocks [2/2] [UU]

md1 : active raid1 sdb2[1] sda2[0]
      10482304 blocks [2/2] [UU]

unused devices: <none>

If your array has a "[Bitmap]"-entry then the following (line) describes the bitmap state (Example 1 reported):

     bitmap: 0/10 pages [0KB], 16384KB chunk

What's a bitmap? It's a portion of data which keeps an "picture" of "data on disk". Write intent bitmaps means every time some data is about to be written, the RAID array region is marked as dirty. Then, say, after a power failure, all that needs to be done to ensure that all disks in the array have matching data it is to check the regions that are listed as dirty (Vs checking the entire disk surface for dirty bits).

This way, instead of waiting for an hour or more, only a few seconds of work are required.
here: http://etbe.coker.com.au/2008/01/28/write-intent-bitmaps/
here: http://www.linuxfoundation.org/collaborate/workgroups/linux-raid/mdstat#bitmap_line

Bitmaps are of two types: internal (stored inside the array) & external (stored outside the array).

How to add an internal bitmap?

mdadm -G /dev/md0 --bitmap=internal

(Quoted from the linked sites):"The down-side to this feature is that it will slightly reduce performance: When I enabled internal bitmaps a simple copy operation performa(n)ce dropped to HALF! So if you really can’t afford slowdown during RAID rebuild try to use external bitmap if at all possible. For example use fast SSDs, usb flash drives or CF".

External bitmap example:

mdadm --grow --bitmap=/boot/md2-raid5-bitmap /dev/md2

(Quoted from the above linked sites):"If the array has a write-intent bitmap, it is strongly recommended that you remove the bitmap before increasing the size of the array. Failure to observe this precaution can lead to the destruction of the array if the existing bitmap is insufficiently large, especially if the increased array size necessitates a change to the bitmap's chunksize".

How to remove a bitmap?

mdadm --grow /dev/mdX --bitmap none

Before following my procedure, I assume you don't have a bitmap (or if you do, you disabled it).

Preliminary checks: gathering info on your actual hdds status.

My hdds were partitionet this way:

md0 = boot    (sda1+sdb1)
md1 = /        (sda2+sdb2)
swap        (sda3 = sdb3)
md2 = var    (sda4+sdb4)

Device         Boot      Start   End   #cyls    #blocks   Id  System

/dev/sda1   Active      0+     32      33-    265041   fd  Autorilevamento raid di Linux

/dev/sda2         33    1337    1305   10482412+  fd  Autorilevamento raid di Linux

/dev/sda3       1338    1598     261    2096482+  82  Linux swap / Solaris

/dev/sda4       1599    9729    8131   65312257+  fd  Autorilevamento raid di Linux

Take notes on how your hdds are partitioned (i.e.: how swap was defined, etc.).

I've used this setup and it just worked^TM, but I reckon it's not the best, since it doesn't supports views, groups and other (interesting) features.

You may use it to get an introduction to net-SNMP. I'm also assuming you're willing to configure it on CentOS or Ubuntu (the sudo / apt commands refer to Ubuntu).

Install NET-SNMP.

Follow those steps to install net-snmp on your system (1st command = CentOS/2nd command = Ubuntu):

yum install net-snmp-utils net-snmp-devel
apt-get install snmp libsnmp-base snmpd sysv-rc-conf

Configure NET-SNMP.

My article uses a Read-only snmp user (remove "-ro" for a rw-one). Name:"user"; Password: "p@ssword":

net-snmp-config --create-snmpv3-user -A MD5 -a p@ssword -x DES -X p@ssword -ro user
sudo net-snmp-config --create-snmpv3-user -A MD5 -a p@ssword -x DES -X p@ssword -ro user

The following autostarts snmpd:

chkconfig snmpd on
sudo sysv-rc-conf snmpd on

Make a backup copy of the original snmpd.conf:

sudo cp /etc/snmp/snmpd.conf /etc/snmp/snmpd.conf.ori

Then edit it /etc/snmp/snmpd.conf:

###########################################################################
# SECTION: Access Control Setup
#
# rouser: a SNMPv3 read-only user
# arguments: user [noauth|auth|priv] [restriction_oid]
rouser user authpriv
# rocommunity: a SNMPv1/SNMPv2c read-only access community name
# arguments: community [default|hostname|network/bits] [oid]
# read-only SNMPv2c naw from all IP addresses/networks:
rocommunity public
interface eth0
agentaddress 192.168.2.11:161
interface lo0
agentaddress 127.0.0.1:161
syslocation "hq"
syscontact "and...@pwrusr.com"

Verify NET-SNMP.

Launch snmpd:

/etc/init.d/snmpd start

Verify it works on your loopback I/F:

snmpwalk -v 3 -a sha -A p@ssword -x des -X p@ssword -u user -l authPriv localhost | less

Verify it works on your physical I/F adapter:

snmpwalk -a MD5 -A p@ssword -x DES -X p@ssword -u user -c public -l authPriv 192.168.2.11 | less

SNMP iptables rules.

# SNMP
iptables -A INPUT -p udp -m udp -s 0/0 --dport 161:162 -j ACCEPT

Verify user is inside /usr/share/snmp/snmpd.conf:

rouser user

Enable net-snmp logging.

[CentOS-only - for Ubuntu adjust accordingly] Enable net-snmp logging to snmpd.log:

sed -i 's|OPTIONS="-LS0-6d -Lf /dev/null -p /var/run/snmpd.pid"|OPTIONS="-LS0-6d -Lf /var/log/snmpd.log -p /var/run snmpd.pid"|g' /etc/init.d/snmpd

• Also, put snmpd.log into logrotate.d, for example:

/var/log/snmpd.log {
    notifempty
    missingok
    size 32M
    yearly
    compress
    compresscmd /usr/bin/bzip2
    compressext .bz2
    postrotate
       /sbin/service snmpd condrestart 2> /dev/null > /dev/null || true
    endscript
}

Debug NET-SNMP.

From a terminal, stop the snmpd daemon and start it manually with:

snmpd -f -L -Dread_config

Enable SNMP debug option as a default, edit /etc/init.d/snmpd:

OPTIONS="-Lsd -Lf /var/log/snmpd.log -p /var/run/snmpd.pid -a -Dread_config"

NET-SNMP AES Support.

[Bonus]: Check if your net-snmp instance supports AES (from http://www.zenoss.com/community/docs/zenoss-guide/2.4.2/apbs02.html).

snmpwalk -x AES 2>&1 | head -1

IF "No hostname specified." -> AES Supported.

IF "Invalid privacy protocol specified after -x flag: AES" -> AES NOT supported.

Top is an historic monitoring tool to get a fast overview of a running Linux system. It is especially useful when you're at limited linux consoles and terminals (i.e.: busybox and the like).

By default it doesn't look so great, but there's always fast ways to customize it and use it fruitfully.

Custom top Color Scheme.

Edit ~/.toprc, paste the following then Save:

RCfile for "top with windows"           # shameless braggin'
Id:a, Mode_altscr=0, Mode_irixps=1, Delay_time=3.000, Curwin=0
Def     fieldscur=AEHIOQTWKNMbcdfgjplrsuvyzX
        winflags=63800, sortindx=10, maxtasks=0
        summclr=2, msgsclr=1, headclr=3, taskclr=1
Job     fieldscur=ABcefgjlrstuvyzMKNHIWOPQDX
        winflags=62777, sortindx=0, maxtasks=0
        summclr=6, msgsclr=6, headclr=7, taskclr=6
Mem     fieldscur=ANOPQRSTUVbcdefgjlmyzWHIKX
        winflags=62777, sortindx=13, maxtasks=0
        summclr=5, msgsclr=5, headclr=4, taskclr=5
Usr     fieldscur=ABDECGfhijlopqrstuvyzMKNWX
        winflags=62777, sortindx=4, maxtasks=0
        summclr=3, msgsclr=3, headclr=2, taskclr=3

top running with the previous color scheme applied to it (green|white|red is not casual):

My favourite top options.

Some of my top tricks…

top -cSb n 1 > running.log

Shows you the full top text output, very useful to take a snapshot of a running system to analyze afterwards. Please note the output of the previous command is very similar to the ps command.

top -cSb n 1 -p PID

Useful to monitor a process (check also my fav ps options to know how to get the PID of a daemon).

top -cSb -p PID n 1 | tail -n+8 | sort -rn -k5 | awk 'NR > 0 { s +=$9 }; END {print "cpu %",s}'

To monitor the CPU usage (in percent) of a single process, identified by a PID, with top.

What's the difference between the previous 2 commands? Check this screenshot:

As you can see by the previous screenshot, the two commands above differ by the full top header output.

The second one is very useful on bash scripts.

Also, some of the previous commands are extremely useful when used in conjunction with the watch command.

Intro.

Say you have a LINUX dir full of thousands of insignificant small files, like a dir full of mail messages that you want to clean fast.

You are 100% sure you can delete all of those small pesky files without consequences for anyone, because they are just uninformative messages (say you were monitoring your devices and you’ve put your host down for maintenance).

Issue.

This is what happens when you try to delete this huge files list:

root@srvr008 [/home/user/mail/domain.com/hostmaster/cur]# rm -rf *
-bash: /bin/rm: Argument list too long

Fix.

To successfully remove them, use instead:

ls | xargs rm

Some of my favourite ps options.

ps fauxe

Gives you a complete list of all your running processes, complete of path, PID & UID. You may wish to remove some options like u (u stands for user -- just type "faxe" or "fax" (very easy to remember indeed!)).

Useful in conjunction with the watch command (i.e.:when you want to ask a Linux System "how are you!").

Screenshot:

ps aux|awk 'NR > 0 { s +=$3 }; END {print "cpu %",s}'

Prints the Global CPU usage (in percent) by using ps. Useful with bash scripts.

Who's sucking my CPU?

ps fauxe  | sort -r -k 3 | head -7

The previous ps command takes everything you'll need to know, its output then gets sorted by cpu consumption with the sort command and head filters the output by showing you the first 7 cpu hogs.

ClamAV is "the (only…) Antivirus" when referring to Linux.

To scan from the cli is very easy:

cd /dir/to/scan
root@srvr101 [/home/guido]# clamscan -r -v -i --bell  > clamscan-guido-infected.log 2>clamscan-guido-error.log &

-r Stands for recursive.

-v Is for verbose.

-i Prints the infected files only.

--bell sounds a bell for every positive result found.

The infected scan result will be put inside the clamscan-guido-infected.log and the standard error log will be written inside clamscan-guido-error.log.

The & will execute the scan on the background and the prompt will allow you to do whatever you wish during the scan.