'

my local DEBIAN mirror.

October 20th, 2008 by Andrea Matesi 1930 Views

At work, for business needs, we decided it would have been better to adopt a local debian mirror.

After some search, I choose total flexibility, by adopting the anonftpsync shell script. It is a self-descriptive-log-and-mail kind of script. Just customize and use it. Here for convenience I'm gonna post the one I used (it's just commented, there are some parts that need to be uncommented out):

#! /bin/sh set -e  
# This script originates from http://www.debian.org/mirror/anonftpsync  
# modified by Andrea Matesi  
# CVS: cvs.debian.org:/cvs/webwml - webwml/english/mirror/anonftpsync 
# Version: $Id: anonftpsync,v 1.30 2007/09/06 18:05:44 joy Exp $  
# Note: You MUST have rsync 2.6.4 or newer, which is available in sarge 
# and all newer Debian releases, or at http://rsync.samba.org/  
# Don't forget: 
# chmod u+x anonftpsync  
# Set the variables below to fit your site. You can then use cron to have 
# this script run daily to automatically update your copy of the archive.  
# TO is the destination for the base of the Debian mirror directory 
# (the dir that holds dists/ and ls-lR). 
# (mandatory)  TO=/mnt/backup/mirror  
# RSYNC_HOST is the site you have chosen from the mirrors file. 
# (http://www.debian.org/mirror/list-full) 
# (mandatory)  RSYNC_HOST=debian.fastweb.it  
# RSYNC_DIR is the directory given in the "Packages over rsync:" line of 
# the mirrors file for the site you have chosen to mirror. 
# (mandatory)  RSYNC_DIR=debian/  
# LOGDIR is the directory where the logs will be written to 
# (mandatory)  LOGDIR=/var/log  
# ARCH_EXCLUDE can be used to exclude a complete architecture from 
# mirrorring. Please use as space seperated list. 
# Possible values are: 
# alpha, amd64, arm, hppa, hurd-i386, i386, ia64, m68k, mipsel, mips, powerpc, s390, sh and sparc 
# 
# There is one special value: source 
# This is not an architecture but will exclude all source code in /pool 
# 
# eg. ARCH_EXCLUDE="alpha amd64 arm hppa hurd-i386 ia64 m68k mipsel mips powerpc s390 sh sparc" 
# 
# With a blank ARCH_EXCLUDE you will mirror all available architectures 
# (optional)  #ARCH_EXCLUDE=  
# EXCLUDE is a list of parameters listing patterns that rsync will exclude, in 
# addition to the architectures excluded by ARCH_EXCLUDE. 
# 
# Use ARCH_EXCLUDE to exclude specific architectures or all sources 
# 
# --exclude stable, testing, unstable options DON'T remove the packages of 
# the given distribution. If you want do so, use debmirror instead. 
# 
# The following example would exclude mostly everything: EXCLUDE="\ --exclude stable/ --exclude testing/
 --exclude unstable/ \ --exclude source/ \ --exclude *.orig.tar.gz --exclude *.diff.gz --exclude *.dsc 
\ --exclude /contrib/ --exclude /non-free/ \ "  
# With a blank EXCLUDE you will mirror the entire archive, except the 
# architectures excluded by ARCH_EXCLUDE. 
# (optional)  #EXCLUDE=  
# MAILTO is the address to send logfiles to; 
# if it is not defined, no mail will be sent 
# (optional)  MAILTO=myself@mywork-co.it  
# There should be no need to edit anything below this point, unless there 
# are problems.  
#-----------------------------------------------------------------------------
#  
# If you are accessing a rsync server/module which is password-protected, 
# uncomment the following lines (and edit the other file).  
# . ftpsync.conf 
# export RSYNC_PASSWORD 
# RSYNC_HOST=$RSYNC_USER@$RSYNC_HOST  
#-----------------------------------------------------------------------------
#  
# Check for some environment variables if [ -z $TO ] || [ -z $RSYNC_HOST ] || [ -z $RSYNC_DIR ] || [ -z $LOGDIR ]; 
then echo "One of the following variables seems to be empty:" echo "TO, RSYNC_HOST, RSYNC_DIR or LOGDIR" exit 2 fi
if ! [ -d ${TO}/project/trace/ ]; then 
# we are running mirror script for the first time umask 002 mkdir -p ${TO}/project/trace fi  
# Note: on some non-Debian systems, hostname doesn't accept -f option. 
# If that's the case on your system, make sure hostname prints the full 
# hostname, and remove the -f option. If there's no hostname command, 
# explicitly replace `hostname -f` with the hostname.  HOSTNAME=`hostname`  
# The hostname must match the "Site" field written in the list of mirrors. 
# If hostname doesn't returns the correct value, fill and uncomment below 
# HOSTNAME=mirror.domain.tld  LOCK="${TO}/Archive-Update-in-Progress-${HOSTNAME}"  
# The temp directory used by rsync --delay-updates is not 
# world-readable remotely. It must be excluded to avoid errors. TMP_EXCLUDE="--exclude .~tmp~/"  
# Exclude architectures defined in $ARCH_EXCLUDE for ARCH in $ARCH_EXCLUDE; do EXCLUDE=$EXCLUDE"\ --exclude binary-$ARCH/
\ --exclude disks-$ARCH/ \ --exclude installer-$ARCH/ \ --exclude Contents-$ARCH.gz \ --exclude Contents-$ARCH.diff/ 
\ --exclude *_$ARCH.deb \ --exclude *_$ARCH.udeb " if [ "$ARCH" == "source" ]; then SOURCE_EXCLUDE="\ --exclude *.tar.gz 
\ --exclude *.diff.gz \ --exclude *.dsc " fi done  
# Logfile LOGFILE=$LOGDIR/debian-mirror.log  
# Get in the right directory and set the umask to be group writable 
# cd $HOME umask 002  
# Check to see if another sync is in progress if [ -f "$LOCK" ]; then if [ "`find $LOCK -maxdepth 1 -amin -360`" = "" ]; then 
# Note: this requires the procps ps; for other ps', adjust as necessary if ps ax | grep '[r]'sync | grep -q $RSYNC_HOST; 
then echo "stale lock found, but a rsync is still running, aiee!" exit 1 
else echo "stale lock found (not accessed in the last 6 hours), forcing update!" rm -f $LOCK fi 
else echo "current lock file exists, unable to start rsync!" exit 1 fi fi  touch $LOCK 
# Note: on some non-Debian systems, trap doesn't accept "exit" as signal # specification. 
If that's the case on your system, try using "0". trap "rm -f $LOCK" exit  set +e  
# First sync /pool rsync --recursive --links --hard-links --times --verbose \ $TMP_EXCLUDE $EXCLUDE $SOURCE_EXCLUDE \ 
$RSYNC_HOST::$RSYNC_DIR/pool/ $TO/pool/ >> $LOGFILE 2>&1 result=$?  if [ 0 = $result ]; then 
# Now sync the remaining stuff rsync --recursive --links --hard-links --times --verbose --delay-updates --delete-after \
 --exclude "Archive-Update-in-Progress-${HOSTNAME}" \ --exclude "project/trace/${HOSTNAME}"
 \ $TMP_EXCLUDE $EXCLUDE $SOURCE_EXCLUDE \ $RSYNC_HOST::$RSYNC_DIR $TO >> $LOGFILE 2>&1  LANG=C 
date -u > "${TO}/project/trace/${HOSTNAME}" else echo "ERROR: Help, something weird happened" | tee -a $LOGFILE 
echo "mirroring /pool exited with exitcode" $result | tee -a $LOGFILE fi  if ! [ -z $MAILTO ]; 
then mail -s "debian archive synced" $MAILTO < $LOGFILE fi  savelog $LOGFILE >/dev/null  rm $LOCK 

I've put this script auto-executing inside crontab, then I configured apache http for the file-serving purpose at a different standard name. When I had to manually install a debian distribution, I pointed the installer to grab deb packages from this http server and everything worked as expected, but especially, it was fast, since it was inside our lan.

Posted in LINUX, System Administration | Comments Off on my local DEBIAN mirror.

Comments are closed.