Narkisr - blog

Narkisr.com where code writes itself.


08
Jan 11

apt-mirror, local speedup

One of the biggest advantages of Linux are package manager like apt yum and pacman, its really easy to get your software installed via a single command.
Still installing package via your WAN can be time/bandwidth consuming, there are two main solutions:
  • Caching tools like Apt-Cacher that cache installed packages in a local proxy server, the main drawback is that the first one who installs a package waits the most.
  • Mirroring tools like apt-mirror that mirror an entire repository onto a local server, the main drawback of mirroring is the initial sync operation bandwidth (can take 50Gb) and storage capacity.

Iv choose to use mirroring mainly because the time a single machine will have to wait the first time, lets start by installing apt-mirror:
$ sudo aptitude install apt-mirror
The main configuration file is /etc/apt/mirror.list:
############# config ##################
set base_path    /media/ubuntu-mirror
set limit_rate 15k
set mirror_path  $base_path/mirror
set skel_path    $base_path/skel
set var_path     $base_path/var
set nthreads     10
set _tilde 0
############# end config ##############

deb-i386 http://archive.ubuntu.com/ubuntu maverick main restricted universe multiverse
deb-i386 http://archive.ubuntu.com/ubuntu maverick-security main restricted universe multiverse
deb-i386 http://archive.ubuntu.com/ubuntu maverick-updates main restricted universe multiverse
#deb-i386 http://archive.ubuntu.com/ubuntu maverick-proposed main restricted universe multiverse
#deb-i386 http://archive.ubuntu.com/ubuntu maverick-backports main restricted universe multiverse

deb-i386-src http://archive.ubuntu.com/ubuntu maverick main restricted universe multiverse
deb-i386-src http://archive.ubuntu.com/ubuntu maverick-security main restricted universe multiverse
deb-i386-src http://archive.ubuntu.com/ubuntu maverick-updates main restricted universe multiverse
#deb-i386-src http://archive.ubuntu.com/ubuntu maverick-proposed main restricted universe multiverse
#deb-i386-src http://archive.ubuntu.com/ubuntu maverick-backports main restricted universe multiverse


deb-amd64 http://archive.ubuntu.com/ubuntu maverick main restricted universe multiverse
deb-amd64 http://archive.ubuntu.com/ubuntu maverick-security main restricted universe multiverse
deb-amd64 http://archive.ubuntu.com/ubuntu maverick-updates main restricted universe multiverse
#deb-amd64 http://archive.ubuntu.com/ubuntu maverick-proposed main restricted universe multiverse
#deb-amd64 http://archive.ubuntu.com/ubuntu maverick-backports main restricted universe multiverse

deb-amd64-src http://archive.ubuntu.com/ubuntu maverick main restricted universe multiverse
deb-amd64-src http://archive.ubuntu.com/ubuntu maverick-security main restricted universe multiverse
deb-amd64-src http://archive.ubuntu.com/ubuntu maverick-updates main restricted universe multiverse
#deb-i386-src http://archive.ubuntu.com/ubuntu maverick-proposed main restricted universe multiverse
#deb-i386-src http://archive.ubuntu.com/ubuntu maverick-backports main restricted universe multiverse

clean http://archive.ubuntu.com/ubuntu
Notice limit_rate and nthreads, both will affect how much bandwidth will be used during sync, iv also disabled source downloads, we can now run the initial sync:
$ apt-mirror
  Downloading 90 index files using 10 threads...
The initial sync will take some time, feel free to stop it at any stage (it will resume once restarted), once its done will head on to set regular update job, edit /etc/cron.d/apt-mirror to the following:
 #
 # Regular cron jobs for the apt-mirror package
 #
 0 4 * * * apt-mirror  /usr/bin/apt-mirror >> /var/spool/apt-mirror/var/cron.log 2>&1
Unlike the default setting Im redirecting stderr and also appending to the log file, in order to serve apt requests from this server will use Nginx:
$ cd /var/www
$ sudo ln -s /media/ubuntu-mirror/mirror/archive.ubuntu.com/ubuntu ubuntu
Last thing left is to edit /etc/apt/sources.list to use our new local mirror:
#deb cdrom:[Ubuntu 10.10 _Maverick Meerkat_ - Release amd64 (20101007)]/ maverick main restricted
# See http://help.ubuntu.com/community/UpgradeNotes for how to upgrade to
# newer versions of the distribution. 

deb http://mirror-host/ubuntu/ maverick main restricted

## Major bug fix updates produced after the final release of the
## distribution.
deb http://mirror-host/ubuntu/ maverick-updates main restricted

...
apt-getting was never faster.


Rebeca
Rebeca

Hi, great post! help me a lot, but i have a doubt, i mirrored the three last versions of ubuntu. apt-mirror has created the pool but did not create the folder /dists... so my local computers are unable to pull from the mirror in sources.list. what i did wrong? and how fix that? Thanks..and sorry my "google english"!

ronen
ronen

Hey Rebeca, Iv taken a look in my folder structure and it should contain dists, did you notice any error in the logs?

This website content by Ronen Narkis is licensed under a Creative Commons Attribution 2.5 Israel License, based on a work at narkisr.com, Permissions beyond the scope of this license may be available at narkisr.com.