Create a Local Ubuntu Repository using Apt-Mirror and Apt-Cacher

How can a company or organization minimize bandwidth costs when maintaining multiple Ubuntu installations? With bandwidth becoming the currency of the new millennium, being responsible with the bandwidth you have can be a real concern. In this article by Christer Edwards, we will learn how to create, maintain and make available a local Ubuntu repository mirror, allowing you to save bandwidth and improve network efficiency with each machine you add to your network.

(For more resources on Ubuntu, see here.)

Background

Before I begin, let me give you some background into what prompted me to initially come up with these solutions.

I used to volunteer at local university user groups whenever they held "installfests". I always enjoyed offering my support and expertise with new Linux users. I have to say, the distribution that we helped deploy the most was Ubuntu. The fact that Ubuntu's parent company, Canonical, provided us with pressed CDs didn't hurt! Despite having more CDs than we could handle we still regularly ran into the same problem. Bandwidth. Fetching updates and security errata was always our bottleneck, consistently slowing down our deployments.

Imagine you are in a conference room with dozens of other Linux users, each of them trying to use the internet to fetch updates. Imagine how slow and bogged down that internet connection would be. If you've ever been to a large tech conference you've probably experienced this first hand!

After a few of these "installfests" I started looking for a solution to our bandwidth issue. I knew that all of these machines were downloading the same updates and security errata, therefore duplicating work that another user had already done. It just seemed so inefficient. There had to be a better way. Eventually I came across two solutions, which I will outline below.

Apt-Mirror

The first method makes use of a tool called “apt-mirror”. It is a perl-based utility for downloading and mirroring the entire contents of a public repository. This may likely include packages that you don't use and will not use, but anything stored in a public repository will also be stored in your mirror.

To configure apt-mirror you will need the following:

• apt-mirror package (sudo aptitude install apt-mirror)
• apache2 package (sudo aptitude install apache2)
• roughly 15G of storage per release, per architecture

Once these requirements are met you can begin configuring the apt-mirror tool. The main items that you'll need to define are:

• storage location (base_path)
• number of download threads (nthreads)
• release(s) and architecture(s) you want

The configuration is done in the /etc/apt/mirror.list file. A default config was put in place when you installed the package, but it is generic. You'll need to update the values as mentioned above.

Below I have included a complete configuration which will mirror Ubuntu 8.04 LTS for both 32 and 64 bit installations. It will require nearly 30G of storage space, which I will be putting on a removable drive. The drive will be mounted at /media/STORAGE/.

# apt-mirror configuration file



##

## The default configuration options (uncomment and change to override)

##

#

set base_path /media/STORAGE/

# set mirror_path $base_path/mirror

# set skel_path $base_path/skel

# set var_path $base_path/var

#

# set defaultarch <running host architecture>

set nthreads 20

#



# 8.04 "hardy" i386 mirror

deb-i386 http://us.archive./ubuntu hardy main restricted universe 
multiverse

deb-i386 http://us.archive./ubuntu hardy-updates main restricted 
universe multiverse

deb-i386 http://us.archive./ubuntu hardy-security main restricted 
universe multiverse

deb-i386 http://us.archive./ubuntu hardy-backports main restricted 
universe multiverse

deb-i386 http://us.archive./ubuntu hardy-proposed main restricted 
universe multiverse



deb-i386 http://us.archive./ubuntu hardy main/debian-installer 
restricted/debian-installer universe/debian-installer multiverse/debian-installer

deb-i386 http://packages./ hardy free non-free



# 8.04 "hardy" amd64 mirror

deb-amd64 http://us.archive./ubuntu hardy main restricted universe 
multiverse

deb-amd64 http://us.archive./ubuntu hardy-updates main restricted 
universe multiverse

deb-amd64 http://us.archive./ubuntu hardy-security main restricted 
universe multiverse

deb-amd64 http://us.archive./ubuntu hardy-backports main restricted 
universe multiverse

deb-amd64 http://us.archive./ubuntu hardy-proposed main restricted 
universe multiverse



deb-amd64 http://us.archive./ubuntu hardy main/debian-installer 
restricted/debian-installer universe/debian-installer multiverse/debian-installer

deb-amd64 http://packages./ hardy free non-free



# Cleaning section

clean http://us.archive./

clean http://packages./

It should be noted that each of the repository lines within the file should begin with deb-i386 or deb-amd64. Formatting may have changed based on the web formatting.

Once your configuration is saved you can begin to populate your mirror by running the command:

apt-mirror

Be warned that, based on your internet connection speeds, this could take quite a long time. Hours, if not more than a day for the initial 30G to download. Each time you run the command after the initial download will be much faster, as only incremental updates are downloaded.

You may be wondering if it is possible to cancel the transfer before it is finished and begin again where it left off. Yes, I have done this countless times and I have not run into any issues.

You should now have a copy of the repository stored locally on your machine. In order for this to be available to other clients you'll need to share the contents over http. This is why we listed Apache as a requirement above. In my example configuration above I mirrored the repository on a removable drive, and mounted it at /media/STORAGE. What I need to do now is make that address available over the web. This can be done by way of a symbolic link.

cd /var/www/
sudo ln -s /media/STORAGE/mirror/us.archive./ubuntu/ ubuntu

The commands above will tell the filesystem to follow any requests for “ubuntu” back up to the mounted external drive, where it will find your mirrored contents. If you have any problems with this linking double-check your paths (as compared to those suggested here) and make sure your link points to the ubuntu directory, which is a subdirectory of the mirror you pulled from. If you point anywhere below this point your clients will not properly be able to find the contents.

The additional task of keeping this newly downloaded mirror updated with the latest updates can be automated by way of a cron job. By activating the cron job your machine will automatically run the apt-mirror command on a regular daily basis, keeping your mirror up to date without any additional effort on your part.

To activate the automated cron job, edit the file /etc/cron.d/apt-mirror. There will be a sample entry in the file. Simply uncomment the line and (optionally) change the “4” to an hour more convenient for you. If left with its defaults it will run the apt-mirror command, updating and synchronizing your mirror, every morning at 4:00am.

The final step needed, now that you have your repository created, shared over http and configured to automatically update, is to configure your clients to use it. Ubuntu clients define where they should grab errata and security updates in the /etc/apt/sources.list file. This will usually point to the default or regional mirror. To update your clients to use your local mirror instead you'll need to edit the /etc/apt/sources.list and comment the existing entries (you may want to revert to them later!)

Once you've commented the entries you can create new entries, this time pointing to your mirror. A sample entry pointing to a local mirror might look something like this:

deb http://192.168.0.10/ubuntu hardy main restricted universe multiverse
deb http://192.168.0.10/ubuntu hardy-updates main restricted universe multiverse
deb http://192.168.0.10/ubuntu hardy-security main restricted universe multiverse

Basically what you are doing is recreating your existing entries but replacing the archive. with the IP of your local mirror. As long as the mirrored contents are made available over http on the mirror-server itself you should have no problems. If you do run into problems check your apache logs for details.

By following these simple steps you'll have your own private (even portable!) Ubuntu repository available within your LAN, saving you future bandwidth and avoiding redundant downloads.

Asterisk Gateway Interface 1.4 and 1.6 Programming

Design and develop Asterisk-based VoIP telephony platforms and services using PHP and PHPAGI

(For more resources on Ubuntu, see here.)

Apt-Cacher

The second method of mirroring Ubuntu packages is through the use of a tool called Apt-Cacher.

Apt-Cacher differs from apt-mirror in that it does not mirror the entire repository contents, but simply caches (stores) any packages requested by clients within the network. In other words, it works as a middle man between the public repository and the LAN clients, sharing any common downloads internally.

Each time any client requests a package, whether it be a security update or a new application to install, the apt-cache server will request the package from the public repository, store it and pass it back to the original requesting client. The package is then made available to any future requests from the internal network.

This method is best suited for those with minimal storage availability, or those that may simply want to make updates more efficient.

To configure this method you will need the following:

• apt-cacher package (sudo aptitude install apt-cacher)
• apache2 package (sudo aptitude install apache2)
• available storage (size varies based on the number of requested packages stored)

Once you've installed the required packages you'll need to activate the service. This is done by editing the /etc/default/apt-cacher file and changing the value of AUTOSTART:

"AUTOSTART=0" to "AUTOSTART=1"

You can also limit access to the cache within the /etc/apt-cacher/apt-cacher.conf file. Update the value for allowed_hosts to match the individual hosts or subnets that you want to allow. Once these access rules are set you can start the caching system:

sudo /etc/init.d/apt-cacher restart

The final step in configuring apt-cacher is the configuration of the clients. Each machine within your network will need to be configured to work with this caching system, otherwise it will be ineffective.

The way that I prefer to do this is by defining an apt proxy setting within the /etc/apt/apt.conf.d/ directory. The way you can do this is by creating a new file called 90-apt-proxy.conf and entering the line:

Acquire::http::Proxy "http://repository-cache:3142";

Note, make sure you replace "repository-cache" with the IP address or URL of the machine you installed apt-cacher on.

Now, any time that client is able to access the URL (repository-cache:3142) it will use the cache. If the URL is not available it should default back to the entries in the /etc/apt/sources.list file. If it does not, and you see 404 errors or similar, simply comment out the "Acquire::http::Proxy" line that you populated in the file above.

This step should be repeated for any and all machines that you want to access the cache. With each additional system you configure you're saving potential bandwidth and adding to the pool of cached packages. There is no limit on the number of machines you can point to an apt-cache.

Summary

In conclusion, creating a mirror or caching updates can have a dramatic impact on the use of bandwidth within your organization. Whether it is a full blown mirror, holding every last package available, or simply a cache of requested updates, both of these tools are simple to configure and a breeze to maintain.

I highly recommend using one of these solutions if you have multiple Ubuntu installations.