To download via HTTP or FTP, use either the wget or lftp command. The wget command recurses subdirectories automatically and pulls down entire trees of data with a single command. If you are not careful, however, it is possible to pull down much more data than you intended. The following commands mirror the entire current Fedora Core distribution:

cd /var/www/mirror 
wget --mirror -np -nH --cut-dirs=2 http://mirror.example.com/pub/mirror/fedora/linux/core/5/

Note the options used above:

The same syntax works for both HTTP and FTP upstream mirrors. It is possible that you may download some extraneous files if the HTTP site formats its pages for browser viewing. These files can be safely deleted, but return each time the mirror updates unless you exclude them using special options. See the wget man pages for more information.

The lftp command works like the wget command, and mirrors the content of a HTTP or FTP server. The wget command, however, does not delete old files locally. This feature is important for update repository mirrors to stay synchronized to upstream mirrors. New files are created and old files are automatically removed from the upstream mirrors on a frequent basis.

The lftp command synchronizes files and directories from a remote host like rsync, but uses HTTP or FTP protocols. Use the following command to mirror the entire Fedora Core distribution with lftp:

cd /var/www/mirror && \
lftp -c "open http://mirror.example.com/pub/mirror/linux/core/5/i386/ && \
mirror --delete --verbose"

The -c parameter executes a set of commands in a lftp process. Commands are separated with && to prevent the lftp command from executing if the cd command fails. The commands in the lftp command set work the same way. The command syntax A && B is often shorthand for "if A returns success, run B." An explanation of the lftp commands follows:

The lftp command above mantains an exact copy of the directory for you. It downloads only new or changed files, and deletes only those that no longer exist on the upstream mirror.

As with wget, it is possible you may download some unwanted files. The lftp command supports regular expressions for excluding files within a mirror command. The command below shows how to mirror an current Fedora Core distribution updates repository, excluding debug and repodata directories:

cd /var/www/mirror && \
lftp -c "set mirror:exclude-regex 'debug\/|repodata\/' && \
open http://mirror.example.com/pub/mirror/linux/core/updates/5/i386/ && \
mirror --delete --verbose"

Consult the lftp man pages for more details and usage options.

	Using Proxy for HTTP or FTP retrieval
	If you are behind a proxy or firewall, you may need to use a HTTP proxy to mirror files. To do this, export the environment variables http_proxy and ftp_proxy before you run the wget or lftp commands: export http_proxy=http://username:password@host:port export ftp_proxy=http://username:password@host:port

Use the rsync command to synchronize a set of files and/or directories with a remote host. It operates in much the same way as rcp, but it is usually faster. One reason for the speed is that rsync has a special protocol that evaluates and skips files (or portions of files) that are already downloaded.

Begin by identifying the modules available on the upstream mirror site you have chosen. Note that the double colon "::" is always used after the host name to separate it from the rest of the rsync path. The following command generates a list of "modules" on the upstream mirror.

rsync mirror.example.org::

These modules are roughly equivalent to top-level directories, and they follow the same rules. To list any subdirectory of the upstream mirror, add the directory path to the command above. For example, on many mirrors, the fedora-linux-core module is equivalent to the fedora/linux/core path found at the Fedora Project main download server. To list the contents of the Fedora Core 5 distribution folder on the upstream server, issue the following command. Do not forget the trailing slash "/". Without it, you only receive a listing of a folder name that matches the last component of the remote path.

rsync mirror.example.org::fedora-linux-core/5/

To download via rsync, add a destination path on your system to the end of the command line. The resulting tree of files from the listing you perform are downloaded to the local path you specify. Remember, if you leave off the trailing slash on the remote path, then the last component of that path is created as a folder, and its contents are copied.

rsync filehouse.example.org::files/misc/ /var/www/misc/

When downloading using rsync for mirror purposes, use some of the command line switches to improve performance and feedback. The switches -PHav enable the following rsync features:

Remove the -v switch if you run this mirroring process as part of a script, or have no need to monitor progress. The following example mirrors all available versions of Fedora Core from an upstream site.

	Example command downloads many gigabytes of files
	This command downloads many gigabytes of files, and is intended for use as an example only. Do not run this command if you do not understand the consequences.

rsync -PHav mirror.example.org::fedora-linux-core/5/ /var/www/mirror/fedora/linux/core/5

The -n switch performs a "dry run" using the other given parameters. Use this switch to test any rsync command if you are unsure what files you will receive. See also Possible data loss.

The -z switch enables compression during the rsync process. The server compresses data before transmission, and the client decompresses the data before writing it to disk.

	Compression using rsync
	The vast majority of the Fedora Core distribution consists of RPM files, which are already compressed data. Therefore, additional compression does not save time, and instead induces an unnecessary load on the upstream mirror CPU. As a courtesy, do not use the -z switch for this purpose.

The next section features some additional switches that can be used to automatically trim branches from the tree of downloaded folders. With proper usage, they result in a mirror that is exactly as organized and full-featured as any high-volume public upstream site.

Possible data loss

If you are not exceedingly careful in using these switches, it is possible to delete large portions of your mirrored data. Fixing this problem might require performing the copying steps outlined in Section 2.2, “Copying the Original Distribution” above. On the other hand, if you are also careless about your destination path, and you are running as root, you could put your entire system at risk. Know your environment before using these switches: What is your current working directory? Use pwd to find out, if you are unsure. Are you logged in as root? If you are using SELinux extensions, what is your current security context? Have you tested this command using the -n switch (see Section 2.4.3, “Downloading Using rsync”)?

Use the --exclude switch, along with a simple pattern, to disallow download of certain files and/or folders. For instance, --exclude "*.iso" excludes the download of any file whose name ends with the string ".iso".

Use the --delete switch, again with a pattern, to remove any file from the local system which does not have a match on the upstream mirror. This switch prevents unwanted file debris from cropping up in your mirror. You can also use it to retroactively trim branches of the tree which you no longer wish to maintain or download.

Wildcards are permitted with rsync commands, including the asterisk *, question mark ?, and brackets [ ]. The question mark and brackets work as in the shell; the former matches any single character, while the brackets define a set of characters to be matched. Asterisks are especially powerful when combined with a portion of a file name. The double asterisk ** pattern matches any character, including slashes; a single asterisk * matches any character, but stops at a slash. Therefore, be judicious about using either. The double asterisk is very useful for mirroring a tree that includes multiple instances of directories and files that contain a pattern. A good example is mirroring several versions of Fedora Core, where certain folder names appear in every version.

Pattern matching wildcards

Use double asterisks to trim out directories that repeat throughout a mirrored tree. For example, when mirroring for a site that only uses i386 architecture machines, you may trim all files and folders marked for x86_64 architecture, using the switch --exclude "**x86_64**". This matches not only folders marked x86_64, but also files such as ISO images for x86_64, which are indicated by file names such as FC5-x86_64-disc1.iso.

Process a long list of exclusions and deletions with the --exclude-from and --delete-from options. Follow each tag with a file name that includes a list of patterns, one per line, to be matched by the appropriate option.

These syntax hints only scratch the surface of rsync, but suffice to make your first mirror. Once you have selected your site and formulated your excludes and deletes, run your rsync command with the -n option. Redirect output to a file so you can examine the resulting list of files in the editor or pager of your choice.

The following example mirrors the entire Fedora Core 5 distribution, with --exclude options that avoid downloading:

The -n switch is included for testing purposes. Backslashes at the ends of lines indicate this example is a single command line.

rsync -Pan --delete --exclude "**x86_64**" --exclude "**headers**" \
  --exclude "**debug**" --exclude "**iso**" \
  mirror.example.com::fedora-linux-core/5/ \
  /var/www/mirror/fedora/core/5