### Setting up a Devuan Package mirror -- walkthrough ### This document includes a simple walkthrough to help setting up Devuan package mirrors. We assume that you know how to create and use an ssh key pair, how to use rsync, and how to setup a web server. (Evilham -- 20190522 -- added instructions for self-checks) (Evilham -- 20190505 -- added lighttpd instructions) (KatolaZ -- 20171224 -- included information about deb.devuan.org) (KatolaZ -- 20171031 -- first edition) == INDEX == 0) INTRODUCTION 1) YOU NEED AN SSH KEY PAIR 2) LET US KNOW ABOUT YOUR MIRROR, AND STAY INFORMED 3) EVERYBODY: PULL! 4) WHAT USERS WILL LOOK FOR 5) SETTING UP THE NEEDED REWRITE RULES 6) ENTERING THE deb.devuan.org DNS ROUND-ROBIN 7) CHECKING YOUR MIRROR ====== 0) INTRODUCTION The full Devuan package repository is available at pkgmaster.devuan.org All the mirrors will rsync from that main repo. The repo is contained in a folder "devuan/" which contains two more folders: merged/ : this is the folder containing the merged repo, i.e., the list of packages available in Devuan, where forked Devuan packages superseed the corresponding Debian package (if available) devuan/ : this folder contains the repos that are specific to Devuan, and the Devuan's /pool/ folder (i.e., the actual Devuan packages). At the time of writing (July 2019) the full Devuan package mirror requires about 50GB of space. Notice that this size is much smaller than the size of a full Debian repo (which is about 2TB). The reason is that, thanks to amprolla, Devuan can use all the standard Debian packages (coming from a standard Debian repo) together with Devuan-specific packages. This is achieved by rewriting the Filename: property of each package description, and implemented on the repository side by appropriately rewriting the URLs required by the client. Hence, any mirror needs to provide a few rewrite rules in order to be able to serve Devuan packages. See SECTION 5 below for further details. N.B.: The size of Devuan package repositories will most probably keep increasing in the future, so please take that into account when you decide to host a package mirror. 1) YOU NEED AN SSH KEY PAIR rsync access to pkgmaster.devuan.org is through ssh with public key. You can use an existing ssh key pair or instead create an ad-hoc key pair for your new Devuan mirror using: $ ssh-keygen 2) LET US KNOW ABOUT YOUR MIRROR, AND STAY INFORMED In order to be able to pull from pkgmaster.devuan.org, you need to have your ssh public key added to the list of authorised keys. Please send the public ssh key you wish to use to mirrors@devuan.org, together with the following information: - Name of the mirror contact person - Contact person email - mirror IP - mirror FQDN - Bandwidth limits (if any) - Disk limits (if any) - Mirror Country - supported protocols (HTTP/HTTPS/RSYNC/FTP) We try to guarantee a quick setup of new mirrors, and we will get back to you as soon as possible. Just please be patient :) In addition, we require that the mirror contacts subscribe the devuan-mirrors mailing list at the following address: devuan-mirrors@lists.dyne.org This is a low-traffic mailing list where all the important announcements about mirror configuration and issues are posted. 3) EVERYBODY: PULL! Once your ssh key has been set up, you should be able to rsync the whole package repository using: $ cd MIRROR_DIR $ rsync -avz vesta@pkgmaster.devuan.org:~/devuan ./ where MIRROR_DIR is the directory on your mirror where the "devuan/" folder will be placed. By doing so, you will have the new folders: ${MIRROR_DIR}/devuan/devuan ${MIRROR_DIR}/devuan/merged The same command should be run periodically (e.g., as a cron job), and normally no more than once every hour. Each mirror might be asked to set slightly different update rates, if the need arises. (N.B.: upon setting up your mirror, and only then, you might need to run rsync twice, since amprolla3 is merging every few minutes and in theory some files could have been changed during your first rsync of the whole stuff.) 4) WHAT USERS WILL LOOK FOR apt users will connect to your mirror via http/https. So you need to have a web server setup at your mirror FQDN. If ${MIRROR_DIR}/devuan/ is the ServerRoot of: your.dev.uan.doma.in the users of your mirror will look for stuff into: your.dev.uan.doma.in/merged your.dev.uan.doma.in/devuan In most of the cases, Devuan is not the only thing mirrored by a mirror site, hence your ServerRoot might be set to the parent folder of "devuan/" (i.e., ${MIRROR_DIR}). In these cases, the users of your mirror will look for stuff into: your.dev.uan.doma.in/devuan/merged your.dev.uan.doma.in/devuan/devuan Your configuration might differ a bit from either of the two above. But if that is the case, you know what we are talking about and you should be able to find a way around it. If you are unsure, just ask (e.g., either on the devuan-mirrors ML, or on the devuan-dev ML, or on #devuan-dev IRC channel. 5) SETTING UP THE NEEDED REWRITE RULES Now you have to configure your webserver to rewrite some of the destinations under ${MIRROR_DIR}/devuan/ so that users can find the packages. There are two cases: either your mirror has also a local Debian mirror under '${MIRROR_DIR}/debian/', or it does not. 5a) IF YOU HAVE A LOCAL DEBIAN MIRROR AT ${MIRROR_DIR}/debian/ If you have a local DEBIAN mirror and it is placed at $MIRROR_DIR/debian, and ${MIRROR_DIR} is your ServerRoot, then you will need to rewrite the following destinations: /devuan/merged/pool/DEVUAN/(.*) -> /devuan/devuan/pool/$1 /devuan/merged/pool/DEBIAN/(.*) -> /debian/pool/$1 /devuan/merged/pool/DEBIAN-SECURITY/(.*) -> /debian-security/pool/$1 For nginx, you will want to add the following section to the relevant server config: location /devuan/merged { autoindex on; rewrite /devuan/merged/pool/DEVUAN/(.*) /devuan/pool/$1; rewrite /devuan/merged/pool/DEBIAN-SECURITY/(.*) /debian-security/pool/$1; rewrite /devuan/merged/pool/DEBIAN/(.*) /debian/pool/$1; } For apache2, you will want to use something like the following instead: RewriteEngine on RewriteRule /devuan/merged/pool/DEVUAN/(.*) /devuan/pool/$1 RewriteRule /devuan/merged/pool/DEBIAN-SECURITY/(.*) /debian-security/pool/$1 RewriteRule /devuan/merged/pool/DEBIAN/(.*) /debian/pool/$1 (N.B.: These rewrite rules are just indicative. If you have already a Debian mirror, you know better than us where the stuff is to be found in your server. If you are unsure, just shout). 5b) IF YOU DON'T HAVE A LOCAL DEBIAN MIRROR If you don't have a local DEBIAN mirror, you will need to rewrite all the DEBIAN destinations to an appropriate DEBIAN MIRROR. We assume below that the "devuan/" folder is directly under your ServerRoot. We also assume that you choose "deb.debian.org" as Debian mirror. This is the recommended choice, since "deb.debian.org" is the entry point of the Debian mirror infrastructure, and should automatically redirect to the appropriate local Debian mirror. You need the following rewrites: /merged/pool/DEVUAN/(.*) -> /devuan/pool/$1 /merged/pool/DEBIAN/(.*) -> http://deb.debian.org/debian/pool/$1 /merged/pool/DEBIAN-SECURITY/(.*) -> http://deb.debian.org/debian-security/pool/$1 In nginx you will want to add the following section to the relevant server config: location /merged { autoindex on; rewrite /merged/pool/DEVUAN/(.*) /devuan/pool/$1; rewrite /merged/pool/DEBIAN-SECURITY/(.*) http://deb.debian.org/debian-security/pool/$1; rewrite /merged/pool/DEBIAN/(.*) http://deb.debian.org/debian/pool/$1; } For apache, instead: RewriteEngine on RewriteRule /devuan/merged/pool/DEVUAN/(.*) /devuan/pool/$1 RewriteRule /devuan/merged/pool/DEBIAN-SECURITY/(.*) http://deb.debian.org/debian-security/pool/$1 RewriteRule /devuan/merged/pool/DEBIAN/(.*) http://deb.debian.org/debian/pool/$1 For lighttpd, please check section 6.3) of this file. Again, those rewrites are just indicative. Similar instructions hold for setting up https mirrors. 6) ENTERING THE deb.devuan.org DNS ROUND-ROBIN We have put in place a DNS Round-Robin for the domain deb.devuan.org, which points to all the available package mirrors which can serve requests for the domanin "deb.devuan.org". The easiest to have your mirror added to the Round-Robin is to add a named VirtualHost to your web server conf to serve files for deb.devuan.org. Sample configuration files for apache and nginx are available under Section 6.1) and 6.2) below. Please amend them as necessary, and incorporate them in your webserver configuration. In particular, be careful in setting the document root and rewrite rules appropriately. *** IMPORTANT: THE DNS ROUND-ROBIN WILL NOT WORK FOR HTTPS *** It is nevertheless recommended to keep your mirror reachable in *both* ways, i.e., directly through your own URL and via deb.devuan.org, since we will also advertise a list of existing mirrors with their corresponding URL (and HTTPS is not supported through deb.devuan.org). As usual, please shout if you need help with this configuration. 6.1) Sample apache conf for a deb.devuan.org named virtual host ServerName deb.devuan.org #### the root must be the folder containing "amprolla.txt" DocumentRoot /home/mirror/devuan RewriteEngine on RewriteRule /merged/pool/DEVUAN/(.*) /devuan/pool/$1 RewriteRule /merged/pool/DEBIAN-SECURITY/(.*) http://deb.debian.org/debian-security/pool/$1 RewriteRule /merged/pool/DEBIAN/(.*) http://deb.debian.org/debian/pool/$1 6.2) Sample nginx conf for a deb.devuan.org named virtual host server { listen 80; listen [::]:80; server_name deb.devuan.org; ### the root must be the folder containing "amprolla.txt" root /home/mirror/devuan; default_type "text/plain"; location / { autoindex on; } location /merged { autoindex on; rewrite /merged/pool/DEVUAN/(.*) /devuan/pool/$1; rewrite /merged/pool/DEBIAN-SECURITY/(.*) http://deb.debian.org/debian-security/pool/$1; rewrite /merged/pool/DEBIAN/(.*) http://deb.debian.org/debian/pool/$1; } } 6.3) Sample lighttpd conf for a deb.devuan.org named virtual host $HTTP["scheme"] == "http" { $HTTP["host"] == "deb.devuan.org" { server.document-root = "/home/mirror/devuan" dir-listing.activate = "enable" url.redirect = ( "^/merged/pool/DEVUAN/(.*)" => "/devuan/pool/$1", "^/merged/pool/DEBIAN/(.*)" => "http://deb.debian.org/debian/pool/$1", "^/merged/pool/DEBIAN-SECURITY/(.*)" => "http://deb.debian.org/debian-security/pool/$1" ) } } 7) CHECKING YOUR MIRROR Check that your rewrite rules are working properly by using the provided scripts/test_rewrite.sh script. 7.a) UNDER YOUR OWN HOSTNAME # If $HOSTNAME contains both /devuan and /merged directories: $ scripts/test_rewrite.sh $HOSTNAME # It is also common to serve the Devuan mirror under a /devuan directory # that contains both devuan and merged directories: $ scripts/test_rewrite.sh $HOSTNAME/$BASEURL # To test HTTPS support, run as above making the protocol explicit: $ scripts/test_rewrite.sh https://$HOSTNAME[$BASEURL] 7.b) AS deb.devuan.org (ROUND ROBIN) # Regardless of how you serve the mirror under your hostname, # test as follows: $ scripts/test_rewrite.sh $HOSTNAME deb.devuan.org 7.c) TESTING RSYNC # Output of $ rsync $HOSTNAME::devuan # Should list this file along with the devuan, merged and scripts dirs.