Project

General

Profile

Mirroring Subversion Repositories

Since version 1.5, Subversion provides the write-through proxying feature, permitting to setting up local repository mirrors, while committing changes to the master repository.

Since version 1.4, Subversion provides the svnsync tool, permitting to synchronise subversion repositories. Though, integrating svnsync into the subversion hook scripts post-commit, pre-revprop-change and post-revprop-change, any changes on the master repository can be propagated to all mirror repositories immediately.

The write-through proxying feature of version 1.5 is not supported in version 1.4, but it can be emulated using some Apache rewrite rules, when subbversion is implemented within the Apache web server.

The most useful hints regarding the implementation of write-through proxying in version 1.4 have been found in the article Subversion DIY write-through-proxy

This document decribes the current setup of the repository synchronisation between the EDIT Developers repository":http://dev.e-taxonomy.eu/svn (master) and its "Shibboleth protected mirror (slave).

Create a mirror repository

First, we need to setup subversion on the mirror site and to mirror the current master repository to the slave site.

Instructions on how to setup am initial subversion repository can be found here. Since, we want to setup a Trac mirror too, we should use the same repository path as on the master repository.

svnadmin create /var/lib/svn/edit

Next, we can start mirroring the master repository to the slave. Therefore, we run the svnsync command on the slave.

svnsync init file:///var/lib/svn/edit http://dev.e-taxonomy.eu/svn

Depending on the size of your master repository, running this command will take some time. Finally, all revisions should be copied into the slave repository.

Since, we want to use the repository via the Apache web server, we should make the repository writable for the www-data group:

chown -R root:www-data /var/lib/svn/edit
chmod -R 770 /var/lib/svn/edit

Configure Apache's Subversion module

Prior to continue, please check if you have Apache2 and the corresponding Subversion module installed on the slave machine.

Additionally, we need to install a specific user account on the slave (e.g. svnsync) to restrict write access to the repository to the master repositories svnsync tool only. While, Shibboleth authentication is not supported by svnsync, the easiest way is to provide it with a local Apache user database. But, if you have already implemented another HTTP Basic authentication or SSL Certificate based authentication mechanism, you may create the new user there as well.

Here is how to create an Apache passwordfile.

htpasswd -cm /etc/subversion/svnsync-auth svnsync

Ensure, that your passwordfile is readable to the Apache web server(usually the user/group www-data).

Further on, we will have to configure Apache's subversion module in order to provide two access points to our subversion repository. One is for general user access to the slave repository, and one is for internale use of the svnsync command from the master repository.

The configuration file can be located either of in the subversion module configuration file /etc/modules-available/dav_svn.conf or within a site configuration file like /etc/sites-available/svn. Remember, you will need to enable either the module (a2enmod dav_svn) or the site (a2ensite svn) to activate your settings!

# Access to the subversion mirror and write-through-proxy
<Location /svn>
  DAV svn

  # Set this to the path to your repository
  SVNPath /var/lib/svn/edit

  # Use Shibboleth authentication
  AuthType           shibboleth
  ShibRequireSession On
  require            valid-user

  # Rewrite rules to enable write-through proxying to the master repository
  RewriteEngine on

  RewriteCond %{REQUEST_METHOD} !^GET$
  RewriteCond %{REQUEST_METHOD} !^PROPFIND$
  RewriteCond %{REQUEST_METHOD} !^OPTIONS$
  RewriteCond %{REQUEST_METHOD} !^REPORT$

  RewriteRule ^(.*)$ http://dev.e-taxonomy.eu%{REQUEST_URI} [P]
  ProxyPassReverse http://dev.e-taxonomy.eu/svn
</Location>

# Internal access for the master's svnsync
<Location /svn-internal>
  DAV svn

  # Set this to the path to your repository
  SVNPath /var/lib/svn/edit

  # Basic authentication using password file
  AuthType Basic
  AuthName "SVN sync only"
  AuthUserFile /etc/subversion/svnsync-auth

In order to use Apache's RewriteRule and ProxyPassReverse directives, enabling the Apache modules rewrite, proxy and proxy_http is required.

a2enmod rewrite
a2enmod proxy
a2enmod proxy_http

Propagating repository changes to the slave

The propagation of any changes to the master repository to the mirrors can be realised by activating the following subversion hooks on the master:

  • post-commit

  • pre-revprop-change

  • post-revprop-change

Subversion hooks are simply shell scripts run by subversion when certain events occur. They have to be located within the hooks subdirectory of the relating repository(i.e. /var/lib/svn/edit). There should be templates for every supported hook event within this directory, which can be used as a base.

Furthermore, they have to be readable and executable by the user running subversion. In our case, this is www-data, the user running the Apache web server.

chown -R root:www-data /var/lib/svn/edit/hooks
chmod -R 750 /var/lib/svn/edit/hooks

Next, hook scripts are started without any environment information. So, any paths to commands must be given completely.

The post-commit hook will be used to synchronise the repositories, whenever a svn commit command has succeeded successfully. Both *-revprop-change hooks will be executed, whenever changes to revision properties have been made.

Here are the relevant parts of our hooks

post-commit::

whoami

The hook synchronises master and slave repositories executing svnsync against the internal access path to the slave repository.

Uncommenting the interactive once may be useful when initialising the SSL connection to the slave repository.

pre-revprop-change::

#!/bin/sh

REPOS="$1"
REV="$2"
USER="$3"
PROPNAME="$4"

/bin/echo "pre-revprop $REPOS $REV $USER $PROPNAME" >> /var/log/svnsync.log
exit 0

Usually, this hook is used for access control before executing property changes. We don't need it, but the hook must be present in order to enable the post-revprop-change hook.

post-revprop-change::

#!/bin/sh

REPOS="$1"
REV="$2"
USER="$3"
PROPNAME="$4"
ACTION="$5"

/bin/echo "post-revprop $REPOS $REV $USER $PROPNAME $ACTION" >> /var/log/svnsync.log

/usr/bin/svnsync copy-revprops https://sp.e-taxonomy.eu/svn-internal $REV --username "svnsync" --password "secret" >>/var/log/svnsync.log 2>&1

This hook progates any successful property modifications to the slaves using the svnsync copy-revprops directive.

SSL configuration

Since, our Shibboleth repository mirror is accessible via SSL only, svnsync has to be configured to accept the SSL certificate sent by SSL enabled slave respositories. There are at least three possible ways doing so.

  1. Using svnsync in interactive mode

Running svnsync interactively provokes an user input request to accept the presented certificate or not. Entering 'p' instructs svnsync to permanently accept the certificate. This will be done automatically by activating the interactive mode and deactivating the non-interactive by uncommenting/commenting the lines within the post-commit hook script when activating the synchronisation hooks for the first time. This should be reverted for any subsequent commit actions.

  1. Configure /var/www/.subversion/auth

  2. Trust the certificates configured with OpenSSL (TODO: check this)

This can be activated setting the ssl-trust-default-ca variable to "yes" within subversions server configuration file. Then, any CA-certificates locating in the SSL certificate directory (usually /etc/ssl/certs) will be used to verify and accept certificates (e.g. of our SSL protected subversion mirror).

Internal Access Control on slaves

On slave side, the start-commit, pre-revprop-change hooks should be activated for security reasons. The following hook scripts enable the actions commit and revision property changes for the internal user svnsync only.

/var/lib/svn/edit/pre-revprop-change::

#!/bin/sh

REPOS="$1"
REV="$2"
USER="$3"
PROPNAME="$4"
ACTION="$5"

if [ "$USER" = "svnsync" ]; then exit 0; fi

# Just for test, comment it in production !!!
#exit 0

echo "Only the svnsync user can change revprops" >&2
exit 1

/var/lib/svn/edit/start-commit::

#!/bin/sh

REPOS="$1"
USER="$2"

if [ "$USER" = "svnsync" ]; then exit 0; fi

echo "Only the svnsync user may commit new revisions" >&2
exit 1

Check Installation

Finally, you can check your installation by committing new revisions to the master or slave repository.

Troubleshooting

Modifying repository's uuid

The repository's uuid is stores within the file _REPO_PATH/db/uuid.

You can retrieve the uuid of a repository with this command

svnlook uuid /var/lib/svn/edit

You can change the uuid of a repository by modifying the file db/uuid.

Add picture from clipboard (Maximum size: 40 MB)