mirror public repos gitlab

[frank-dspeed]

So for example, to mirror Linux, I first create a new project in the GitLab mirror group called linux (this would be accessed at something like https://gitlab/mirror/linux.git).

Then as the mrmirror user on GitLab I run a mirror clone:

[mrmirror@gitlab ~]$ cd ~/git/mirror
[mrmirror@gitlab mirror]$ git clone --mirror git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git

Then the script takes care of future updates and pushes them directly to GitLab via localhost:

#!/bin/bash
 
# Setup proxy for any https remotes
export http_proxy=http://proxy:3128
export https_proxy=http://proxy:3128
 
cd ~/git/mirror
 
for x in $(ls -d *.git) ; do
    pushd ${x}
    git remote prune origin
    git remote update -p
    git push --mirror git@localhost:mirror/${x}"
    popd
done
 
echo $(date) > /tmp/git_mirror_update.timestamp

That’s managed by a simple cronjob that the mrmirror user has on the GitLab server:

[mrmirror@gitlab mirror]$ crontab -l
0 */4 * * * /usr/local/bin/git_mirror_update.sh

And that seems to be working really well.

[frank-dspeed]

I installed git-daemon which includes systemd [email protected] and git.socket files. I copied these to make a new service called git-daemon, like so:

[root@gitlab ~]# cp /usr/lib/systemd/system/[email protected] \
/etc/systemd/system/[email protected]
[root@gitlab ~]# cp /usr/lib/systemd/system/git.socket \
/etc/systemd/system/git-daemon.socket
[root@gitlab ~]# systemctl daemon-reload

Then I edited the git-daemon.socket to point it to the git repositories, /var/opt/gitlab/git-data/repositories/, which is the default location when using the GitLab omnibus package.

[Unit]
Description=Git Repositories Server Daemon
Documentation=man:git-daemon(1)
 
[Service]
User=git
ExecStart=-/usr/libexec/git-core/git-daemon \
--base-path=/var/opt/gitlab/git-data/repositories/ \
--syslog --inetd --verbose
StandardInput=socket

start enable the service

[root@gitlab ~]# systemctl start git-daemon.socket
[root@gitlab ~]# systemctl enable **git-daemon.socket**

As per the git-daemon.service systemd file, you should now have git-daemon listening on port 9418, however you may need to open the port through the firewall:

# case firewalld
firewall-cmd --permanent --zone=public --add-port=9418/tcp
systemctl reload firewalld
# case ufw

Now, to enable git:// access to any given repository, you need to touch a file called git-daemon-export-ok in that repo’s git dir (it should be owned by your gitlab user, which is probably git). For example, a mirror of the Linux kernel:

 touch /var/opt/gitlab/git-data/repositories/mirror/linux.git/git-daemon-export-ok

Success!

If you wanted to, you could set up a cron job to make sure that any new mirrors that come along are exported without manual intervention.

First, create an executable script somewhere, like /usr/local/bin/export_git-daemon_repos.sh (note, this excludes any wiki git repos).

#!/bin/bash
 
set -eo pipefail
 
if [[ "$USER" != "git" ]]; then
    echo "Only run this as the git user."
    exit 1
fi
 
cd /var/opt/gitlab/git-data/repositories/mirror
for x in $(ls -d * |grep -v \.wiki\.git) ; do
    pushd ${x}
    if [[ ! -e "git-daemon-export-ok" ]]; then
        touch git-daemon-export-ok
    fi
    popd
done

Then add it as a cron job for the git user on your gitlab server to run every two hours, or whatever suits you, e.g.:

-sh-4.2$ crontab -l
0 */2 * * * /usr/local/bin/export_git-daemon_repos.sh >/dev/null