docker-registry-proxy/docs/kops

How to use docker-registry-proxy with kops

Install docker-registry-proxy

For running docker-registry-proxy with kops you will need to run it outside the cluster you want to configure, you can either use and EC2 instance and run:

docker run --rm --name docker_registry_proxy -it \
       -p 0.0.0.0:3128:3128 -e ENABLE_MANIFEST_CACHE=true \
       -v $(pwd)/docker_mirror_cache:/docker_mirror_cache \
       -v $(pwd)/docker_mirror_certs:/ca \
       rpardini/docker-registry-proxy:0.6.0

or you can run it from another cluster, maybe a management/observability one with provided yaml, in this case, you will need to change the following lines:

  annotations:
    external-dns.alpha.kubernetes.io/hostname: docker-registry-proxy.<your_domain>
    service.beta.kubernetes.io/aws-load-balancer-internal: "true"

with the correct domain name, so then you can reference the proxy as http://docker-registry-proxy.<your_domain>:3128

Test the connection to the proxy

A simple curl should return:

❯ curl docker-registry-proxy.<your_domain>:3128
docker-registry-proxy: The docker caching proxy is working!%

Configure kops to use the proxy

Kops has the option to configure a cluster wide proxy, as explained here but this wont work, as nodeup will fail to download the images, what you need is to use additionalUserData, which is part of the instance groups configuration.

So consider a node configuration like this one:

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  labels:
    kops.k8s.io/cluster: spot.k8s.local
  name: spotgroup
spec:
  image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20200528
  machineType: c3.xlarge
  maxSize: 15
  minSize: 2
  mixedInstancesPolicy:
    instances:
    - c3.xlarge
    - c4.xlarge
    - c5.xlarge
    - c5a.xlarge
    onDemandAboveBase: 0
    onDemandBase: 0
    spotAllocationStrategy: capacity-optimized
  nodeLabels:
    kops.k8s.io/instancegroup: spotgroup
  role: Node
  subnets:
  - us-east-1a
  - us-east-1b
  - us-east-1c

you will need to add the following:

  additionalUserData:
    - name: docker-registry-proxy.sh
      type: text/x-shellscript
      content: |
        #!/bin/sh

        # Add environment vars pointing Docker to use the proxy
        # https://docs.docker.com/config/daemon/systemd/#httphttps-proxy

        mkdir -p /etc/systemd/system/docker.service.d
        cat << EOD > /etc/systemd/system/docker.service.d/http-proxy.conf
        [Service]
        Environment="HTTP_PROXY=http://docker-registry-proxy.<your_domain>:3128/"
        Environment="HTTPS_PROXY=http://docker-registry-proxy.<your_domain>:3128/"
        EOD

        # Get the CA certificate from the proxy and make it a trusted root.
        curl http://docker-registry-proxy.<your_domain>:3128/ca.crt > /usr/share/ca-certificates/docker_registry_proxy.crt
        echo "docker_registry_proxy.crt" >> /etc/ca-certificates.conf
        update-ca-certificates --fresh

        # Reload systemd
        systemctl daemon-reload

        # Restart dockerd
        systemctl restart docker.service

so the final InstanceGroup will look like this:

apiVersion: kops.k8s.io/v1alpha2
kind: InstanceGroup
metadata:
  labels:
    kops.k8s.io/cluster: spot.k8s.local
  name: spotgroup
spec:
  additionalUserData:
    - name: docker-registry-proxy.sh
      type: text/x-shellscript
      content: |
        #!/bin/sh

        # Add environment vars pointing Docker to use the proxy
        # https://docs.docker.com/config/daemon/systemd/#httphttps-proxy

        mkdir -p /etc/systemd/system/docker.service.d
        cat << EOD > /etc/systemd/system/docker.service.d/http-proxy.conf
        [Service]
        Environment="HTTP_PROXY=http://docker-registry-proxy.<your_domain>:3128/"
        Environment="HTTPS_PROXY=http://docker-registry-proxy.<your_domain>:3128/"
        EOD

        # Get the CA certificate from the proxy and make it a trusted root.
        curl http://docker-registry-proxy.<your_domain>:3128/ca.crt > /usr/share/ca-certificates/docker_registry_proxy.crt
        echo "docker_registry_proxy.crt" >> /etc/ca-certificates.conf
        update-ca-certificates --fresh

        # Reload systemd
        systemctl daemon-reload

        # Restart dockerd
        systemctl restart docker.service
  image: 099720109477/ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20200528
  machineType: c3.xlarge
  maxSize: 15
  minSize: 2
  mixedInstancesPolicy:
    instances:
    - c3.xlarge
    - c4.xlarge
    - c5.xlarge
    - c5a.xlarge
    onDemandAboveBase: 0
    onDemandBase: 0
    spotAllocationStrategy: capacity-optimized
  nodeLabels:
    kops.k8s.io/instancegroup: spotgroup
  role: Node
  subnets:
  - us-east-1a
  - us-east-1b
  - us-east-1c

Now all you need is to upgrade your cluster and do a rolling-update of the nodes, all images will be cached from now on.