RPJosh
/
docker-registry-proxy
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

3-tier implementation of manifest caching (#57) 

* implement manifest caching; refactor config with includes, and generate from ENVs in entrypoint.sh
- disabled by default; enable with -e ENABLE_MANIFEST_CACHE=true
- default times and regexes are a wild guess, make sure to tune for your use case.
-  add manifest caching/anti-ratelimit usage note to README
- add -e ENABLE_MANIFEST_CACHE=true to examples, some wording changes
- add -e ENABLE_MANIFEST_CACHE=true to one the steps in test workflow.
pull/64/head
Ricardo Pardini 2020-10-30 18:54:10 +01:00 committed by GitHub
parent 227a397225
commit a726f88049
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
7 changed files with 138 additions and 47 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
.github/workflows/test.yaml vendored
View File

@ -53,10 +53,10 @@ jobs:
cache-from: type=local,src=/tmp/.buildx-cache/release
# this only reads from the cache
- name: Start proxy instance in docker
- name: Start proxy instance in docker (ENABLE_MANIFEST_CACHE=false)
run: |
docker run -d --rm --name docker_registry_proxy \
-p 0.0.0.0:3128:3128 \
-p 0.0.0.0:3128:3128 -e ENABLE_MANIFEST_CACHE=false \
-v $(pwd)/docker_mirror_cache:/docker_mirror_cache \
-v $(pwd)/docker_mirror_certs:/ca \
sanity-check/docker-registry-proxy:latest
@ -115,10 +115,10 @@ jobs:
run: |
sudo systemctl restart docker.service
- name: Start proxy instance in docker again
- name: Start proxy instance in docker again (ENABLE_MANIFEST_CACHE=true)
run: |
docker run -d --rm --name docker_registry_proxy \
-p 0.0.0.0:3128:3128 \
-p 0.0.0.0:3128:3128 -e ENABLE_MANIFEST_CACHE=true \
-v $(pwd)/docker_mirror_cache:/docker_mirror_cache \
-v $(pwd)/docker_mirror_certs:/ca \
sanity-check/docker-registry-proxy:latest

24
Dockerfile
View File

@ -42,6 +42,8 @@ VOLUME /ca
# Add our configuration
ADD nginx.conf /etc/nginx/nginx.conf
ADD nginx.manifest.common.conf /etc/nginx/nginx.manifest.common.conf
ADD nginx.manifest.stale.conf /etc/nginx/nginx.manifest.stale.conf
# Add our very hackish entrypoint and ca-building scripts, make them executable
ADD entrypoint.sh /entrypoint.sh
@ -70,5 +72,27 @@ ENV DEBUG_HUB="false"
# Enable nginx debugging mode; this uses nginx-debug binary and enabled debug logging, which is VERY verbose so separate setting
ENV DEBUG_NGINX="false"
# Manifest caching tiers. Disabled by default, to mimick 0.4/0.5 behaviour.
# Setting it to true enables the processing of the ENVs below.
# Once enabled, it is valid for all registries, not only DockerHub.
# The envs *_REGEX represent a regex fragment, check entrypoint.sh to understand how they're used (nginx ~ location, PCRE syntax).
ENV ENABLE_MANIFEST_CACHE="false"
# 'Primary' tier defaults to 10m cache for frequently used/abused tags.
# - People publishing to production via :latest (argh) will want to include that in the regex
# - Heavy pullers who are being ratelimited but don't mind getting outdated manifests should (also) increase the cache time here
ENV MANIFEST_CACHE_PRIMARY_REGEX="(stable|nightly|production|test)"
ENV MANIFEST_CACHE_PRIMARY_TIME="10m"
# 'Secondary' tier defaults any tag that has 3 digits or dots, in the hopes of matching most explicitly-versioned tags.
# It caches for 60d, which is also the cache time for the large binary blobs to which the manifests refer.
# That makes them effectively immutable. Make sure you're not affected; tighten this regex or widen the primary tier.
ENV MANIFEST_CACHE_SECONDARY_REGEX="(.*)(\d|\.)+(.*)(\d|\.)+(.*)(\d|\.)+"
ENV MANIFEST_CACHE_SECONDARY_TIME="60d"
# The default cache duration for manifests that don't match either the primary or secondary tiers above.
# In the default config, :latest and other frequently-used tags will get this value.
ENV MANIFEST_CACHE_DEFAULT_TIME="1h"
# Did you want a shell? Sorry, the entrypoint never returns, because it runs nginx itself. Use 'docker exec' if you need to mess around internally.
ENTRYPOINT ["/entrypoint.sh"]

53
README.md
View File

@ -6,6 +6,48 @@
## TL,DR
A caching proxy for Docker; allows centralised management of (multiple) registries and their authentication; caches images from *any* registry.
Caches the potentially huge blob/layer requests (for bandwidth/time savings), and optionally caches manifest requests ("pulls") to avoid rate-limiting.
### NEW: avoiding DockerHub Pull Rate Limits with Caching
Starting November 2nd, 2020, DockerHub will
[supposedly](https://www.docker.com/blog/docker-hub-image-retention-policy-delayed-and-subscription-updates/)
[start](https://www.docker.com/blog/scaling-docker-to-serve-millions-more-developers-network-egress/)
[rate-limiting pulls](https://docs.docker.com/docker-hub/download-rate-limit/),
also known as the _Docker Apocalypse_.
The main symptom is `Error response from daemon: toomanyrequests: Too Many Requests. Please see https://docs.docker.com/docker-hub/download-rate-limit/` during pulls.
Many unknowing Kubernetes clusters will hit the limit, and struggle to configure `imagePullSecrets` and `imagePullPolicy`.
Since version `0.6.0`, this proxy can be configured with the env var `ENABLE_MANIFEST_CACHE=true` which provides
configurable caching of the manifest requests that DockerHub throttles. You can then fine-tune other parameters to your needs.
Together with the possibility to centrally inject authentication (since 0.3x), this is probably one of the best ways to bring relief to your distressed cluster, while at the same time saving lots of bandwidth and time.
Note: enabling manifest caching, in its default config, effectively makes some tags **immutable**. Use with care. The configuration ENVs are explained in the [Dockerfile](./Dockerfile), relevant parts included below.
```dockerfile
# Manifest caching tiers. Disabled by default, to mimick 0.4/0.5 behaviour.
# Setting it to true enables the processing of the ENVs below.
# Once enabled, it is valid for all registries, not only DockerHub.
# The envs *_REGEX represent a regex fragment, check entrypoint.sh to understand how they're used (nginx ~ location, PCRE syntax).
ENV ENABLE_MANIFEST_CACHE="false"
# 'Primary' tier defaults to 10m cache for frequently used/abused tags.
# - People publishing to production via :latest (argh) will want to include that in the regex
# - Heavy pullers who are being ratelimited but don't mind getting outdated manifests should (also) increase the cache time here
ENV MANIFEST_CACHE_PRIMARY_REGEX="(stable|nightly|production|test)"
ENV MANIFEST_CACHE_PRIMARY_TIME="10m"
# 'Secondary' tier defaults any tag that has 3 digits or dots, in the hopes of matching most explicitly-versioned tags.
# It caches for 60d, which is also the cache time for the large binary blobs to which the manifests refer.
# That makes them effectively immutable. Make sure you're not affected; tighten this regex or widen the primary tier.
ENV MANIFEST_CACHE_SECONDARY_REGEX="(.*)(\d|\.)+(.*)(\d|\.)+(.*)(\d|\.)+"
ENV MANIFEST_CACHE_SECONDARY_TIME="60d"
# The default cache duration for manifests that don't match either the primary or secondary tiers above.
# In the default config, :latest and other frequently-used tags will get this value.
ENV MANIFEST_CACHE_DEFAULT_TIME="1h"
```
## What?
@ -37,6 +79,7 @@ for this to work it requires inserting a root CA certificate into system trusted
- Map volume `/docker_mirror_cache` for up to `CACHE_MAX_SIZE` (32gb by default) of cached images across all cached registries
- Map volume `/ca`, the proxy will store the CA certificate here across restarts. **Important** this is security sensitive.
- Env `CACHE_MAX_SIZE` (default `32g`): set the max size to be used for caching local Docker image layers. Use [Nginx sizes](http://nginx.org/en/docs/syntax.html).
- Env `ENABLE_MANIFEST_CACHE`, see the section on pull rate limiting.
- Env `REGISTRIES`: space separated list of registries to cache; no need to include DockerHub, its already done internally.
- Env `AUTH_REGISTRIES`: space separated list of `hostname:username:password` authentication info.
- `hostname`s listed here should be listed in the REGISTRIES environment as well, so they can be intercepted.
@ -46,7 +89,7 @@ for this to work it requires inserting a root CA certificate into system trusted
### Simple (no auth, all cache)
```bash
docker run --rm --name docker_registry_proxy -it \
-p 0.0.0.0:3128:3128 \
-p 0.0.0.0:3128:3128 -e ENABLE_MANIFEST_CACHE=true \
-v $(pwd)/docker_mirror_cache:/docker_mirror_cache \
-v $(pwd)/docker_mirror_certs:/ca \
rpardini/docker-registry-proxy:0.5.0
@ -60,7 +103,7 @@ For Docker Hub authentication:
```bash
docker run --rm --name docker_registry_proxy -it \
-p 0.0.0.0:3128:3128 \
-p 0.0.0.0:3128:3128 -e ENABLE_MANIFEST_CACHE=true \
-v $(pwd)/docker_mirror_cache:/docker_mirror_cache \
-v $(pwd)/docker_mirror_certs:/ca \
-e REGISTRIES="k8s.gcr.io gcr.io quay.io your.own.registry another.public.registry" \
@ -88,7 +131,7 @@ For GitLab.com itself the authentication domain should be `gitlab.com`.
```bash
docker run --rm --name docker_registry_proxy -it \
-p 0.0.0.0:3128:3128 \
-p 0.0.0.0:3128:3128 -e ENABLE_MANIFEST_CACHE=true \
-v $(pwd)/docker_mirror_cache:/docker_mirror_cache \
-v $(pwd)/docker_mirror_certs:/ca \
-e REGISTRIES="reg.example.com git.example.com" \
@ -109,7 +152,7 @@ Example with GCR using credentials from a service account from a key file `servi
```bash
docker run --rm --name docker_registry_proxy -it \
-p 0.0.0.0:3128:3128 \
-p 0.0.0.0:3128:3128 -e ENABLE_MANIFEST_CACHE=true \
-v $(pwd)/docker_mirror_cache:/docker_mirror_cache \
-v $(pwd)/docker_mirror_certs:/ca \
-e REGISTRIES="k8s.gcr.io gcr.io quay.io your.own.registry another.public.registry" \
@ -172,7 +215,7 @@ This allows very in-depth debugging. Use sparingly, and definitely not in produc
```bash
docker run --rm --name docker_registry_proxy -it
-e DEBUG_NGINX=true -e DEBUG=true -e DEBUG_HUB=true -p 0.0.0.0:8081:8081 -p 0.0.0.0:8082:8082 \
-p 0.0.0.0:3128:3128 \
-p 0.0.0.0:3128:3128 -e ENABLE_MANIFEST_CACHE=true \
-v $(pwd)/docker_mirror_cache:/docker_mirror_cache \
-v $(pwd)/docker_mirror_certs:/ca \
rpardini/docker-registry-proxy:0.5.0-debug

43
entrypoint.sh
View File

@ -78,6 +78,49 @@ CACHE_MAX_SIZE=${CACHE_MAX_SIZE:-32g}
# Set to 32gb which should be enough
echo "proxy_cache_path /docker_mirror_cache levels=1:2 max_size=$CACHE_MAX_SIZE inactive=60d keys_zone=cache:10m use_temp_path=off;" > /etc/nginx/conf.d/cache_max_size.conf
# Manifest caching configuration. We generate config based on the environment vars.
echo -n "" >/etc/nginx/nginx.manifest.caching.config.conf
[[ "a${ENABLE_MANIFEST_CACHE}" == "atrue" ]] && [[ "a${MANIFEST_CACHE_PRIMARY_REGEX}" != "a" ]] && cat <<EOD >>/etc/nginx/nginx.manifest.caching.config.conf
# First tier caching of manifests; configure via MANIFEST_CACHE_PRIMARY_REGEX and MANIFEST_CACHE_PRIMARY_TIME
location ~ ^/v2/(.*)/manifests/${MANIFEST_CACHE_PRIMARY_REGEX} {
set \$docker_proxy_request_type "manifest-primary";
proxy_cache_valid ${MANIFEST_CACHE_PRIMARY_TIME};
include "/etc/nginx/nginx.manifest.stale.conf";
}
EOD
[[ "a${ENABLE_MANIFEST_CACHE}" == "atrue" ]] && [[ "a${MANIFEST_CACHE_SECONDARY_REGEX}" != "a" ]] && cat <<EOD >>/etc/nginx/nginx.manifest.caching.config.conf
# Secondary tier caching of manifests; configure via MANIFEST_CACHE_SECONDARY_REGEX and MANIFEST_CACHE_SECONDARY_TIME
location ~ ^/v2/(.*)/manifests/${MANIFEST_CACHE_SECONDARY_REGEX} {
set \$docker_proxy_request_type "manifest-secondary";
proxy_cache_valid ${MANIFEST_CACHE_SECONDARY_TIME};
include "/etc/nginx/nginx.manifest.stale.conf";
}
EOD
[[ "a${ENABLE_MANIFEST_CACHE}" == "atrue" ]] && cat <<EOD >>/etc/nginx/nginx.manifest.caching.config.conf
# Default tier caching for manifests. Caches for ${MANIFEST_CACHE_DEFAULT_TIME} (from MANIFEST_CACHE_DEFAULT_TIME)
location ~ ^/v2/(.*)/manifests/ {
set \$docker_proxy_request_type "manifest-default";
proxy_cache_valid ${MANIFEST_CACHE_DEFAULT_TIME};
include "/etc/nginx/nginx.manifest.stale.conf";
}
EOD
[[ "a${ENABLE_MANIFEST_CACHE}" != "atrue" ]] && cat <<EOD >>/etc/nginx/nginx.manifest.caching.config.conf
# Manifest caching is disabled. Enable it with ENABLE_MANIFEST_CACHE=true
location ~ ^/v2/(.*)/manifests/ {
set \$docker_proxy_request_type "manifest-default-disabled";
proxy_cache_valid 0s;
include "/etc/nginx/nginx.manifest.stale.conf";
}
EOD
echo "Manifest caching config: ---"
cat /etc/nginx/nginx.manifest.caching.config.conf
echo "---"
# normally use non-debug version of nginx
NGINX_BIN="/usr/sbin/nginx"

44
nginx.conf
View File

@ -267,57 +267,27 @@ echo "Docker configured with HTTPS_PROXY=$scheme://$http_host/"
# For blob requests by digest, do cache, and treat redirects.
location ~ ^/v2/(.*)/blobs/sha256:(.*) {
set $docker_proxy_request_type "blob-by-digest";
add_header X-Docker-Registry-Proxy-Cache-Upstream-Status "$upstream_cache_status";
add_header X-Docker-Registry-Proxy-Cache-Type "$docker_proxy_request_type";
proxy_pass https://$targetHost;
proxy_cache cache;
proxy_cache_key $uri;
proxy_intercept_errors on;
error_page 301 302 307 = @handle_redirects;
include "/etc/nginx/nginx.manifest.common.conf";
}
# For manifest requests by digest, do cache, and treat redirects.
# These are some of the requests that DockerHub will throttle.
location ~ ^/v2/(.*)/manifests/sha256:(.*) {
set $docker_proxy_request_type "manifest-by-digest";
add_header X-Docker-Registry-Proxy-Cache-Upstream-Status "$upstream_cache_status";
add_header X-Docker-Registry-Proxy-Cache-Type "$docker_proxy_request_type";
proxy_pass https://$targetHost;
proxy_cache cache;
proxy_cache_key $uri;
proxy_intercept_errors on;
error_page 301 302 307 = @handle_redirects;
include "/etc/nginx/nginx.manifest.common.conf";
}
# Cache manifest requests that are not by digest (e.g. tags)
# Since these are mutable, we invalidate them immediately and keep them only in case the backend is down
# These are some of the requests that DockerHub will throttle.
location ~ ^/v2/(.*)/manifests/ {
set $docker_proxy_request_type "manifest-mutable";
add_header X-Docker-Registry-Proxy-Cache-Upstream-Status "$upstream_cache_status";
add_header X-Docker-Registry-Proxy-Cache-Type "$docker_proxy_request_type";
proxy_pass https://$targetHost;
proxy_cache cache;
proxy_cache_key $uri;
proxy_intercept_errors on;
proxy_cache_use_stale error timeout http_500 http_502 http_504 http_429;
proxy_cache_valid 0s;
error_page 301 302 307 = @handle_redirects;
}
# Config for manifest URL caching is generated by the entrypoint based on ENVs.
# Go check it out, entrypoint.sh
include "/etc/nginx/nginx.manifest.caching.config.conf";
# Cache blobs requests that are not by digest
# Since these are mutable, we invalidate them immediately and keep them only in case the backend is down
location ~ ^/v2/(.*)/blobs/ {
set $docker_proxy_request_type "blob-mutable";
add_header X-Docker-Registry-Proxy-Cache-Upstream-Status "$upstream_cache_status";
add_header X-Docker-Registry-Proxy-Cache-Type "$docker_proxy_request_type";
proxy_pass https://$targetHost;
proxy_cache cache;
proxy_cache_key $uri;
proxy_intercept_errors on;
proxy_cache_use_stale error timeout http_500 http_502 http_504 http_429;
proxy_cache_valid 0s;
error_page 301 302 307 = @handle_redirects;
include "/etc/nginx/nginx.manifest.stale.conf";
}
location @handle_redirects {

8
nginx.manifest.common.conf 100644
View File

@ -0,0 +1,8 @@
# nginx config fragment included in every manifest-related location{} block.
add_header X-Docker-Registry-Proxy-Cache-Upstream-Status "$upstream_cache_status";
add_header X-Docker-Registry-Proxy-Cache-Type "$docker_proxy_request_type";
proxy_pass https://$targetHost;
proxy_cache cache;
proxy_cache_key $uri;
proxy_intercept_errors on;
error_page 301 302 307 = @handle_redirects;

3
nginx.manifest.stale.conf 100644
View File

@ -0,0 +1,3 @@
# Just like the common block, but adds proxy_cache_use_stale
include "/etc/nginx/nginx.manifest.common.conf";
proxy_cache_use_stale error timeout http_500 http_502 http_504 http_429;