ENABLE_MANIFEST_CACHING make it impossible to do roolups more freq than
once per manifest cache interval. Add exclude list, so that manifest
caching isn't applied for registries in this list.
After building the debug image and trying to run it, the mitproxy
would fail to start:
Traceback (most recent call last):
File "/usr/bin/mitmweb", line 8, in <module>
sys.exit(mitmweb())
File "/usr/lib/python3.8/site-packages/mitmproxy/tools/_main.py", line 172, in mitmweb
from mitmproxy.tools import web
File "/usr/lib/python3.8/site-packages/mitmproxy/tools/web/__init__.py", line 1, in <module>
from mitmproxy.tools.web import master
File "/usr/lib/python3.8/site-packages/mitmproxy/tools/web/master.py", line 5, in <module>
from mitmproxy import addons
File "/usr/lib/python3.8/site-packages/mitmproxy/addons/__init__.py", line 12, in <module>
from mitmproxy.addons import onboarding
File "/usr/lib/python3.8/site-packages/mitmproxy/addons/onboarding.py", line 2, in <module>
from mitmproxy.addons.onboardingapp import app
File "/usr/lib/python3.8/site-packages/mitmproxy/addons/onboardingapp/__init__.py", line 3, in <module>
from flask import Flask, render_template
File "/usr/lib/python3.8/site-packages/flask/__init__.py", line 14, in <module>
from jinja2 import escape
File "/usr/lib/python3.8/site-packages/jinja2/__init__.py", line 12, in <module>
from .environment import Environment
File "/usr/lib/python3.8/site-packages/jinja2/environment.py", line 25, in <module>
from .defaults import BLOCK_END_STRING
File "/usr/lib/python3.8/site-packages/jinja2/defaults.py", line 3, in <module>
from .filters import FILTERS as DEFAULT_FILTERS # noqa: F401
File "/usr/lib/python3.8/site-packages/jinja2/filters.py", line 13, in <module>
from markupsafe import soft_unicode
ImportError: cannot import name 'soft_unicode' from 'markupsafe' (/usr/lib/python3.8/site-packages/markupsafe/__init__.py)
Fixed the issue by explicitly requiring an older version of MarkupSafe.
The issue is related to:
https://github.com/pallets/markupsafe/issues/282
The proxy refused to connect to a registry that was hosted on
a port other than 443. For example, I was not able to connect
to my registry that is hosted on port 5002:
$ https_proxy=proxy.lab.example.com:3128 curl -v https://registry.lab.example.com:5002
* Uses proxy env variable https_proxy == 'proxy.lab.example.com:3128'
* Trying 192.168.140.1:3128...
* Connected to proxy.lab.example.com (192.168.140.1) port 3128 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to registry.lab.example.com:5002
> CONNECT registry.lab.example.com:5002 HTTP/1.1
> Host: registry.lab.example.com:5002
> User-Agent: curl/7.74.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 403 Forbidden
< Server: nginx/1.20.1
< Date: Thu, 11 Aug 2022 15:12:23 GMT
< Content-Type: text/html
< Content-Length: 153
< Connection: keep-alive
<
* Received HTTP code 403 from proxy after CONNECT
* CONNECT phase completed!
* Closing connection 0
curl: (56) Received HTTP code 403 from proxy after CONNECT
The proxy refused to pass through connections to URLs that used
port other than 443. For example, trying to connect to port 8443:
$ https_proxy=proxy.lab.example.com:3128 curl -v https://google.com:8443
* Uses proxy env variable https_proxy == 'proxy.lab.example.com:3128'
* Trying 192.168.140.1:3128...
* Connected to proxy.lab.example.com (192.168.140.1) port 3128 (#0)
* allocate connect buffer!
* Establish HTTP proxy tunnel to google.com:8443
> CONNECT google.com:8443 HTTP/1.1
> Host: google.com:8443
> User-Agent: curl/7.74.0
> Proxy-Connection: Keep-Alive
>
< HTTP/1.1 403 Forbidden
< Server: nginx/1.20.1
< Date: Thu, 11 Aug 2022 16:05:52 GMT
< Content-Type: text/html
< Content-Length: 153
< Connection: keep-alive
<
* Received HTTP code 403 from proxy after CONNECT
* CONNECT phase completed!
* Closing connection 0
curl: (56) Received HTTP code 403 from proxy after CONNECT
This commit fixes the issue by configuring the proxy_connect_allow
paramater to allow connecting to any destination port number. By
default only port 443 and 563 were allowed. See also documentation
here:
https://github.com/chobits/ngx_http_proxy_connect_module#proxy_connect_allow
If push is allowed, buffering requests can cause issues on slow upstreams.
The put requests pile up, and since `proxy_ignore_client_abort` is on...
...the push fails.
If you have trouble pushing, set `PROXY_REQUEST_BUFFERING`to false, then fix
remainig issues/timeouts.
Default is true to not change default behavior.
* implement manifest caching; refactor config with includes, and generate from ENVs in entrypoint.sh
- disabled by default; enable with -e ENABLE_MANIFEST_CACHE=true
- default times and regexes are a wild guess, make sure to tune for your use case.
- add manifest caching/anti-ratelimit usage note to README
- add -e ENABLE_MANIFEST_CACHE=true to examples, some wording changes
- add -e ENABLE_MANIFEST_CACHE=true to one the steps in test workflow.
- injects a mitmproxy between the caching layer and DockerHub's registry-1.docker.io
- it also disables VERIFY_SSL so the mitmproxy can be used;
- DEBUG_HUB's mitmweb is exposed on port 8082
- both DEBUG and DEBUG_HUB can be enabled independently
Alex Sheplyakov
Ales Nosek
saces
Mahmoud Saada
Ricardo Pardini
Léo
Akos Dudas
Alex Thomae
Mateusz Kowalski
Sezer Bülent
Federico Gimenez
hishamanver
Jye Lee
Marcelo Grebois
naftulee