Sep 05, 2018 Hi, I have been trying out the BuildKit feature built into Docker 18.06, but building with DOCKERBUILDKIT=1 seems to cause my builds to fail since I am behind a firewall and need to access the docker hub registry through a local mirror. The first time you request an image from your local registry mirror, it pulls the image from the public Docker registry and stores it locally before handing it back to you. On subsequent requests, the local registry mirror is able to serve the image from its own storage. What if the content changes on the Hub? If you run the registry as a container, consider adding the flag -p 443:5000 to the docker run command or using a similar setting in a cloud configuration. You should also set the hosts option to the list of hostnames that are valid for this registry to avoid trying to get certificates for random hostnames due to malicious clients connecting. Docker Content Trust (DCT) and Registry Mirrors? 29th April 2021 docker, docker-proxy, docker-registry Does someone know, if a client that pulls from a registry mirror of DockerHub can have DCT enabled? The problem is the command: $ docker -registry-mirror=-d Is intended for configuring the Docker daemon, not the Docker client.

TL;DR:

• to create a local pull-through registry to speed up image pulling in a Kind cluster, run:

• you can’t use this pull-through proxy registry to push your own images (e.g. to speed up Tilt builds), but you can create two registries (one for caching, the other for local images). See this section for more context; the lines are:

• in case you often create & delete Kind clusters, using a local registry that serves as a proxy avoids redundant downloads

• KIND_EXPERIMENTAL_DOCKER_NETWORK is useful but remember that the default network (bridge) doesn’t have DNS resolution for container hostnames

• the Docker default network (bridge) has limitations as detailed by Docker.

• If you play with ClusterAPI with its Docker provider, you might not be able to use a local registry due to the clusters being created on the default network, which means the “proxy” hostname won’t be resolved (but we could work around that).

Kind is an awesome tool that allows you to spin up local Kubernetes clusters locally in seconds. It is perfect for Kubernetes developers or anyone who wants to play with controllers.

One thing I hate about Kind is that images are not cached between two Kind containers. Even worse: when deleting and re-creating a cluster, all the downloaded images disappear.

In this post, I detail my discoveries around local registries and why the default Docker network is a trap.

Contents:

Kind has no image caching mechanism

Whenever I re-create a Kind cluster and try to install ClusterAPI, all the (quite heavy) images have to be re-downloaded. Just take a look at all the images that get re-downloaded:

That’s a total of 418 MB that get re-downloaded every time I restart both clusters!

Unfortunately, there is no way to re-use the image registry built into your default Docker engine (both on Linux and on macOS). One solution to this problem is to spin up an intermediary Docker registry in a side container; as long as this container exists, all the images that have already been downloaded once can be served from cache.

Creating a caching proxy registry

We want to create a registry with a simple Kind cluster; let’s start with the registry:

Details:

• --net kind is required because Kind creates its containers in a separate network; it does that the because the “bridge” has limitations and doesn’t allow you to use container names as DNS names:

  By default, a container inherits the DNS settings of the host, as defined in the /etc/resolv.conf configuration file. Containers that use the default bridge network get a copy of this file, whereas containers that use a custom network use Docker’s embedded DNS server, which forwards external DNS lookups to the DNS servers configured on the host.

  which means that the container runtime (containerd) that runs our Kind cluster won’t be able to resove the address proxy:5000.

• REGISTRY_PROXY_REMOTEURL is required due to the fact that by default, the registry won’t forward requests. It simply tries to find the image in /var/lib/registry/docker/registry/v2/repositories and returns 404 if it doesn’t find it.

  Using the pull-through feature (I call it “caching proxy”), the registry will proxy all requests coming from all mirror prefixes and cache the blobs and manifests locally. To enable this feature, we set REGISTRY_PROXY_REMOTEURL.

  Other interesting bit about REGISTRY_PROXY_REMOTEURL: this environement variable name is mapped from the registry YAML config API. The variable

  is equivalent to the following YAML config:

  ⚠️ The registry can’t be both in normal mode (“local proxy”) and in caching proxy mode at the same time, see below.

Creating a Kind cluster that knows about this caching proxy registry

The second step is to create a Kind cluster and tell the container runtime to use a specific registry; here is the command to create it:

Note:containerdConfigPatches is a way to semantically patch /etc/containerd/config.conf. By default, this file looks like:

Note 2: the mirror prefix (docker.io) can be omitted for images stored on Docker Hub. For other registries such as gcr.io, this mirror prefix has to be given. Here is a table with some examples of image names that are first prepended with “docker.io” if the mirror prefix is not present, and we get the final address by mapping these mirror prefixes with mirror entries:

Check that the caching proxy registry works

Let’s see if the proxy registry works by running a pod:

We can also see through the registry logs that everything is going well:

Docker proxy vs. local registry

Docker Registry Mirrors

A bit later, I discovered that you can’t push to a proxy registry. Tilt is a tool I use to ease the process of developping in a containerized environment (and it works best with Kubernetes); it relies on a local registry in order to cache build containers even when restarting the Kind cluster.

Either the registry is used as a “local registry” (where you can push images), or it is used as a pull-through proxy. So instead of configuring one single “proxy” registry, I configure two registries: one for local images, one for caching.

Note that we do use a port-forwarding proxy (-p 5000:5000) so that we can push images “from the host”, e.g.:

If you use Tilt, you might also want to tell Tilt that it can use the local registry. I find it a bit weird to have to set an annotation (hidden Tilt API?) but whatever. If you set this:

then Tilt will use docker push localhost:5000/you-image (from your host, not from the cluster container) in order to speed up things. Note that there is a proposal (KEP 1755) that aims at standardizing the discovery of local registries using a configmap. Tilt already supports it, so you may use it!

Improving the ClusterAPI docker provider to use a given network

Docker Registry Mirrors For Sale

When I play with ClusterAPI, I usually use the CAPD provider (ClusterAPI Provider Docker). This provider is kept in-tree inside the cluster-api projet.

Docker Registry Repository

I want to use the caching mechanism presented above. But to do that, I need to make sure the clusters created by CAPD are not created on the default network (current implementation creates CAPD clusters on the default “bridge” network).

I want to be able to customize the network on which the CAPD provider creates the container that make up the cluster. Imagine that we could pass the network name as part of a DockerMachineTemplate (the content of the spec is defined in code here):

Update 26 July 2020: added a section about local registry vs. caching proxy. Reworked the whole post (less noise, more useful information).