How does Docker Layer Cache (DLC) work on CircleCI
This is an attempt to explain CircleCI's Docker Layer Cache (DLC) feature in simple terms.
What is Docker Layer Cache
Docker creates an image via layers.
Each layer is created via Dockerfile commands like RUN, COPY and ADD, and they represent a unit of filesystem change.
When Docker builds an image, the built layers are stored.
Rebuilding the same image becomes faster because Docker can now retrieve the stored layers as cache.
This is known as layer caching on Docker, or Docker Layer Cache (DLC).
DLC is useful when we are rebuilding the same layers.
It works well when we build images on the same machine;
The stored layers are still there when we build a second time.
However, this becomes tricky on CI/CD systems, since most CI/CD pipelines run on ephemeral environments. The execution environment is destroyed once your pipeline completes.
How does CircleCI implement DLC
For DLC to work on CI/CD systems, we need to ensure stored/cached layers can be shared between CI/CD pipelines.
To achieve this, CircleCI assigns an external volume (store) to a job when we enable DLC.
Think of it like passing around an external hard drive amongst friends.
This external volume will store the built Docker images' layers in a job.
When CircleCI assigns same external volume to the next job, we should see faster image builds since the cached layers can be found.
An external volume can be assigned only to one job at a time.
Hence, a job may see cache misses when building images due to a different volume being assigned.
A cache miss will mean a longer time taken to build the Docker image, which translates to a longer job duration.
However, it does not and should not cause a job on CircleCI to fail.