Gitlab Ci Install Docker

  1. Different approaches to building Docker images with Gitlab CI, drawbacks, and solutions. My most popular blog post so far is Setting up a Docker image builder with Gitlab CI Runner. The focus of that post was how you could set up a relatively secure Docker build environment using Gitlab CI and a host with a Docker engine.
  2. Gitlab-runner exec docker can only be used when Docker is installed locally. This is needed because GitLab Runner is using host-bind volumes to access the Git sources. Internal commands. GitLab Runner is distributed as a single binary and contains a few internal commands that are used during builds. Gitlab-runner artifacts-downloader.
  3. Hi, did you build that setup yourself, or where did you copy the parts? From a first peek, it looks like as if the GitLab runner is configured to use the ruby image as default, and none of.gitlab-ci.yml references a different Docker image.
- 9 mins

For more information how to install Docker Engine on different systems checkout the Supported installations. Add gitlab-runner user to docker group: sudo usermod -aG docker gitlab-runner Verify that gitlab-runner has access to Docker: sudo-u gitlab-runner -H docker info You can now verify that everything works by adding docker info to.gitlab.

Photo by Rinson Chory on Unsplash

Gitlab allows seamlessly using docker image from public and private hubs. I bet that most of you uses docker executors. All works great and without a hassle until you need to build your own docker image. Fortunately, you can build your docker image automatically in pipeline by leveraging docker-in-docker image build.
I’ll show you how to include docker image build in Gitlab CI Pipeline, push it to Gitlab Repo and use it in another job.

  • Gitlab account
  • Docker basic knowledge
  1. Create Gitlab blank project
  2. Create Dockerfile
  3. Create Gitlab CI pipeline (.gitlab-ci.yml)
    1. Build and push docker image
    2. Use custom docker image
    3. Be efficient!
  4. Summary

I’ll start from scratch. Create new blank project which we will use throughout this blog post. Login into GitLab and navigate to New project -> Create blank project/repository. Give it a project name and hit Create project.

Blank project

Clone the project and we are ready to go.

We need to create a Dockerfile, which will be used to build an docker image. Let’s use Alpine Linux as base image. It is a minimal linux distribution ~5MB. Alpine Linux is great choice when you have specific task to accomplish and you want to use less storage, have fast build times.
Alpine doesn’t have java installed by default - command java -version would fail. We will create Dockerfile to create new docker image based on Alpine with openjdk11 installation. After building and pushing image to repo, we will use it in another job and run command java -version which should run successfully.
Creating new image is not the only option. You can use base Alpine image in your pipeline job, then in script section of the job install java with regular linux command. It will also work. Just keep in mind that that our example is simple. Real world scenario could include multiple software installation and configuration. You wouldn’t want to run it every time pipeline is trigged. Time is precious in CI/CD pipeline. It’s more efficient to build image at first and rebuild it only if Docker file changes.

Dockerfile for alpine with openjdk installation

Our Dockerfile couldn’t be more simple:

Gitlab Ci Install Docker Download

  • FROM alpine:3 - use Alpine image with tag 3 as base image
  • RUN apk add openjdk11 - run command to install openjdk11
  • CMD ['/bin/sh'] - specifies what command to run in container

You can try build the image locally and test if java is available:

  • docker build -t alpine-java . - build docker image and tag it alpine-java
  • docker run --rm alpine-java java -version - run container with alpine-java image and execute java -version command

Local Dockerfile test

Dockerfile is ready. Push it to Gitlab repo or create Dockerfile directly in Gitlab.

We will now create Gitlab CI pipeline and there are two options we could use:

  1. Create a .gitlab-ci.yml file in the root of the repository
  2. Use Gitlab CI/CD editor (in Gitlab, CI/CD -> Editor)

Option 1 is probably used more often, especially in project using a git branch strategy.
Option 2 is more than enough for our scenario and I’ll go with it.

Create New CI/CD Pipeline

Click on Create new CI/CD pipeline.
You will directed to editor with an example pipeline. We can use it, but we only need stage section including build and test stages. Remove the rest of the pipline and we can start creating jobs.

Build and push docker image

Pipeline job for building docker image must have 3 main actions:

  • login into container registry
  • build docker image
  • push image to the docker registry
Install

Pipeline job - docker image build

Currently our pipeline has job for building docker image. Let’s explain it:

  • build:docker-alpine-java - A job’s name
  • stage - Stage in which job runs.
  • image - An image which will be used to create container for running our script
  • services - Defines docker image that runs during the job linked to the docker image specified in image. We are using docker-in-docker (job is running docker as image and as service)
  • variables - Defines variables for the job. We only define one variable, image name.
  • before_script - Defines commands to be run before commands in script section. We use it to login into Gitlab container registry. The variables used are predefined Gitlab variables.
  • script - Defines the main commands to be run during job. There are 3 main steps in the script:
    • docker pull $TAG:latest true
      Pulls image with latest tag or returns true. It will ensure that job will not fail if there is no image with latest tag.
    • docker build --cache-from $TAG:latest --tag $TAG:$CI_COMMIT_REF_NAME --tag $TAG:latest .
      Builds image using Dockerfile in repo root directory and tags it twice (branch name and latest). Uses option --cache-from to use cache from latest avaiable image in container registry. Since we are running docker-in-docker, each time in fresh environment, current runtime has no data from previous runs. The docker runtime can’t use cache from previous build, unlesss we manually pull latest image and supply it in --cache-from option.
    • docker push $TAG:$CI_COMMIT_REF_NAME
      Push image tagged with branch name to container repository.
    • docker push $TAG:latest
      Push image tagged with latest to container repository.
      Hit button Commit changes below editor and the pipeline should start automatically. Go to CI/CD -> Pipelines.

First pipeline run for docker image build

You can click on Status to check the logs and more detailed information. Pipeline run succesfully. Let’s check the container registry. Go to Packages & Registry -> Container Registry.

Images in container registry

You should see an image with a name the same as defiend in $TAG variable inside pipeline job. Under the name there is information that two tagged image exisit. Click on image name.

Tagged images

Two images exisit as expected: one tagged latest and second tagged with branch name (master in mine case).

Use custom docker image

Docker image build is working now as expected and is part of the pipeline. Now we will use newly created image in another pipeline job. Second job will use docker-alpine-java image and run script java -version to see that in fact we are using our custom alpine image with java installed.

Pipeline job - use custom image

Testing custom image job is quite simpler.

  • test:alpine-java - Job name.
  • image - Image used by job. It’s path to our custom image in Gitlab container registry
  • stage - Stage in which job runs.
  • script - Defines the main commands to be run during job. Java version check in our case.

Go to the CI/CD -> Pipelines, then click on last one to see that two jobs were done succesfully (build and test). Click on test job to check the logs. / We are looking for two interesting parts:

  • logs showing which image is being used

  • logs showing java -version command

Be efficient!

You might have noticed a flaw in pipeline. When pipeline run at first it had built alpine-java image and pushed it to repository. When pipeline has run the second time it built alpine-image again, even though it was not necessary. It’s casued by not having any rules defining when to run a job in piepline.
Fortunately it is a quick fix. Add a rule section in build:docker-alpine-java job. The rule will allow job to run only if there is any change in Dockerfile.

Pipeline job - rule


If you now check latest pipeline run you will notice that only test:alpine-java was run in pipeline.

Pipeline with rule


Moreover, thanks to the flaw we have just fixed, you can compare diffrence in duration of two docker build jobs.

Build jobs duration


Maybe you already figured why the build job took 1 min 30 sec when was run first time, but only 1 min when run second time? The answer is cache. Second run built exactly the same image as the first run, beacause there was no diffrence in Dockerfile. Since we implemented cache usage in docker image build job, the second run was much quicker.

Throughtout the blog post we have succesfully:

  • Created new Gitlab project
  • Created Dockerfile
  • Implemented docker image build in Gitlab pipeline job
  • Used the newly created image in another job

Thanks for reading!

Related Posts

Please enable JavaScript to view the comments powered by Disqus.comments powered by Disqus

Configuring a Linux-based Gitlab runner to support Docker-based builds is relatively straight-forward and well-documented. Doing the same with Windows is a bit less so. Here’s how to configure a Windows Server 2019 VM to host Docker-based builds with Visual Studio or other Windows-based tools.

Before we begin, it’s worth spending a little time to plan your setup - there are a couple of issues that may impact how you proceed:

Gitlab Ci Install Docker

Container Compatibility

Windows

The first issue is Windows container compatibility. If you’re just getting started with Windows containers, you might be surprised to get an error that looks something like this when you try to run a Windows container:

This is because a Windows container host can only run a container image that is of the same OS version or older. For example, if you are running Windows 10 or Server version 1909, you will not be able to run a version 2004 or newer image.

Fortunately, Microsoft provides most of their base container images in a variety of versions, but you should research which base images you are going to need and plan for a container host that will support them.

Process Isolation vs. Hyper-V Isolation

Windows containers support two isolation modes - process isolation, which is very similar to how containers normally work in a Linux environment and Hyper-V isolation that runs the container in a highly-optimized virtual machine using Windows Hyper-V. There are pros and cons to both:

  • Process isolation is the “traditional” container isolation model and is more efficient because the container host and container both share the same OS/kernel.
  • In order to use process isolation, the host and container image must be the same version. Otherwise you will get the version mismatch error above.
  • While Hyper-V isolation offers better security and a wider range of compatibility, containers running Hyper-V isolation will be somewhat slower and consume more resources than process-isolation containers.

Gitlab / Gitlab Runner Version

When Gitlab runs a Docker-based CI job, it uses a “helper” Docker image to bootstrap the execution environmnent for the build. In the case of Windows and due to the container compatibilty constraints above, this helper image must match the Windows version of your container. This means the Gitlab team has to create an updated helper image for each new version of Windows and this helper image is included in the Gitlab release.

As of this blog post date, 13.6 is the latest official release of Gitlab and it supports Windows hosts/containers up to version 1909. Check the documentation for your version of Gitlab to get the latest supported Windows container version.

Normally you will be limited to whatever version of Windows container host your version of Gitlab supports. However, if you really want to run later container versions, there is a simple, but somewhat ugly work-around. See this issue comment by Cédric Menzi. Basically, Cédric’s work-around allows you to use a Windows 2004 (or later) container host by fooling Gitlab it into thinking the server is an older version. Ugly but effective.

Virtual/Physical Machine Configuration

If you plan to use Hyper-V isolation, you will need to enable virtualization features in your BIOS settings.

OS Install

I’m using a headless Windows Server 2019 VM. Windows 10 will also work but there are Microsoft licensing restrictions to be aware of for production use.

We’re going to be doing everything from the command-line using Powershell. Be sure your command prompt window or remote session is running as Adminstrator.

Install OpenSSH

Unless you are working with a physical machine or prefer or have some other remote access solution already in place, you will need some method for remotely Administering your server to install and configure the runner. While you can use Powershell remoting, OpenSSH is simpler and more flexible for me.

Follow these instructions to install the OpenSSH option on your server and perform the initial configuration.

Public key auth for Adminstrator logins

If you don’t want to enter the Administrator password each time you open an SSH session to the server, add your SSH public key to C:ProgramDatasshadministrators_authorized_keys (note that ProgramData is a hidden directory). If you login as another user, you will add your public key to %USERPROFILE%.sshauthorized_keys.

Install Docker

For a server installation, you will not be installing the regular Docker Desktop release for docker.com, instead follow the Microsoft instructions for adding the Docker Windows component from the command-line.

If you want to test out your Docker installation, remember the constraints above - by default, Docker for Windows will attempt to run containers using process isolation, which means that the container image version must match your server version.

Install Hyper-V

Unless you plan to only run containers with process isolation, you will need to install the Hyper-V service. Check to see if it is already installed with:

If it is not installed, install it with:

And reboot.

Enable Hyper-V isolation in Docker config

If you plan to use Hyper-V isolation, you will need to tell Docker to use it by default instead of process isolation. Edit (or create if it does not exist) C:ProgramDatadockerconfigdaemon.json and add:

Restart the Docker service with:

Install & Register Gitlab Runner

Now you are ready to install and register the Gitlab runner. Be sure to select the docker-windows executor.

Install Docker Linux

Depending on your setup, you may need to edit config.toml to configure additional options. One option in particular you may want to set is the pull_policy in the [runners.docker] section. For example:

Gitlab Runner Install Docker

This setting controls the policy used by the Gitlab runner for handling Docker images. The default is always which means that the runner will request images from the Docker repository every time. With the new Docker pull-rate limits implemented recently, this could cause your jobs to fail with an error like the following unless you have configured a private mirrored repo or have a paid Docker Hub account:

Gitlab Ci Install Docker-compose

Refer to the Advanced Configuration section of the Gitlab Runner docs for more info on pull_policy and other settings.