Yarn Install Docker

Prerequisites

FAQ: Why Binci over Docker-Compose? Installation YARN/NPM. The best way to install Binci and keep it updated is through NPM, included with Node.js, or Yarn. Yarn global add binci or npm install binci -g. #!/bin/bash # install nodemodules and deploy application to production yarn npm run deploy # build your docker image docker build. # push your docker image docker push. # run cleanup commands. Jan 26, 2017 Npm (or Yarn) Install within a Docker Container, the Right Way Published on 26 January 2017 Working as a web agency (or more specifically at marmelab, as an innovation workshop), we have to deal with several different customers and projects. The only Yarn-related files that are not captured in the Docker image are yarn.lock and.pnp.js because their locations are not configurable. The presence of these two files in the codebase directory means that it’s not possible to use both Yarn v2 on the container side and Yarn v1 on the host side although there currently is a ticket to to.

(Please note that all following prerequisites are just an example for you to install. You can always choose to install your own version of kernel, different users, different drivers, etc.).

Operating System

The operating system and kernel versions we have tested are as shown in the following table, which is the recommneded minimum required versions.

Enviroment Verion
Operating System centos-release-7-3.1611.el7.centos.x86_64
Kernal 3.10.0-514.el7.x86_64

User & Group

Yarn Install Docker Software

As there are some specific users and groups recommended to be created to install hadoop/docker. Please create them if they are missing.

GCC Version

Check the version of GCC tool (to compile kernel).

Yarn install docker software

Kernel header & Kernel devel

GPU Servers (Only for Nvidia GPU equipped nodes)

Nvidia Driver Installation (Only for Nvidia GPU equipped nodes)

To make a clean installation, if you have requirements to upgrade GPU drivers. If nvidia driver/cuda has been installed before, They should be uninstalled firstly.

Yarn Install Docker Slow

To check GPU version, install nvidia-detect

Pay attention to This device requires the current xyz.nm NVIDIA driver kmod-nvidia. Download the installer like NVIDIA-Linux-x86_64-390.87.run.

Some preparatory work for nvidia driver installation. (This is follow normal Nvidia GPU driver installation, just put here for your convenience)

Check whether nouveau is disabled

Some options during the installation

Check nvidia driver installation

Yarn

Reference: https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html

Docker Installation

We recommend to use Docker version >= 1.12.5, following steps are just for your reference. You can always to choose other approaches to install Docker.

Reference:https://docs.docker.com/cs-engine/1.12/

Docker Configuration

Add a file, named daemon.json, under the path of /etc/docker/. Please replace the variables of image_registry_ip, etcd_host_ip, localhost_ip, yarn_dns_registry_host_ip, dns_host_ip with specific ips according to your environments.

Restart docker daemon:

Docker EE version

Nvidia-docker Installation (Only for Nvidia GPU equipped nodes)

Submarine depends on nvidia-docker 1.0 version

According to nvidia-driver version, add folders under the path of /var/lib/nvidia-docker/volumes/nvidia_driver/

Test docker, nvidia-docker, nvidia-driver installation

Reference: https://github.com/NVIDIA/nvidia-docker/tree/1.0

Tensorflow Image

There is no need to install CUDNN and CUDA on the servers, because CUDNN and CUDA can be added in the docker images. we can get basic docker images by following WriteDockerfile.md.

Yarn install docker software

The basic Dockerfile doesn’t support kerberos security. if you need kerberos, you can get write a Dockerfile like this

Install

Test tensorflow in a docker container

After docker image is built, we can check Tensorflow environments before submitting a yarn job.

If there are some errors, we could check the following configuration.

  1. LD_LIBRARY_PATH environment variable
  1. The location of libcuda.so.1, libcuda.so
Install

Etcd Installation

etcd is a distributed reliable key-value store for the most critical data of a distributed system, Registration and discovery of services used in containers. You can also choose alternatives like zookeeper, Consul.

To install Etcd on specified servers, we can run Submarine-installer/install.sh

Yarn Install Docker Tutorial

Check Etcd cluster health

Install Yarn On Node Docker

Calico Installation

Calico creates and manages a flat three-tier network, and each container is assigned a routable ip. We just add the steps here for your convenience. You can also choose alternatives like Flannel, OVS.

To install Calico on specified servers, we can run Submarine-installer/install.sh

Check Calico Network

Dockerfile Install Yarn

Create containers to validate calico network