Sharing files with host machine

Mac

The ability to share volumes with Mac hosts is built into Docker >= 1.3. Just link any volume (under /Users/) as you would on Linux:

docker run -v /Users/bob/myapp/src:/src [...]

Still need to test if this requires any special handling of permissions.

Windows

boot2docker provides two current workarounds for sharing directories on Windows, though more native sharing might follow. The official instructions are here but are more complicated than our preferred method of using the -v flag, which you can do like so:

docker run -d -p 8787:8787 -v /c/Users/foobar:/home/rstudio/foobar rocker/rstudio

In this case, /c/Users/foobar corresponds to an existing folder on your computer at C:/Users/foobar, and foobar can be anything. You can now connect to RStudio at http://192.168.59.103:8787 and log in with rstudio/rstusio. With this -v method you can read and write files both ways between Windows and RStudio.

The above approach may not always work, as described in this bug report. The workaround is to either use a double slash or $(pwd):

docker run -d -p 8787:8787 -v //c/Users/foobar:/home/rstudio/foobar rocker/rstudio

or

docker run -d -p 8787:8787 -v /$(pwd):/home/rstudio/foobar rocker/rstudio

Linux

As docker runs natively on Linux, you can always link any volume to the container with the -v or --volume flag, as described in the docker documentation. Note that this overwrites anything on the container that already exists in the target location, and will create the necessary directories if they do not already exist. We recommend running as a non-root user and linking to a subdirectory in the that user's home directory as shown below.

Avoiding permission changes when sharing volumes

By default, our docker containers run as the root user. (See managing users in docker for details.) Files created or modified by the container will thus become owned by the root user, even after quitting the container. To avoid this problem, it is necessary to run the container using a non-root user.

RStudio-server logins require a non-root user whether or not the container is sharing any volumes with the host. A default user is created automatically when these containers are run with the default command. When sharing volumes with the host, it may be necessary to make sure that the UID of the user created (whether the it is the default user, rstudio or a custom username) matches the UID of the host user (the username is irrelevant, only the UID must match). If your host machine has only one non-root user, chances are this is already fine, though you can still set the UID explicitly as described below.

Anyone running our containers as a non-root user can link a directory, including ones at or above ~/, to the corresponding user's directory on the container. The container will only ever have one user.

If the host machine user has a UID other than 1000 (or 0, for root), the user should specify their UID when running docker, e.g.

docker run -d -P -v $(pwd):/home/$USER/foo -e USERID=$UID rocker/rstudio

to avoid changing the permissions in the linked volume on the host. This is designed for rstudio-based logins and runs only when the container is executed without a default command. (Note, optionally a custom user name can be given with the environmental variable argument e USER=someuser, instead of the default rstudio user, but this is purely an aesthetic change. For matching permissions, merely ensure the UID is consistent.

Interactive containers

For interactive containers, it is sufficient to run with the flag --user docker, e.g. the interactive R container:

docker run --rm -it --user docker -v $(pwd):/home/docker/foo -w /home/docker/foo rocker/r-base R

note this command links the current working directory to /home/docker/foo on the container, where our user docker can write to, and also sets this as the working directory for the container. This is analogous to just running R in the working directory.

The only limitation is that the interactive method doesn't handle alternate UIDs. If the user running docker has a different UID, they have to do this more manually. Run a terminal in docker (using rstudio image or above):

docker run --rm -it -v $(pwd):/home/$USER/foo \
-e USER=$USER  -e USERID=$UID rocker/rstudio bash

and then in the bash shell run:

userconf.sh
su $USER
R

Deploy war file to docker image

Deploy war file to docker image

This tutorial demonstrate how to deploy a war file into docker image.

I’ll be using 2 approach:

1. Embedded war file to build into docker image.
2. Externalize war file by mounting with docker tomcat path.

Structure

Approach 1

Let’s get started.

Step 1) Prepare a Dockerfile
Step 2) Run build custom image base on docker hub tomcat image
Step 3) Start docker containers

Dockerfile

1. Prepare a Dockerfile with the following content.
2. Copy the war file from out from the target folder.

# Pull base image
From tomcat:8-jre8

# Maintainer
MAINTAINER "xxx <xxx@gmail.com">

# Copy to images tomcat path
ADD dockerwar.war /usr/local/tomcat/webapps/

Build docker image

Now run the follow command to docker image name webserver with your Dockerfile  in current directory. First time to build docker image will require download and may take longer times.

docker build -t webserver .

Run docker container

Run docker container with interactive mode.

docker run -it --rm -p 8080:8080 --name dockerwar webserver

Test your container

Open a browser with URL http://192.168.59.103:8080/dockerwar/

Approach 2

Approach 2 are slightly different, let’s modified the Dockerfile, we just extends the tomcat based image.

Dockerfile

# Pull base image
From tomcat:8-jre8

# Maintainer
MAINTAINER "xxx <xxx@gmail.com">

Build docker file

docker build -t webserver .

Eclipse classpath

In this example, I’m using eclipse classpath “target” folder to mount with docker container’s /webapps/ directory.
 

Delete all files in “target folder”

Before you start the container run “mvn clean package” to your project.
 

Run docker

Run the following command in interactive mode, mount your eclipse build path to your docker container tomcat webapps folder.
 

docker run -it --rm -p 8080:8080 -v /Users/mingch/workspace/dockerwar/target:/usr/local/tomcat/webapps/ --name dockerwar webserver

Test your container

Open a browser with URL http://192.168.59.103:8080/dockerwar/
 

Tails container logs

You can either start your container without interactive mode -it or run another terminal to interact with your container. Eg, my container instance is e6b2.

To list all the log files:

docker exec -it e6b2 ls /usr/local/tomcat/logs/

To tail the log file:

docker exec -it e6b2 tail -f /usr/local/tomcat/logs/localhost_access_log.2015-10-07.txt

The Easy Way To Deploy Java Tomcat With Docker

If you are trying to deploy a Java Tomcat app on a Docker system, you might find other tutorials which will take you hours of heartache to get just right. How about deploying a full custom Tomcat app in just 30 seconds?

See For Yourself

Don't believe me? Just try these 10 simple steps:

$ brew install maven boot2docker # on OS X
$ boot2docker init ; boot2docker up # on OS X
$ sudo gem install bundler highline building

$ git clone https://github.com/jesperfj/webapp-with-jndi
$ cd webapp-with-jndi
$ mvn package
$ cd target/my-webapp
$ building -b http://github.com/jesperfj/buildpack-tomcat.git myapp
create Dockerfile
building docker build -t hhvm-app:latest .
hint: To run your app, try: docker run -d -p 8080 -e "PORT=8080" myapp
hint To re-build your app, try: docker build -t myapp .
$ JAVA_APP_ID=$(docker run -d -p 8080 -e "PORT=8080" myapp)
$ curl `docker port $JAVA_APP_ID 8080`/hello Hello World

The basic Heroku Java buildpack doesn't work with every Java app. If you run it from the jesperfj/webapp-with-jndi base directory it will fail. But if you use the jesperfj/buildpack-tomcat custom buildpack within the app's directory, everything works smoothly.

Conclusion

This is just a simple example of how much better the developer tooling around Docker has become over the last year. With Docker 1.0 just around the corner, there has never been a better time to start incorporating it into your daily workflow. If you are ready to get your feet wet with CoreOS, try our Building Your First App on CoreOS tutorial.

Deploying Java applications with Docker

Docker allows you to create containers from your applications, for more information see: https://www.docker.io/. Docker can be used for many things, one of the options is to replace virtual machines with Docker containers. This article will explain how Docker can be used to setup Tomcat and deploy a Java application.

Why Docker?

Of course there are many tools available to automate the installation of servers and applications. Virtual machines can be used together with tools like Vagrant, Chef, Puppet and many more. But most of them  have their own DSL and some are not for free. That makes it harder to use them as developers and operations are unfamiliar with the tools and DSL’s. Docker has some specific commands, but for the installation of applications like the JDK or Tomcat standard OS commands can be used like apt-get, wget etcetera. Also Docker containers require less resources and startup faster than virtual images as this article will show.

Docker can be used to setup Java application servers for instance test and production machines (although for production it is better to wait until release 1). It is also possible to create a build environment with Jenkins, Nexus, Git, Sonar etc. One container should be used for every application to separate concerns.

The advantages of Docker are quite useful to overcome some of the technical challenges of continuous delivery. Versioning of Docker configuration files is quite easy. Another advantage is that environments can be setup from scratch for every deployment because Docker containers are quickly created and started.

Installing Docker

It is quite easy to install Docker. Instructions can be found on https://docs.docker.io/en/latest/installation. For this article Ubuntu 13.10 was chosen and the commands to install Docker are shown below.

sudo apt-get update
sudo apt-get install linux-image-extra-`uname -r`
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9
sudo sh -c "echo deb http://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list"
sudo apt-get update
sudo apt-get install lxc-docker
sudo docker run -i -t ubuntu /bin/bash

The last step will start a Docker container based on Ubuntu and start bash.

Adding your user to the Docker group

Root access is necessary to work with Docker, but you can also add your user to the docker group. After adding yourself to that group you no longer have to use the ‘sudo’ command.

sudo groupadd docker
sudo gpasswd -a your_username docker
sudo service docker restart

It is necessary to logout and login, restart Ubuntu or enter the command ‘newgrp docker’ before this works correctly.

Create a Docker container with Java and Tomcat

Create a file called ‘Dockerfile’ and add the content below. This is the configuration to install Java and Tomcat on a Ubuntu Saucy (13.10) Docker container that is provided by Docker. There are a lot of official and user supplied container configurations available at https://index.docker.io/. These containers can be used as the basis for your own container.

Dockerfile content:

FROM ubuntu:saucy
# Update Ubuntu
RUN apt-get update && apt-get -y upgrade
# Add oracle java 7 repository
RUN apt-get -y install software-properties-common
RUN add-apt-repository ppa:webupd8team/java
RUN apt-get -y update
# Accept the Oracle Java license
RUN echo "oracle-java7-installer shared/accepted-oracle-license-v1-1 boolean true" | debconf-set-selections
# Install Oracle Java
RUN apt-get -y install oracle-java7-installer
# Install tomcat
RUN apt-get -y install tomcat7
RUN echo "JAVA_HOME=/usr/lib/jvm/java-7-oracle" >> /etc/default/tomcat7
EXPOSE 8080
# Download Slashdot homepage
RUN mkdir /var/lib/tomcat7/webapps/slashdot
RUN wget http://www.slashdot.org -P /var/lib/tomcat7/webapps/slashdot
# Start Tomcat, after starting Tomcat the container will stop. So use a 'trick' to keep it running.
CMD service tomcat7 start && tail -f /var/lib/tomcat7/logs/catalina.out

Build the container using: ‘docker build –t tomcat7 .’. Do not forget to use the ‘.’ at the end. Instead of ‘tomcat7’ you can pick another name if you want. Now start the container using ‘docker run -p 8080:8080 -d tomcat7’. This command makes sure that port 8080 of the container is forwarded to your local port 8080.  After a few seconds the Slashdot page should be available at http://localhost:8080/slashdot/.

Container startup speed

A small test was executed to see how fast the container actually started. This test starts the Docker container and uses ‘wget’ to retrieve a page from within the container.

The commands are placed in a file called ‘dockerSpeed’, the content of the file is shown below:

docker run -p 8080:8080 -d tomcat7
wget localhost:8080/slashdot

The ‘time’ command can be used to see how fast the commands above are executed. Executing ‘time sh dockerSpeed’ gave the following result:

real        0m4.367s
user        0m0.011s
sys         0m0.008s

A few of these tests showed that the execution times were always between 4 and 5 seconds. That’s really quick if you compare that to starting virtual machines.

Container size

Docker creates a container for every command in the Dockerfile. If you have 12 commands in the Dockerfile, then Docker will create 12 containers. This allows the user to select an older container, for instance if you only need a container with Java, you could use that container instead of the one including Tomcat. Luckily only the difference between the containers is stored, which saves quite some diskspace.

Where Docker really wins from virtual machines is if you use multiple containers with the same base. For instance if you have 8 containers each using Ubuntu and Java. You could make a base container with Ubuntu and Java and create containers based on that base image. That way Ubuntu and Java are stored only once in a container, this results in fewer diskspace being used. With virtual machines, you would need that diskspace 8 times as it is not possible to create a common base.

This results in another good point of Docker. Separation of concerns is quite important and can be achieved easily, every application can be deployed in a separate container. So for a buildserver, you could make separate containers for Nexus, Jenkins, Git etcetera.

There are a few options to view the containers. The docker ‘ps’ command shows all active containers. Adding ‘-a’ to the ‘ps’ command also shows the stopped containers. Adding ‘-s’ to the ‘ps’ command shows the size of the containers. The output below from the ‘tomcat7’ container that was created in this article shows that most containers are quite small. Only the containers where bigger things are added like the JDK result in larger containers.

$ docker ps -a -s
CONTAINER ID        IMAGE               COMMAND                CREATED             STATUS              PORTS                    NAMES                    SIZE
3d9bd89b5ace        tomcat7:latest      /bin/sh -c service t   10 seconds ago      Up 10 seconds       0.0.0.0:8080->8080/tcp   grave_hawking            51.64 kB
fc116050d899        1ed13b7f9eb1        /bin/sh -c #(nop) CM   15 minutes ago      Exit 0                                       drunk_babbage            0 B
b788f8373ae7        5eb0489aed66        /bin/sh -c wget http   15 minutes ago      Exit 0                                       sleepy_nobel             4.613 kB
f465ff7c6cdf        a8febaa9ed55        /bin/sh -c wget http   15 minutes ago      Exit 0                                       hungry_lumiere           121.3 kB
d4dee2fd7c2f        1938691cd911        /bin/sh -c mkdir /va   15 minutes ago      Exit 0                                       dreamy_euclid            7 B
56f8bff7cfb9        d2e740750084        /bin/sh -c #(nop) EX   15 minutes ago      Exit 0                                       goofy_turing             0 B
1f8639f1841f        52d8cf48f2f3        /bin/sh -c echo "JAV   15 minutes ago      Exit 0                                       grave_bohr               2.074 kB
4cffeada2f59        8811557c9b1b        /bin/sh -c apt-get -   15 minutes ago      Exit 0                                       determined_ptolemy       11 MB
e6a24a05efb3        7d4e8d307140        /bin/sh -c apt-get -   16 minutes ago      Exit 0                                       nostalgic_wozniak        450 MB
090b2652f31e        9f35a5c15127        /bin/sh -c echo "ora   16 minutes ago      Exit 0                                       condescending_brattain   2.764 MB
743d10527376        cce7c0072447        /bin/sh -c apt-get -   16 minutes ago      Exit 0                                       sleepy_ritchie           179 kB
9da7cf4f4ca7        cea346235f02        /bin/sh -c add-apt-r   16 minutes ago      Exit 0                                       jovial_engelbart         863 B
d8c359bdd163        0dcc4e24ae2f        /bin/sh -c apt-get -   17 minutes ago      Exit 0                                       jolly_albattani          33.06 MB
7b184e8b6ece        ubuntu:13.10        /bin/sh -c apt-get u   17 minutes ago      Exit 0                                       trusting_shockley        25.18 MB

Another option is to view the images tree from Docker, the output for the ‘tomcat7’ container is shown below.

$ docker images -tree
??511136ea3c5a Virtual Size: 0 B
??1c7f181e78b9 Virtual Size: 0 B
??9f676bd305a4 Virtual Size: 178 MB Tags: ubuntu:13.10, ubuntu:saucy
??0dcc4e24ae2f Virtual Size: 215.6 MB
??cea346235f02 Virtual Size: 249 MB
??cce7c0072447 Virtual Size: 249 MB
??9f35a5c15127 Virtual Size: 249.2 MB
??7d4e8d307140 Virtual Size: 251.9 MB
??8811557c9b1b Virtual Size: 702.3 MB
??52d8cf48f2f3 Virtual Size: 713.7 MB
??d2e740750084 Virtual Size: 713.7 MB
??1938691cd911 Virtual Size: 713.7 MB
??a8febaa9ed55 Virtual Size: 713.7 MB
??5eb0489aed66 Virtual Size: 713.8 MB
??1ed13b7f9eb1 Virtual Size: 713.8 MB
??f846774ccd37 Virtual Size: 713.8 MB Tags: tomcat7:latest

Deploy a Java application in Tomcat

The example above used a simple HTML file from Slashdot. Of course that’s not enough, we want to deploy Java applications! That can be done quite easily, just add the following line to the Dockerfile just below the ‘wget’ command for Slashdot.

RUN wget http://tomcat.apache.org/tomcat-7.0-doc/appdev/sample/sample.war -P /var/lib/tomcat7/webapps

Build a Docker container based on the new Dockerfile and run the container. Browsing to http://localhost:8080/sample/ will show the sample Tomcat web application.

Some useful Docker commands

Although Docker basics are fairly simple it still has a lot of options and useful commands.

Testing a bit with Docker will result in quite some containers. Stopping and starting them one by one is not really convenient. The following commands can be used to stop and remove all containers

docker stop $(docker ps -a -q)
docker rm $(docker ps -a -q)

Sometimes you create a container using a Dockerfile but then you realize the container is not working properly. With the command below you can login to the container and see what is going wrong.

docker run -i -t tomcat7 /bin/bash

Docker Architecture

Docker follows client-server architecture. Its architecture consists mainly three parts.

1) Client: Docker provides Command Line Interface (CLI) tools to client to interact with Docker daemon. Client can build, run and stop application. Client can also interact to Docker_Host remotely.

2) Docker_Host: It contains Containers, Images, and Docker daemon. It provides complete environment to execute and run your application.

3) Registry: It is global repository of images. You can access and use these images to run your application in Docker environment.

The Docker daemon

It is a process which is used to listen for Docker API requests. It also manages Docker objects like: images, container, network etc. A daemon can also communicate with other daemons to manage Docker services.

The Docker client

The Docker client is the primary way that many Docker users interact with Docker. When we use commands such as docker run, the client sends these commands to docker d, which carries them out. The docker command uses the Docker API.

Docker Registries

Docker registry is used to store Docker images. Docker provides the Docker Hub and the Docker Cloud which are public registries that anyone can use. Docker is configured to look for images on Docker Hub by default.

When we use the docker pull or docker run commands, the required images are pulled from your configured registry. When you use the docker push command, your image is pushed to your configured registry.

Understand images, containers, and storage drivers

Images and layers

Each Docker image references a list of read-only layers that represent filesystem differences. Layers are stacked on top of each other to form a base for a container’s root filesystem. The diagram below shows the Ubuntu 15.04 image comprising 4 stacked image layers.

The Docker storage driver is responsible for stacking these layers and providing a single unified view.

When you create a new container, you add a new, thin, writable layer on top of the underlying stack. This layer is often called the “container layer”. All changes made to the running container - such as writing new files, modifying existing files, and deleting files - are written to this thin writable container layer. The diagram below shows a container based on the Ubuntu 15.04 image.

Content addressable storage

Docker 1.10 introduced a new content addressable storage model. This is a completely new way to address image and layer data on disk. Previously, image and layer data was referenced and stored using a randomly generated UUID. In the new model this is replaced by a secure content hash.

The new model improves security, provides a built-in way to avoid ID collisions, and guarantees data integrity after pull, push, load, and save operations. It also enables better sharing of layers by allowing many images to freely share their layers even if they didn’t come from the same build.

The diagram below shows an updated version of the previous diagram, highlighting the changes implemented by Docker 1.10.

As can be seen, all image layer IDs are cryptographic hashes, whereas the container ID is still a randomly generated UUID.

There are several things to note regarding the new model. These include:

1. Migration of existing images
2. Image and layer filesystem structures

Existing images, those created and pulled by earlier versions of Docker, need to be migrated before they can be used with the new model. This migration involves calculating new secure checksums and is performed automatically the first time you start an updated Docker daemon. After the migration is complete, all images and tags will have brand new secure IDs.

Although the migration is automatic and transparent, it is computationally intensive. This means it can take time if you have lots of image data. During this time your Docker daemon will not respond to other requests.

A migration tool exists that allows you to migrate existing images to the new format before upgrading your Docker daemon. This means that upgraded Docker daemons do not need to perform the migration in-band, and therefore avoids any associated downtime. It also provides a way to manually migrate existing images so that they can be distributed to other Docker daemons in your environment that are already running the latest versions of Docker.

The migration tool is provided by Docker, Inc., and runs as a container. You can download it from https://github.com/docker/v1.10-migrator/releases.

While running the “migrator” image you need to expose your Docker host’s data directory to the container. If you are using the default Docker data path, the command to run the container will look like this

$ sudo docker run --rm -v /var/lib/docker:/var/lib/docker docker/v1.10-migrator

If you use the devicemapper storage driver, you will need to include the --privileged option so that the container has access to your storage devices.

MIGRATION EXAMPLE

The following example shows the migration tool in use on a Docker host running version 1.9.1 of the Docker daemon and the AUFS storage driver. The Docker host is running on a t2.micro AWS EC2 instance with 1 vCPU, 1GB RAM, and a single 8GB general purpose SSD EBS volume. The Docker data directory (/var/lib/docker) was consuming 2GB of space.

$ docker images

REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
jenkins             latest              285c9f0f9d3d        17 hours ago        708.5 MB
mysql               latest              d39c3fa09ced        8 days ago          360.3 MB
mongo               latest              a74137af4532        13 days ago         317.4 MB
postgres            latest              9aae83d4127f        13 days ago         270.7 MB
redis               latest              8bccd73928d9        2 weeks ago         151.3 MB
centos              latest              c8a648134623        4 weeks ago         196.6 MB
ubuntu              15.04               c8be1ac8145a        7 weeks ago         131.3 MB

$ sudo du -hs /var/lib/docker

2.0G    /var/lib/docker

$ time docker run --rm -v /var/lib/docker:/var/lib/docker docker/v1.10-migrator

Unable to find image 'docker/v1.10-migrator:latest' locally
latest: Pulling from docker/v1.10-migrator
ed1f33c5883d: Pull complete
b3ca410aa2c1: Pull complete
2b9c6ed9099e: Pull complete
dce7e318b173: Pull complete
Digest: sha256:bd2b245d5d22dd94ec4a8417a9b81bb5e90b171031c6e216484db3fe300c2097
Status: Downloaded newer image for docker/v1.10-migrator:latest
time="2016-01-27T12:31:06Z" level=debug msg="Assembling tar data for 01e70da302a553ba13485ad020a0d77dbb47575a31c4f48221137bb08f45878d from /var/lib/docker/aufs/diff/01e70da302a553ba13485ad020a0d77dbb47575a31c4f48221137bb08f45878d"
time="2016-01-27T12:31:06Z" level=debug msg="Assembling tar data for 07ac220aeeef9febf1ac16a9d1a4eff7ef3c8cbf5ed0be6b6f4c35952ed7920d from /var/lib/docker/aufs/diff/07ac220aeeef9febf1ac16a9d1a4eff7ef3c8cbf5ed0be6b6f4c35952ed7920d"
<snip>
time="2016-01-27T12:32:00Z" level=debug msg="layer dbacfa057b30b1feaf15937c28bd8ca0d6c634fc311ccc35bd8d56d017595d5b took 10.80 seconds"

real    0m59.583s
user    0m0.046s
sys     0m0.008s

The Unix time command prepends the docker run command to produce timings for the operation. As can be seen, the overall time taken to migrate 7 images comprising 2GB of disk space took approximately 1 minute. However, this included the time taken to pull the docker/v1.10-migrator image (approximately 3.5 seconds). The same operation on an m4.10xlarge EC2 instance with 40 vCPUs, 160GB RAM and an 8GB provisioned IOPS EBS volume resulted in the following improved timings:

real    0m9.871s
user    0m0.094s
sys     0m0.021s

This shows that the migration operation is affected by the hardware spec of the machine performing the migration.

Container and layers

The major difference between a container and an image is the top writable layer. All writes to the container that add new or modify existing data are stored in this writable layer. When the container is deleted the writable layer is also deleted. The underlying image remains unchanged.

Because each container has its own thin writable container layer, and all changes are stored in this container layer, this means that multiple containers can share access to the same underlying image and yet have their own data state. The diagram below shows multiple containers sharing the same Ubuntu 15.04 image.

The Docker storage driver is responsible for enabling and managing both the image layers and the writable container layer. How a storage driver accomplishes these can vary between drivers. Two key technologies behind Docker image and container management are stackable image layers and copy-on-write (CoW).

The copy-on-write strategy

Sharing is a good way to optimize resources. People do this instinctively in daily life. For example, twins Jane and Joseph taking an Algebra class at different times from different teachers can share the same exercise book by passing it between each other. Now, suppose Jane gets an assignment to complete the homework on page 11 in the book. At that point, Jane copies page 11, completes the homework, and hands in her copy. The original exercise book is unchanged and only Jane has a copy of the changed page 11.

Copy-on-write is a similar strategy of sharing and copying. In this strategy, system processes that need the same data share the same instance of that data rather than having their own copy. At some point, if one process needs to modify or write to the data, only then does the operating system make a copy of the data for that process to use. Only the process that needs to write has access to the data copy. All the other processes continue to use the original data.

Docker uses a copy-on-write technology with both images and containers. This CoW strategy optimizes both image disk space usage and the performance of container start times. The next sections look at how copy-on-write is leveraged with images and containers through sharing and copying.

Sharing promotes smaller images

This section looks at image layers and copy-on-write technology. All image and container layers exist inside the Docker host’s local storage area and are managed by the storage driver. On Linux-based Docker hosts this is usually located under /var/lib/docker/.

The Docker client reports on image layers when instructed to pull and push images with docker pull and docker push. The command below pulls the ubuntu:15.04 Docker image from Docker Hub.

$ docker pull ubuntu:15.04

15.04: Pulling from library/ubuntu
1ba8ac955b97: Pull complete
f157c4e5ede7: Pull complete
0b7e98f84c4c: Pull complete
a3ed95caeb02: Pull complete
Digest: sha256:5e279a9df07990286cce22e1b0f5b0490629ca6d187698746ae5e28e604a640e
Status: Downloaded newer image for ubuntu:15.04

From the output, you’ll see that the command actually pulls 4 image layers. Each of the above lines lists an image layer and its UUID or cryptographic hash. The combination of these four layers makes up the ubuntu:15.04 Docker image.

Each of these layers is stored in its own directory inside the Docker host’s local storage area.

Versions of Docker prior to 1.10 stored each layer in a directory with the same name as the image layer ID. However, this is not the case for images pulled with Docker version 1.10 and later. For example, the command below shows an image being pulled from Docker Hub, followed by a directory listing on a host running version 1.9.1 of the Docker Engine.

$  docker pull ubuntu:15.04

15.04: Pulling from library/ubuntu
47984b517ca9: Pull complete
df6e891a3ea9: Pull complete
e65155041eed: Pull complete
c8be1ac8145a: Pull complete
Digest: sha256:5e279a9df07990286cce22e1b0f5b0490629ca6d187698746ae5e28e604a640e
Status: Downloaded newer image for ubuntu:15.04

$ ls /var/lib/docker/aufs/layers

47984b517ca9ca0312aced5c9698753ffa964c2015f2a5f18e5efa9848cf30e2
c8be1ac8145a6e59a55667f573883749ad66eaeef92b4df17e5ea1260e2d7356
df6e891a3ea9cdce2a388a2cf1b1711629557454fd120abd5be6d32329a0e0ac
e65155041eed7ec58dea78d90286048055ca75d41ea893c7246e794389ecf203

Notice how the four directories match up with the layer IDs of the downloaded image. Now compare this with the same operations performed on a host running version 1.10 of the Docker Engine.

$ docker pull ubuntu:15.04
15.04: Pulling from library/ubuntu
1ba8ac955b97: Pull complete
f157c4e5ede7: Pull complete
0b7e98f84c4c: Pull complete
a3ed95caeb02: Pull complete
Digest: sha256:5e279a9df07990286cce22e1b0f5b0490629ca6d187698746ae5e28e604a640e
Status: Downloaded newer image for ubuntu:15.04

$ ls /var/lib/docker/aufs/layers/
1d6674ff835b10f76e354806e16b950f91a191d3b471236609ab13a930275e24
5dbb0cbe0148cf447b9464a358c1587be586058d9a4c9ce079320265e2bb94e7
bef7199f2ed8e86fa4ada1309cfad3089e0542fec8894690529e4c04a7ca2d73
ebf814eccfe98f2704660ca1d844e4348db3b5ccc637eb905d4818fbfb00a06a

See how the four directories do not match up with the image layer IDs pulled in the previous step.

Despite the differences between image management before and after version 1.10, all versions of Docker still allow images to share layers. For example, If you pull an image that shares some of the same image layers as an image that has already been pulled, the Docker daemon recognizes this, and only pulls the layers it doesn’t already have stored locally. After the second pull, the two images will share any common image layers.

You can illustrate this now for yourself. Starting with the ubuntu:15.04 image that you just pulled, make a change to it, and build a new image based on the change. One way to do this is using a Dockerfile and the docker build command.

1. In an empty directory, create a simple Dockerfile that starts with the ubuntu:15.04 image.

    FROM ubuntu:15.04
   

2. Add a new file called “newfile” in the image’s /tmp directory with the text “Hello world” in it.
   When you are done, the Dockerfile contains two lines:

    FROM ubuntu:15.04
   
    RUN echo "Hello world" > /tmp/newfile
   

3. Save and close the file.
4. From a terminal in the same folder as your Dockerfile, run the following command:

    $ docker build -t changed-ubuntu .
   
    Sending build context to Docker daemon 2.048 kB
    Step 1 : FROM ubuntu:15.04
     ---> 3f7bcee56709
    Step 2 : RUN echo "Hello world" > /tmp/newfile
     ---> Running in d14acd6fad4e
     ---> 94e6b7d2c720
    Removing intermediate container d14acd6fad4e
    Successfully built 94e6b7d2c720
   

   Note: The period (.) at the end of the above command is important. It tells the docker build command to use the current working directory as its build context.

   The output above shows a new image with image ID 94e6b7d2c720.
5. Run the docker images command to verify the new changed-ubuntu image is in the Docker host’s local storage area.

    REPOSITORY       TAG      IMAGE ID       CREATED           SIZE
    changed-ubuntu   latest   03b964f68d06   33 seconds ago    131.4 MB
    ubuntu           15.04    013f3d01d247   6 weeks ago       131.3 MB
   

6. Run the docker history command to see which image layers were used to create the new changed-ubuntu image.

    $ docker history changed-ubuntu
    IMAGE               CREATED              CREATED BY                                      SIZE        COMMENT
    94e6b7d2c720        2 minutes ago       /bin/sh -c echo "Hello world" > /tmp/newfile    12 B
    3f7bcee56709        6 weeks ago         /bin/sh -c #(nop) CMD ["/bin/bash"]             0 B
    <missing>           6 weeks ago         /bin/sh -c sed -i 's/^#\s*\(deb.*universe\)$/   1.879 kB
    <missing>           6 weeks ago         /bin/sh -c echo '#!/bin/sh' > /usr/sbin/polic   701 B
    <missing>           6 weeks ago         /bin/sh -c #(nop) ADD file:8e4943cd86e9b2ca13   131.3 MB
   

   The docker history output shows the new 94e6b7d2c720 image layer at the top. You know that this is the new image layer added because it was created by the echo "Hello world" > /tmp/newfile command in your Dockerfile. The 4 image layers below it are the exact same image layers that make up the ubuntu:15.04 image.

Note: Under the content addressable storage model introduced with Docker 1.10, image history data is no longer stored in a config file with each image layer. It is now stored as a string of text in a single config file that relates to the overall image. This can result in some parent image layers showing as missing in the output of the docker history command. This is normal behavior and can be ignored. missing means that there is no local image associated with this history chain.

You may hear images like these referred to as flat images.

Notice the new changed-ubuntu image does not have its own copies of every layer. As can be seen in the diagram below, the new image is sharing its four underlying layers with the ubuntu:15.04 image.

The docker history command also shows the size of each image layer. As you can see, the 94e6b7d2c720 layer is only consuming 12 Bytes of disk space. This means that the changed-ubuntu image we just created is only consuming an additional 12 Bytes of disk space on the Docker host - all layers below the 94e6b7d2c720 layer already exist on the Docker host and are shared by other images.

This sharing of image layers is what makes Docker images and containers so space efficient.

Copying makes containers efficient

You learned earlier that a container is a Docker image with a thin writable, container layer added. The diagram below shows the layers of a container based on the ubuntu:15.04 image:

All writes made to a container are stored in the thin writable container layer. The other layers are read-only (RO) image layers and can’t be changed. This means that multiple containers can safely share a single underlying image. The diagram below shows multiple containers sharing a single copy of the ubuntu:15.04 image. Each container has its own thin RW layer, but they all share a single instance of the ubuntu:15.04 image:

When an existing file in a container is modified, Docker uses the storage driver to perform a copy-on-write operation. The specifics of operation depends on the storage driver. For the AUFS and OverlayFS storage drivers, the copy-on-write operation is pretty much as follows:

• Search through the image layers for the file to update. The process starts at the top, newest layer and works down to the base layer one layer at a time.
• Perform a “copy-up” operation on the first copy of the file that is found. A “copy up” copies the file up to the container’s own thin writable layer.
• Modify the copy of the file in container’s thin writable layer.

Btrfs, ZFS, and other drivers handle the copy-on-write differently. You can read more about the methods of these drivers later in their detailed descriptions.

Containers that write a lot of data will consume more space than containers that do not. This is because most write operations consume new space in the container’s thin writable top layer. If your container needs to write a lot of data, you should consider using a data volume.

A copy-up operation can incur a noticeable performance overhead. This overhead is different depending on which storage driver is in use. However, large files, lots of layers, and deep directory trees can make the impact more noticeable. Fortunately, the operation only occurs the first time any particular file is modified. Subsequent modifications to the same file do not cause a copy-up operation and can operate directly on the file’s existing copy already present in the container layer.

Let’s see what happens if we spin up 5 containers based on our changed-ubuntu image we built earlier:

1. From a terminal on your Docker host, run the following docker run command 5 times.

    $ docker run -dit changed-ubuntu bash
   
    75bab0d54f3cf193cfdc3a86483466363f442fba30859f7dcd1b816b6ede82d4
   
    $ docker run -dit changed-ubuntu bash
   
    9280e777d109e2eb4b13ab211553516124a3d4d4280a0edfc7abf75c59024d47
   
    $ docker run -dit changed-ubuntu bash
   
    a651680bd6c2ef64902e154eeb8a064b85c9abf08ac46f922ad8dfc11bb5cd8a
   
    $ docker run -dit changed-ubuntu bash
   
    8eb24b3b2d246f225b24f2fca39625aaad71689c392a7b552b78baf264647373
   
    $ docker run -dit changed-ubuntu bash
   
    0ad25d06bdf6fca0dedc38301b2aff7478b3e1ce3d1acd676573bba57cb1cfef
   

   This launches 5 containers based on the changed-ubuntu image. As each container is created, Docker adds a writable layer and assigns it a random UUID. This is the value returned from the docker run command.
2. Run the docker ps command to verify the 5 containers are running.

    $ docker ps
    CONTAINER ID    IMAGE             COMMAND    CREATED              STATUS              PORTS    NAMES
    0ad25d06bdf6    changed-ubuntu    "bash"     About a minute ago   Up About a minute            stoic_ptolemy
    8eb24b3b2d24    changed-ubuntu    "bash"     About a minute ago   Up About a minute            pensive_bartik
    a651680bd6c2    changed-ubuntu    "bash"     2 minutes ago        Up 2 minutes                 hopeful_turing
    9280e777d109    changed-ubuntu    "bash"     2 minutes ago        Up 2 minutes                 backstabbing_mahavira
    75bab0d54f3c    changed-ubuntu    "bash"     2 minutes ago        Up 2 minutes                 boring_pasteur
   

   The output above shows 5 running containers, all sharing the changed-ubuntu image. Each CONTAINER ID is derived from the UUID when creating each container.
3. List the contents of the local storage area.

    $ sudo ls /var/lib/docker/containers
   
    0ad25d06bdf6fca0dedc38301b2aff7478b3e1ce3d1acd676573bba57cb1cfef
    9280e777d109e2eb4b13ab211553516124a3d4d4280a0edfc7abf75c59024d47
    75bab0d54f3cf193cfdc3a86483466363f442fba30859f7dcd1b816b6ede82d4
    a651680bd6c2ef64902e154eeb8a064b85c9abf08ac46f922ad8dfc11bb5cd8a
    8eb24b3b2d246f225b24f2fca39625aaad71689c392a7b552b78baf264647373
   

Docker’s copy-on-write strategy not only reduces the amount of space consumed by containers, it also reduces the time required to start a container. At start time, Docker only has to create the thin writable layer for each container. The diagram below shows these 5 containers sharing a single read-only (RO) copy of the changed-ubuntu image.

If Docker had to make an entire copy of the underlying image stack each time it started a new container, container start times and disk space used would be significantly increased.

Data volumes and the storage driver

When a container is deleted, any data written to the container that is not stored in a data volume is deleted along with the container.

A data volume is a directory or file in the Docker host’s filesystem that is mounted directly into a container. Data volumes are not controlled by the storage driver. Reads and writes to data volumes bypass the storage driver and operate at native host speeds. You can mount any number of data volumes into a container. Multiple containers can also share one or more data volumes.

The diagram below shows a single Docker host running two containers. Each container exists inside of its own address space within the Docker host’s local storage area (/var/lib/docker/...). There is also a single shared data volume located at /data on the Docker host. This is mounted directly into both containers.

Data volumes reside outside of the local storage area on the Docker host, further reinforcing their independence from the storage driver’s control. When a container is deleted, any data stored in data volumes persists on the Docker host.