Category Archives: Uncategorized

Building docker containers with make on CoreOS

In this post, I’ll show how you can use a Makefile to build docker containers, and how to orchestrate the builds of containers which have build-time dependencies. Finally, I’ll show you an easy way to install make into CoreOS.

When I began my docker journey, I started out with shell scripts which would prepare the build context before running docker build, then performing cleanups.

Any non-trivial dockerized application will be made up of many containers, and it wasn’t long before these scripts started to get more elaborate, and I felt that using a Makefile would help me accelerate build times by eliminating unchanged steps in the process.

Not only that, a Makefile communicates the intent and the dependencies better to other developers.

I’m not an expert on make, but I’ve picked up a few useful tricks and I’ve boiled them down into this sample (note, Makefiles are finicky about tabs, so I’ve put this up on a gist)

#-----------------------------------------------------------------------------
# configuration - see also 'make help' for list of targets
#-----------------------------------------------------------------------------

# name of container 
CONTAINER_NAME = myregistry.example.com:5000/mycontainer:latest

# list of dependencies in the build context - this example just finds all files
# in the 'files' subfolder 
DEPS = $(shell find myfiles -type f -print)

# name of instance and other options you want to pass to docker run for testing
INSTANCE_NAME = mycontainer
RUN_OPTS =

#-----------------------------------------------------------------------------
# default target
#-----------------------------------------------------------------------------

all   : ## Build the container - this is the default action
all: build

#-----------------------------------------------------------------------------
# build container
#-----------------------------------------------------------------------------

.built: . $(DEPS)
	docker build -t $(CONTAINER_NAME) .
	@docker inspect -f '{{.Id}}' $(CONTAINER_NAME) > .built

build : ## build the container
build: .built
 
clean : ## delete the image from docker
clean: stop
	@$(RM) .built
	-docker rmi $(CONTAINER_NAME)

re    : ## clean and rebuild
re: clean all

#-----------------------------------------------------------------------------
# repository control
#-----------------------------------------------------------------------------

push  : ## Push container to remote repository
push: build
	docker push $(CONTAINER_NAME)

pull  : ## Pull container from remote repository - might speed up rebuilds
pull:
	docker pull $(CONTAINER_NAME)

#-----------------------------------------------------------------------------
# test container
#-----------------------------------------------------------------------------

run   : ## Run the container as a daemon locally for testing
run: build stop
	docker run -d --name=$(INSTANCE_NAME) $(RUN_OPTS) $(CONTAINER_NAME)

stop  : ## Stop local test started by run
stop:
	-docker stop $(INSTANCE_NAME)
	-docker rm $(INSTANCE_NAME)

#-----------------------------------------------------------------------------
# supporting targets
#-----------------------------------------------------------------------------

help  : ## Show this help.
	@fgrep -h "##" $(MAKEFILE_LIST) | fgrep -v fgrep | sed -e 's/\\$$//' | sed -e 's/##//'

.PHONY : all build clean re push pull run stop help

Once this is configured and sitting alongside your Dockerfile, all you need to to is run make and it will build your container. If you’re feeling confident, you can `make push` to build and push in one step.

The default build target depends on a file called .built. You can see that .built depends on the current directory, plus any files defined in the DEPS variable. If any of those dependencies are newer than the build, then docker build will be executed, and the .built file is updated with a new timestamp.

This file also has a push target, which ensures the build is up to date, and pushes to your remote registry.

There’s also some simple targets for run and stop so you can quickly test a container after building.

Finally, there’s a neat help target, which simply parses the comments preceded by ## to provide some simple help on the available targets!

Orchestrating multiple container builds

I set up one Makefile for each container, then higher up I have a simpler Makefile which I can use to build everything, or selective subsets.

all: common web
	$(MAKE) -C myapp-mysql
	$(MAKE) -C myapp-redis

web: common
	$(MAKE) -C myapp-web-base
	$(MAKE) -C myapp-web

common:
	$(MAKE) -C myapp-common

.PHONY : all web common

Pretty simple – if I want to just build the web containers, I make web, and it will first ensure the common containers are up to date before running the Makefiles for myapp-web-base followed by myapp-web

How to install make on CoreOS

If you’re using CoreOS for your development environment, you might be thinking “but I don’t have make, and no means to install it!”. Well here is a one-liner to brighten your day

docker run -ti --rm -v /opt/bin:/out ubuntu:14.04 \
  /bin/bash -c "apt-get -y install make && cp /usr/bin/make /out/make"

That just runs a temporary ubuntu container, installs make, then copies out of the container into the host /opt/bin directory.

Credits

Thanks to Guillaume Charmes for his blog post on docker Makefiles, and to Payton White for his neat one-liner for adding help to Makefiles.

There’s a bit of me in PHP!

Last October I found a bug in PHP’s SOAP module. It was pretty obscure bug in the way PHP used Digest authentication. As it was a showstopper for me, I submitted a bug report and wrote a patch to fix it.

It took a while, but my patch has finally been merged into the 5.* and 6.0 sources!

I’ve been using PHP for ten years, and so it’s perhaps a little surprising it’s taken me this long to give anything back. Truth is, this was the first time I’ve come across a bug which halted my development work.

It’s a tiny fix, 6 lines of code, but it makes me happy that after all this time, finally, there’s a bit of me in PHP! Hurrah for open source software!

Geograph Torrents

Something Geograph has needed for some time is a way to get at the entire archive without causing us huge bandwidth bills – it’s currently around 100GB in size.

So I’m please to announce http://torrents.geograph.org.uk/ where you will be able to download the entire archive via BitTorrent!

More details…

We’re going to release the archive as a series of volumes, each containing around 50,000 images and weighing in at around 5GB. The first volume is available now, with a smaller sample volume which allows you to preview how the metadata is delivered.

All the metadata for each volume is delivered as a RDF formatted file called manifest.rdf, which should be self explanatory, here is a sample fragment:
Look

<rdf:Description rdf:about=”00/00/000014.jpg“>
<dc:identifier>http://www.geograph.org.uk/photo/14</dc:identifier>
<dc:title>Durdle Door from the east</dc:title>
<dc:subject>Coastline/Beaches</dc:subject>
<dc:creator>Helena Downton</dc:creator>
<dc:dateSubmitted>2005-03-06T10:20:26Z</dc:dateSubmitted>
<georss:point relationshiptag=”is-picture-of”>50.6205085825182 -2.27398572077417</georss:point>
<georss:point relationshiptag=”was-taken-from”>50.6211552007986 -2.26663784055893</georss:point>
<dc:licence>http://creativecommons.org/licenses/by-sa/2.0/</dc:licence>
<dc:format>image/jpeg</dc:format>
<dc:type>http://purl.org/dc/dcmitype/StillImage</dc:type>
<dc:publisher>www.geograph.org.uk</dc:publisher>
</rdf:Description>

How to download

If you are new to using bittorrent, you will need some software, such as Vuze (aka Azureus) or uTorrent. Once installed, go to http://torrents.geograph.org.uk/ and click a torrent link to set it downloading – easy!

Please seed!

Please seed these torrents if you are able – simply leave your torrent client software running after your download has completed. You’ll be helping others to complete their downloads faster!