gee-whiz

State of the Art CI

Sun, 30 Sep 2018 16:55:21 +0000

Here you can find the slides of my talk at the JUG Saxony Day 2018. If you have any question, email us :)

Extending build containers: Generic caching

Wed, 26 Sep 2018 00:00:00 +0000

Declaring your build environment by using Dockerfiles and creating separate containers for each and every build is good practice. But starting in a fresh environment each time prevents various caching mechanisms from having any effect. The result is an increase in build time as well as network traffic, where especially the latter can cause some bottlenecking to occur. This affects mainly git repositories, build tools like Maven and npm, but also Docker builds itself. The git repository has to be cloned from scratch. Dependencies for Maven and npm have to be fetched from their official repositories on every build, even if the majority did not change. In case of Docker builds, at least the base image must be pulled. The Docker build cache is also empty.

This post covers a couple of methods to overcome these limitation while still keeping isolation and reproducibility guarantees.

tl;dr: We create a Docker image that contains a generic template for building caches for git repositories, Maven artifacts and npm dependencies by (a) cloning relevant git repositories into a single bare reference repository and (b) searching for pom.xml and package.json files and pre-fetching their dependencies, thus presenting depending builds with an immediately available cache. That image must be build either on a regular basis or on demand in order to keep the cache up to date.

The details and full implementation can be found on Github.

Building a generic caching solution for each tool

The following sections show the taken approach for git, maven and npm. The caching of Docker images and builds will be discussed in a future blog post.

Setup and strategy

Let’s make some assumptions before we begin. Assume we have a Docker image hub.gee-whiz.de/build-env:latest where builds are executed in. This contains our default build environment with a set of standard tools. This would also be the starting point for new users who want to use a build container in general.

There’s also a central git server where multiple projects, each with multiple repositories, reside. Those are stored in hierarchical form like git.gee-whiz.de/project/repository. We’re aiming to build a generic caching solution for each project for all its repositories. To do so, we create a separate so called project specific Docker image with an embedded cache by extending our default build image and creating the actual cache as part of the docker build process. Say we have a project named website, the resulting image will be named hub.gee-whiz.de/build-env-website:latest. That image will contain a cache for every repository of that project.

The cache is now being build as follows. Depending on your environment and requirements, you may of course deviate in certain steps or add additional mechanisms:

Select a project we want to build an image with a suitable cache for.
Determine available git repositories of said project and clone them all in a single bare reference (git doc) repository. This will act as the git cache.
Temporarily checkout the default branches of every repository. On each we
1. Search for Maven projects (pom.xml) and build them with --fail-never and the dependency:go-offline goal. The fetched artifacts will be stored in the local .m2 folder.
2. Search for npm projects (package.json) and install them. The fetched dependencies will be stored in the local npm cache.

We now dive into the implementation details of each step involved. Note that we also support to optionally ignore repositories by setting the environment variable IGNORE_REPOS. There are also a bunch of other variables being used that have to be set appropriately. We come back to those later.

Git

Determine available git repositories for the project website from a central Atlassian Bitbuck instance:

FILTER="$(echo "(\"${IGNORE_REPOS}\")" | sed 's/,/"|"/g')" \
&& REPOS=$(curl -S -u ${BITBUCKET_HTTP_CREDENTIALS} https://bitbucket.gee-whiz.de/rest/api/1.0/projects/${PROJECT}/repos?limit=100 \
    | jq -cM '.values[] | {name: .name, url: .links.clone[].href} | select(.url | contains("ssh://"))' \
    | grep -Ev "${FILTER}") \
&& echo -n "Considering $(echo ${REPOS} | jq -cMs '. | length') repositories for buildcontainer creation process: " \
&& echo "$(echo ${REPOS} | jq -cM '.name ' | jq -cMs '.')"

Clone all git repositories into a single bare reference repository:

cd /var/tmp/cache/git \
&& git init --bare \
&& for REPO in ${REPOS}; do \
    REPO_NAME=$(echo ${REPO} | jq -cMr '.name') \
    && REPO_URL=$(echo ${REPO} | jq -cMr '.url') \
    && git remote add ${REPO_NAME} ${REPO_URL} \
    ; done \
&& git fetch --all --quiet \
&& cd /

Temporarily checkout the default branch of every repository:

mkdir -p /tmp/git \
&& for REPO in ${REPOS}; do \
    REPO_NAME=$(echo ${REPO} | jq -cMr '.name') \
    && REPO_URL=$(echo ${REPO} | jq -cMr '.url') \
    && git clone --reference /var/tmp/cache/git ${REPO_URL} /tmp/git/${REPO_NAME} \
    ; done

Prefetch Maven dependencies

for REPO_DIR in /tmp/git/*; do \
    echo "Scanning for Maven project in ${REPO_DIR}" \
    && if [ -f ${REPO_DIR}/pom.xml ]; then \
        echo "Found pom.xml in ${REPO_DIR}" \
        && cd ${REPO_DIR} \
        && JAVA_HOME=${JDK_HOME} ${MAVEN_HOME}/bin/mvn -gs /tmp/build/maven-global-settings.xml -B -V -q --fail-never org.apache.maven.plugins:maven-dependency-plugin:3.0.2:go-offline \
        ; fi \
    ; done \

Prefetch npm dependencies

PATH=${PATH}:${NODEJS_HOME}/bin \
&& for REPO_DIR in /tmp/git/*; do \
    echo "Scanning for npm projects in ${REPO_DIR}/*" \
    && for NODE_PACKAGE in $(find ${REPO_DIR} -maxdepth 2 -iname package.json); do \
        NODE_PACKAGE_DIR=$(dirname ${NODE_PACKAGE}) \
        && echo "Found package.json in ${NODE_PACKAGE_DIR}" \
        && cd ${NODE_PACKAGE_DIR} \
        && npm --globalconfig /tmp/build/npm-global-rc install \
        ; done \
    ; done \

Using ONBUILD

Instead of copying the above bash lines in each and every project specific Dockerfile, in practice we use Dockers ONBUILD instruction: We create a separate template image called hub.gee-whiz.de/build-env-template:latest which inherits from the default build image and contains instructions in the form of ONBUILD RUN build_cache.sh. Those instructions are then being executed when a dependent child image is built. Thus moving the generic caching implementation as some kind of template into a central place, thereby improving maintainability. The project the cache should be build for must be given as a build parameter.

We’re now able to create an arbitrary amount of project specific images by just inheriting from that single template image. If no further customization is needed, the Dockerfile for the image hub.gee-whiz.de/build-env-website:latest would just be:

FROM hub.gee-whiz.de/build-env-template:latest

Creating the project specific image on Jenkins

The previous bash snippets were using a couple of environment variables we’re now going to address. Included where locations for specific tools, configuration files, credentials and of course the projects name. As these kind of artifacts should not be included directly within the Docker image (to maintain a single location for configurations and for security reasons), we inject them during build time with the help of Jenkins.

We also implement the pipeline as a shared library. Thus allowing a simple usage and again, maintainability. It’s implemented as a function named buildProjectSpecificDockerImage and looks like this:

buildProjectSpecificDockerImage {
    triggeredBy = '../build-env-template/master'
    project = 'website'
    ignoreRepos = [
        'testing',
        'old'
    ]
}

The build is triggered whenever the extended template-image was built. We pass the project name the cache should be build for, but also the repositories to ignore. Optionally, the function also allows to specify custom configuration files, credentials or specific tool to use for building the image. You can see the full implementation on Github.

Conclusion

On the premise these project specific build images are build either on a regular basis or on demand, builds running inside those containers have immediate access to the cache for git, Maven and npm. If the image is not quite up to date, for instance when a dependency was bumped recently, only that specific artifact musst be fetched. Same principle goes for the git repository.

Thanks to Dockers image layering technique, the cache can be shared by multiple in parallel running builds on the same host while still being isolated. To further improve build time, the build image might also be distributed to build nodes in advance, right after the build image is created. This alleviates possibly time consuming on demand image pulls.

Remote debugging containerized Jenkins builds: debugShell()

Mon, 24 Sep 2018 00:00:00 +0000

Providing build environments by employing some kind of container technology is quite common nowadays. When looking at Docker as an example, it offers various benefits: Declaring the environment in a textual way by using a Dockerfile , you can combine it with a versioning tool like git and track every change. Each build is running inside its own fresh container and is therefore isolated and also reproducible. If you choose to use a clustering solution like Docker Swarm or Kubernetes, you’re also able to dynamically scale available build power at run time.

Any downsides?

Of course this comes at a cost. Besides building, maintaining and supporting additional infrastructure, from a perspective of developers and DevOps engineers, the usability is definitely lowered. If one is using a central build server whose builds are executed on that same machine, debugging problems seems way easier: When a build fails, we simply browse its work space and check on relevant files to investigate potential causes. If that’s not enough, we can always SSH into the machine and continue debugging locally.

On the other hand, there are some challenges involved when executing a build inside a container on some node somewhere on a container platform. In the nature of ephemeral build environments where separate containers are created and teared down for each individual build, accessing build artifacts or work space files becomes impossible. Using SSH is also problematic: Usually there is no SSH daemon listening inside the container. It would also only be reachable from the outside, if you expose the daemons port. But even then, you don’t even know exactly on which node the container is running on.

Proposed solution and tl;dr

We’ll create a TCP shell server inside the container a client can connect to. To overcome the connectivity issues we’ll setup a dynamic reverse proxy on a public reachable machine that tunnels the connection inside the container. Following this approach, we’ll actually gain some additional features along the road. Using the right combination of tools enable us to:

Set breakpoints within a Jenkinsfile
Gain an interactive shell inside the buildcontainer
Inherit the actual build environment within that shell: Setting a breakpoint inside withMaven() for example, would leave us with the corresponding configuration when calling mvn.
Modify the workspace during build time

Tool introduction

This is a brief introduction of the tools and their respective role to achieve the laid out features above.

frp (fast reverse proxy): Exposes TCP sockets behind a NAT, i.e. inside a container. There is a server part that runs on a public reachable machine. The client runs inside the container and connects to the server and thereby establishes a reverse tunnel to a socket inside the container. The socket inside the container is now reachable via the public machine.
socat (SOcket CAT): Multipurpose socket relay utility. Used to let a developer connect to the buildcontainer and to establish the interactive shell session.
Jenkins shared library: To create a custom build step which functions as a break point that halts the running build, constructs the shell and establishes the reverse tunnel.

Setup

The following shows the basic idea on how to implement the remote debugging functionality. For those of you who want to dive in deeper: Everything laid out here here can be fully explored by checking out the corresponding repository over at Github.

frp

As frp is responsible for establishing a communication link between the client and the buildcontainer, it needs to be setup first. As already mentioned, we require a public reachable node the server component (frpd) must be installed on. Let’s call it jenkins.gee-whiz.de. frp’s feature set is quite extensive, but a minimal server configuration that fit our needs would look like this:

[common]
bind_port = 7000
privilege_allow_ports = 17000-17100

We define a listening port of 7000 where the clients inside of buildcontainers are supposed to connect to, as well as a range of ports frpd may use on the public node for inbound tunneled connections. On the client side of things, i.e. inside the buildcontainer, a suitable frpc configuration would be:

[common]
server_addr = jenkins.gee-whiz.de
server_port = 7000

[shell]
local_ip = 127.0.0.1
local_port = 22222
remote_port = 17000

Beside the servers address and port in the common section, we define the local endpoint that should be exposed. We’re planning to let socat listen on 127.0.0.1:22222. socat will provide the interactive shell a client is able to connect to. In this case, the remote port is explicitly set to 17000. To let frpd automatically choose a free port within the specified range on the public node, the remote port must be set to 0.

When running the client against the server, the resulting tunnel would basically look like this:

[ jenkins.gee-whiz.de:17000 ] --> [ some_docker_server ] --> [ inside_container:22222 ]

socat

Socat will be used in two ways: First, it will act as a server inside the container that is responsible for providing the interactive shell. Second, it will be run on the client to connect to said server via the reverse tunnel created by frp.

On the server side of things we’re going to spawn a TCP socket on port 22222 and redirect incoming traffic to a bash instance. We also need to set some additional options as laid out in this blog post to create a fully interactive PTY. This lets you do things you would expect from a terminal, like signal handling (ctrl+c), tab-completion or using a text editor like vim. This is the command to spawn the server process:

$ socat exec:'bash -li',pty,stderr,setsid,sigint,sane tcp-listen:22222

Once we created both the frp reverse tunnel and the socat server we’re able to establish the connection from a client to the container. This can be done from an arbitrary Linux system with socat installed, or by using something like Mintty with MSYS2 or Cygwin on Windows. The native Windows cmd.exe technically also works, but misses a few interactive shell features mentioned above. As on the server side, we also set a few additional options to make the shell fully interactive:

$ socat file:$(tty),raw,echo=0,escape=0x0f tcp:jenkins.gee-whiz.de:17000
$ ls -al
total 4
drwxr-xr-x 1 jenkins jenkins  30 Feb 14 08:46 .
drwxr-xr-x 1 jenkins jenkins 300 Feb 14 08:46 ..
drwxr-xr-x 1 jenkins jenkins 136 Feb 14 08:46 .git
-rw-r--r-- 1 jenkins jenkins 676 Feb 14 08:46 Jenkinsfile

Side note: Instead of socat you may also use a real SSH daemon to allows every standard SSH client to connect. You would also automatically gain features like authentication, encryption and file transfer.

Jenkins shared library

Doing everything above manually inside a sh step would be way to cumbersome. Instead we’ll use the Jenkins shared library mechanism to neatly encapsulate all of those actions into a single custom step. This enables us to halt the execution of the Jenkinsfile and start the debug shell with just a single command. We name that custom step we’re about to create debugShell().

We’re not diving into all the details about creating shared libraries here. Also note that this version is way stripped down to bring across the basic idea. Among other things, the full version adds automatic port determination and therefore multi user capability.

This is the content of the file debugShell.groovy inside our shared library:

#!/usr/bin/env groovy

def call() {
    echo "------------------------------"
    echo "Spawning debug shell socket"
    echo "------------------------------"
    echo "Reverse connection endpoint to localhost:22222 created on: jenkins.gee-whiz.de:17000"
    echo "Connect via: socat file:\$(tty),raw,echo=0,escape=0x0f tcp:jenkins.gee-whiz.de:17000"
    echo "Listening for incoming connection and pausing execution until connection terminates"
    echo "------------------------------"
    
    sh '/tmp/debugshell/create.sh'
    sh 'frpc -c /etc/frp/frpc.ini &'
    sh 'sleep 1'
    sh 'socat exec:"bash -li",pty,stderr,setsid,sigint,sane tcp-listen:22222'
}

Upon calling the debugShell() function, we can use socat as a client to connect to the buildcontainer. To do so, just execute the echoed command. As the created shell inherits the environment of the current build on the exact location the function is called, access to every environment variable, injected configurations and credentials is being gained. We’re also able to edit files inside the workspace with our favorite editor. As an example, we could modify the content of a pom.xml file, and run Maven on our changes. Upon exiting the debug shell, the build resumes.

Putting it all together

Surly the laid out approach was suitable to understand the basic gist of the setup. Running it in production though, we add a few additional things:

Automatically determine an unused port for socats listening socket inside the container
Use a remote_port of 0 to let the frp server automatically determine an unused port within the specified port range on the public reachable machine
Generating a corresponding frp client configuration
Feed back the chosen port and the endpoint as a whole to the user as a log statement

Use a custom bashrc to print some useful information when a client connects:

  Connected to buildcontainer a163fc77fac0 on Swarm-a163fc77fac0 for job https://jenkins.gee-whiz.de/some_project/some_repository/master/75/ from 127.0.0.1:57964
  Building commit b6050f00bbd7c9a10ebabbd1151dad7130e32e71 on branch master originating from ssh://git@git.gee-whiz.de:2222/some_project/some_repository.git

Be able to run the debugShell in the background while the build continues

Go investigate all the details on Github, if you like. Or drop us an email if you have any questions or suggestions.

Inhalte von Git Projekten verschlüsseln

Fri, 27 Apr 2018 16:10:23 +0000

Inhalte von Git Projekten verschlüsseln

Um z.B. auf github auch sicherheitsrelevante Sachen ablegen zu können, gibt es die Möglichkeit Inhalte eines git Projekt zu verschlüsseln. Ein Beispiel hierfür ist git-crypt. Damit git-crypt auch unter Windows funktionert, muss man folgendes tun:

git-crypt Installation in Msys2

Folgende Pakete installieren pacman -S make gcc openssl
git-crypt clonen: git clone https://github.com/AGWA/git-crypt.git
cd git-crypt
Das Makefile wie in Pull Request beschrieben anpassen (ist noch nicht germerged).
make
make install

Projekt für git-crypt konfigurieren

In bestehendes/neues git projekt wechseln
git-crypt init ausführen, es wird ein default Key angelegt
vim .gitattributes

Folgendes eintragen:

      * filter=git-crypt diff=git-crypt
       .gitattributes !filter !diff
       README.md !filter !diff

Ab diesem moment werden nun alle Dateien verschlüsselt. Außer die .gitattributes und README.md .
Key exportieren git-crypt export-key ../exportKey
Der Key wird von denjenigen benötigt die das Projekt clonen und es auch entschlüsseln sollen

Encryptetes Projekt nutzen

Projekt clonen
Den Key bekommen
git-crypt installieren
git-crypt unlock path/to/exportKeyausführen

Hello World!

Sat, 21 Jan 2017 23:05:23 +0000

Everything starts with a Hello World… So here we go :-)

public class HelloWorld
{
       public static void main (String[] args)
       {
             System.out.println("Hello World!");
             System.out.println("it's so geee");
       }
}

See you on github.