Checking if a Firewall is ready before starting an application

October 4, 2021
devops kubernetes digitalocean

Sometimes, you may want your containers to wait for another service to be reachable before starting them. The initContainers property allows you to define a set of containers to start before your normal containers (1). In a real world uses case, it can be really convenient to check the connections specified in your configMap/Secret files are ready. It could be a MySQL, MongoDB, Redis instance, etc.

You can leverage the init container with a TCP ping to test if a database port is reachable (2). To reduce repetion, it is possible to specify the same configMap file as the one used by your container.

[...]				
        spec:
          initContainers:
          - name: init-mysql
            image: gcr.io/kubernetes-e2e-test-images/dnsutils:1.3
            command: ['sh', '-c', "until nc -z $MYSQL_HOST $MYSQL_PORT; do echo waiting for mysql; sleep 3; done; echo $?"]
            envFrom:
            - configMapRef:
                name: config
            containers:
[...]

The config file could look like the following:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: "config"

data:
  MYSQL_HOST: "my.host.managed.mysql.com"
  MYSQL_PORT: "3306"

Kubernetes will wait for each init container to terminate with an exit code 0 (successful) before starting your container(s). The command will ping the database port every 3 seconds, and will finish once the port is reachable.

Digital Ocean Managed database "trusted source"

In our case, we never had the need to check if a connection was ready, until... few weeks ago.

With our growing infrastructure, we started to notice a timeout connection error on some of our kubernetes jobs.

We start several time per day different batches of pods, and our Managed Kubernetes Provider (Digital Ocean) is auto-scaling nodes to match the load request (~50 servers).

All the pods encountering a timeout problem were spawned on new nodes, which raised suspicion.

After digging into the problem, it seems like Digital Ocean is setting the nodes in a Ready state without checking if the firewall of their managed database was updated. Our nodes have a tag allow-mysql, and our managed mysql database allows connections from resources with the former same tag.

Depending of the number of nodes requiring the firewall access, it can take up to 4-5 minutes for the firewall to be ready. Therefore, the init container is a very good work-around to make sure all firewalls are open before starting the containers.

Not only there to ping: Setting permissions

Another example of init container is used by the Traefik helm chart, to set the correct permissions on a volume mounted in the container.

  initContainers: []
    # The "volume-permissions" init container is required if you run into permission issues.
    # Related issue: https://github.com/traefik/traefik/issues/6972
    - name: volume-permissions
      image: busybox:1.31.1
      command: ["sh", "-c", "chmod -Rv 600 /data/*"]
      volumeMounts:
        - name: data
          mountPath: /data
  • 1: As the blocking behavior of the init container can be useful, it can also be harmful and block for a long duration the start of your containers. Monitor careful how it behaves to prevent future troubles.
  • 2: Using a docker image with mysql tools could allow you to mysqladmin ping a mysql instance for instance. TCP ping is only one way among others.
  • 3: Official documentation
Back to Blog