Checking if a Firewall is ready before starting an application

October 4, 2021 · 3 min read
devopskubernetesdigitalocean

Sometimes, you may want your containers to wait for another service to be reachable before starting them. The initContainers property allows you to define a set of containers to start before your normal containers (1). In a real world uses case, it can be really convenient to check the connections specified in your configMap/Secret files are ready. It could be a MySQL, MongoDB, Redis instance, etc.

You can leverage the init container with a TCP ping to test if a database port is reachable (2). To reduce repetion, it is possible to specify the same configMap file as the one used by your container.

[...]				
        spec:
          initContainers:
          - name: init-mysql
            image: gcr.io/kubernetes-e2e-test-images/dnsutils:1.3
            command: ['sh', '-c', "until nc -z $MYSQL_HOST $MYSQL_PORT; do echo waiting for mysql; sleep 3; done; echo $?"]
            envFrom:
            - configMapRef:
                name: config
            containers:
[...]

The config file could look like the following:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: "config"

data:
  MYSQL_HOST: "my.host.managed.mysql.com"
  MYSQL_PORT: "3306"

Kubernetes will wait for each init container to terminate with an exit code 0 (successful) before starting your container(s). The command will ping the database port every 3 seconds, and will finish once the port is reachable.

Digital Ocean Managed database “trusted source”#

In our case, we never had the need to check if a connection was ready, until… few weeks ago.

With our growing infrastructure, we started to notice a timeout connection error on some of our kubernetes jobs.

We start several time per day different batches of pods, and our Managed Kubernetes Provider (Digital Ocean) is auto-scaling nodes to match the load request (~50 servers).

All the pods encountering a timeout problem were spawned on new nodes, which raised suspicion.

After digging into the problem, it seems like Digital Ocean is setting the nodes in a Ready state without checking if the firewall of their managed database was updated. Our nodes have a tag allow-mysql, and our managed mysql database allows connections from resources with the former same tag.

Depending of the number of nodes requiring the firewall access, it can take up to 4-5 minutes for the firewall to be ready. Therefore, the init container is a very good work-around to make sure all firewalls are open before starting the containers.

Not only there to ping: Setting permissions#

Another example of init container is used by the Traefik helm chart, to set the correct permissions on a volume mounted in the container.

  initContainers: []
    # The "volume-permissions" init container is required if you run into permission issues.
    # Related issue: https://github.com/traefik/traefik/issues/6972
    - name: volume-permissions
      image: busybox:1.31.1
      command: ["sh", "-c", "chmod -Rv 600 /data/*"]
      volumeMounts:
        - name: data
          mountPath: /data
  • 1: As the blocking behavior of the init container can be useful, it can also be harmful and block for a long duration the start of your containers. Monitor careful how it behaves to prevent future troubles.
  • 2: Using a docker image with mysql tools could allow you to mysqladmin ping a mysql instance for instance. TCP ping is only one way among others.
  • 3: Official documentation
Back to Blog