Sometimes, you may want your containers to wait for another service to be reachable before starting them. The initContainers property allows you to define a set of containers to start before your normal containers (1). In a real world uses case, it can be really convenient to check the connections specified in your configMap/Secret files are ready. It could be a MySQL, MongoDB, Redis instance, etc.
You can leverage the init container with a TCP ping to test if a database port is reachable (2). To reduce repetion, it is possible to specify the same configMap file as the one used by your container.
[...] spec: initContainers: - name: init-mysql image: gcr.io/kubernetes-e2e-test-images/dnsutils:1.3 command: ['sh', '-c', "until nc -z $MYSQL_HOST $MYSQL_PORT; do echo waiting for mysql; sleep 3; done; echo $?"] envFrom: - configMapRef: name: config containers: [...]
The config file could look like the following:
--- apiVersion: v1 kind: ConfigMap metadata: name: "config" data: MYSQL_HOST: "my.host.managed.mysql.com" MYSQL_PORT: "3306"
Kubernetes will wait for each init container to terminate with an exit code 0 (successful) before starting your container(s). The command will ping the database port every 3 seconds, and will finish once the port is reachable.
In our case, we never had the need to check if a connection was ready, until... few weeks ago.
With our growing infrastructure, we started to notice a timeout connection error on some of our kubernetes jobs.
We start several time per day different batches of pods, and our Managed Kubernetes Provider (Digital Ocean) is auto-scaling nodes to match the load request (~50 servers).
All the pods encountering a timeout problem were spawned on new nodes, which raised suspicion.
After digging into the problem, it seems like Digital Ocean is setting the nodes in a Ready state without checking if the firewall of their managed database was updated. Our nodes have a tag
allow-mysql, and our managed mysql database allows connections from resources with the former same tag.
Depending of the number of nodes requiring the firewall access, it can take up to 4-5 minutes for the firewall to be ready. Therefore, the init container is a very good work-around to make sure all firewalls are open before starting the containers.
Another example of init container is used by the Traefik helm chart, to set the correct permissions on a volume mounted in the container.
initContainers:  # The "volume-permissions" init container is required if you run into permission issues. # Related issue: https://github.com/traefik/traefik/issues/6972 - name: volume-permissions image: busybox:1.31.1 command: ["sh", "-c", "chmod -Rv 600 /data/*"] volumeMounts: - name: data mountPath: /data
mysqladmin pinga mysql instance for instance. TCP ping is only one way among others.