How to debug RDS database from a Kubernetes cluster

August 5, 2023
AWS Kubernetes

When running Kubernetes workflows, most people tend to keep the stateless applications like applications on the cluster, and the stateful applications like databases outside of the cluster, on the same VPC. This is a good practice, but it makes debugging a bit more complicated.

Note: I use AWS as an example, but this can be applied to any cloud provider with managed database & K8s using private networks

I assume your database is private and only accessible from a VPC. Maybe for some obscure reasons, you still need to access this database to debug something (Staging database of course cough cough). Well, it might be a bit annoying.

One way would be to spawn an EC2 instance inside the same VPC, install a bunch of tools like netcat, mysql, postgres and debug from the VM. Not very convenient, you might left resources running and it's not very reproducible.

What I would advise, is to use a "reverse-proxy" container inside your cluster. Being on the same VPC as your managed database, you will be able to access it, and this container port can be port-forwarded to your machine. Of course, all the following expect you to have enough privileges to create a pod inside your cluster.

----------        -------        ------------
| Laptop |  <-->  | Pod |  <-->  | Database |
----------        -------        ------------

You will need 2 terminal windows:

  • One to create the pod opening the connection to the database
  • One to port-forward the pod port to your local machine
DB_PORT=5432
DB_HOST=64eddd8e4b61-postgresql.993191f72591.eu-central-1.rds.amazonaws.com

kubectl run psql-tunnel -it --image=alpine/socat --tty --rm --expose=true --port=$DB_PORT tcp-listen:$DB_PORT,fork,reuseaddr tcp-connect:$DB_HOST:$DB_PORT

Here is an explanation of each argument that might be useful to understand what is happening:

  • socat: This image allows to establish two bidirectional byte streams and transfers data between them, perfect for our use case!
  • --rm: Removes the container automatically once it exits. (It helps in cleanup and avoids leaving unused containers.)
  • tcp-listen:$DB_PORT,fork,reuseaddr: Configures socat to listen on TCP port 5432 inside the container. The fork option allows handling multiple connections, and reuseaddr allows reusing the address.
  • tcp-connect:$DB_HOST:$DB_PORT: Specifies the destination address and port where the incoming connections to 5432 inside the container will be forwarded. It connects to the host 64eddd8e4b61-postgresql.993191f72591.eu-central-1.rds.amazonaws.com on port 5432.

and the second command to port-forward the container port to localhost:

kubectl port-forward psql-tunnel 54323:5432

You can now use any GUI to connect to the DB and visualize what is happening (TablePlus, DBeaver, PhpMyAdmin, etc)!

Back to Blog