At our company, we primarily operate on bare-metal infrastructure, which provides us with very low network latency within a zone or datacenter. When we began migrating services to Kubernetes (also running on bare-metal), we needed to ensure that certain daemons running on each bare-metal server were also present on Kubernetes nodes in a Kubernetes-native manner. To achieve this, we utilized DaemonSets, which are designed for this purpose.

In this article, we will focus on a specific service we use, mcrouter, a proxy that enhances memcached’s reliability by facilitating read/write operations to multiple memcached servers.

During my work, I stumbled upon some little problems that often require to deploy several component for just a simple task.

For example, we had to validate with our CDN that our endpoint was own by us. Fair enough, let’s set a TXT record on our DNS servers! Well, it doesn’t always work that way. Those providers required an HTTP validation with a specific URL and content.

Context

We have two providers requesting an HTTP endpoint with a specific url/content.

I have been strugling with Kafka recently, or at least log4j. The default configuration output a lot of logs to stdout. In my actual configuration, stdout logs from an application are captured by systemd-journald and then given to rsyslog. This can come handy when the application doesn’t know how to log with syslog. However, I had an issue with Kafka and journald where logs were also written to the filesystem until the partition is full. So output log directly to rsyslog was necessary.