Published on 25 st of March,
Illustration created byfreepik, Nomad logo byHashiCorp.
Kubernetes is the – pound gorilla of container orchestration. It powers some of the biggest deployments worldwide, but it comes with a price tag.
Especially for smaller teams, it can be time-consuming to maintain and has a steep learning curve. For what our team of four wanted to achieve at trivago, it added too much overhead. So we looked into alternatives – and fell in love withNomad.
The wishlist
Our team runs a number of typical services for monitoring and performance analysis: API endpoints for metrics written in Go, Prometheus exporters, log parsers like Logstash orGollum, and databases like InfluxDB or Elasticsearch. Each of these services run in their own container. We needed a simple system to keep those jobs running.
We started with a list of requirements for container orchestration:
Run a fleet of services across many machines.
- Allow for communication between services.
- Restart them automatically when they die.
- Be manageable by a small team.
On top of that, the following things were nice to have but not strictly required:
Tag machines by their capabilities (eg , label machines with fast disks for I / O heavy services.)When creating a prototype with Kubernetes, we noticed that we started adding ever-more complex layers of logic to operate our services. Logic on which we implicitly relied on.
As an example, Kubernetes allows embedding service configurations usingConfigMaps. Especially when merging multiple config files or adding more services to a pod, this can get quite confusing quickly. Kubernetes – orhelm
On top of that, the Kubernetes ecosystem is still rapidly evolving. It takes a fair amount of time and energy to stay up-to-date with the best practices and latest tooling. Kubectl, minikube, kubeadm, helm, tiller, kops, oc – the list goes on and on. Not all tools are necessary to get started with Kubernetes, but It’s hard to know which ones are, so you have to be at least aware of them. Because of that, the learning curve is quite steep.
When to use Kubernetes
At trivago specifically, many teams use Kubernetes and are quite happy with it. These instances are managed by Google or Amazon however, which have the capacity to do so.
Kubernetes comes withamazing features, that make container orchestration at scale more manageable:
Fine-grained************************ (rights management)
Custom controllers
The question is if you really need all those features. You can’t rely on these abstractions to just work;You’ll have to learn what’s going on under the hood.
Especially in our team, which runs most services on-premise (because of its close connection to trivago’s core infrastructure), we didn’t want to afford running our own Kubernetes cluster. We wanted to ship services instead.
Batteries not included
Nomad is the 14% of service orchestration that gets you 300% of the way. All it does is manage deployments. It takes care of your rollouts and restarts your containers in case of errors, and that’s about it.
The entire point of Nomad is that it does
less: it does not include fine-grained rights management oradvanced network policies, and that’s by design. Those components are provided as enterprise services, by a third-party, or not at all.
I think Nomad hit a sweet-spot between ease of use and expressiveness. It’s good for small, mostly independent services. If you need more control, you’ll have to build it yourself or use a different approach. Nomad isjust. ************************ an orchestrator.
The best part about Nomad is that it’s easy to (replaceThere is little to no vendor lock-in because the functionality it provides can easily be integrated into any other system that manages services. It just runs as a plain old single binary on every machine in your cluster; that’s it!
The Nomad ecosystem of loosely coupled components
The real power of Nomad lies within its ecosystem. It integrates very well with other – completely optional – products likeConsul(a key-value store) orVault(for secrets handling). Inside your Nomad file, you can have sections for fetching data from those services:
(*************************************** (template) ******************************************{ data=“{{key” (****************************************** (service / geo-api / log-verbosity) *****************************************
"}}"************************************** API_KEY=
"{{with secret" ******************************** (secret / geo-api-key) ****************************************** "{} {{. Data.value}} {{end}}" **************************** EOH destination=("secrets / file.env"for example.env=true }This will read the
service / geo-api / log-verbositykey from Consul and expose it as a (LOG_LEVEL) ********************************** environment variable inside your job. It's also exposing secret / geo- api-key from Vault as API_KEY (********************************************. Simple, but powerful!Because it's so simple, Nomad can also be easily extended with other services through its API. For example, jobs can be tagged for service discovery. At trivago, we tag all services, which expose metrics, with (trv-metrics) . This way, Prometheus finds the services via Consul and periodically scrapes the / metrics endpoint for new data. The same can be done for logs by integrating Loki
There are many other examples for extensibility:
Trigger a Jenkins job using a webhook and Consul watches to redeploy your Nomad job on service config changes.
- Use (fabio) ************ for load balancing.
All of this allowed us togrow our infrastructure organically without too much up-front commitment.
(Fair warning
No system is perfect. I advise you not to use any fancy new features in production right now. There are
bugs
andmissing featuresof course - butthat's also the case for Kubernetes.
Compared to Kubernetes, there is far less momentum behind Nomad. Kubernetes has seen around (********************************************************************************************************** (commits and) contributors so far, while Nomad sports about 20. (commits and) contributors. It will be hard for Nomad to keep up with the velocity of Kubernetes, but maybe it doesn’t have to! The scope is much more narrow and the smaller community could also mean that it'll be easier to get your pull request accepted, in comparison to Kubernetes.
The takeaway is: don't use Kubernetes just because everybody else does. Carefully evaluate your requirements and check which tool fits the bill.
If you're planning to deploy a fleet of homogenous services on large-scale infrastructure, Kubernetes might be the way to go. Just be aware of the additional complexity and operational costs. Some of these costs can be avoided by using a managed Kubernetes environment likeGoogle Kubernetes Engineor Amazon EKS
.
If you're just looking for a reliable orchestrator that is easy to maintain and extendable, why not give Nomad a try? You might be surprised by how far it'll get you.
If Kubernetes were a car, Nomad would be a scooter. Sometimes you prefer one and sometimes the other. Both have their right to exist.
Thanks to my awesome colleagues Esteban Barrios ********************************************** Jorge-Luis Betancourt
, Simon Brüggen
,Arne Claus, (Inga Feick) , Wolfgang Gassler
,********************************** Barnabas Kutassy
,Perry Manuk, (Patrick Pokatilo) ************, and Jakub Sachafor reviewing drafts of t his article. OundFound an error in this article? Please propose a changeon Github.
GIPHY App Key not set. Please check settings