In this article, we’ll take a look at the features of Consul — the swiss knife by Hashicorp. We will demonstrate the power of this tool with simple, close to real life examples.
The materials for this article may be found on GitHub. There you can also find a
consul_utils.sh script which contains a set of Bash functions that may simplify your life greatly!
When people talk about Consul they often refer to it as a “Service discovery tool”. So, why would we call it a swiss knife? The answer may be found both in Consul’s capabilities and the right usage of its features.
Basically, here is the list of things Consul is able to do:
- Key-Value store
- Distributed locking system
- Service discovery
- Event triggers
- Remote execution on nodes in a cluster
- Metrics collection for monitoring
But, before diving into this, we will run a basic Consul cluster.
Consul is distributed as a single binary. To install it, just go to the download page and pick the appropriate build.
If you are using Linux — you can run the command from this gist:
curl -s -L https://git.io/fxJJR | sudo bash -s
Starting the agent
Consul can be started in two modes — server and client. When started as a server, Consul participates in consensus, stores data and responds to RPC queries. In client mode, an agent becomes a lightweight stateless process which forwards queries to servers.
In this guide, we will run our Consul cluster in a single-agent dev environment, which is evidently not recommended for production. But this may be fine for our educational purpose.
To start the agent, you may simply run:
~$ consul agent -dev ==> Starting Consul agent... ==> Consul agent running! Version: 'v1.2.3' Node ID: '756801d1-0cd0-f576-eedc-00956c712bee' Node name: 'bogdan-comp' Datacenter: 'dc1' (Segment: '<all>') Server: true (Bootstrap: false) Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, DNS: 8600) Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302) Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false
The core of Consul is distributed key-value storage. This storage is replicated across all servers in a Consul cluster. To read and write data into the storage, you can use CLI or the HTTP API directly. A quick example of writing the data:
$ consul kv put kibana/config/elastic_host 'http://elastic:9200' Success! Data written to: kibana/config/elastic_host
n the example above, the key is
http://elastic:9200 is the value.
You’re able to do the same with
$ curl --request PUT \ --data 'http://elastic:9200' \ http://127.0.0.1:8500/v1/kv/kibana/config/elastic_host
To read the value you can use the following:
$ consul kv get kibana/config/elastic_host http://elastic:9200
Distributed locking system
Distributed locks may be a very useful tool for systems where shared resources may be operated by different processes in a mutually exclusive way.
Let’s consider a quick example: we have one DB server with data we want to backup and three separate servers — workers. To achieve this goal we created a bash script to export and save data on S3:
Basically, we can now set up a cron job on one of the workers to run this script. But if this one particular server fails — the job won’t be executed and the backups won’t be performed.
This is where Consul comes for help. Let’s copy our script on all workers and make some modifications — add logic for lock acquirement:
Consul allows you to acquire the lock to a key. It is similar to simply writing data to key from the previous part. The only difference — you need to create a session first and perform an
acquire action with this session. In the example above we are using
locks/backup/.lock as a key for our lock.
Now we can set up the cron job on every worker. Everyone tries to acquire a lock, but only one worker succeeds. The succeeded one performs a backup and releases a lock. Now if at least one worker is alive — the backup job will be performed.
In this part, we described two features of Consul — Key-Value store and Distributed Locking system. The demonstration we provided is quite simple and not suitable for production, but this may help you to understand the potential of this tool quickly.
In the next part, we’ll continue describing the features of Consul diving deeper into this great tool from Hashicorp. We will show how to benefit from Consul in common examples which you may face on a daily basis.
If you have any questions, don’t be shy drop us a line. We’re always happy to help!