Service Discovery with Consul

Introduction

Any developer that has worked with wiring together multiple tools has probably run across the following sequence of thoughts. Initial setup is done using hard coded IP addresses, no time for setting up names. System crash or IP change, now we switch to DNS after spending a day finding everywhere the IP was set over the course of a day or two. Next the company wants to scale the solution; work needs to be done to set up load balancers and other tooling to remove the single point of failure. If only there was an easier way to handle keeping services connected and available without having to fall through the same painful sequence.

The good news is that there is a way to handle that, service discovery. In this article I’ll walk through how Consul can be used to set up a simple service discovery setup. Once Consul is setup services can find each other without having to worry about updating multiple locations of a change, or dealing with load balancers.

What is Service Discovery?

Service discovery is the ability for a client to be able to find a service that it wishes to connect to without having to manually configure the connection. The way this I done is through a Service Registry, which is a database of services and their location. The registry provides a mechanism for registering services, removing services, and a way to query for services.

The Service Registry is then used in either a client-side or server-side service discovery. With client-side the client is who makes the query call to the registry and uses the returned information directly. In a server-side service discovery setup the client makes a call to a router, like a load balancer, and the router handles querying for the service and forwarding the traffic along.

Depending on the use case and constraints either method can be an effective way to handle reducing the configuration overhead. For this post the focus will be on using client-side discovery to find the service in question. To do that we will be using HashiCorp’s Consul and it’s built in DNS Service Discovery feature.

What is Consul?

Consul is a Service Mesh solution provided by Hashicorp. In addition to service discovery it also provides a key/value store, and health checking. It’s designed to be easy to scale in both a single data-center as well as cross site. Part of the ease of scaling and high availability comes from the fact that it has an agent running alongside the services that are registered. This allows for the health checking in addition to ensuring a distributed mesh of consul agents.

The features of Consul are designed to be easy to use and easy to integrate with your projects. Consul has a fairly intuitive UI where you can view registered services, their health checks if enabled, as well as viewing and editing Key/Value pairs. This is not only a good place to see the state of the information in Consul, but also a good first stop when trying to troubleshoot issues.

Information about Consul can also be gathered from its robust APIs. These allow you to view information like the UI as well as registering new services, and providing for service discovery. This means that if you are building a new micro-service you can build integration for it to both register and discover other services using the Consul API.

The Consul command line interface provides a number of tools around the agent and cluster. You can use it for basic tasks like starting consul as well as getting information about the running agent or cluster it is part of. There is also functionality for interacting with the Key/Value store and viewing service information.

The final interaction method with Consul is the DNS interface. This is the one I have been using primarily as it allows you to get the advantages of service discovery without going through heavy changes. Any app or micro-service that can be configured to locate other services via a DNS call can take advantage of DNS based service discovery.

Demo

To show how service discovery with Consul works I’ll be putting together a simple demo. For this demo there will be a basic nodejs app that will display some basic text. There will be three instances deployed. All three will be on different IP addresses and one will be on a different port. As part of the demo I’ll show how the information for the port and IP can be retrieved.

Demo App

The app I will be using is the following simple web server that runs in nodejs. It takes in two arguments. The first is the port to run on. The second is the message to display.

const http = require('http');

var myArgs = process.argv.slice(2);

var port = myArgs[0]
var message = myArgs[1]

const requestListener = function (req, res) {
    res.writeHead(200);
    res.end(message);
}

const server = http.createServer(requestListener);
server.listen(port);

It can be run simply with the following command. This will produce a simple “hello, world” text when you browse to 127.0.0.1:9002.

node app.js 9002 "hello, world"

Service Definition

Next is to create the service definition that Consul will use to register the service. In our case a very simple definition will do. The service definition contains the name of the service, the port it is deployed to, and the health check. The health check will submit a GET request to the URL in the “http” section. Any 2XX response will be considered up, 429 as warning, and anything else as failure. This check will occur every 30 seconds based on the set interval.

{
  "service": {
    "name": "web",
    "port": 9002,
    "check": {
      "id": "http",
      "http": "http://127.0.0.1:9002/",
      "interval": "30s"
    }
  }
}

Installing Consul

Installing Consul is fairly simple since you can get pre-compiled binaries. For CentOS 7 I used the following commands to quickly and easily install Consul. You will need to be sure to chown the opt/consul directory to the user that you plan to run Consul as.

cd /usr/local/bin
sudo curl -o consul.zip https://releases.hashicorp.com/consul/1.9.3/consul_1.9.3_linux_amd64.zip
sudo unzip consul.zip
sudo rm consul.zip
sudo mkdir -p /etc/consul.d
sudo mkdir -p /opt/consul

Once installed you can check that everything is working by running the consul command. For checking the version you should get an output like below.

Configuring Consul

Consul can be run from just command line arguments or a configuration file. I tend to use the configuration file as it simplifies things if I want to run Consul as a service. There are a large number of options that can be passed to Consul, but this demo will use a basic configuration.

{
  "datacenter": "lab",
  "client": "192.168.1.72",
  "data_dir": "/opt/consul",
  "server": true,
  "bootstrap_expect": 3,
  "retry_join": [
    "192.168.1.72",
    "192.168.1.98",
    "192.168.1.99"
  ],
  "ui_config": {
    "enabled": true
  }
} 

This file and the service definition files will both be placed in the /etc/consul.d/ directory. That will be passed to Consul on startup as the config directory.

Starting Consul

Consul can be easily started with the following command. If you see an error due to the machine you are using having multiple addresses you will need to add the -bind option with the address you wish to use.

consul agent -config-dir /etc/consul.d/

Once Consul and the app are up and running you can navigate to the Consul UI and should be able to see two services listed.

Viewing the Service Status

Once everything is up and running you can view the status of a service in the UI by clicking on it. You will see a list of all instances registered along with some basic information.

If you click on an individual instance you can see a more in depth view of the service information and status.

If a health check starts to fail the UI is updated accordingly.

DNS Discovery

The last part of this demo is showing how you can use DNS to locate the services that are registered to Consul. As part of the Consul deployment it has a DNS server. If you point to that server when making DNS requests you can look up services. The default style of names for services is [service_name]service.consul. You can see the result of a dig query below.

In addition to general DNS queries Consul will also respond to SRV queries. These are a special type of DNS query where not only the address, but also the port can be discovered. The result of such a query can be seen below.

Conclusion

As you can see service discovery can be quite powerful when trying to interconnect multiple services. Hashicorp Consul provides an easy, well documented way to get started with service discovery. If you are working with micro-services or distributed architecture I would encourage you to give it a try.