Envoy Discovery “hello world”

A simple app demonstrating a small part of Envoy’s Endpoint Discovery Service. THis is a sample walkthough of a trivial envoy config that sets up:

  • Envoy with EDS bootstrap (both envoy v1 and v2 APIs)
  • EDS Server to provide service discovery info for upstream back to Envoy
  • N upstream instances envoy will proxy back.

Some of the configurations are hardcoded in the envoy_config.yaml file just as a demonstration. Specifically, the service, cluster and bootstrap endpoint to get discovery information.

References

Prerequsites

You can find the source in the following repo:

Start Envoy with EDS

Bootstraping EDS within Envoy is relatively simple:

Note the api_type: is set to v2 REST endpoint. If you want to swtich to v1 simply use api_type: REST_LEGACY

So start envoy with debug enabled:

envoy -c envoy_config.yaml --v2-config-only -l debug

At this point, envoy attempts to connect to the upstream EDS cluster at 127.0.0.1:8080 but since your SDS isn't running yet, nothing additional config takes place.

Start Upstream services

Now in a new window, start the upstream service on a given the default port for the script (:8081)

cd upstream/virtualenv env
source env/bin/activate
pip install -r requirements.txt
$ python server.py -p 8081

Right now envoy doens’t know aboutt his endpoint:

Start EDS

Now start EDS without any bootstrapped config:

cd eds_server/virtualenv env
source env/bin/activate
pip install -r requirements.txt
python main.py

You should see the following output on SDS stdout indicating an inbound Envoy discovery request:

Inbound v2 request for discovery.  POST payload: {u'node': {u'build_version': u'fd44fd6051f5d1de3b020d0e03685c24075ba388/1.6.0-dev/Clean/RELEASE', u'cluster': u'mycluster', u'id': u'test-id'}, u'resource_names': [u'myservice']}
127.0.0.1 - - [29/Apr/2018 22:59:04] "POST /v2/discovery:endpoints HTTP/1.1" 200 -

then on the envoy proxy stdout, something like:

Basically, this shows no updates were recieved from the endpoint

You can verify that envoy doesn’t know anything about this endpoint by attempting to connect through to it:

$ curl -v  http://localhost:10000/
...
< HTTP/1.1 503 Service Unavailable
< content-length: 19
< content-type: text/plain
< date: Mon, 30 Apr 2018 06:06:20 GMT
< server: envoy
<
* Connection #0 to host localhost left intact
no healthy upstreams

Add endpoint to EDS

Now we’re ready to add an upstream service configuration to the SDS server. This sample uses Flask-RESTplus framework which delivers a convenient API console (you can, ofcourse, use curl)

connect to EDS servers UI console at (..and just to clarify, this is the Flask-RestPlus UI, nothing to do with envoy)

http://localhost:8080/

From there, you can register a service endpoint by selecting POST and the default payload.

Create Endpoint

Since we defined the service as myservice in the envoy_config.yaml, we can need to register an endpoint against it:

curl -X POST --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
"hosts": [
{
"ip_address": "127.0.0.1",
"port": 18080,
"tags": {
"az": "us-central1-a",
"canary": false,
"load_balancing_weight": 50
}
}
]
}' http://localhost:8080/edsservice/myservice

What this will do is set some endpoints for myservice. Now, envoy will query SDS for membership so on the next poll, you'll see some lines like:

Check client connectivity via envoy

Since we already started the upstream service above, you can connect to it via envoy:

$ curl -v  http://localhost:10000/

< HTTP/1.1 200 OK
< content-type: text/html; charset=utf-8
< content-length: 36
< server: envoy
< date: Mon, 30 Apr 2018 06:21:43 GMT
< x-envoy-upstream-service-time: 3
<
* Connection #0 to host localhost left intact
40b9bc6f-77b8-49b7-b939-1871507b0fcc

(note the server: envoy part in the header)

Delete Endpoint

Ok, so now we’ve dynamically added in an endpoint…lets remove it by the EDS server’s custom API and emptying out its hosts: []

curl -X PUT --header 'Content-Type: application/json' --header 'Accept: application/json' -d '{
"hosts": [ ]
}' http://localhost:8080/edsservice/myservice

Now try the endpoint, you should see no healthy upstream message from envoy

$ curl -v  http://localhost:10000/
< HTTP/1.1 503 Service Unavailable
< content-length: 19
< content-type: text/plain
< date: Mon, 30 Apr 2018 06:23:40 GMT
< server: envoy
<
* Connection #0 to host localhost left intact
no healthy upstream

Rinse and repeat

Ok, you can continue to play with the endpoints by adding and removing new upstream services on differnet ports:

eg:

$ python server.py -p 8082
$ python server.py -p 8083

and then using the API to add hosts to the SDS server (use the PUT endpoint to do that)

Healthcheck

A note about healthchecks: I noticed that once i added in healthchecks to the endpoint, even if i deleted the upstream service from the SDS server (i.,e removed it from hosts: list), the healthchecks continued and we can still connect to it via envoy. Since i'm new to envoy config, i suspect this is working as intended..

health_checks: 
- timeout: 1s
interval: 5s
unhealthy_threshold: 1
healthy_threshold: 1
http_health_check:
path: /healthz

Conclusion

I wrote this up just in an effort to play around with envoy i'm pretty much new to this so i likely have numerous misunderstanding on what i just did here...if you see something amiss, please do let me know.