Envoy control plane ‘hello world’

A couple of weeks ago i wanted to program and understand how the control plane for Envoy Proxy works. I know its used in various comprehensive control systems like Istio and ofcourse at Lyft.

This repo/article describes a sample golang control plane for an Envoy Proxy. It demonstrates its dynamic configuration by getting a specific predetetermined setting set push to each proxy at runtime.

That is, once Envoy is started, it reads in an empty configuration which only tells it where the control plane gRPC server exists.

After connecting to the control plane, it receives configuration information to setup an upstream cluster and listener set. The specific listener and cluster is trivial: it merely proxies a request for (and only for) https://www.bbc.com/robots.txt.

To run this sample, you need to install golang and Envoy binary itself.

Again, this repo/articleis just how I worked through setting this up…it not best practices but simply a ‘hello world’ config. If i’ve done something incorrectly or bit that i can improve on, please do let me know.

As a bonus, the control plane also launches an Access Log gRPC service. This service will receives access log stats dirctly from the proxy. Setting up the access log is not the primary focus of this article but I’ll describe it in the appendix.

This repo uses the control APIs as of: docker pull envoyproxy/envoy:v1.12.2

Note: much of the code and config i got here is taken from the Envoy integration test suite

Additional Reading

Start Control Plane

go run src/main.go

Which should startup the control plane, access log service and REST->gRPC gateway (the latter again not in scope of this article)

The code is almost entirely contained in src/main.go. The code there starts the control plane and proceeds to setup a static config to proxy to a set of /robots.txt files from three sites:

[]string{"www.bbc.com", "www.yahoo.com", "blog.salrashid.me"}

Every 60 seconds, the host will rotate over which means for the first 60 seconds, you’ll see the robots.txt file from bbc, then yahoo then google.

Note, we increment the snapshot verision number and the host as well:

$ go run src/main.go 
INFO[0000] Starting control plane
INFO[0000] gateway listening HTTP/1.1 port=18001
INFO[0000] access log server listening port=18090
INFO[0000] management server listening port=18000
INFO[0003] OnStreamOpen 1 open for type.googleapis.com/envoy.api.v2.Cluster
INFO[0003] OnStreamRequest
INFO[0003] cb.Report() callbacks fetches=0 requests=1
INFO[0003] >>>>>>>>>>>>>>>>>>> creating cluster service_bbc with remoteHost%!(EXTRA string=www.bbc.com)
INFO[0003] >>>>>>>>>>>>>>>>>>> creating listener listener_0
INFO[0003] >>>>>>>>>>>>>>>>>>> creating snapshot Version 1
INFO[0003] OnStreamResponse...
INFO[0003] cb.Report() callbacks fetches=0 requests=1
INFO[0003] OnStreamRequest
INFO[0004] OnStreamOpen 2 open for
INFO[0007] OnStreamOpen 3 open for type.googleapis.com/envoy.api.v2.Listener
INFO[0007] OnStreamRequest
INFO[0007] OnStreamResponse...
INFO[0007] cb.Report() callbacks fetches=0 requests=3
INFO[0007] OnStreamRequest
INFO[0063] >>>>>>>>>>>>>>>>>>> creating cluster service_bbc with remoteHost%!(EXTRA string=www.yahoo.com)
INFO[0063] >>>>>>>>>>>>>>>>>>> creating listener listener_0
INFO[0063] >>>>>>>>>>>>>>>>>>> creating snapshot Version 2
INFO[0063] OnStreamResponse...
INFO[0063] cb.Report() callbacks fetches=0 requests=4
INFO[0063] OnStreamResponse...
INFO[0063] cb.Report() callbacks fetches=0 requests=4
INFO[0063] OnStreamRequest
INFO[0063] OnStreamRequest
INFO[0123] >>>>>>>>>>>>>>>>>>> creating cluster service_bbc with remoteHost%!(EXTRA string=blog.salrashid.me)
INFO[0123] >>>>>>>>>>>>>>>>>>> creating listener listener_0
INFO[0123] >>>>>>>>>>>>>>>>>>> creating snapshot Version 3
INFO[0123] OnStreamResponse...
INFO[0123] OnStreamResponse...
INFO[0123] cb.Report() callbacks fetches=0 requests=6
INFO[0123] cb.Report() callbacks fetches=0 requests=6
INFO[0123] OnStreamRequest
INFO[0123] OnStreamRequest

You can review the code to see how the structure is nested and initialized.

If you just set the value to bbc and not iterate, the code will behave as if bbc.yaml config file was passed to envoy.

Create Cluster

Create Listener

Commit Snapshot

snap := cache.NewSnapshot(fmt.Sprint(version), nil, c, nil, l, nil)
config.SetSnapshot(nodeId, snap)

Start Envoy Proxy

$ envoy -c baseline.yaml --v2-config-only -l info

You can verify the cluster was dynamically added in by viewing the envoy admin console at http://localhost:9000. A sample output of that console:

Access enpoint thorough proxy

Sample output

curl

Control Plane

Envoy Proxy

AccessLog

The accesslog config is taken fron the test suite resource.go given by the accesslog.proto.

Basically, when you configure a listener, you can configure a target to emit access_logs as shown below.

The sample service provided in this sample also starts a gRPC service implementing AccessLogService.

Equivalent yaml configuration

To run this config, pass --onlyLogging switch to the control plane

$ envoy -c logs.yaml

and then the control plane (which also starts the access_log server)

$ go run src/main.go --onlyLogging
INFO[0000] Starting control plane
INFO[0000] access log server listening port=18090

and access the listener on

$ curl -vk http://localhost:10000/robots.txt

you should see access_logs emitted to on the same stdout as before:

Conclusion