2/4: Message Payload Encryption in Google Cloud Pub/Sub (Part 2: Service Account Public Keys)

Introduction

This is the second in a series exploring how to encrypt the message data payload within a Google Pub/Sub message. This will not go into any real detail on the situations why you want to ensure confidentiality or integrity of a given message but many of them were discussed in the first part of the series. If you haven’t already, please do read atleast the introduction section in the first part.

Anyway, what we’ve covered so far is a form of simple symmetric encryption where the sender and receiver are both using a static key There are clear drawbacks to this which was outlined in that article. We’re here to see if we can improve on it or develop a different technique.

So, instead of sharing a symmetric key, can we use the public/private key pairs already provided by google as part of their Service Account Credential?

What exactly are we encrypting and signing?

This technique relies on the publisher have a service_account private key in JSON format. What that will allow you to do is sign a message payload locally for integrity/authentication. The public key for most GCP service accounts is available externally for any recipient to verify with. For example, here is a snippet for a private key json:

which has the public key available at a well-known URL (client_x509_cert_url):

Since the subscriber can download these certs to verify a message, the issue of key-distribution is a bit easier.

Wait….we’ve got a public key on our hands now…we can use that for encryption too. In this mode, the publisher uses the public key for the for the subscriber_ to encrypt a message (in our case, one of the available keys here). The subscriber on the other end must have in possession or have the ability to verify the message. With this technique, you are ensuring confidentiality by encrypting the payload using public/private keys

Using Service Account Token Creator Role

What we’ve done here is use a service account to sign the payload. GCP has another mechanism where one service account can impersonate another and sign data on its behalf. This is useful if you want to mint a message on behalf of another service account. If you want more information on this variation and code samples, see:

Simple message Encryption

Ok, Now that we’re on the same field, lets run through a sample run.

Encryption

We’re going to use the SAME service accounts as the publisher and subscriber already use as authorization to GCP. You are ofcourse do not have to use the same ones..infact, for encryption, you can use the public key for any recipient on any project!

Anyway..

Output

  • Publisher
  1. Lookup the public key for the subscribers service account
  2. Use the subscribers public key to encrypt a message
  3. Send the encrypted message in the pub/sub data: field
  • Subscriber:
  1. Decode the pub/sub message data using the private key for the subscriber’s service account

Signing

For signing, we do something similar where we’re singing just what we would put into the ‘data:’ field and placing that in a specific PubSubMessage.attribute called signature=

Output

  • Publisher
  1. Use the publishers local service account json file or iam.signBlob() to sign the pub/sub message data
  2. Transmit the signed message as a pub/sub attribute
  • Subscriber
  1. Lookup the publishers public key
  2. Verify the signature payload attribute using public key

the good and the bad

Ok, now that we went using GCP -provided service account public/private keys

Plus:

  • Doesn’t involve full key distribution; only requires key access at one end of the operation
  • If using local .json certificate, there is no network round-trip just to sign; everything is done locally
  • IAM impersonation can allow a publisher to sign as a different service_account.
  • Signed messages can be always verified since the public key is available online and referenceable.

Minus

  • Atleast one participant needs to acquire a key remotely (eg. on verify, download public cert; on encrypt for publisher, acquire public cert at subscriber).
  • Adds some network latency
  • Asymmetric key verification is slightly slower than symmetric for verification and decryption
  • Requires coordination between publisher and subscriber on which public key_id to use for encryption; Signing key_id can be transmitted in line.
  • Have to coordinate Key distribution for thejsoncertificate file.
  • Possible to encrypt or sign a message using a key that has been revoked on the GCP console. If you have a keyjson file in possession, you can use that to sign a message still even if that specific key was revoked on the GCP console and not valid for GCP-API authentication.
  • Possible to encrypt a message using the wrong service account public key.
  • Message attributes are not encrypted or signed (you can work around this by signing recreated canonical json format of the message)
  • RSA messages are larger than symmetric keys (note, Pub/Sub maximum message size is 10MB)

Conclusion

This is a more complex way to encrypt or sign your messages using GCP’s PKI for service accounts. Its nothing new and quite obvious, its basically encrypting an arbitrary message with a specific scheme. There are clear drawbacks as outlined above among several others.

Can we improve on this? Lets see if using GCP’s Key Management System (KMS) helps with this in the next set of articles in the series

Appendix

Code

References

Config/Setup

Here is the setup i used in the screenshots below and testing

  • Project: esp-demo-197318
  • Service Accounts with JSON certificates
  • Publisher identity: publisher@esp-demo-197318.iam.gserviceaccount.com
  • Subscriber identity: subscriber@esp-demo-197318.iam.gserviceaccount.com
  • KeyRing+Key: projects/esp-demo-197318/locations/us-central1/keyRings/mykeyring/cryptoKeys/key1
  • PubSub:
  • PUBSUB_TOPIC: projects/esp-demo-197318/topics/my-new-topic
  • PUBSUB_SUBSCRIPTION projects/esp-demo-197318/subscriptions/my-new-subscriber

Disclaimer

The techniques and code contained here is not supported by google and is provided as-is (the corresponding git repo is under Apache license). It seeks to provide some options you can investigate, evaluate and employ if you choose to.