TLS with openssl trace logging and openssl FIPS

…or how to get way more info from a TLS connection than you ever wanted to know….now with FIPS (sort of)

This article basically talks about a simple docker container with openssl 1.1.1i which has TLS trace flags enabled.

What that allows you to do see is the very low-level TLS traffic between a client and server for both TLS, mTLS and OCSP traffic.

I recently needed to find out if the TLS Server i was connecting to would unilaterally support client-side certificate stapling (which is in of itself incredibly rare)…but in the course of doing that bit, i create the following repo and here the corresponding article:

The dockerhub image for the bit above is docker.io/salrashid123/openssl s_server

Anyway, before you get started, to stop the containers that are running, just execute the following in a new shell…docker rm -f client server

FIPS or NO FIPS

The FIPS version uses openssl-1.0.1e and openssl-fips-2.0.16

$ docker run  docker.io/salrashid123/openssl:fips version
OpenSSL 1.0.1e-fips 11 Feb 2013
$ docker run -e "OPENSSL_FIPS=0" docker.io/salrashid123/openssl:fips version
OpenSSL 1.0.1e-fips 11 Feb 2013

With server TLS

First test one way TLS traffic with just server TLS

# server
docker run \
--name server \
-p 8081:8081 \
--net=host \
-v `pwd`/html:/apps/ \
-v `pwd`/certs:/certs \
-t docker.io/salrashid123/openssl s_server \
-cert /certs/http_server.crt \
-key /certs/http_server.key \
-port 8081 \
-CAfile /certs/tls-ca-chain.pem \
-tlsextdebug \
-tls1_3 \
-trace \
-WWW
# client
docker run \
--name client \
--net=host \
-v `pwd`/certs/:/certs \
-t docker.io/salrashid123/openssl s_client \
-connect localhost:8081 \
-servername http.domain.com \
-CAfile /certs/tls-ca-chain.pem \
-tls1_3 \
-tlsextdebug \
-trace
# curl
curl -vvvvv \
-H "host: http.domain.com" \
--resolve http.domain.com:8081:127.0.0.1 \
--cacert certs/tls-ca-chain.pem \
https://http.domain.com:8081/index.html

With mTLS

Now graduate to the 200' level course and use mtls

#server
docker run \
--name server \
-p 8081:8081 \
--net=host \
-v `pwd`/html:/apps/ \
-v `pwd`/certs:/certs \
-t docker.io/salrashid123/openssl s_server \
-cert /certs/http_server.crt \
-key /certs/http_server.key \
-port 8081 \
-CAfile /certs/tls-ca-chain.pem \
-Verify 5 \
-tlsextdebug \
-tls1_3 \
-trace \
-WWW
# client
docker run \
--name client \
--net=host \
-v `pwd`/certs/:/certs \
-t docker.io/salrashid123/openssl s_client \
-connect localhost:8081 \
-servername http.domain.com \
-CAfile /certs/tls-ca-chain.pem \
-cert /certs/client.crt \
-key /certs/client.key \
-tls1_3 \
-tlsextdebug \
--verify 5 \
-trace
# curl
curl -vvvvv \
-H "host: http.domain.com" \
--resolve http.domain.com:8081:127.0.0.1 \
--cert certs/client.crt \
--key certs/client.key \
--cacert certs/tls-ca-chain.pem \
https://http.domain.com:8081/index.html

With OCSP Stapling:

Make the server return a stapled cert for OCSP

# server
docker run \
--name server \
-p 8081:8081 \
--net=host -v `pwd`/html:/apps/ \
-v `pwd`/certs:/certs \
-t docker.io/salrashid123/openssl s_server \
-status_file /certs/http_server_ocsp_resp_valid.bin \
-cert /certs/http_server.crt \
-key /certs/http_server.key \
-port 8081 \
-CAfile /certs/tls-ca-chain.pem \
-Verify 5 \
-tlsextdebug \
-tls1_3 \
-status \
-status_verbose \
-trace \
-extended_crl \
-WWW
# client
docker run \
--name client \
--net=host \
-v `pwd`/certs/:/certs \
-t docker.io/salrashid123/openssl s_client \
-connect localhost:8081 \
-servername http.domain.com \
-CAfile /certs/tls-ca-chain.pem \
-cert /certs/client.crt \
-key /certs/client.key \
-tls1_3 \
-tlsextdebug \
-status \
--verify 5 \
-trace
# curl
curl -vvvvv \
-H "host: http.domain.com" \
--resolve http.domain.com:8081:127.0.0.1 \
--cert certs/client.crt \
--key certs/client.key \
--cacert certs/tls-ca-chain.pem \
--cert-status \
https://http.domain.com:8081/index.html

Note, http_server_ocsp_resp_valid.bin is a valid OCSP response while http_server_ocsp_resp_revoked.bin is an expired one.

To see the details of an OCSP response, run

docker run \
-v `pwd`/certs:/certs \
docker.io/salrashid123/openssl ocsp \
-CA /certs/tls-ca-ocsp-chain.pem \
-CAfile /certs/tls-ca-ocsp-chain.pem \
-respin /certs/http_server_ocsp_resp_valid.bin \
-text

if you want to generate and use your own CA, see Create Root CA Key and cert

TLS Decryption with wireshark

If you want to decrypt the actual TLS traffic using wireshark, set an env-var for curl/openssl pointing to the keys SSLKEYLOGFILE

Start Wireshark, under Preferences->Protocols->TLS set "(Pre)-Master-Secret log filename" to /tmp/keylog.log

Begin capture on lo (localhost) interface and filter on tcp.port==8081

Then run

# server
docker run \
--name server \
-p 8081:8081 \
--net=host \
-v `pwd`/html:/apps/ \
-v `pwd`/certs:/certs \
-t docker.io/salrashid123/openssl s_server \
-cert /certs/http_server.crt \
-key /certs/http_server.key \
-port 8081 \
-CAfile /certs/tls-ca-chain.pem \
-tlsextdebug \
-tls1_3 \
-trace \
-WWW
# curl
export SSLKEYLOGFILE=/tmp/keylog.log
curl -vvvvv \
-H "host: http.domain.com" \
--resolve http.domain.com:8081:127.0.0.1 \
--cert certs/client.crt \
--key certs/client.key \
--cacert certs/tls-ca-chain.pem \
https://http.domain.com:8081/index.html

You should see the decrypted traffic (in this case the HTTPS response from the server, ok)

I’ve left a sample keylog file and associated tls capture (just load tls.pcapng in wireshark and then specify the path to keylog.log as the master-secret log file)

FIPS Ciphers

$ docker run   -e "OPENSSL_FIPS=1"   -t docker.io/salrashid123/openssl:fips ciphers -v 'kRSA+FIPS'
AES256-GCM-SHA384 TLSv1.2 Kx=RSA Au=RSA Enc=AESGCM(256) Mac=AEAD
AES256-SHA256 TLSv1.2 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA256
AES256-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(256) Mac=SHA1
DES-CBC3-SHA SSLv3 Kx=RSA Au=RSA Enc=3DES(168) Mac=SHA1
AES128-GCM-SHA256 TLSv1.2 Kx=RSA Au=RSA Enc=AESGCM(128) Mac=AEAD
AES128-SHA256 TLSv1.2 Kx=RSA Au=RSA Enc=AES(128) Mac=SHA256
AES128-SHA SSLv3 Kx=RSA Au=RSA Enc=AES(128) Mac=SHA1
docker run \
--name server \
-p 8081:8081 \
--net=host \
-e "OPENSSL_FIPS=1" \
-v `pwd`/html:/apps/ \
-v `pwd`/certs:/certs \
-t docker.io/salrashid123/openssl:fips s_server \
-cert /certs/http_server.crt \
-key /certs/http_server.key \
-port 8081 \
-cipher kRSA+FIPS \
-CAfile /certs/tls-ca-chain.pem \
-tlsextdebug \
-WWW
docker run \
--name client \
--net=host \
-e "OPENSSL_FIPS=1" \
-v `pwd`/certs/:/certs \
-t docker.io/salrashid123/openssl:fips s_client \
-connect localhost:8081 \
-cipher kRSA+FIPS \
-servername http.domain.com \
-CAfile /certs/tls-ca-chain.pem \
-tlsextdebug

— -

fin.