Empty Handshakes

13 Nov 2021
Tags: bugfix networking protocols

When attempting to make a https request from a Qt app, a terse error was returned:

Which seemed odd, given that curl had no issue doing the same request, without the user specifying any additional certificates. So, what was different?


With strace -f -k, we don’t find the message text verbatim, but we can search for the last instance of “handshake”, then look up for application specific functions:

1984033 write(5, "\1\0\0\0\0\0\0\0", 8) = 8
 > /usr/lib64/libQt5Widgets.so.5.15.2(QPushButton::QPushButton(QString const&, QWidget*)+0x18) [0x34c448]
 > /usr/lib64/libQt5Widgets.so.5.15.2(QMessageBox::warning(QWidget*, QString const&, QString const&, int, int, int)+0x5f) [0x3f933f]
 > /home/foo/opt/zeal-dev/build/bin/zeal(Zeal::WidgetUi::DocsetsDialog::downloadCompleted()+0x273) [0x4c7e81]
1984262 write(163, "\1\0\0\0\0\0\0\0", 8) = 8
 > /usr/lib64/libQt5Network.so.5.15.2(QAbstractSocket::disconnectFromHost()+0xc8) [0xfe128]
 > /usr/lib64/libQt5Network.so.5.15.2(QSslSocketBackendPrivate::checkSslErrors()+0x11b) [0x13f99b]
 > /usr/lib64/libQt5Network.so.5.15.2(QSslSocketBackendPrivate::startHandshake()+0x3ea) [0x143eaa]

On an earlier call stack with the “handshake” function, we see OpenSSL specific functions, from libssl.so:

 > /usr/lib64/libssl.so.1.1.1l(state_machine.part.0+0x43a) [0x53f3a]
 > /usr/lib64/libQt5Network.so.5.15.2(QSslSocketBackendPrivate::startHandshake()+0x4e4) [0x143fa4]

This shows us what Qt ends up using for the SSL connection. We can use the openssl s_client tool to compare validation results, since it will use the same library (checked with ldd).

Before testing with openssl s_client, we should clarify who is actually reporting a handshake error: the client (our app) or the server (the remote host)?

Let’s sniff the traffic from curl, filtering by the ip given by nslookup api.zealdocs.org:

Compare it with the traffic from our client app:

We get to see that it’s the client app that decides to terminate the connection with a TCP FIN packet.

There’s some “Encrypted Handshake Message” packets, which can be decrypted by instrumenting OpenSSL functions1, so that the private key is logged in this format:

RSA Session-ID:f310e2aefbb422b9e7ab02afa2fbc3bdfd00107d6d5dd8653340c98b3ee9db36 Master-Key:1f52347840128456110317cb312a38985b2fd212afa6da2793caaa30b374790eb8043ec665cce0599159b4575a6b0415

Which is then loaded in wireshark: Right click on a TLS packet > Protocol Preferences > Transport Layer Security > Pre-Master-Secret Log


Oh, it was just a “Finished” message…

Certificate Verification

Now that we suspect that our client is the one originating the error, let’s check if we can actually verify the certificate chain successfully with other tools.

Let’s start with a minimal Qt app that downloads a file from a user provided URL. Keywords qt example http direct us to such an example, built with:

cd $path/qtbase
cmake .
cmake --build . --parallel
cd $path/qtbase/examples/network/http
LD_LIBRARY_PATH=$path/qtbase/lib $path/qtbase/bin/qmake -o Makefile *.pro

Running the example app with our URL gives us the same error2, but with a clearer description:

One or more SSL errors has occurred:
The issuer certificate of a locally looked up certificate could not be found

Moving on to curl, we trace our request:

curl --head https://api.zealdocs.org/v1/docsets --trace /dev/stderr >/dev/null

Which shows us the system certificates that are loaded:

* successfully set certificate verify locations:
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt

curl does not send any (client) certificate:

== Info: TLSv1.3 (OUT), TLS handshake, Client hello (1):
=> Send SSL data, 512 bytes (0x200)
<= Recv SSL data, 5 bytes (0x5)
0000: 16 03 03 00 6c                                  ....l
== Info: TLSv1.3 (IN), TLS handshake, Server hello (2):
<= Recv SSL data, 108 bytes (0x6c)
<= Recv SSL data, 5 bytes (0x5)
0000: 16 03 03 10 da                                  .....
== Info: TLSv1.2 (IN), TLS handshake, Certificate (11):
<= Recv SSL data, 4314 bytes (0x10da)
<= Recv SSL data, 5 bytes (0x5)
0000: 16 03 03 00 04                                  .....
== Info: TLSv1.2 (IN), TLS handshake, Server finished (14):
<= Recv SSL data, 4 bytes (0x4)
0000: 0e 00 00 00                                     ....
=> Send SSL data, 5 bytes (0x5)
0000: 16 03 03 00 46                                  ....F
== Info: TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
=> Send SSL data, 70 bytes (0x46)

The request then proceeds without issues.

Now, let’s check the server’s certificate chain:

openssl s_client -showcerts -connect api.zealdocs.org:443 </dev/null

The root CA certificate should be the last issuer:

Certificate chain
 0 s:CN = api.ams2-01.zealdocs.org
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3

And we know they are valid:

SSL handshake has read 5071 bytes and written 436 bytes
Verification: OK

Let’s check if our system certificate bundle contains these CA certificates:

openssl crl2pkcs7 -nocrl -certfile /etc/pki/tls/certs/ca-bundle.crt \
    | openssl pkcs7 -print_certs -text -noout

Seems like they are present:

Subject: O=Digital Signature Trust Co., CN=DST Root CA X3
Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1

Therefore, loading this bundle should be enough to verify these CA certificates.

Is our Qt client doing it?

Filtering with strace -e file -f -k, we don’t find any read operations of that certificate bundle! Instead, it tries to load some other files that don’t exist:

1984262 newfstatat(AT_FDCWD, "/etc/openssl/certs//8d33f237.0",  <unfinished ...>
1984262 <... newfstatat resumed>0x7f0d9bffd080, 0) = -1 ENOENT (No such file or directory)
1984262 newfstatat(AT_FDCWD, "/etc/ssl/certs//4042bcee.0", 0x7f0d9bffd080, 0) = -1 ENOENT (No such file or directory)

Turns out that it’s a known issue. To summarize, some environments generate c_rehash symlinks (which should reference certificate files). When these are present, Qt will parse them, skipping any existing bundles, even if the symlinks don’t reference any valid files.


Keywords qnetworkaccessmanager add ssl ca cert eventually lead to a snippet that hinted at how to set the certificates for requests:

QSslConfiguration sslconfig( pReq->sslConfiguration() );
sslconfig.setCaCertificates( QgsAuthManager::instance()->getTrustedCaCertsCache() );
// [...]
pReq->setSslConfiguration( sslconfig );

Now, how to get the system certificates? One way to find out is to download the qtbase sources, which contain the QSslConfiguration class (adapt to your favourite distro):

sudo dnf debuginfo-install qt5-qtbase

Then, grep -rin cacertificate matches this function definition:

    \since 5.5

    This function provides the CA certificate database
    provided by the operating system. The CA certificate database
    returned by this function is used to initialize the database
    returned by caCertificates() on the default QSslConfiguration.

    \sa caCertificates(), setCaCertificates(), defaultConfiguration(),
    addCaCertificate(), addCaCertificates()
QList<QSslCertificate> QSslConfiguration::systemCaCertificates()
    // we are calling ensureInitialized() in the method below
    return QSslSocketPrivate::systemCaCertificates();

Finally, we have all the parts to go to our client app’s request building function and add these system certificates:

diff --git a/src/libs/core/networkaccessmanager.cpp b/src/libs/core/networkaccessmanager.cpp
index 95200f9..26b88ed 100644
--- a/src/libs/core/networkaccessmanager.cpp
+++ b/src/libs/core/networkaccessmanager.cpp
@@ -71,5 +71,9 @@ QNetworkReply *NetworkAccessManager::createRequest(QNetworkAccessManager::Operat
         op = QNetworkAccessManager::GetOperation;
+    QSslConfiguration sslConfig = overrideRequest.sslConfiguration();
+    sslConfig.setCaCertificates(QSslConfiguration::systemCaCertificates());
+    overrideRequest.setSslConfiguration(sslConfig);

     return QNetworkAccessManager::createRequest(op, overrideRequest, outgoingData);

With these changes, requests are now successful.

  1. Alternatively, we could compile Qt sources with some debug macro definitions, maybe a lot of them to understand where exactly the validation logic fails. [return]

  2. SSL errors seem to be frequent enough in Qt apps that someone bothered to write slides on this topic. More generic guides can also be found. [return]