Vulnerable OpenSSL Handshake Renegotiation Can Trigger Denial of Service

OpenSSL, the popular general-purpose cryptographic library that implements SSL/TLS protocols for web authentication, has recently suffered from several vulnerabilities. We have written about “CVE-2017-3731: Truncated Packets Can Cause Denial of Service in OpenSSL” and “SSL Death Alert (CVE-2016-8610) Can Cause Denial of Service to OpenSSL Servers” among others. Today we examine the high-severity bug CVE-2017-3733, the Encrypt-Then-MAC renegotiation crash that can cause a denial of service.

Before SSL/TLS encrypts data, it runs the Handshake and ChangeCipherSpec protocols.

During the Handshake phase, the client and server decide which encryption algorithms to use. Once the negotiation is done, the client and the server send each other a ChangedCipherSpec message, after which the traffic is encrypted with the negotiated algorithms.

Encrypted data is sent in one of two ways along with the message authentication code (MAC) in SSL/TLS.

1. MAC-then-encrypt: This method calculates the MAC of the plain text, concatenates it with the plain text, and runs the encryption algorithm over it.
2. Encrypt-then-MAC: The cipher-text is generated by encrypting the plaintext and then appending a MAC of the encrypted plaintext.

If the ClientHello message does not contain an Encrypt-Then-Mac extension, then the default is MAC-then-encrypt mode. If ClientHello has an Encrypt-Then-Mac extension, the server will compute the MAC after encrypting the data.

If the client or server wish to change the algorithms used for encryption, they can renegotiate the Cipher_Suites that they have already agreed upon. This can occur any time during data transfer by initiating a new Handshake, which takes place over an existing SSL connection.

Triggering the vulnerability

OpenSSL offers this explanation:

“During a renegotiation handshake if the Encrypt-Then-Mac extension is negotiated where it was not in the original handshake (or vice-versa) then this can cause OpenSSL to crash (dependent on ciphersuite). Both clients and servers are affected.”

Say the client starts a TLS handshake with the server using the default MAC-then-encrypt mode. If the client later renegotiates with the Encrypt-then-MAC extension enabled and sends encrypted data in that mode before the ChangeCipherSpec message, the server will crash, causing a denial of service.

When the client triggers this vulnerability, the server crashes at the ssl3_get_record function, in the ssl3_record.c file:

The crash occurs at line no. 352, when checking to see if mac_size is less than EVP_MAX_MD_SIZE (64 bytes):

The if statement preceding the assertion checks whether the Encypt-then-MAC flag is set in the server. The macro in the if condition:

The flag TLS1_FLAGS_ECRYPT_THEN_MAC is already set when the ClientHello packet is sent with the Encrypt-then-MAC extension at the time of renegotiation. So the control will go inside the if condition. But because the ChangeCipherSpec message has not yet passed to the server, it does not know it must use Encrypt-then-MAC.

Putting a break point at line no. 352 and checking the mac_size variable shows us the value 0xffffffff, which is greater than EVP_MAX_MD_SIZE (64). Thus the assertion fails and the server crashes.

Let’s go to the code and find how the mac_size variable gets the value 0xffffffff. The EVP_MD_CTX_size function calculates the mac_size.

It returns -1 when the message digest value is null. 0xffffffff is the two’s complement of -1. This means “s->read_hash” returns null as the server tries to calculate the hash using the MAC-then-encrypt mode.

Users of McAfee products are protected from this attack by signature 0x45c09700. All administrators should update OpenSSL to the latest version.

Thanks to Hardik Shah for helping me with this post.

Introducing McAfee+

Identity theft protection and privacy for your digital life

Download McAfee+ Now