May 30 Sectigo Expiry Incident: Recap and Analysis

On May 30, 2020, users using Sectigo certificates experienced website and service downtime due to an ‘expired certificate’, despite their certificates being completely valid and un-expired. A vast majority were not affected by this strange occurrence at all – however, consumers who were on old, legacy operating systems, or hadn’t updated their browsers since 2015, were more likely to have run into problems with certificate validity. Websites leveraging services such as APIs, cURL, and OpenSSL reported having difficulties, too.

Let’s take a closer look at what happened.

Technical Background

To better understand this outage, we need to examine how SSL/TLS certificates work, in the first place. When an application (let’s say, a browser), attempts to connect with a web service (say, a website), the browser first has to verify the authenticity of the server, and vice-versa. This is done by examining the other entity’s digital certificate, which in turn, has been issued by a body called a Certificate Authority (or CA, for short). Sectigo is one such CA. This whole verification process is necessary to ensure that the web service you’re communicating with is legitimate, and isn’t just some rogue service that’s masquerading as another website to possibly steal sensitive information and credentials. In technical terms, it’s called public key cryptography, but that’s a story for another day.

Now, these certificates installed on websites (or servers) aren’t solely responsible for their trusted status. The certificate sold to you (called a leaf certificate) is actually signed by something called an intermediate certificate, which in turn, is signed by something called a root certificate. Now, a Certificate Authority like Sectigo controls this entire chain, called a chain of trust, and is responsible for renewing the validity of root certificates when they expire. It’s important to note that root certificates are self-signed, which means they sign themselves in order to establish their legitimacy (as opposed to intermediate and leaf certificates), and since it’s assumed that a CA is trusted, these self-signed root certificates are universally trusted as well.

The whole reason a chain of trust exists is to act as an extra measure of security – leaf certificates need to be renewed often, while Root certificates maintain their validity for decades, and can be used to issue millions of certificates over their lifetime.

Now, Operating Systems (and by extension, Browsers) consist of a ‘trust store’ which is a repository of root certificates which they use to cross-check against the certificate presented by a website they’re attempting to connect with. Now, operating systems/browsers that are regularly updated keep receiving software updates that renew the trust stores on them – if a root certificate has expired, it is replaced in the trust store by a new, valid one that has been issued by the CA in place of the old one. This way, browsers will switch the trust path to the new CA, and continue to function properly.

However, consider old, legacy software and hardware that is not being updated (or is unable to receive updates due to hardware limitations or lack of developer support). They continue to trust old root certificates, even past their expiration dates. Of course, the CA will have provided a new root certificate that systems can trust instead of the old one, but the legacy system isn’t being updated, and it does not have this information. Thus, an expired root will lead them to distrust all certificates issued by that CA, even if your external-facing leaf certificate is valid.

Furthermore, there remains the fact that older chains of trust are considered more reliable by browsers. Hence, they are disposed to distrust newer chains of trust (which are inevitable, due to newer root certificates being valid).

This poses a significant problem to OEMs and vendors who are unable or unwilling to roll out constant updates – their systems will inevitably stop trusting the CA when the root certificate expires, since it will not have received information about a new root.

There is a solution to this snag, and it’s called cross-signing.

Cross-signing

In simple terms, it’s the certificate version of being backwards compatible.

To avoid the problem of being reliant on one, inevitably-expiring root certificate, a leaf certificate is linked to two different intermediates – one path is to an old root certificate, and the other is to a new root, issued recently by the CA.

In theory, here’s how it works. The leaf certificate is associated with the older root certificate until the end of its validity. When the verifying software (for instance, the browser), detects that it has an expired root, it automatically checks the other path to the new root certificate. When it checks out as valid, the certificate is accepted as well.

There’s one little problem here. Some legacy software systems fail to carry out the trust-switching examination entirely. They check the first root certificate, and when they find that it is invalid, they assume that the trust chain is invalid in its entirety, without bothering to check the secondary path to the new root.

This ties directly into the problem Sectigo users have been having with their certificates.

The Sectigo Expiry

At present, Sectigo controls three root certificates with varying expiration dates:

• AddTrust External CA Root (Issued 2000 ; Expires 2020)
• COMODO/Sectigo USERTrust RSA/ECC Certification Authority (Issued 2010 ; Expires 2038)
• AAA Certificate Services (Issued 2004)

Now, the AddTrust root certificate (the root used to sign the majority of Sectigo certificates in circulation today) was set to expire by the end of May 2020, and Sectigo pre-emptively issued the USERTrust root in 2010 (set to expire in 2038). To ensure maximum compatibility, AddTrust-issued certificates were cross-signed with USERTrust certificates, and the changes were distributed via software updates in 2015. This meant all AddTrust-reliant certificates would automatically switch their trust paths to the newer root in their trust store, which, on paper, would eliminate any possibility of service disruption on the 30th of May.

However, while major operating systems and browsers received this update, some applications such as cURL, which use different methods to verify the chain of trust, did not. Neither did legacy browsers and operating systems receive them. These systems were configured to trust the AddTrust root, and either did not possess the new USERTrust root in their trust stores, or were explicitly programmed to accept only the expired AddTrust root certificate.

There was also the case of very old legacy systems which, in some cases, refused to accept the USERTrust root, as they did not have it in their trust store, and also did not trust it due to it being issued relatively recently. To counter this potential issue, Sectigo made a third path available – a legacy root for cross-signing, called AAA Certificate Services. Since it was issued earlier than the USERTrust root, there was a chance it would be accepted more readily than the other. However, Sectigo advised caution prior to taking this approach.

While Sectigo and most other service providers were fully aware of the impending expiration and were well prepared to handle it without disruptions, they did not anticipate the sheer number of legacy systems and TLS libraries that were not configured to handle this change.

Hence, several services experienced downtime and traffic to websites diminished significantly until administrators could replace CA bundles on their systems and include the new root certificate.

Next Steps

If you’ve been affected by this incident, Sectigo has issued instructions on how you can get around it here.

If you’re a network owner/administrator, and have several devices reliant on Sectigo/COMODO certificates that have been affected by this issue, AppViewX can help with detecting the devices that are likely to be affected by the expiry. We can also help you automate the process of updating certificate bundles on all your devices with new, valid certificates. You can request a demo with our experts to learn more about how this is possible – register here.