Eliminate Application Outages Caused by Root Certificate Changes


X.509 certificates are digital certificates that use the X.509 public key infrastructure (PKI) specification to derive private and public key pairs for various applications. They’re used by web-application systems, and mobile and IoT devices for authentication and encryption purposes. The most common application of this specification is SSL/TLS certificates. The private keys of these certificates are used to authenticate the identity of the web-application owner as well as decrypt communication between the application and the end-user online. Accepted by operating systems and browsers worldwide, these certificates represent the foundation of trust online.

Since 2018, the industry has seen a 37% increase in SSL/TLS certificates and by the end of 2019, a projected 97% of all web pages loaded in Google Chrome will use HTTPS. Most enterprises now manage close to hundreds – or even thousands – of certificates regularly in their network infrastructure. Given their finite lifespans, it is crucial that these certificates be monitored, tracked and renewed on-time to avoid expensive application outages. However, the maintenance required isn’t the only challenge posed by the growing number of SSL/TLS certificates – security of private keys and the trust associated with the entire certificate chain is also a significant concern.

What is Certificate Chaining?

Before we get to what a certificate chain is, here’s a recap on how a certificate request and signing works. The following are the steps:

1) First, you receive a certificate request from your application team.
2) Then, you generate a CSR (Certificate Signing Request) with necessary parameters such as the Common Name (your application’s FQDN), Organization, its Business Unit and Location (if needed) and also the Email Address to contact you.
3) This CSR, along with your public key information, will be sent to the Certificate Authority (CA) for signing.
4) After validating the request, the CA will create a certificate from the CSR and sign it with their private key (which is trusted by 99.9% trust stores)
5) The certificate (along with the intermediate certificate, if any) is finally pushed to the server and associated with an SSL profile

So, technically, anyone with a publicly trusted private key can issue legitimate certificates for “any” domain. This makes it extremely harmful in the wrong hands. When a root certificate’s private key is compromised, the CA needs to immediately revoke all certificates (chaining to that particular root) and should get a new private key installed in all trust stores – which is a time-consuming and cumbersome process. This is where an intermediate certificate plays a significant role. The root certificate signs a handful of intermediate certificates, which then signs the end-server certificates – thereby limiting exposure. In case of a compromise, the root CA can just revoke the intermediate certificate’s private key and the certificates it signed and can install a new intermediate certificate. And, this chain is ultimately validated as the certificate’s chain of trust. First, you have a root certificate’s private key that signs the intermediate certificate’s private key, which is then used to sign the server certificates. Thus, in practice, the trust chain first identifies and validates the private key that signed the server certificate – which is the intermediate certificate. If it doesn’t identify it in the trust store, it moves up a chain and validates the private key that signed the intermediate certificate – which is the root certificate. If the trust fails in any part of this chain, then there will be an unnecessary service disruption.

Few Reasons Why a Certificate Chain Can Change

In their 2018 SSL Threat Report, zScaler reported a 30 percent increase in encrypted payloads that used fake certificates with valid digital signatures.1 Most of it being Domain validated certificates, which isn’t surprising, as DV certificates do not require a stringent vetting process. However, what was surprising is that there is a thriving market in the dark web that sells valid certificates for just a couple of hundred dollars.2 This has made access to a fake digital certificate much easier. That’s why there is a huge onus on the security task force, made up of browsers and operating systems, to maintain the certificate trust store and actively prevent compromised certificates from being misused. The security task force can identify these compromised certificates through Certificate Transparency programs that monitor and audits SSL certificates in near real-time or by having teams working round the clock to identify vulnerabilities in certificates before bad actors.

If you look closely into these certificate deprecations or change in certificate chain cases, they can be grouped primarily into four categories:

1) Cryptographic Vulnerability – A group of certificates can be deprecated when any one of the three pieces of a key’s cryptography – Key Algorithm (DSA, RSA), Key Length (1024, 2048) or Hash Function (SHA1, SHA2) – is broken. SHA-1, MD5, RC4, and DSA are a few of the many cryptographic techniques that have already been compromised, and many others could be next. Now that computing power is becoming more and more accessible, hackers can easily crack algorithms in a relatively short amount of time. What is now accepted as the industry-standard could become obsolete in only a couple years, forcing you to shift to stronger PKIs.

2) Private Key Compromise – Hackers don’t only focus on breaking complex cryptographic algorithms – they also focus on compromising certificate authorities and their corresponding registration authorities (intermediate certificates). By gaining access to either system, hackers can circumvent the stringent vetting process that certain high-value certificates require and automatically get valid, trusted certificates issued for malicious use. While rare, in a more damaging attack, the hacker may also gain access to a certificate authority’s Private Key (root certificate). The certificate authority is then forced to revoke all certificates signed using the compromised private key, generate a new set of keys, apply for admission to all trust stores, and finally, regenerate certificates for its Few Reasons Why a Certificate Chain Can Change 3 customers. In such cases, the lead time needed to replace compromised certificates increases drastically, if you plan to stick with your original CA.

3) Faulty Certificate Authority – In this scenario, a hacker doesn’t even need to be directly involved to get their hands on a mis-issued certificate. In the past, there have been numerous incidents where certificate authorities have been incapable of vetting certificate requests properly. As a result, they issue certificates to people other than the rightful owners of a particular domain, even impersonating popular web apps like Google. Usually, when this happens, certificate authorities are bound to re-evaluate their PKI environment and plug the gaps that caused the mis-issue. However, in extreme cases with repeat offenders, the security task force aggressively deprecates each and every certificate ever issued by that certificate authority – forcing enterprises to migrate a massive amount of certificates in a short period of time.

4) Change in Root Ownership – This happens when one company sells off its Certificate Authority division to another company, mostly due to a shift in focus. For example, let’s take the recent case of a major CA deprecation. Symantec sold its website security and related PKI solutions to DigiCert for $950 million, post revocation of its root certificates.3 Thus, any certificate that is chained to the deprecated root will fail the trust chain validation. In this case, DigiCert will cross-sign the intermediate certificate with its trusted root certificate to re-instate trust.4 The impact is relatively less for CAs with root and intermediate certificates already available in the trust stores. But, for enterprises with pinned intermediate or root certificates, there’ll be huge service disruptions due to certificate trust. There’ll be an even higher impact for CAs that’ve spun off as a separate entity and trying to change the Root CA abruptly.

Apart from just certificate expiries, enterprises using certificates from deprecated roots or intermediates can also suffer severe disruptions. And, to successfully avoid such disruptions, migrating to a trusted PKI plays a major role. Enterprises often admit to delaying PKI migrations due to downtime concerns, resourcing issues and lack of planning. But, they seldom remember: migration is always cheaper than remediation. And, with a well-documented process, such as the one below, any migration can become easier.

6-step migration template

Whether you are migrating from a weak signature to a stronger one, or from one vendor to another, if you’re updating your signatures manually, consider following these six steps to minimize errors:

Step 1: Run a discovery of all certificates with weak keys

The first step is to identify all SSL certificates with vulnerable digital signatures across the infrastructure. It is essential that every certificate that has this weak signature (all certificates in the chain of trust, including intermediate) is tracked down regardless of the nature of the server (internal or public-facing).

Step 2: Perform an inventory assessment of existing certificates

Next, assess the certificates within the discovered inventory. Group and prioritize them according to the organization’s requirements. For example, replace weak certificates on mission-critical and public-facing applications before updating certificates on internal servers.

Step 3: Perform impact analysis of the impending PKI migration

Loop in each key stakeholder who might be affected by the impending update and keep them apprised of the progress. Once the plan is in place, the migration team should perform an impact analysis to assess the system’s compatibility with the certificates set to be updated. Split multi-domain certificates (like wildcard certificates) into multiple certificates for supporting legacy applications that do not support the updated signature. All external-facing legacy systems that do not support the latest PKI must be updated. In the case of legacy systems that are being used internally, proceed with the weaker PKI (at your own risk) and schedule a phase-out at your convenience.

Step 4: Trigger a full PKI migration

After the impact analysis, update certificates in the order of their priority. The updated certificates can be reissued, renewed or purchased from the vendor(s) of your choice. If necessary, the intermediate certificates should also be updated to complete the trust chain. Before replacing the weak certificates with the updated ones, backup your old (and new) certificates in a secure place. Manual migration can be tedious and error-prone, so it is important to ensure each step is documented and accounted.

Step 5: Validate the migration

Once the migration is complete and you’ve rechecked your environment for old certificates, create a detailed and holistic migration report. Use this report to validate and ensure successful completion. Share the status and progress of the migration plan with key stakeholders.

Step 6: Create enforceable policies

The migration team should create policies to guide the post-migration process and ensure standardized deployment across the infrastructure in the future.

Pillars of a successful PKI update

How do you measure the success of your PKI update? It should be estimated based on your answers to the following questions:

  • Continuity of operations: Have any of your applications suffered an outage?
  • Compatibility: Is it backward-compatible with legacy systems?
  • Scalability: Can your infrastructure scale to include future enhancements in PKI?
  • Recovery: Can you restore your older PKI if something goes wrong?
  • Budget: Did the process fall within your budget and timeline?

Automate Crypto-agility using AppViewX CERT+

Cryptographic agility refers to the ability of an IT infrastructure to evolve and adopt alternatives to their current cryptographic primitive quickly. When needed, AppViewX CERT+ can help users automate the entire six-step migration process to accelerate the PKI update process. Whether it’s the process of transitioning to a new vendor or updating revoked certificates with valid ones, CERT+ provides the high levels of visibility users need to keep certificates secure.

One-Step migration Process

With AppViewX, we have essentially consolidated all six steps in the migration process into one single-step automation workflow.

First, start with Discovery. The “Discover” function allows users to search for and display the list of all available SSL certificates within the network. The AppViewX certificate discovery search engine allows users to search using any of the following methods:

  • IP range
  • Subnet
  • URL
  • Upload
  • Managed devices
  • Certificate authorities

Once the certificates are retrieved, they are stored in an AES 256-bit encrypted certificate inventory. The updated inventory ensures that no deprecated certificates are missed during the migration. The pre-built automation workflow can then be triggered from a catalog of available automation workflows within AppViewX. This workflow retrieves the list of certificates (server and intermediate) with a particular characteristic (such as CA, cipher-suite, etc.) and allows you to choose the group that must be migrated during the first phase, based on your business needs. Then, choose your preferred certificate authority and type of certificates in the ‘Cert Replace Self-Service Form.”

Next, a pre-validation check ensures that the given information is within the acceptable parameters. After the check, the CSR is generated and submitted to the CA, and corresponding certificates are stored back in the inventory. Users can then either push these certificates manually into the end-devices, or push these certificates automatically.

The post-validation check ensures that all certificates chosen in the previous step have been issued a replacement certificate. The final step of the workflow triggers an email with a report of the entire migration to the necessary stakeholders. This automation workflow is purely plug-and-play and fully customizable. This allows users to face any certificate deprecation with confidence. Users can also choose to integrate a change management system in this workflow by adding your ServiceNow or BMC Remedy systems. This will enable users to create a change management ticket automatically whenever this workflow is triggered. These endless opportunities are what makes automation workflows in CERT+ extremely powerful.


All cryptographic signatures have vulnerabilities and any CA can make a mistake, making some level of a certificate deprecation inevitable. Regardless of the reason for migration, the amount of time you take between identifying deprecated certificates and patching it is crucial. Never take a vulnerability for granted and never put off an update because of its complexity. You can’t wait for external entities like browsers to set the timeline for your update. It is recommended to use an automation tool as manual processes can be error-prone. Any undocumented or mislabelled certificate can nullify the entire process. By following this guide and using the right tools, you can be completely protected against weak signatures, faulty CAs and CAs that abruptly change the certificate chain.

Learn more

An automation tool can help you achieve unlimited possibilities with limited resources. To learn more about our solutions, please visit AppViewX CERT+.


  • Application Outages
  • Certificate Outages
  • Root Certificate