A leading energy and utilities holding company that acts as an energy supplier in eight states across the USA. It operates multiple coal plants, wind farms, and nuclear power stations.
Lack of defined PKI processes and visibility resulted in outages
Public certificate authority (CA)s were used to provide transport layer security (TLS) certificates for hundreds of external access points. The cyber-security identity and access management (IAM) team was responsible for certificate management, issuance support, and PKI processes. They would assist application support teams by responding to certificate task requests raised via tickets, as necessary. There were thousands of servers and employees that made use of PKI, but there was no well-defined process that dictated how certificates and keys were managed.
- Manual PKI configuration errors
- Certificate outages caused due to lack of visibility
- Scattered ownership of certificates
- Cumbersome key regeneration/re-keying
- Lack of defined PKI process
- Lack of PKI self-service resulting in increased reliance on IT
Primary Business Challenges
Lack of PKI Visibility
A lack of clear visibility into where every certificate was located resulted in frequent expiry-related outages, certificate duplication, cumbersome troubleshooting and complicated maintenance. Detecting the presence of all self-signed certificates and certificates with weak keys and deprecated algorithms was quite difficult to achieve manually, exposing the firm to vulnerabilities.
Manual, Decentralized Certificate Operations
Certificate tasks such as expiry monitoring and installations were done manually by the PKI team. There was a need for an automated system to inventory and group certificates, which would serve the purpose of providing visibility post-discovery. Most importantly, the customer desired a centralized system using which all aspects of PKI could be managed in a secure manner.
Insecure Endpoint Deployment
There were several different device types that certificates needed to be deployed to Windows servers, Red Hat servers, F5 LTMs, and so on. Key distribution was done in an un-encrypted fashion, and pushing the certificate to its respective endpoints required significant work due to its decentralized nature. The entire process of certificate deployment needed to be streamlined and made fully secure, as it was a critical component of the certificate lifecycle.
The AppViewX deployment worked seamlessly with our customer’s IT infrastructure, and started delivering results right from the start.
Certificate Discovery and Inventory
AppViewX scanned and located certificates on a multitude of devices and servers, and across multiple CAs. The discovered certificates were automatically added to the inventory, and AppViewX allowed for grouping based on certain criteria. The discovery process could be carried out using a range of parameters – including scanning by subnet/IP, or by issuing CA, device, and so on. AppViewX also integrated with Rapid7 in order to query the asset group identified by the customer, in order to discover certificates.
Alerts and Monitoring
AppViewX provided constant visibility into certificate health with reports that displayed validity statuses. Periodic alerts for imminent certificate expirations could be configured to be sent via email to the respective certificate/group owner, ensuring that a renewal was never missed. AppViewX also permitted the transfer of certificate ownership to solve the issue of alerts being sent to the wrong people (people who were no longer employed by the firm, for instance).
Self-Service of PKI
AppViewX made a self-service portal accessible to application maintenance teams that could be used to directly requisition certificates as necessary. This minimized their reliance on the PKI security team for trivial certificate tasks, and was a huge time-saver. Role-based control was also applied, ensuring that only authorized personnel would be able to make changes to PKI. Most importantly, AppViewX’s low-code page builder was used to design self-service forms in such a way that different teams were exposed to only the information that was relevant to them.
Tasks such as certificate signing request (CSR) generation, email notifications, certificate signing, and CLM (more on that below) were completely abstracted and automated. AppViewX’s automation engine tied together disparate tasks and was able to execute them in an orderly fashion based on activity triggers from users, minimizing significant manual effort.
End-to-end Certificate Lifecycle Management
The AppViewX platform integrates with most endpoints and commercial CAs available on the market. In this case, teams were able to discover, request, renew, revoke, deploy, and create certificates from right within the AppViewX console, without having to switch between various CA and device vendor portals. SSL policy could be defined and enforced across the organization as well.
The client is a leading global insurance organization, operating across more than 80 countries and jurisdictions. Also, a Fortune 500, providing insurance and other financial services to support their clients in business and in life.
With a rapidly-growing customer base of over 100 million and thousands of internal and public-facing applications, the company’s IT infrastructure team was constantly challenged with endless service requests. Despite having one of the most powerful ADC infrastructures supporting their applications, the team found its core processes to be painfully manual, slow, and inefficient. They realized the immediate need for a more comprehensive, scalable and automated solution that could catapult them into the digital world.
Key Technical Objectives
- Streamline F5 BIG IP configuration management
- Automate LTM provisioning and VIP lifecycle management
- Enable Application and Security teams with self-servicing capabilities
- Obtain granular visibility into Network Infrastructure
- Perform device backups on regular basis and restore when required
With over 120 ADC devices and no management & automation platform, the network and application teams were unable to efficiently leverage the full potential of their F5 ADC infrastructure
- Long queue of tickets and subsequent service delays – Every change request had to go through the network team. Application teams had to wait long hours even for a simple enable/disable operation, which again had to be done manually.
- Outages due to implementation of faulty configurations – ADC configuration change requests were raised by multiple teams across organization without being vetted thoroughly at multiple levels (completely manual).The change records were manually opened, approved and executed.
- 3 – 4 days to provision an LTM instance – Backup, rollback, migration, and provisioning of device configurations were all manual and error-prone.
- Time-consuming software upgrades – The team was manually handling the high CVE upgrades, which was highly time-consuming thus were unable to run adequate validation checks, causing production outages.
- No VIP clean-up process – Utilizing Orion and custom SNMP discovery to generate reports and were analyzed and validated manually. The change records were manually opened, approved and executed.
AppViewX ADC+ as a Solution
AppViewX’s ADC+, application delivery automation solution provided role-based management, automation, and orchestration of F5 BIG-IP services. It simplified version upgrades and enabled self-service capabilities to multiple lines of business for the client.
- Self-Servicing with Controlled Network Dashboards
Multiple teams could get real-time visibility into the state, status, health, and performance of devices and applications from the Controlled Network Dashboards. Network engineers could create automation workflows for application-centric tasks like application enable/disable for rerouting traffic or spinning up virtual instances for testing. These workflows could then be shared with application owners using role-based access controls to self-service application-centric tasks without relying on network teams, leading to a significant drop in tickets.
- Configuration Management with Out-of-the-Box Automation Flows
AppViewX ADC+ fully automated the migration of configurations across devices and reduced configuration errors with out-of-the-box automation flows, change control through ITSM, pre- and post-validations, and built-in approval management process. The Visual Workflow module of the ADC+ solution generated templates of existing device configurations that could be updated with the required variables and automatically pushed to the new devices after the automated mandatory checks. Visual Workflow also supports bulk migrations, eliminating the need to type out configurations from scratch. These templates could also be self-serviced by the application teams, further saving time and effort.
- VIP/WIP Lifecycle Management
It automated the lifecycle of VIP/WIP management on BIG-IP LTM and DNS – from creation, modification, deletion, and decommissioning. AppViewX ADC+ is integrated with BlueCat to reserve & fetch free IPs and map them to the virtual server(s). It enabled teams with configurable parameters to track the VIPs/WIPs that are up/down for ‘X’ time frame. It also automated the approval and validation processes involved in creating virtual IPs.
- Software Version Upgrades
Software version upgrades, too, could be easily accomplished with the APS templates. Configurations could be migrated to a new/unused instance where the upgrade could be applied and tested, and finally brought to production. AppViewX automated the whole gamut of pre- and post-validation checks, ensuring zero possibility of outages and other service disruptions.
- Backup and Restore
AppViewX ADC+ enabled engineers to take on-demand or scheduled backups of device configurations and attributes and store them in a centralized repository. It also facilitates easy rollbacks to the last working configuration in case of failure during migration.
- End-to-End detailed Reporting
The platform enabled F5 administrators with app-centric topology views through customized reports and dashboards. Leveraging REST APIs, it helped the client optimize application and ADC performance with real-time auto-generated reports on CPU utilization, application traffic statistics, and unused VIPs.
- 99% reduction in time taken to provision LTM/GTM configurations
- 98% reduction in application service delays with self-serviceability
- 75% reduction in time taken for software upgrades – 1 GB or bigger files copied to multiple devices via single workflow in minutes
- 10X increase in application availability with almost zero outages
- Total Cost of Ownership reduced by 92%
Nationwide Building Society is the seventh-largest cooperative financial institution and the largest building society in the world, with over 15 million members. It is headquartered in Swindon, England.
Business Challenges in Certificate Management
Prior to AppViewX, Nationwide was using a generic solution to manage certificates. The solution was manually-intensive – certificate request, issuance, renewal, etc. required multiple steps and several back-and-forth exchanges between stakeholders. Manual certificate management resulted in considerable delays and inefficiencies.
The AppViewX Advantage
AppViewX provides Nationwide with a unified, automated solution to manage its certificates end-to-end. The certificate team receives automated certificate alerts and can request and download certificates from a single point. Issuing and renewing certificates have also got a lot easier, as AppViewX collects all the data necessary for requesting a certificate, which means users no longer need to go back and forth to gather missing information. This has significantly reduced enquiries.
The solution sends notifications to the team to enable access whenever someone requests a certificate and allows them to quickly approve or decline requests after viewing, eliminating multiple steps from the previous process.
AppViewX comes with a made-to-measure workflow which integrates the customer’s Active Directory, simplifying access and creating more efficient processes for users. The interactive GUI makes navigation easy and intuitive for users, adding to efficiency.
“The implementation of AppViewX has materially saved time and effort for users across the whole certificate management lifecycle, which is a great outcome.”
The customer is a Fortune 500 financial services company in the US, specializing in credit cards. It has over 110 million customers worldwide.
As a leading provider of financial services, the company handles tens of thousands of transactions every day, most of which happens online. The company has thousands of application instances running in distributed data centers across the globe to support the huge traffic. Application teams continuously develop new applications and feature updates to deliver seamless customer service.
Each new application instance or upgrade requires a configuration change on the LTM and DNS devices, which lands into the NetOps pipeline as a service request.
- NetOps teams received over ten thousand service requests a month on average. The requests included creating/modifying/deleting VIPs, WIPs, and DNS records, enabling/disabling pools for server rotations, creating configuration items in ServiceNow, generating performance and resource utilization reports, etc.
- NetOps engineers had to resolve each service request manually. Manually resolving a service request at any given time took up to 10 days, owing to long backlogs.
- There were no standardized workflows for the above processes. NetOps engineers scripted different workflows for the same process, leading to inconsistencies and compliance issues
Solutions Delivered by ADC+
AppViewX ADC+ provided out-of-the-box solutions that helped NetOps teams standardize, automate, and orchestrate service requests end-to-end.
- With ready-to-use workflows, NetOps teams could automate all repetitive service requests and resolve them in minutes as opposed to days, doing away with busywork.
- App owners could self-service tasks such as enabling/disabling pool members for blue-green/canary deployments and server rotation within the change window by triggering pre-built workflows with RBAC.
- Since workflows were templated, all NetOps engineers and app owners followed a standardized procedure to execute any task, such as creating a VIP or modifying a DNS record.
The company began using ADC+ in 2017.
- With automation and self-servicing, the number of service requests that the company could handle within the same period grew dramatically. It automated around 400,000 service requests a year.
- Automation cleared backlogs and accelerated service delivery. The time required to process a service request fell from days to minutes.
- Time saved with automation and resource optimization resulted in the company saving $20 million annually.
The customer is a mass media and entertainment giant headquartered in the US, with offices worldwide. They generate annual revenue to the tune of $35 billion on average.
In 2015, the company had around 1500 applications, 70 F5 ADC devices, and 9 network field technicians managing them. Year by year, the application count kept going up, warranting more ADC devices.
- The company needed to employ more network technicians on contract to manage the growing ADC numbers, resulting in an exorbitantly high TCO (Total Cost of Ownership) projection.
- Although the company had only 70 ADC devices initially, they lacked centralized management.
- With no centralized management and strict access controls, the devices ran a high risk of getting compromised.
- The team stored and exchanged certificates in spreadsheets and emails, worsening security issues.
- ADC management was chiefly manual, resulting in significantly high TTM (Time To Market) for applications, which in turn impacted business.
The company realized that the only way to lower TCO and improve operational
efficiency was through automation, which led them to ADC+
Solutions Delivered by ADC+
ADC+ provided end-to-end management and automation of F5 ADC devices.
- It provided centralized management of ADCs across data centers, allowing engineers to view and control ADCs from a single, GUI-based pane of glass.
- Network engineers could gain real-time visibility into the health and performance of applications, as well as historical data, helping in optimization.
- ADC+ allowed the company to schedule and automate both device and object-level
backups of F5, improving efficiency.
- External teams could use API for bi-weekly maintenance activity to bring applications up/down. This provides significant time saving while rolling out the maintenance patches.
- Application owners could self-service the management of their respective applications with RBAC-powered dashboards, reducing the reliance on network engineers for repetitive tasks such as enable/disables.
- RBAC also significantly brought down instances of unauthorized access,
- ADC+ provided complete, application-centric visibility into the application’s underlying infrastructure, aiding lighting-quick detection and resolution of incidents.
- ADC+ automated the Certificate Lifecycle Management of F5 devices
end-to-end, from request to renewal. Spreadsheets out, security in.
The company began using ADC+ in 2017.
- The company’s ADC fleet grew from 70 devices in 2015 to 200 devices in 2020.
- Their application count went up from 1500 in 2017 to 6500 in 2020.
- Due to centralized management, automation, and self-servicing capabilities offered by ADC+, the number of contract network technicians required dropped from 9 for 70 ADC in 2015 to 4 for 200 devices in 2020.
- As a result, the company was able to lower their TCO by 55% while increasing their F5 ADC footprint by almost 300%.
- With automation, the TTM for applications reduced by over 80%
- The MTTR (Mean Time To Resolution) of incidents reduced by 85% with visibility, monitoring, and context-aware troubleshooting.
A leading U.S.-based life sciences company that specializes in providing instrumentation, equipment, software, services, and consumables to the healthcare, pharmaceutical, and biotechnology sectors.
This customer is a firm that has multiple internal public key infrastructures (PKIs) for issuing certificates. The organization also has both on- and off-network client computers that required regular system updates and patches. This is a critical infrastructure security, especially for supporting increased remote work scenarios used during the COVID-19 pandemic.
Microsoft System Center Configuration Manager (SCCM) performs the patch and software deployments. SCCM requires machine certificates to authenticate and establish connectivity with the hosts.
Machine-identity certificates used for application security purposes are issued to client computers, services, and servers. This PKI system required extensive management, particularly the acquisition, enrollment, and management of certificates.
Primary Business Challenges
The IT team sought an abstraction tool that could achieve the following objectives:
Uniform Certificate Auto-Enrollment: While computers connected to Active Directory can leverage the Group Policy-based Windows Auto-Enrollment feature, those without regular connectivity to Active Directory cannot; hence, there was no way to enroll certificates on these devices. SCCM could not function because it requires certificates to authenticate.
Furthermore, certificates had to be renewed, enrolled, and installed on their respective clients periodically, and new computers were continually onboarded onto the network. This necessitated use of a tool that could standardize enrollment and connect to computers primarily running off-network.
Inventory and Reporting: Frequent acquisitions, ad-hoc deployments, and use of multiple certificate authorities and vendors posed a challenge for obtaining a comprehensive overview of certificates and their respective endpoints. This leading life sciences company identified the need for a centralized inventory for certificates deployed across the network. The company also wanted a transparent view of the certificate infrastructure.
Delivering a solution with AppViewX
After careful analysis, the AppViewX team crafted a solution for each of the aformentioned challenges. By helping implement an EST-based enrollment agent and a full-cycle certificate management suite, the solution met all objectives and delivered rapid results, which are detailed below.
Standardized Auto-Enrollment: An agent leveraging the EST protocol for certificate enrollment was deployed. It enabled AppViewX to act as an EST server, thus automating the enrollment and provisioning process. This also established a standard means of enrolling certificates across all machines, where AppViewX acted as a single, uniform interface for auto-enrollment. This enabled smooth patch and software management on SCCM client computers. Furthermore, new certificates were configured automatically on the end devices without human intervention. This particular use case was a distribut- ed, multi-node deployment across multiple Amazon Web Services data centers across the U.S. and Europe.
Controlled Access to PKI: To preserve PKI confidentiality and integrity, a role-based access control system was enforced across the network. It restricted access to infrastructure components, and, when necessary, provisioned them on an ad hoc basis. AppViewX’s audit trail feature also helped in this regard.
Full-cycle visibility, management, and automation: AppViewX’s environment scanning and inventory consolidation tool helped IT Operations build comprehensive inventories of certificates on file, complete with endpoint maps, statuses, and cryptographic details. AppViewX’s workflow automation capabilities enabled automation of certificate request/renewal processes while its reporting capabilities provided clear visibility into critical details such as validity. This increased visibility and control help prevent outages and contributed toward upholding organization-wide business continuity.