Managing Citrix ADC vulnerabilities with Citrix ADM Security Advisory

Vulnerabilities are inevitable when dealing with any system or process, computer based or otherwise. This isn’t a problem in itself, providing the people who are developing and maintaining the systems or processes are continually looking to uncover issues, develop ways to prevent the vulnerability; and implementing these changes to the systems in use.

This can feel like a lot of work when you have multiple systems and processes to manage at once.

Thankfully, our friends at Citrix take security seriously and are constantly looking to discover vulnerabilities in their systems, develop ways to mitigate or resolve their causes and provide us with appropriate fixes. Anyone managing a Citrix ADC (NetScaler) can attest to that, given the number of critical security updates that are being released.

Last week Citrix published a security bulletin for 2 new CVEs that impact Citrix ADC. This seemed like a good opportunity to demonstrate how the Citrix Cloud ADM Service Security Advisory feature can help us mange Citrix ADC vulnerabilities at each stage of the vulnerability lifecycle.

More info regarding the latest CVEs published by Citrix for Citrix ADC: Citrix Application Delivery Controller, Citrix Gateway, and Citrix SD-WAN WANOP Edition appliance Security Update Update 14/6/2021: After writing this blog an issue was found with the latest firmware version released to fix the 2 new CVEs – they have since released new versions to fix this issue: Customers who are using 13.0-82.41 and 12.1-62.23 versions and are experiencing functionality issues logging into Citrix Workspace App are recommended to upgrade to the following versions:  – Citrix ADC and Citrix Gateway 13.0-82.42 and later releases of 13.0  – Citrix ADC and NetScaler Gateway ADC 12.1-62.25 and later releases of 12.1

Firstly, it’s important to mention that the Instance Advisory feature is only available in Citrix Cloud ADM Service.

Vulnerability Discovery & Assessment

Citrix ADM Service will perform a scheduled scan on each of the appliances you have enrolled. By default this is run at weekly intervals and currently the time or date of the scan is not configurable; however you are able to run scans on demand. The on-demand scanning is obviously useful after the remediation stage to validate vulnerabilities are no longer present.

Assessing whether your ADC is vulnerable is not as clear cut as knowing the version you’re running. It is a combination of the firmware version, features that are in use and the configuration of these features. For this reason the Security Advisory scan is not just a version check against a list of firmware versions that have known vulnerabilities, Citrix ADM service also scans the configuration of your appliance to determine whether your configuration is using vulnerable features, or is configured in a way that enables the vulnerability.

This is achieved by running 2 scans periodically:

• Version scan: checks if the ADC version is a vulnerable version. The logic used is if a CVE is fixed on ADC release xx.xx, all the releases and builds lower than xx.xx build are considered vulnerable.
• Config scan – scanning the ADC config to see if there exists a specific config pattern that makes it vulnerable.

Having the config scan run on a schedule is an important feature as your config is not static, and any changes could introduce vulnerabilities, so a one-time check or assessment is not good enough.

Detailed information on the scans can be found by browsing to ‘Security Advisory’ from the ‘Networks > Instance Advisory’ menu and selecting the ‘Scan Log’ tab.

Each scan that is run generates a CSV report that you can download that details each CVE that has been checked for, plus what method of detection was used to determine whether the appliance was vulnerable.

Note: In the below report (columns with sensitive data hidden and IPs obscured) the Detection Method (Column Q) for CVE-2020-8299 is just a version check, whereas the detection method for CVE-2020-8300 is a shell script run against the instance config file:

My testing has shown that an on-demand scan can take anywhere between 5 – 45 minutes. I suspect this is dependent on the demand, as the scans I ran within an hour of the 2 CVEs being published by Citrix, took over 40 minutes.

I’ve had a look at the ADM Service rest API guide and cannot see a way to automate the scan on demand.

It is important to note that the CVE repository only contains CVEs published after December 2019. A full list of CVEs in the repository can be seen by browsing to ‘Security Advisory’ from the ‘Networks > Instance Advisory’ menu and selecting the ‘CVE Repository’ tab.

If you are running appliances on firmware version released prior to December 2019 it is strongly recommended that you upgrade to a later version.

Reporting

Once the scan has been run, either as per the schedule or kicked off on-demand, your Citrix ADM admins are alerted to the discovery of vulnerability on any appliances in a few ways:

• ADM Administrators will receive an email alerting them to instances with known CVEs

• When a Citrix Cloud Admin logs into https://citrix.cloud.com they will receive a notification

• When a Citrix ADM Service admin logs into ADM Service they can browse to ‘Security Advisory’ from the ‘Networks > Instance Advisory’ menu

From the Security Advisory console we get a good overview of the vulnerabilities that have been found:

• CVEs discovered on instances enrolled to Citrix ADM Service
• Date of the CVE publication
• Severity type (high/medium/critical) and score (hovering over the severity reveals the score)

• Vulnerability type
• Number of instances affected by the CVE. If you hover over the ‘ADC Details’ you will get a quick view of the appliance IPs (IPs obscured to protect the innocent).

• Steps required to remediate this CVE

From this page select ‘View affected instances’ at the bottom to see which appliances are vulnerable to which CVEs

Note: Currently there are no configurable notification settings for Security Advisory alerts, for example the ability to send vulnerability notifications to a certain address like you can with other ADM or ADC events. I have made a request for this so we shall see what comes back.

Remediation & Verification

Now to take action. From the ‘Security Advisory’ page we can start to kick off firmware upgrades and configuration job templates which are provided by Citrix where the vulnerability requires additional configuration – This is very cool.

Give you upgrade job a name and select your instance(s):

ADM will then perform the following pre upgrade checks to make sure the appliance is ready for an upgrade:

• Disk Space Check –
• HDD Errors
• User Customisation detection

If there is insufficient disk space, as was the case on this dev appliance, clicking ‘Check disk space’ allows us to see the used space on each partition and do some disk cleanup from ADM itself, rather than having to SSH onto the appliance

In this case just a few old firmware versions in the /var/nsinstall directory needed deleting and we we’re back on track.

We do have the option to run pre or post update commands as part of this configuration job. There are lots of reasons you may want to do this, for example to ensure the appliance or pair are not participating in a GSLB configuration before upgrading, enabling/disabling services or applying customisations post upgrade.

For this appliance we don’t need any pre or post commands to be run.

We can either run the job right after it’s been created, or schedule for another time.

We can now select the firmware version we wish to upgrade to. This is a brilliant enhancement as previously, you would have to go to the Citrix website and download the packages then upload them to ADM. another small update that saves us time

Specify reporting setting to be alerted when the job is complete and create the job.

You can track the progress of your upgrade job from the ‘Networks > Configuration Jobs > Maintenance Jobs’ screen

From here you can drill down into the Execution Summary > Instance Log > Command Logs for each upgrade:

Once complete, as per my request for an email upgrade report, I get an email telling me my job is completed along with a detailed report:

Attached report:

At this stage we have detected 2 vulnerabilities and performed a remediation action on our development appliance however, this is not the end, we still need to verify that we have resolved the vulnerabilities.

To do this we run an on-demand scan from Citrix ADM Security Advisory.

Note: When I first ran the on-demand scan, Security Advisory told me there were no CVEs impacting any appliances. Knowing this to be a lie I logged out of ADM and logged back in again, at which point I could see the results I was expecting.

The dev appliance (IP ending .240) is now no longer showing as vulnerable to CVE-2020-8299, only CVE-2020-8300.

We can see from the on-demand scan report for this appliance, that the CVE-2020-8300 script was run and assessed that this appliance was still vulnerable.

This is because CVE-2020-8300 requires additional configuration to mitigate the vulnerability so we still have some work to be done. Never fear, ADM can help us with this too.

This time we select ‘Proceed to a configuration job workflow’ from the Security Advisory console and a new configuration job will be created, with a template specific to the CVE you are remediating.

In this job template we have 6 variables declared, this is because ADM needs some information from you about your ADC configuration, to ensure the mitigation is applied correctly:

For Citrix ADCs that are configured to act as a SAML SP, authenticating users for a service against an external SAML iDP the first 3 lines are relevant.

• $saml_action_patset$ – This variable sets the name for the SAML action pattern set which will contain all the domains that should be authenticated using this SAML Action.
• $saml_action_domain1$ – This variable is to add the domain of a service to be authenticated against this SAML action (including trailing slash example: https://<example1.com>/)

If you have more then one domain that should use this SAML action to authenticate against an external iDP this line will have to be replicated for each domain with a unique domain variable name. For example:

bind patset $saml_action_patset$ "$saml_action_domain1$"
bind patset $saml_action_patset$ "$saml_action_domain2$"
bind patset $saml_action_patset$ "$saml_action_domain3$"

• $saml_action_name$ – This variable should reference the existing SAML Action to apply the RelayStateRule to.

If you have more then one SAML action configured the above lines should be replicated for each with unique varibale names. For example:

add patset $saml_action1_patset$
bind patset $saml_action1_patset$ "$saml_action1_domain1$"
bind patset $saml_action1_patset$ "$saml_action1_domain2$"
bind patset $saml_action1_patset$ "$saml_action1_domain3$"
set samlaction $saml_action1_name$ -relaystateRule AAA.LOGIN.RELAYSTATE.CONTAINS_ANY("$saml_action1_patset$")
add patset $saml_action2_patset$
bind patset $saml_action2_patset$ "$saml_action2_domain1$"
bind patset $saml_action2_patset$ "$saml_action2_domain2$"
set samlaction $saml_action2_name$ -relaystateRule AAA.LOGIN.RELAYSTATE.CONTAINS_ANY("$saml_action2_patset$")

For Citrix ADCs that are configured to act as a SAML iDPs, used by external services as an identify provider to authenticate users via SAML Profiles lines 4 – 5 are relevent.

• $saml_profile_patset$ – This variable sets the name for the SAML profile pattern set which will contain all the domains that should be authenticated using this SAML Profile.
• $saml_profile_url1$ – This variable is to add valid service profiles URLs that will use this SAML iDP profile – The URL format depends on the SAML SP.
• $saml_profile_name$ – This variable should reference the existing SAML Profile to apply the acsURLRule parameter to.

Again is multiple SAML iDP profiles are configured or multiple acs URLs are required lines will need to be duplicated in this configuration job as demonstrated in the SAML SP example.

In this case our appliance has no SAML iDP profile and a single SAML SP domain so we don’t need too much config.

More information on how to determine which lines of config you require can be found in the following article: Remediate vulnerabilities for CVE-2020-8300 (citrix.com)

This simple config job will:

1. Create a pattern set
2. Bind domains in which we have SAML SP config to the patset
3. Set the SAML action relaystaterule parameter to check the patset for allowed domains

Select the instance we wish to run this remediation config on:

Specify variable values specific to your configuration

A preview is then created of the configuration job

Specify when this configuration job should run and whether a report should be generated

After the job has completed we again go to verify whether our appliance is still vulnerable by running an on-demand security scan.

No more dev appliance showing up in the advisory console!

To double check, we can take a look at our scan log report and can see we are no longer vulnerable to either CVE. Fantastic

Hopefully, this has demonstrated how ADM Service can Discover a vulnerability on appliances across your ADC estate, accurately Assess whether this vulnerability affects your firmware versions, enabled features and configuration, Remediate the vulnerability by automating firmware upgrades and configuration jobs and then Verify accurately that your ADCs are no longer vulnerable.

Thanks to Stephen Watson for proof reading for me and correcting all my spelling and grammar!