Updating SSL Certificates

Periodically SSL certificates need updating.

Checklist: Domains managed by Digital

• www.boston.gov - city's public website hosted by Acquia,

• registry.boston.gov - registry suite of apps hosted on AWS's ECS service,

• hub.boston.gov - city's intranet hosted on AWS's ElasticBeanstalk service,

• cityofboston.gov - city's legacy website, legacy apps hosted on IIS as ZPCOBWEB01,

• park.boston.gov - certificate for the Park Boston app hosted by Passport (external Vendor),

• microservices/apps - utilities hosted on AWS's ECS service.

Ideally certificate cycling activities should be conducted during the weekly Digital maintenance window. If that is not possible, then a Change Management ticket should be raised, (even if retrospectively) b/c these are production changes.

Digitcert Issued Certificates

See Jira tickets - DIG-3643 Nov 2023

To perform these tasks, you will need to obtain from the Security Team, the following:

• the SSL Certificate (possibly a .crt file),

• the SSL private key (a .key file), and

• one or moreCA intermediate certificates (possibly .pem files). (If there are multiple intermediate certs, you can cut and paste them into a single file with each cert starting on a new line below the old cert. The order is not important.)

All of these files should be readable in a text editor.

Acquia - boston.gov

The new/updated certificate is updated directly on the Acquia Cloud web console.

This process should be repeated on all environments on Acquia. It is recommended that the certificate first be applied to the dev environment so that any issues can be discovered on a non-prod resource ...

To install the new certificate:

1. Login to the cloud console, click Develop in top menu.

2. Select an Organization "City of Boston"), Application ("bostond8") and environment (e.g. "Dev").

3. In the left column menu, select SSL

4. Click the "Install SSL Certificate" button

5. Provide a sensible label (for example *.boston.gov 20xx)

6. Open the files provided and insert their contents into the text boxes. Note: The files will all have text in 80 char wide columns and start with text---- BEGIN CERTIFICATE/RSA KEY ----

7. Click the "Install" button

8. You should see a summary page with the old certificate (active) and the new certificate you just installed (inactive). When you are ready, you can Activate the new certificate, and then Deactivate and Remove the old certificate.

9. Repeat for each Acquia environment, (stage, ci, uat,dev2 & prod)

AWS - WebApps & TheHub

AWS manages SSL certificates in its Certificate Manager service.

WebApps (incl. registry.boston.gov)

The new/updated Digicert certificate can be loaded into the AWS Certificate Manager and then applied to various services within AWS as required.

1. Login to the AWSConsole.

2. Ensure you are in the us-east-1region

3. Navigate to the Certificate Manager.

4. Locate the *.boston.gov certificate in the list. Importantly: The correct entry will have a Type=Imported.

5. Expand the entry and click the "Reimport Certificate" button.

6. Cut and past the certificate parts onto the page and click "Review and Import".

AWS Services which used the old certificate will now use the new one without any further changes required.

04Oct2021. The registry app uses an AWS issued certificate. => This will be changed during 10/2021.

The Hub (hub.boston.gov)

The Hub is hosted on AWS using the ElasticBeanstalk service: It is in region us-east-2.

1. Login to the AWSConsole.

2. Ensure you are in the us-east-2region

3. Navigate to the Certificate Manager.

4. Locate the *.boston.gov certificate in the list. Importantly: The correct entry will have a Type=Imported.

5. Expand the entry and click the "Reimport Certificate" button.

6. Cut and past the certificate parts onto the page and click "Review and Import".

The hub will now use the new certificate without any further changes required.

IIS - cityofboston.gov

The cityofboston.gov certificate is installed on the Imperva load balancer/WAF. Digital do not need to install the certificate on the IIS server at zpcobweb01.

Video streaming

The wowza service on ZSTREAMING requires the updated cityofboston.gov certificate to be added/updated in the keystore.

See here or here

Detailed Instructions

Remote desktop on to zstreaming01.web.cob

Ensure that the path environment variable has the following 2 folders included in it:

C:\Program Files (x86)\Wowza Media Systems\Wowza Streaming Engine 4.8.17+1\bin
C:\Program Files (x86)\Wowza Media Systems\Wowza Streaming Engine 4.8.17+1\jre\bin

In a terminal (cmd), run the following commands (20XX should be the year the new cert will expire i.e. +12 months from now,):

cd C:\Program Files (x86)\Wowza Media Systems\Wowza Streaming Engine 4.8.17+1
mkdir certs20XX
mkdir certs20XX\csr
cd certs20XX\csr
keytool -genkey -keysize 2048 -alias wowzaprivatekey -keyalg RSA -ext SAN=dns:vodstreaming.cityofboston.gov -keystore vodstreaming.cityofboston.gov.jks
keytool -certreq -file vodstreaming.cityofboston.gov.csr -alias wowzaprivatekey -keyalg RSA -ext SAN=dns:vodstreaming.cityofboston.gov  -keystore vodstreaming.cityofboston.gov.jks

❗Note: after the keytool -genkey command you will be prompted for information, the correct field values are:

Enter keystore password: password
Re-enter new password: password
What is your first and last name?
  [Unknown]:  vodstreaming.cityofboston.gov
What is the name of your organizational unit?
  [Unknown]:  DoIT
What is the name of your organization?
  [Unknown]:  CityOfBoston
What is the name of your City or Locality?
  [Unknown]:  Boston
What is the name of your State or Province?
  [Unknown]:  Massachusetts
What is the two-letter country code for this unit?
  [Unknown]:  US
Is CN=vodstreaming.cityofboston.gov, OU=DoIT, O=CityOfBoston, L=Boston, ST=Massachusetts, C=US correct?
  [no]:  y

Once you have executed both keytool commands and have generated the new vodstreaming.cityofboston.gov file, you can send the csr file to IAM to have the certificate issued.

When you get the file back, copy it onto the certs20XX folder on the server.

During the csr generation, a new keystore (vodstreaming.cityofboston.gov.jks) was created in the certs20xx folder. This keystore needs to be copied into the conffolder and the certs loaded into it.

cd C:\Program Files (x86)\Wowza Media Systems\Wowza Streaming Engine 4.8.17+1\conf
del vodstremaing.cityofboston.gov.jks
copy certs20xx\vodstremaing.cityofboston.gov.jks vodstremaing.cityofboston.gov.jks 
keytool -import -alias DigiCertCA -trustcacerts -file ..\certs20xx\DigiCertCA.crt -keystore vodstreaming.cityofboston.gov.jks
keytool -import -alias TrustedRoot -trustcacerts -file ..\certs20xx\TrustedRoot.crt -keystore vodstreaming.cityofboston.gov.jks
keytool -import -alias wowzaprivatekey -trustcacerts -file ..\certs20xx\vodstreaming_cityofboston_gov.crt -keystore vodstreaming.cityofboston.gov.jks

Restart the wowza server.

---

Tip: you can view the currently loaded certs using:

keytool -list -keystore vodstreaming.cityofboston.gov.jks

Tip: If you accidentally load the wrong cert into the keystore, or give the cert the wrong alias, you can remove it with:

keytool -delete -alias xxxxxx -keystore vodstreaming.cityofboston.gov.jks

AWS Issued Certificates

AWS manages and issues SSL certificates in its Certificate Manager service.

AWS hosted webapps use certificates managed by AWS. All webapps without permanent *.boston.gov URLs use AWS issued certificates.

• *.digital-public.boston.gov and

• *.digital-staging.boston.gov.

AWS automatically renews expiring certificates and sends out an email to [email protected]. Simply click on the approval link in the email to allow the cert to be automatically regenerated and installed.

Someone in the Digital Team needs to be in the [email protected] email group, and needs to have the AWS policy AWSCertificateManagerPrivateCAPrivilegedUser or AWSCertificateManagerFullAccess in order to receive the emails and authorize the certificate renewal.

External Vendors

park.boston.gov (Vendor - Passport)

Digital Platform for Mobility Infrastructure & PaymentsPassport

Email: [email protected] - [email protected] Alt: [email protected]. Technical: [email protected]

Need to send Passport a copy of the certificate so that they can add it to the website (and possibly app) they host for parking in boston.