Loading...
Loading...
Loading...
Loading...
Loading...
This has been updated for Drupal 10, as at October 2023.
IDP: Identity Provider - The IDP is Ping Federate which is managed by the City's IAM team. The actual user accounts are stored in Microsoft Active Directory and other City managed domain services which are connected to Ping. The IDP's function is to provide the user "authentication" checking user credentials and reporting security roles - reducing or removing the SP 's responsibility for verifying access and maintaining the security element of user accounts.
SP: Service Provider - The SP can be thought of as as a series of authentication functions within an application. In most implementations, the IDP and SP communicate using a SAML protocol. On successful authentication with the IDP, the SP receive standard and authoritative message (usually via a SAML formatted cookie) defailing the users name, email and (ideally but optionally) the Users security roles/groups.
In the case of boston.gov the SP is Drupal, or more specifically the SamlAuth module in Drupal.
The SamlAuth module (config-UI here) manages the communication with Ping (IDP) and exchanges SAML messages during a standard SAML process.
The IDP is used only to authenticate the user and SamlAuth receives a SAML message back from Ping (IDP) containing the username (cn), firstname, lastname and email of the authenticated user. If Authentication at Ping fails, the user cannot login to Drupal.
SamlAuth maps user accounts from Ping with user accounts in Drupal, where the cn
from Ping equals the username
in Drupal. If no matching account exists, SamlAuth automatically creates an account in Drupal, and assigns the content_author
role.
Security Groups in Ping are not reported to Drupal/SamlAuth, so no attempt is made to synchronize roles between Ping and Drupal. Even though it is not done at this time, this can be simply implemented in the future. Therefore, changes to user groups in Ping and Drupal are independent and never synchronized.
Drupal requires an email address to create an account. The email address does not need to be on the boston.gov domain, and it does not need to be unique. Ping (actually the IAM authoritative record systems behind Ping) cannot guarantee to provide an email address, so SamlAuth has been modified to create a "fake" email address using the pattern username@boston.gov
when there is no email in the SAML response from Ping.
SamlAuth requires an x509 security certificate (issued from or by the IDP (Ping)) in order to verify authenticity of the SAML response, and to decrypt the SAML message contents. This x509 certificate is created and distributed by IAM and is stored in environment variables and accessed by SamlAuth via the Key module.
In COB, we usee 2 IDP providers - one for prod and one for all non-prod.
The prod IDP provider is accessed via to sso.boston.gov.
The non-prod IDP provider is the IAM test provider at sso-test.boston.gov.
Boston.gov users typically use Access Boston to login to boston.gov (Drupal).
The tile on Access Boston Production launches content.boston.gov (prod)
The tile on Access Boston Test launches d8-stg.boston.gov (stage).
To use SSO to login to any other Acquia environment, you cannot use Access Boston and need to use the saml/login link. For example:
Boston.gov dev => https://d8-dev.boston.gov/saml/login
Boston.gov UAT => https://d8-uat.boston.gov/saml/login
Boston.gov local => https://boston.lndo.site/saml/login
From time to time the IDP certificate will expire and need to be re-issued by the Identity and Access Management team.
You should request the IDP Metadata XML
from IAM. This info will come as a single file per environment - with the extension .xml. See the next box to work out which part of the metadata is the certificate.
For Acquia Environments:
Login to the Acquia Cloud Console
Navigate to the environment you wish to update, and select the variables
section
Update the environment variable with the new certificate (cut and paste it over the existing entry) and save.
Note: The change will be immediate on the environment.
For your Local Dev Environment:
Your local dev environment uses the IDP Metadata from the IAM Test environment.
Login to your local Drupal
On this page, edit the key you wish to update
On the key page, in the certificate box, paste the new x509 certificate
Save the page.
Tip
You can check the expiry of the IDP certificate by clicking the info
button the key entry console.
The code used to manage the single sign on process with Ping Federate is SamlAuth.
SAML: /admin/config/people/saml
Keys /admin/config/system/keys
SamlAuth: https://www.drupal.org/project/samlauth
This page documents data sources and pipelines for maps on boston.gov that use the maps Drupal component.
This spreadsheet contains all maps in that have been created. If updating that spreadsheet gets away from you and you want to know what pages on boston.gov are using maps, ask a Drupal dev on the Digital Team to run this query:
That will give you an output that looks something like this JSON file (last ran 10/1/20). A "1" in the status field means they are published pages and publicly accessible, a "0" means they are not published and cannot be seen unless the user can log into the website.
It's important to note that maps will come on and off the website with time as many of them are seasonal or for certain events. Therefore, you may not always be able to see the maps on the pages linked to in the spreadsheet above or in the output of that query. If you need to assist in getting a map that once was on the site but now isn't one of two things can happen:
The Digital Team can get to old JSON configuration file from a past revision of the page and you can work off that to start
You should be able to find the JSON in this Google Drive folder which has every JSON config we've created for a map on boston.gov.
Below are the main steps in getting a map on boston.gov:
Get data
Make sure the data stays up-to-date
Create a map on boston.gov in Drupal
There are two main ways you can set up the data for maps on boston.gov:
Google Sheet that feeds into a hosted feature service
ArcGIS Online hosted feature service
For boston.gov to be able to access and render the dataset, it has to be https. This means anything published to ArcGIS Online via the EGIS database cannot be used for a boston.gov map since they have http feature service urls.
Feature service url for Landmarks published via EGIS database: http://gis.cityofboston.gov/arcgis/rest/services/EnvironmentEnergy/OpenData/MapServer/3
Feature service url for Landmarks hosted feature layer: https://services.arcgis.com/sFnw0xNflSi8J0uh/arcgis/rest/services/BLC_Landmarks_Hosted_Approved_Landmarks/FeatureServer
If the stakeholder you are working with is more comfortable editing and updating information in a Google Sheet, you can create a hosted feature service using this.
The Google Sheet you set up needs to have one of two things:
Latitude and Longitude fields (preferred method)
An address field that contains the locations entire address (e.g. 200 Heath St, Jamaica Plain, MA 02130)
Using latitude and longitude fields allows the connection between the Google Sheet and the created ArcGIS hosted feature service to be automated. If you use the address field, the Google Sheet and the feature service will not be connected and you'll have to manually update it every time it changes.
To create a new feature service using Google Sheets, you need to:
1. Download it as a csv file:
2. Re-name the file to whatever it is you want your feature service to be named.
3. Log into BostonMaps under the ETL developers username.
4. Create a new folder for this feature service.
5. Click on "Add Item" the "From Computer":
6. Choose the file you want to upload. Give it clear tags and confirm that it is using the correct fields for locations. If you are using addresses instead of Latitude and Longitude fields, make sure "ArcGIS World Geocoding Service" is selected.
Below is a screen recording of this process:
7. Once the feature service is created, you need to make sure the share settings are set to "public". If you don't do this, we won't be able to access the service from boston.gov.
If the stakeholder you are working with is comfortable editing hosted feature services in ArcGIS Online, you can use that as well. The only thing needed for the boston.gov maps is that the feature layer be shared publicly.
It is not ideal to have publicly accessible hosted feature services, therefore it is best practice to create a View of the feature service you are using. You can then set that view to be publicly accessible and use that for the map while the editable layer stays internal.
An example of this set up is with the COVID-19 testing locations:
You can create an ArcGIS web application that allows the stakeholder to edit and update the data if you want. Or, if they are comfortable just using the "Open in map viewer with full editing control", you can have them use that.
The pro of setting the map up this way is that when the stakeholder makes an update it is immediately and automatically updated on the boston.gov map.
If you are using an ArcGIS hosted feature service and having the stakeholder edit that, you just need to get them directions on how to add to a map with full editing control, they can then update the service.
If you are using a google sheet that has Lat/Long fields, you need to give the stakeholder instructions on how to update that information. Generally, we point people to google maps. Here is a screen recording of how to get the values.
Once you have that, you can submit a ticket to the Data Engineering team via this form. Be sure to include the link to the Google Sheet, the link to the hosted feature service in your request, and the frequency with which you want the pipeline to run (most if not all are nightly).
It might take a few days for the connection to get automated, and you may have a time sensitive map you are working on (e.g. early voting locations, covid testing sites, etc.). In this case, you can follow the instructions for manually updating a map below.
To manually update a map:
Download the spreadsheet as a csv file.
Rename it to match the hosted feature service.
Navigate to the hosted feature service's "Overview" page in ArcGIS Online.
Click on "Update Data" then "Overwrite Entire Layer"
Chose the csv file you downloaded are renamed.
Date fields can get read into ArcGIS Online as what I think are unix time stamps (number of seconds since Jan 1, 1970).
A hack-y fix for this is the add an additional row to the spreadsheet and just add "test" into the date field for that row. All other fields can be blank.
This extra field with force the dates to be parsed as strings in ArcMap. After you've updated the map, you can delete the test entry so you don't confuse the stakeholder, just remember to put it in again before you update it next time. Again, this is a hack-y solution! I do not believe this issue exists when lat/longs are used and the connection is automated.
You may have to work with your stakeholders to fudge some addresses to get them to show up on the map if you are relying on the ArcGIS World Geocoder. If you open map viewer in ArcGIS Online, you can search for addresses and figure out what one will get you closest to the location you are trying to map.
For example, mapping City hall with the ArcGIS world geocoder can lead to different results depending on the address entered:
The maps component on boston.gov is made for very simple, operational maps. Best practice is to keep the layers as minimal as possible - 1-2 layers max is best.
Any map icons need to be publicly accessible somewhere on the internet for us to be able to use them on a map. In most cases, we use icons available via the Digital Team's pattern library (Fleet). You will see maps the leverage icons that are not in that list, if that happens and you want to use it or you want any of the other experiential icons in this google drive location, email the Digital Team. They should be able to get it onto boston.gov and give you back a link to it.
We can cluster points on maps, we use leaflet's marker clustering functionality to do this. This is important to note, because this is one of the only ways to view points that are on top of each other. The other way is to leverage filtering, discussed below.
If you have a dataset or are mapping two datasets where points over lap, you need to move to location of one of the points if you don't want to use clustering to display them both. We do not have the functionality to click through multiple pop-ups like ArcGIS Online maps do.
For example, in the 2020 early voting map, we display both ballot dropboxes and early voting locations on the same map as two separate layers. There is a ballot dropbox at City Hall as well as an early voting location. The were originally sitting on top of each other until we updated the address for the early voting location.
Polygons and lines can only have one color. For polygons the color will be made transparent and fill the space. The outline of the shape will be the true color.
For lines, the line will be that color but as the user zooms in, it will get transparent so that street names are visible.
Polygons and lines can be set to change color when a user hovers over them:
On these maps we cannot color by attribute (e.g. locations with "Type" of "x" are blue, type "y" is red). If you encounter a situation where this is necessary, the best thing to do is create views in ArcGIS Online that filter by specific type. Then each view can be added to the map as its own layer.
The layers on this maps can be set up to be filtered by the user. Each filter will only work on one layer, but this can be used to help display points that may lay on top of each other.
For example, the food trucks map has multiple trucks in the same location various days of the week. We use filters with a default value of the current day to determine what trucks to show when the map is opened:
These filters can also be aware of each other. For example, the "Truck" filter above will only show trucks that show up on Fridays now that the Day has been set to that.
For certain maps, it is important that we use SAM addresses because the precision is extremely important. This generally the case when we are trying to tell people which polygon their address is inside (e.g. what City Council district you live in, whether or not your address is in a Historic District).
In these cases, the ESRI world geocoder is not precise enough as it may geocode addresses to the street centerline which could mean we tell someone they are represented by the wrong person.
These maps are configured using JSON. This folder contains all the JSONs that have been created for maps on boston.gov.
The best way to get started when creating a new map, is to look through this list and choose a map that has similar functionality to what you are going for. Then grab the JSON configuration linked there, and drop it into this map editor page. You can then start making updates to the JSON and when you want to see the change reflected on the map press the tab key and it will updated.
The embedded maps feature was written as a web component so that it could be added to the Boston.gov Drupal site but also any other web pages we make. It is built using the StencilJS reactive web component compiler.
For info about the <cob-map>
web component and its attributes, see the “Notes” pane in the <cob-map>
Fleet documentation.
The JSON configuration for the web component is specified in map-1.0.json.schema in the CityOfBoston/patterns repo. Auto-generated documentation for it can be found here: map-1.0 schema documentation.
Source code for <cob-map>
and our other web components is in the Fleet repo: CityOfBoston/patterns
Technical and project documentation for charts on boston.gov.
We use the VegaLite and Vega libraries to create our charts on boston.gov. VegaLite is an easier to use version of Vega. Both libraries are built on top of D3. It does a little more guess work for us so we don't have to be so specific in defining our charts. These libraries let users define/create a chart using a JSON schema.
From a technical standpoint, we wrap this library up as a web component that is stored in Fleet, our patterns library. The <cob-chart>
component takes a JSON object as input in a Vega or VegaLite schema as input and creates a chart.
From a functional standpoint, when putting charts on boston.gov, we place them within text components in an effort to bring context to the data and information the chart is displaying. The FY20 Budget Website has good examples of how charts should be used with narrative explanation.
In general, content on boston.gov should be at or below an 8th grade reading level. In an attempt to be in accordance with this and keep our site as easy-to-understand as possible, we push for simple and straight forward charts (e.g. bar charts, line charts, and sometimes pie charts if we have to).
Coordination with both the boston.gov Content Manager (for chart placement) and the Analytics Team (for chart data) will be required to get a chart on boston.gov.
The general overview for getting a chart on boston.gov is:
Get some data
Work with the Analytics Team to get it into a public s3 bucket and set up an automated workflow for it getting updated (if appropriate).
Grab the schema from one of the charts below that looks/is most like the one you want to build
Drop it into the Online Vega Editor to start editing it to use your data
Check that your chart works/looks okay using the COB-chart editor.
Add your chart to boston.gov through Drupal.
Vega/VegaLite can read in data from a publicly available url. We leverage this functionality so that we can set up separate automation practices for the data charts on boston.gov display. This helps ensure the data on the chart stays up-to-date.
For testing purposes, you could upload a csv as a Github Gist and use that url, but when things move to production, all data should be pushing/pulling from an s3 bucket.
The Analytics Team has the ability to create s3 buckets that can store completely public csv files. Once a file has been loaded to a public bucket, you can supply the "Object URL" to Vega.
Storing data for charts on s3, means we can set up separate workflows that will automatically update the data on the chart. For example, for the FY20 Budget website, we set up a workflow in Civis (the Analytics Teams data wharehouse and ETL platform) that:
Pulls data from a publicly available google sheet the Budget Office updates
Drops the data into Civis' Postgres database
Uploads the tables as csv to s3 (e.g. https://s3.amazonaws.com/public-budget-data/fy20_capital_projects_chart_data.csv), replacing existing files of the same name with new information.
With this workflow, we can ensure that when the Budget Office needs to make a change on a chart on boston.gov, they only have to update the agreed upon spreadsheet. More detailed documentation on this process.(add link)
To create the charts, we use a json schema created an maintained by the authors of Vega/VegaLite. We primarily rely on VegaLite as it is easier to use than Vega. The only time Vega is used is to create a pie chart.
A good place to get started for an overall understanding of the libraries is their docs:
The Vega libraries can do much more than what we've implemented at the City (e.g. create maps, box plots, multi-view displays, treemaps, etc.). The charts web component was built and tested for only the following charts:
Each of the above charts can be built with one selection element as well. The selection element can only be built off one field and will always appear as a drop down box. All available types of charts on a boston.gov test page.
When developing a chart, it will likely be easiest to start using the Vega/VegaLite Online editor. Once the chart has largely been built, the COB-chart editor is helpful to make sure the chart will still work when rendered as a web component and for making sure fonts look okay.
The <cob-chart>
web component supports an additional section of the schema called "boston". This section supports three inputs specific to our uses cases:
"chartID" - string, optional: unique id of the chart
"minWidth"- integer, optional: minimum width in pixels of the chart. Due to data constraints, some charts will not be readable on smaller screens. In this case, we can supply a minimum width to the chart and users will be able to horizontal scroll on any screen smaller than the minimum width supplied. Example of this functionality in production.
"defaultSelection": string, optional: value of a drop down's default selection
Below are examples of how each specific type of chart gets implemented at the City of Boston. Links to example schemas are given with call outs of particularly interesting functionality the chart contains.
The height of the chart defined in the schema will be respected and used when the chart is rendered as a web component on a boston.gov page.
The width of the chart will be overwritten based on the screen size the chart is rendered on if "autosize": "fit"
is in the schema. This helps ensure a chart fits on both large and small screens.
Drop the JSON from the file linked below in the COB chart editor to see the chart below (chart in production):
If you want the colored sections of bar charts to display in a specified order, you can use the VegaLite transform to create a new field in the data. In the example below, this field is called "barOrder"
.
You can then use that field to order the bars so that the number of Community-Based Organization seats is always on top.
You can use the "sort" section of the schema or an axis to sort by various metrics. For example, we may want to have the longest bars at the top of a chart on a bar chart.
To achieve that, we add "sort"
and define the field, operation, and order of the bars to the axis definition we want sorted.
Drop the JSON from the file linked below in the COB chart editor to see the chart below (chart in production):
Due to a bug in the version on VegaLite we're using, defining your own tooltip with a "line" mark will break the chart.
When you are not using a selection, you can get around this by layering a line mark and a point mark on top of each other in the chart and defining the tooltip in the point mark's encoding. Use the line chart schema linked above as an example of how to do this.
If you are using a selection, the chart will work if you do not define tooltips, so work with the data owner to ensure field names are appropriate for displaying. An example of this is shown in the "Line Chart with Selection" tab above.
Drop the JSON from the file linked below in the COB chart editor to see the chart below (chart in production):
The main difference between a grouped bar chart and a regular bar chart is that we add a "column"
section, where we define the field to group by and style the header, to the schema's "encoding"
.
The COB web component was not created/tested for supporting "rows", we only support grouping by column.
Pie charts are the only chart type the <cob-chart>
component supports that use the Vega schema.
Drop the JSON from the file linked below in the COB chart editor to see the chart below (chart in production):
The height and width on pie charts should pretty much always be set to 200px. This ensures that the chart fits on all screen sizes.
If the height and width are set to more than 200px, the chart will resize to best fit the container it is loaded in, but the center of the chart will not change if the screen is resized, so it may get cut off if the user adjusts their screen size.
Vega supports both a domain an a ranges an inputs to their "scale"
encoding. This means you can assign individual values to specific colors so they stay consistent as a user interacts with them. This is particularly helpful on charts with a selection in which not every selection will have all the values shown.
For example, we use domain and range to ensure the "Explore Spending" chart on budget.boston.gov always uses the same colors for the seven difference types of Operating Budget spending.
Data from a URL can be parsed to ensure it is read as the correct data type when Vega reads it. This is a good idea to ensure your chart will perform as expected and any issues with the data input will get turned to null
values instead of the entire column being read as a string.
While this is helpful, number fields still need to be formatted without commas in the source data. Even when explicitly parsed as a number, Vega will not read in number fields correctly if there are commas in them. You'll have to work with departments or the Analytics Team to make sure the csv's getting pushed to s3 do not have commas in numeric fields.
Vega and VegaLite support a "fold" transformation that allows you to reshape your data from wide to long.
This can be extremely helpful as Departments may often store and/or conceptualize their data in the "wide" format, while charting libraries and data nerds like data in a "long" format.
Now that our data is formatted correctly, we can use the "Type" column to color our chart so we get three lines showing the trends in the different types of State Aid.
Vega and VegaLite both support d3's number formats allowing you do things like putting a dollar sign in front of money values and inserting back those commas that you had to tell the data owner to take out.
When working with large numbers, you can use Vega's "calculate"
transform to divide the numbers on an axis by some amount (e.g. a million) so that displaying them on a y-axis becomes easier. You can then use the original field for displaying the amount in tooltips.
Defining a tooltip when using a line mark will break the chart.
This has been fixed in later versions of VegaLite, but that release also refactors the entire way selections operate which broke our charts in a bunch of other ways, so yours truly decided to save fixing this for "later".
Pie charts aren't responsive - should always be sized to 200px
We don't support "rows", we only supported vertically grouped bar charts that use columns.
The code for the web component is stored in Fleet, our patterns library.
TDB
Chart Type
Vega or VegaLite
VegaLite
VegaLite
VegaLite
Vega