Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
This document outlines to process for getting the Budget Fiscal Year website together. It has been shared with OBM (Office of Budget Management).
Building Housing allows constituents to see a full inventory of projects and parcels managed by the city of Boston.
This Drupal App allows residents to browse all active housing, open space, commercial, and to be decided (TBD) projects. It also provides information on city-owned land for sale.
Residents can click search for a specific project and/or view a map of all projects. A drill-down into a project page displays goals, a timeline, photos, meetings and more.
Search for Projects on the Building Housing Map
View details of a Project
Auto-create Community Meeting Events from Sales Force on CRON (5 minutes)
Auto-create and update Projects from Sales Force on CRON (5 minutes)
Drupal
Sales Force
Google Maps API
node_buildinghousing (docroot/modules/custom/bos_content/modules/node_buildinghousing)
Views
Webforms
Geolocation
Salesforce
The following Drupal Entities are created to warehouse/cache data which originates in Salesforce.
The following nodes have records which are populated (add, update and delete) by mappings between Salesforce and Drupal which are run each cron cycle.
The following taxonomies have been created and their list items are maintained manually by Drupal developers. Taxonomy items can be added and deleted as needed, but usually will need adjustments to code to work as required.
As at March 2023
Entity | Name | Description |
---|---|---|
node
bh_project
The primary content_type for a Building Housing Property. Contains meta data about the Project and links to updates, attachments, parcels etc.
node
bh_update
Contains information about updates to a project. This includes certain status changes, attached documents, links to community meeting records and comments from CoB Project Managers to insert into the timeline.
node
bh_meeting
Contains information about Community Meetings held by CoB with residents regarding Building Housing Properties.
node
bh_parcel
The official parcel number and top-level info - with GIS coordinates for the parcel.
node
bh_parcel_project_assoc
Possibly deprecated ? I believe the parcel# is now saved in the field_bh_parcel_id field of bh_project.
node
bh_contact
Deprecated (no data)
node
bh_account
Deprecated (no data)
taxonomy
bh_project_stage
The overall project stage. (usually when project status = active).
Linked directly from bh_project
.
taxonomy
bh_project_status
The status of the Project.
Linked directly from bh_project
.
taxonomy
bh_funding_stage
The funding stage for the Project.
Linked directly from bh_project
.
taxonomy
bh_project_type
The broad project type for the Project.
Linked directly from bh_project
.
taxonomy
bh_project_update_type
Update type for a bh_update
.
Linked directly from bh_update
.
taxonomy
bh_property_type
The property type.
Linked directly from bh_parcel
.
taxonomy
bh_public_stage
The Project stage as used in the timeline.
Linked directly from bh_project
.
taxonomy
bh_neighborhood
taxonomy
bh_record_type
taxonomy
bh_disposition_type
List of disposition types - for use in map.
view
building_housing
view
bh_maps
view
building_housing_updates
page
buildinghousing/[propertyname]
This is the landing page for information about a property.
This is a customized page for the node bh_project
and contains a timeline and information about parcels.
From the building housing landing page a user can click and open a map showing all the current projects. The map and sidebar list are both generated via Building Housing View.
Entry point: /buildinghousing (click show map)
Custom CSS:
docroot/modules/custom/bos_content/modules/node_buildinghousing/css/node_bh_landing_page.css
docroot/modules/custom/bos_content/modules/node_buildinghousing/css/views_bh_listings.css
Views Template: docroot/modules/custom/bos_content/modules/node_buildinghousing/templates/views-view--bhmaps--maplist.html.twig
Views Functions: docroot/modules/custom/bos_content/modules/node_buildinghousing/node_buildinghousing.views.inc
Custom Markers: docroot/modules/custom/bos_content/modules/node_buildinghousing/images
Building Housing allows constituents to see a full inventory of projects and parcels managed by the city of Boston.
Entry point: /buildinghousing (click show map)
URL pattern: /buildinghousing/{ProjectName}
Custom CSS: docroot/modules/custom/bos_content/modules/node_buildinghousing/css/node_bh_project.css
Field Templates: docroot/modules/custom/bos_content/modules/node_buildinghousing/templates/snippets
Field Formatters: docroot/modules/custom/bos_content/modules/node_buildinghousing/src/Plugin/Field/FieldFormatter
Helper Functions (Pre-process, alters, fields):
docroot/modules/custom/bos_content/modules/node_buildinghousing/node_buildinghousing.module
docroot/modules/custom/bos_content/modules/node_buildinghousing/src/BuildingHousingUtils.php
Parcel Map - settings
Uses BuildingHousingUtils->setParcelGeoPolyData() and Arcgis to set the Polygon geo data for the parcels
Photo gallery - settings
Building Housing Project Map (same as above Map feature)
Project information
Developer Information - Custom field in node_buildinghousing.module
Project Goals
Project Timeline - Altered field to combine other fields
docroot/modules/custom/bos_content/modules/node_buildinghousing/src/Plugin/Field/FieldFormatter/EntityReferenceTaxonomyTermBSPublicStageFormatter.php
docroot/modules/custom/bos_content/modules/node_buildinghousing/src/BuildingHousingUtils.php
Project Type - Custom field in node_buildinghousing.module
Contact information - Custom field in node_buildinghousing.module
Email sign-up - Custom field in node_buildinghousing.module
Feedback form - settings
The largest on-page component for a Building Housing Project record in Drupal is its timeline. The timeline visually displays a Projects past and projected events & information in a time ordered list.
The main code is contained here:
docroot/modules/custom/bos_content/modules/node_buildinghousing/src/Plugin/Field/FieldFormatter/EntityReferenceTaxonomyTermBSPublicStageFormatter.php
Various templates are defined to create each timeline article, one template per timeline type.
Timeline icons are controlled by css and are defined in function getStageIcon()
in EntityReferenceTaxonomyTermBSPublicStageFormatter.php
The various stages are defined at items in the bh_public_stage
taxonomy
Drupal Building Housing records are synchronized from MOH SalesForce on a schedule. Salesforce is the authoritative source, and data should not be added or changed in Drupal.
There are 6 synchronizations with Salesforce which run in the following order, every cron run (so every 5 mins) The order is important, because Projects must be created before Attachments & Website Updates before Meetings & Chatter postings.
Each synchronization process does the following: A Drupal Application runs a Salesforce API object query to identify any records in the SF object which have been deleted or which have their last updated date after a last updated date stored by Drupal for that SF object. The identified records are then added/updated or deleted in Drupal. At the end of the process Drupal updates its last updated date for that object with the latest SF updated date found in the import. This date is then used as a high-water mark for the next import cycle.
This synchronization imports Project records from Salesforce Project__c
object into Drupals' bh_project
entity.
This synchronization manages project stages, documents and messages to appear on the timeline. It extends and replaces the functionality for the Update__c object which is imported for legacy reasons in Building Housing - Project Update
.
There is only ever 1 Website Update (Website_Update__c
) record per Project (Project__c
) record in Salesforce.
There is a rule in Salesforce to stop multiple records which would potentially create confusion for project stages etc.
There should be no new Update__c records being created in SF. However, there are legacy records containing data which must be included in Drupal. Even though we do not normally expect the sync to process these objects, the code is important if the data is to be recreated accurately and completely (for example if a Salesforce purge is performed).
This handles legacy TextMessages (now use chatter) and document attachments (now use Website Update Attachments).
This synchronization imports Community Meetings event records from Salesforce Community_Meeting_Event__c
object into Drupals bh_meeting
entity.
This is a simple mapping and the import does little except cleaning up any URLs and address fields.
The bh_meeting
record holds a reference to it's parent bh_update
which is linked to the bh_project
.
If a Meeting event is updated, or deleted in SF, then the associated record will be updated in Drupal, and if necessary will move on the timeline.
Timeline items are gathered from various places in the system.
- text posts are items in the field_bh_text_updates
field of the bh_update
entity. Tiles are inserted using the messages create date for ordering. If the message is updated in SF, it continues to use the original created date. If the message is deleted in SF it should be deleted from the timeline. Posts are sourced from Salesforce Chatter and imported using .
- documents are file objects which are referenced as items in the field_bh_attachment
field of the bh_project
. Tiles are inserted using the create date of the attachment for ordering. If Drupal detects an update to the attachment in SF, it continues to use the original created date for ordering. Documents are sourced from Salesforce and imported using .
- a single rfp tile is created if the field_bh_rfp_issued_date
field is not empty Tile is inserted using the date in the field_bh_rfp_issued_date
field. If the date is changed (or deleted) in SF, the rfp tile will be moved or removed from the timeline. RFP date is sourced from Salesforce as part of .
- meetings are bh_meeting
objects which are referenced as items in the field_bh_update_ref
field of the bh_update
entity. Tiles are inserted using the meetings start date. If the meeting is updated in SF, meeting will move accordingly in the timeline. Meetings are sourced from Salesforce and imported using .
- stages
Sync Name | Drupal Destination | SF Origin |
---|
If multiple Website Update records do exist for a Project in Salesforce, then all records will be imported into Drupal, but ONLY the last (when ordered by createdDate) will be used in the .
Building Housing - Projects | bh_project | Project__c |
Building Housing - Website Update | bh_update | Website_Update__c |
Building Housing - Project Update | bh_update | Update__c |
BH Community Meeting Event | bh_meeting | Community_Meeting_Event__c |
Building Housing - Parcels | bh_parcel | Parcel__c |
Building Housing - Parcels-Project Assoc | bh_parcel_project_assoc | ParcelProject_Association__c |
A DND Development Officer is able to create a Meeting object in Sales Force, with all the meeting information, and attach it to a Project in Sales Force. When CRON runs on Drupal it then will sync any new or updated Meetings from Sales Force with a Drupal BH Meeting. After the new meeting is crated in Drupal, we also creat a Drupal Event so that the meeting will be listed on the Boston.gov Events page. The Meeting is also then displayed on the corresponding Drupal BH Project.
BH Meeting Content Type: /admin/structure/types/manage/bh_meeting
Sales Force Mappings: /admin/structure/salesforce/mappings/manage/bh_community_meeting_event
Templates:
docroot/modules/custom/bos_content/modules/node_buildinghousing/templates/snippets/bh-project-meeting-notice.html.twig
docroot/modules/custom/bos_content/modules/node_buildinghousing/templates/snippets/bh-project-timeline-meeting.html.twig
Helper Functions (Pre-process, alters):
docroot/modules/custom/bos_content/modules/node_buildinghousing/node_buildinghousing.module
docroot/modules/custom/bos_content/modules/node_buildinghousing/src/BuildingHousingUtils.php
This feature allows Drupal entities to sync back and forth with Sales Force Objects via the Drupal Sales Force module. It is primarily used by DND to use the data and access that is already on DND's Sales Force server to automatically sync with the Boston.gov Drupal site. This is controlled by field mapping configurations in the Drupal Sales Force module. Currently, all syncing is scheduled to happen on Drupal CRON run, every 5 minutes, with only updated objects.
Sales Force Mappings:
Building Housing - Projects (/admin/structure/salesforce/mappings/manage/building_housing_projects/fields)
bh_project --> Project__c
Building Housing - Website Update (/admin/structure/salesforce/mappings/manage/bh_website_update/fields)
bh_update --> Website_Update__c
Building Housing - Project Update (/admin/structure/salesforce/mappings/manage/building_housing_project_update/fields)
bh_update --> Update__c
BH Community Meeting Event (/admin/structure/salesforce/mappings/manage/bh_community_meeting_event/fields)
bh_meeting --> Community_Meeting_Event__c
Building Housing - Parcels (/admin/structure/salesforce/mappings/manage/building_housing_parcels/fields)
bh_parcel --> Parcel__c
Building Housing - Parcels-Project Assoc (/admin/structure/salesforce/mappings/manage/bh_parcel_project_assoc/fields)
bh_parcel_project_assoc --> ParcelProject_Association__c
Sales Force Settings:
Building Housing - Projects (/admin/structure/salesforce/mappings/manage/building_housing_projects)
Building Housing - Website Update (/admin/structure/salesforce/mappings/manage/bh_website_update)
Building Housing - Project Update (/admin/structure/salesforce/mappings/manage/building_housing_project_update)
BH Community Meeting Event (/admin/structure/salesforce/mappings/manage/bh_community_meeting_event)
Building Housing - Parcels (/admin/structure/salesforce/mappings/manage/building_housing_parcels)
Building Housing - Parcels-Project Assoc (/admin/structure/salesforce/mappings/manage/bh_parcel_project_assoc)
Troubleshoot Sales Force connection issues
If Drupal and Sales Force are not connecting or syncing please check the Authorization from Drupal to Sales Force (/admin/config/salesforce/authorize/list). You may need to Re-auth or even make a new connection if you need to connect to a lower development or testing environment on Sales Force. If you need access to an instance contact DND's Sales Force developer/administrator.
If a single item is not syncing or if you need info about the Drupal to Sales Force connection you can view the list this admin page. If you edit the instance you then have the option to force pull or push the Drupal entity with the Sales Force Object. If there is an issue you should see an error message in the response. You can also find other useful info like timestamps and record ids.
A generic form which is attached to email addresses found on boston.gov, and handles sending emails to those addresses.
A contact us form template is maintained (within script tags) in bos_theme/templates/snippets/contactFormTemplate.html.twig
and is included on every boston.gov (Drupal) page.
The patterns library contact form javascript function start()
(in scripts/components/contact.js
) is executed when a boston.gov (Drupal) page loads.
The start()
function scans the completed page looking for email addresses anywhere in the html being served. Essentially, it:
- replaces the default mailto
directive for each email address with a click event listener which will trigger the handleEmailClick()
function, and
- attaches a click event listener to the forms' submit button and calls the handleFormSubmit()
function when the user clicks the submit button on the contact form.
TODO: The handleEmailClick()
function is called once when the page has finished loading.
It should be extended to also run when ajax events return data to the page (since email addresses could be served by XHR/AJAX as well as traditional document events)..
When an email address is clicked on the page, handleEmailClick()
- copies the template form from the script tags,
- inserts the correct email recipient to a hidden field,
- inserts all this onto the page and displays the contact us form, and
- in the background makes an ajax request to /rest/email_token/create,
generating and saving a unique "session" token in the form.
When the submit button is clicked, handleFormSubmit()
validates the form, and then submits, along with an authorization token to /rest/email_session/contactform
.
The PostmarkAPI.php
in the bos_email
module provides a "guaranteed delivery" type service. It tries to send the email via the postMark API, and if it fails for some reason, queues the email for later delivery in a Drupal queue.
Drupal will retry the email until it is able to send it to the postMark service.
Email tracking with Drupal has been discontinued with the use of the PostMark service. All logging can be obtained from the PostMark UI/Console.
Emails which fail to send can be viewed in the email_contactform queue.
This requirement could be obselete, and a requirement from earler versions of the form. Can consider removing this "feature" and reverting to having the sender be the email address provided by constituent. That way the cob employee/recipient can simply reply to the email.
Emails are sent from an email address that is generated for each email sent. The format of the email address is:
{random_string}@contactform.boston.gov
We don't send the email from the original sender's email address as that could be a vector for an email spoofing attack.
The reply_to header of the email is set to the constituents email first and the unique address of the contact form second. When someone replies to the email the to address of the email is set to the constituents email first and the unique address second. This delivers two copies of the email. One goes directly to the constituent, and one to the contact form API. We log the response time using the copy that gets sent to the contact form API. Once the reply email is delivered to the constituent, further replies will be direct between the constituent and city.
The contact us email which is sent from postMark to the cob recipient is a plain text email.
The PostmarkAPI is capable of generating HTML emails. A nicer experience for cob staff would be to receive an html email.
As well as the tokens etc, we could consider introducing an IP lock for say 60sec after a contact form submission is made (the timer should be managed on the server side, inside the endpoint). It should not affect genuine users but would minimize the impact (and success) of flood attacks and relay exploits on the endpoint. If there were a rapid second submission from the same IPAddress, we would flash a warning back to the user to try again after 60 seconds.
At the moment, there is no confirmation of a successful submission other than the on-screen notification. We have the submitters' email address, so we could send an email confirming the submission (see next enhancement).
There is little in the way of validation of the email the submitter provides as a contact for responses.
The JavaScript validation process does checks that a "visually" valid email pattern is entered, (i.e. it is an email pattern string) but does not validate the email address exists.
If we send email confirmations, then we could use that process to determine if the email address is active and does not have temporary errors (mailbox full etc). Since we are using AJAX it would be simple to end the confirmation email, wait a period of time (say 10sec) and then query postMark to see if the email was delivered. If it was then return success, if not then flag to the submitter on the form. This would not be 100% effective because some errors take time to be reported, and we cannot wait too long during validation. The other two approaches above should still be implemented to try to filter out malicious actions, and to detect innocent errors before consuming email server resources.
-- OUT OF DATE Feb 2023 --
Digital team is responsible for showing unofficial election results on Boston.gov.
On election night, an HTML file of partial results is generated during tabulation and copied to the cityofboston.gov’s web server. The contents of this file are inserted into the Boston.gov unofficial election results page via a client-side AJAX request.
Historically, election night traffic for these results has been more than cityofboston.gov was comfortably able to handle. We previously ran a job on Heroku that copied the page from cityofboston.gov and put it on an S3 bucket, and the Boston.gov page referenced that S3 version.
As of January 2019, we’ve modified both the source file and our Incapsula settings so that Incapsula caches all requests for the file for 60s. This let us turn off the Heroku job.
The Google reCAPTCHA v3 returns a score for each request without user friction. The score is based on interactions with your site and enables you to take an appropriate action for your site.
Install Google reCAPTCHA
Finally navigate to your site and set up form(s) that needs recaptchae added.
reCaptcha analytics
Environment variables
To update environment variables properly, make sure to update private repo with the current variable name and then add to Acquire. Only add Api key to Acquire.
Make sure you add the "recaptcha_v3.settings" in gitignore.
A drupal module that displays various pieces of information about the entered address.
There are two Drupal endpoints associated with this application.one for receiving the updates
NOTES: There are two cards that show data which are hard coded and don't sync with Civis. They are the mayor's name in the "YOUR MAYOR" card found in "mnl_config.js" file and the "YOUR AT-LARGE CITY COUNCILORS" card found in "Representation.js" file.
In addition to the hard coded items, there too is a data dependency on ReCollect. We built an endpoint in Drupal "rest/recollect" found in Drupal module "bos_mnl" to query ReCollect API with the user's inputted address and returns the next trash and recycling date for that specific address.
Jonathan Porter (Analytics division of DoIT) is the best point of contact for the Civis portion. Matt McGowan (Digital division of DoIT) is the best point of contact for the Drupal REST information.
Product
This app uses the bos_email
provided services as .
@see
(needs updating)
2022 this ticket () added a second email address box to try to prevent typos in the email address. While it helped a lot, it is not 100% effective and some emails from innocent errors and malicious actors (e.g. spammers) still slip through.
2023 provided some modules which can check for DNS validity, disposable emails and also email blacklists. This would further help reduce invalid sender email addresses being provided.
Production:
Source:
Make sure to register reCAPTCHA v3 keys (Secret and Site) .
Follow the installation and configuration instructions here to add reCAPTCHA to your Drupal site:
Url to view scores:
All variables are in Acquire:
The my neighborhood application is a Drupal component that can be added to any page on boston.gov. Currently, it is here:
One for receiving the updated records on a nightly basis (updates):
One for receiving the full load of records:
has information on the status of Drupal import scripts that run nightly and once a month.
The data used in this application come from a variety of GIS data sources. lists them all in addition to the workflow that brings each one into Civis.
These datasources are combined with the SAM address dataset in Civis. is the one that combines all the datasets.
Every night, runs and sends the any records that have updated or changed to boston.gov. Once a month on the 1st, the workflow sends the entire load of records to Drupal.
(historical and current)
Historical work that *potentially* isn't reflected above:
Metrolist allows Boston residents to search for affordable housing. The Search and AMI Estimator experiences are a JS WebApp. The rest of the app is built in Drupal (this module), with the underlying data layer provided by Salesforce.
Another feature of this module is allowing property managers to list and update their Metrolist listings without directly contacting DND.
View details of a MetroList Development listing
Submit a new Development listing to DND (Sales Force)
Provide data about Developments via JSON API to the Search and AMI Calculator JS WebApp
AMI data tables
Auto-create and update Developments from Sales Force on CRON (5 minutes)
Drupal
Sales Force
Google Maps API
Postmark
bos_metrolist (docroot/modules/custom/bos_content/modules/bos_metrolist)
Views
Webforms
Geolocation
Salesforce
MetroList Development (/admin/structure/types/manage/metrolist_development)
MetroList Unit (/admin/structure/types/manage/metrolist_unit)
Availability Status
City
Due at signing
Features (MetroList)
Income Eligibility AMI Threshold
MetroList: User Guide Type
Neighborhoods (MetroList)
Occupancy Type
Region
Rent Type
Unit Type
Utilities Included
This feature shows details about a development and its aggregated units. Any new data is pulled from Sales Force on every CRON run. The display for the development page is mainly controlled by the Available Units View that is included via the Development display.
Entry point: /metrolist/search (Use the MetroList Search JS WebApp to find a Development)
URL pattern: /metrolist/search/housing/{Development Name}
Custom CSS: docroot/modules/custom/bos_content/modules/bos_metrolist/css/views_ml_block_avaliable_units.css
Views Template: docroot/modules/custom/bos_content/modules/bos_metrolist/templates/views-view-metrolist-drawers.html.twig
Views Style: docroot/modules/custom/bos_content/modules/bos_metrolist/src/Plugin/views/style/MetroListDrawersStyle.php
Helper Functions (Pre-process and alters): docroot/modules/custom/bos_content/modules/bos_metrolist/bos_metrolist.module
Location Information
View - Available Units - Header
Back to Search
Map - settings
Development Name
Address
Area
Unit Type
Share Listing
Unit Information
View - Available Units - Fields
The Unit entities are aggregated based on type, price, AMI, bedrooms and bathrooms and then related to the proper Development.
Application and lottery information
View - Available Units - Footer
Application type and link
Agent Contact Information
Listing Type Information
Affordable Housing Lean More CTAs
A paragraph from the MetroList landing page that is being included from as paragraph library item
Feedback Form
A webform used to collect user feedback and send an email to DND contacts
This feature allows project managers throughout the state to submit new developments to DND for review and then eventual listing on MetroList. The user starts by navigating to the MetroList listing form landing page. On this page they then enter their email in the MetroList listing request form. When that webform is submitted a new MetroList Listing form draft is crated and an email is sent to the user with a token link (48 hour ttl). THe user then clicks the privet link and are taken to the Listing form. The listing form then communicates with Sales Force on each step to ether pre-fill information about the user and their properties or create new objects on Sales Force using the custom webform submit handler. Once the form is submitted the new objects will ber created and emails sent to the user and DND.
Webforms:
MetroList Listing Request Form
MetroList Listing Form
Webform Handler: docroot/modules/custom/bos_content/modules/bos_metrolist/src/Plugin/WebformHandler/CreateMetroListingWebformHandler.php
Helper Functions (Pre-process and alters): docroot/modules/custom/bos_content/modules/bos_metrolist/bos_metrolist.module
Sales Force Connector (Uses Sales Force module): docroot/modules/custom/bos_content/modules/bos_metrolist/src/MetroListSalesForceConnection.php
This feature used Views REST Export to create a simple JSON API for the MetroList JS WebApp. The application is able to make a request with a number of parameters and receive back MetroList development information and AMI information.
API Paths:
/metrolist/api/v1/developments?_format=json
/metrolist/api/v1/ami/hud/base?_format=json
Views:
/admin/structure/views/view/metrolist/edit/rest_export_nested_1
/admin/structure/views/view/metrolist_ami/edit/ml_ami_hud_base
Views Style:
docroot/modules/custom/bos_content/modules/bos_metrolist/src/Plugin/views/style/MetroListSerializer.php
Helper Functions (Pre-process and alters): docroot/modules/custom/bos_content/modules/bos_metrolist/bos_metrolist.module
This feature uses a View to creat a couple of data tables that are then used on income restricted housing guide. It uses the same base data as the AMI API from the Income Eligibility AMI Threshold taxonomy.
Entry point: /income-restricted-housing-guide
Views:
/admin/structure/views/view/metrolist_ami/edit/ami_table_hud
/admin/structure/views/view/metrolist_ami/edit/ami_table_bpda
Views Field/Validator:
docroot/modules/custom/bos_content/modules/bos_metrolist/src/Plugin/views/field/AMIThresholdTerms.php
docroot/modules/custom/bos_content/modules/bos_metrolist/src/Plugin/views/argument_validator/TermAMIThreshold.php
Views Template: docroot/modules/custom/bos_content/modules/bos_metrolist/templates/views-view-table--metrolist-ami.html.twig
Helper Functions (Pre-process and alters): docroot/modules/custom/bos_content/modules/bos_metrolist/bos_metrolist.module
This feature allows Drupal entities to sync back and forth with Sales Force Objects via the Drupal Sales Force module. It is primarily used by DND to use the data and access that is already on DND's Sales Force server to automatically sync with the Boston.gov Drupal site. This is controlled by field mapping configurations in the Drupal Sales Force module. Currently, all syncing is scheduled to happen on Drupal CRON run, every 5 minutes, with only updated objects.
Sales Force Mappings:
MetroList - Development (/admin/structure/salesforce/mappings/manage/metrolist_development/fields)
metrolist_development --> Development__c
MetroList - Unit (/admin/structure/salesforce/mappings/manage/metrolist_unit/fields)
metrolist_unit --> Development_Unit__c
Sales Force Settings:
MetroList - Development (/admin/structure/salesforce/mappings/manage/metrolist_development)
MetroList - Unit (/admin/structure/salesforce/mappings/manage/metrolist_unit)
Sales Force Mapping Field: docroot/modules/custom/bos_content/modules/bos_metrolist/src/Plugin/SalesforceMappingField/RelatedTermStrings.php
Troubleshoot Sales Force connection issues
If Drupal and Sales Force are not connecting or syncing please check the Authorization from Drupal to Sales Force (/admin/config/salesforce/authorize/list). You may need to Re-auth or even make a new connection if you need to connect to a lower development or testing environment on Sales Force. If you need access to an instance contact DND's Sales Force developer/administrator.
If a single item is not syncing or if you need info about the Drupal to Sales Force connection you can view the list this admin page. If you edit the instance you then have the option to force pull or push the Drupal entity with the Sales Force Object. If there is an issue you should see an error message in the response. You can also find other useful info like timestamps and record ids.
Metrolist allows Boston residents to search for affordable housing. The Search and AMI Estimator experiences are built in React (this repository). The rest of the app is built in Drupal, with the underlying data layer provided by Salesforce. The core UX is composed of the following:
Homepage Links to Search, AMI Estimator, and introductory information. Route: /metrolist/ Controlled by: Drupal Search Lists housing opportunities in a paginated fashion and allows user to filter according to various criteria. Route: /metrolist/search Controlled by: React APIs in use: Developments API AMI Estimator Takes user’s household income and household size, and calculates a recommendation for which housing opportunities to look at. URL: /metrolist/ami-estimator/ Sub-routes:
/metrolist/ami-estimator/household-income
/metrolist/ami-estimator/disclosure
/metrolist/ami-estimator/result
Controlled by: React APIs in use: AMI API Property Pages Route: /metrolist/search/housing/[property]?[parameters] Controlled by: Drupal Developments API Lists housing opportunities as a JSON object. URL: /metrolist/api/v1/developments?_format=json AMI API Lists income qualification brackets as a JSON object, taken from HUD (Department of Housing and Urban Development) data. URL: /metrolist/api/v1/ami/hud/base?_format=json
Prerequisites:
Node.js
Yarn or NPM (These docs use yarn
but it can be substituted for npm
if you prefer.)
Git
Read/write access to CityOfBoston
GitHub
⚠️ Warning: These docs were written for a standalone installation of the Metrolist React codebase, which outputs JavaScript files that can be committed to the Drupal monorepo separately. However, the React codebase has since been subsumed into the monorepo, rendering certain build instructions herein out-of-date. Please refer to the Boston.gov documentation for further instruction.
yarn start
runs:
ipconfig getifaddr en6
(or ipconfig getifaddr en0
if en6
isn’t found), which determines which LAN IP to bind to. This allows testing on mobile devices connected to the same network.
webpack-dev-server
. This compiles the ES6+ JavaScript and starts an HTTP server on port 8080 at the address found in the previous step.
Note: The ipconfig
command has only been tested on a Mac, and it also may not work if your connection isn’t located at en6
or en0
.
This runs webpack-dev-server
without launching a new browser window automatically.
There are Node.js scripts available under _scripts/
to aid development efforts.
Located at _scripts/component.js
, this facilitates CRUD-style operations on components.
This copies everything under _templates/components/Component
to src/components/Widget
and does a case-sensitive find-and-replace on the term “component”, replacing it with your new component’s name. For instance, this index.js
template:
…becomes this:
Subcomponents can also be added. These are useful if you want to encapsulate some functionality inside of a larger component, but this smaller component isn’t useful elsewhere in the app.
This creates the directory src/components/Widget/_WidgetGadget
containing this index.js
:
As you can see, the hierarchical relationship between Widget and Gadget is reflected in the naming. The React display name is WidgetGadget
, and the CSS class name uses a BEM element gadget
belonging to the widget
block, i.e. widget__gadget
.
This renames the directory and does a find-and-replace on its contents.
⚠️ Known issue: The component renaming algorithm does not fully find/replace on subcomponents.
Due to compatibility issues with Google Translate, the AMI API is not fetched live from the AMI Estimator. Instead, it is fetched at compile time using this script, which caches it as a local JSON file at src/components/AmiEstimator/ami-definitions.json
.
The domain from which this data is fetched can be specified with the following environment IDs:
www
or prod
→ https://www.boston.gov
Acquia environment
dev2
→ https://d8-dev2.boston.gov
etc.
The default value is ci
, as that should have the most recent data set in most cases.
Sets the version number for Metrolist in Drupal’s libraries.yml
file and this project’s package.json
file.
Prefer readability for other developers over less typing for yourself.
HTML/CSS:
JavaScript:
Consistent and readable JavaScript formatting is enforced by eslint-config-hughx
+ an ESLint auto-formatter of your choice, such as ESLint for VS Code.
Use Functional Programming principals as often as possible to aid maintainability and predictability. The basic idea is for every function to produce the same output for a given set of inputs regardless of when/where/how often they are called. This means a preference for functions taking their values from explicit parameters as opposed to reading variables from the surrounding scope. Additionally, a function should not produce side-effects by e.g. changing the value of a variable in the surrounding scope.
metrolist/
__mocks__/
: Mocked functions for unit/integration tests.
_scripts/
: CLI tools
_templates/
: Stubbed files for project scaffolding. Used by CLI tools.
coverage/
: Code coverage report. Auto-generated. (.gitignore
’d)
dist/
: Build output. Auto-generated. (.gitignore
’d)
public/
: Static files such as images, favicon, etc. These files are not used by Drupal, which uses its own tempalting; only in development. Thus, images have to be copied to the appropriate directory prior to deployment.
src/
: React source.
components/
: React components.
globals/
: SASS variables, mixins, etc. which are used cross-component.
util/
: Utility functions.
index.js
: React entrypoint.
index.scss
: App-wide styles. (Use sparinginly; prefer component-scoped.)
serviceWorker.js
: Service Worker code from Create React App; not currently used.
setupTests.js
: Jest configuration.
_redirects
: Netlify redirects.
.env
, .env.development
, .env.production
: Dotenv configuration (environment variables).
.eslintrc.js
: ESLint configuration.
.travis.yml
: Travis CI configuration.
babel.config.js
: Babel configuration.
DEVNOTES.md
: Notes taken during development.
package.json
: Project metadata/NPM dependencies.
postcss.config.js
: PostCSS configuration. Used to postprocess CSS output.
README.md
: Project documentation (this file).
webpack.config.js
, webpack.production.js
, webpack.staging.js
: Webpack configurations for different environments.
yarn.lock
/package-lock.json
: Yarn/NPM dependency lock file.
Every React component consists of the following structure:
Component/
__tests__
: Integration tests (optional)
Component.scss
: SASS styling
Component.test.js
: Unit test
index.js
: React component
methods.js
: Any methods that don’t need to go in the render function, for tidiness. (optional)
All classes namespaced as ml-
for Metrolist to avoid collisions with main Boston.gov site and/or third-party libraries.
Vanilla BEM (Block-Element-Modifier):
Blocks: Lowercase name (block
)
Elements: two underscores appended to block (block__element
)
Modifiers: two dashes appended to block or element (block--modifier
, block__element--modifier
).
When writing modifiers, ensure the base class is also present; modifiers should not mean anything on their own. This also gives modifiers higher specificity than regular classes, which helps ensure that they actually get applied.
An exception to this would be for mixin classes that are intended to be used broadly. For example, responsive utilities to show/hide elements at different breakpoints:
Don’t reflect the expected DOM structure in class names, as this expectation is likely to break as projects evolve. Only indicate which block owns the element. This allows components to be transposable and avoids extremely long class names.
Avoid parent selectors when constructing BEM classes. This allows the full selector to be searchable in IDEs. (Though there is a VS Code extension, CSS Navigation, that solves this problem, we can’t assume everyone will have it or VS Code installed.)
Always include parentheses when calling mixins, even if they have no arguments.
Don’t declare margins directly on components, only in wrappers.
Rucksack is installed to enable the same CSS helper functions that are used on Patterns, such as font-size: responsive 16px 24px
.
Currently this is used for previewing on Netlify, to get a live URL up without going through the lengthy Travis and Acquia build process.
This first runs a production Webpack build (referencing webpack.config.js
), then copies the result of that build to ../boston.gov-d8/docroot/modules/custom/bos_components/modules/bos_web_app/apps/metrolist/
, replacing whatever was there beforehand. This requires you to have the boston.gov-d8
repo checked out and up-to-date one directory up from the project root.
To make asset URLs work both locally and on Drupal, all references to /images/
get find-and-replaced to https://assets.boston.gov/icons/metrolist/
when building for production. Note that this requires assets to be uploaded to assets.boston.gov
first, by someone with appropriate access. If you want to look at a production build without uploading to assets.boston.gov
first, you can run a staging build instead.
This is identical to the production build, except Webpack replaces references to /images/
with /modules/custom/bos_components/modules/bos_web_app/apps/metrolist/images/
. This is where images normally wind up when running yarn copy:drupal
.
Aliases exist to avoid long pathnames, e.g. import '@components/Foo'
instead of import '../../../components/Foo'
. Any time an alias is added or removed, three configuration files have to be updated: webpack.config.js
for compilation, jest.config.js
for testing, and .eslintrc.js
for linting. Each one has a slightly different syntax but they all boil down to JSON key-value pairs of the form [alias] → [full path]. Here are the same aliases defined across all three configs:
webpack.config.js
:
jest.config.js
:
.eslintrc.js
:
All mailto:
links require the class hide-form
to be set, otherwise they will trigger the generic feedback form.
We’re using Jest + React Testing Library to ensure that future development doesn’t break existing functionality.
Every component should have its own unit test in the same directory. This is enforced by the Component test stub (_templates/components/Component/Component.test.js
), which contains the following:
So when running yarn component add
, you automatically generate a test that fails by default. You have to manually uncomment the call to render
(and ideally write more specific tests) in order to pass. This is designed to be annoying so it isn’t neglected.
When testing interactions between two or more components, or for utility functions (src/util
), put tests in a nested __tests__
directory.
One example of this is the Search
component, which contains a separate test file for every FiltersPanel
+ ResultsPanel
interaction,:
You have to run a browser without CORS restrictions enabled. For Chrome on macOS, you can add this to your ~/.bash_profile
, ~/.zshrc
, or equivalent for convenience:
This will prevent you from running your normal Chrome profile. To run both simultaneously, install an alternate Chrome such as Canary or Chromium. For Canary you would use this command instead:
Then in a terminal, just type chrome-insecure
and you will get a separate window with no security and no user profile attached. Sometimes Google changes the necessary commands to disable security, so check around online if this command doesn’t work for you. Unfortunately no extensions will be installed for this profile, and if you install any they will only exist for that session since your data directory is under /tmp/
.
We’re using React Router for routing, which provides a Link
component to use in place of a
. Link
uses history.pushState
under the hood, but this will fail inside the Google Translate iframe due to cross-domain security features in the browser. (For an in-depth technical explanation of why this happens, see DEVNOTES). So in order to make app navigation work again, we have to hack around the issue like so:
Change base.href
to the Google Translate iframe domain,
Perform the navigation,
Change base.href
back to boston.gov immediately afterward to make sure normal links and assets don’t break.
To do this automatically, there is a custom Metrolist Link
which wraps the React Router Link
and attaches a click handler with the workaround logic. So, anytime you want to use React Router’s Link
, you need to import and use @components/Link
instead. This is the technique used by the Search component to link to the different pages of results.
If instead you want to use React Router’s history.push
(or the browser-native history.pushState
) manually, you can import these helper functions individually:
This is the technique used by the AMI Estimator component to navigate between the different steps in the form.
Option
Description
-m
, --major
Sets the left version part, e.g. 2.x.x. If omitted, major will be taken from existing Metrolist version.
-n
, --minor
Sets the middle version part, e.g. x.5.x. If omitted, minor will be a hash of index.bundle.js for cache-busting.
-p
, --patch
Sets the right version part, e.g. x.x.3289. If omitted while minor is set, patch will be a hash of index.bundle.js for cache-busting. If omitted while minor is not set, patch will not be set.
-f
, --force
Allow downgrading of Metrolist version.
--help
This screen.