- Repository Purpose and Scope
- Requirements and Setup
- Sources of Data
- Uses of Data
- HMDA Platform Census Files
- Provide an ETL for geographic and Census data used by the HMDA Platform
- Check to ensure the accuracy of Census data in the HMDA Platform
Install Requirements The code is built in Python3.X which can be found at the link below. The following packages are also required and can be installed using the commands listed.
- Python 3.6 or greater
- set up a virtual environment if desired:
virtualenv venv- turn on virtual environment:
source venv/bin/activate - turn off virtual environment:
deactivate
- turn on virtual environment:
- install requirements packages:
pip install -r requirements.txt - Note: to load data files to a database, you must have one installed locally. This code has been tested with PostgreSQL
- Update the
python/census_config.yamlto include the relevant years census file in msa_md_delinations section. - Update the
yearvariable in thepython/create_ffiec_census_file.pyfile to be the year for which you want to generate the platform census file. - Run the
python/create_ffiec_census_file.pyfile. - The file will be created in
output/asffiec_census_msamd_names_<year>.txt - Move the file in the HMDA-Platform repo as
common/src/main/resources/ffiec_census_<year>.txt
Configuration: Determines which years of data to use, allows selection of fields in both data files, and contains data specifications and URLs relevant.
The configuration is used in the census_functions.py class. The test.py script contains examples that use the class to download, cut, merge, and load to database the resulting census data.
- MSA to tract mapping verification needs to be updated for the new codebase
- MSA delineation files pre-2000 are in a different format that is yet to be parsed
The HMDA Platform uses data the combines elements of the FFIEC Census Flat File and the OMB MSA delineation files. The FFIEC Census file contains over 1,000 data elements, of which the HMDA Platform uses a small subset. The OMB MSA bulletines are primarily used for names.
The Office of Management and Budget produces MSA data. Updates can include changes to an MSA's boundaries or creation of new MSAs. These data have no regular publication cycle. HMDA Operations uses the MSA definitions in effect on 12/31 of the year preceding collection, this aligns with other Regulation C criteria.
The Census delineation files are used to map names to MSA/MD geographies.
The FFIEC produces an annual Census Flat File containing demographic data and a mapping of MSA data to Census tract.
- FFIEC Census Flat Files by Year
- Note: the Census Flat File is indexed starting at 1
Additional Census data is available, but not used in this project: The Census reference files contain MSA/MD, micropolitan statistical area definitions, names, and maps to county and tract codes.
The HMDA Platform uses data during data submission and publication.
During submission Census data are used to verify the relationship between reported geographic identifiers for loans and applications.
In publication the Census demographic and geographic data are used to add demographic information to LAR datasets. The variables added include:
- Total Population
- Minority Population Percentage
- FFIEC Median Family Income
- Tract to MSA/MD Income Percentage
- Number of Owner Occupied Units
- Number of 1 to 4 Family Units
- MSA (new in 2018, was previously submitted by FIs)
Census geographic data are used to map MSAs to county and tract areas in the Aggregate and Disclosure reports and for geographic lookup features in HMDA data tools web interfaces.
See here for the HMDA-Platform logic mapping Census to LAR data.
HMDA Publication Products
- Aggregate Reports: contain MSA level data on application and lending activity for all institutions reporting HMDA data.
- Disclosure Reports: contain MSA level data on application and lending activity for a single institution.
- LAR snapshot publication: contains the entire dataset of loans and applications submitted in accordance with Regulation C.