To further expand its fair lending supervision and enforcement activities outside of the traditional mortgage area, the Consumer Financial Protection Bureau ("CFPB") has been using the Bayesian Improved Surname Geocoding ("BISG") proxy methodology since 2012 to infer the likelihood that a credit applicant is a member of one of six racial / ethnic groups. Since that time, a number of analyses have identified significant shortcomings in the BISG proxy's use for fair lending analysis - including my own comprehensive assessment of the BISG methodology's risks, limitations, and biases. However, despite these important shortfalls, the CFPB's race/ethnicity proxy methodology has remained essentially unchanged for the last decade -- until now?
Buried within the CFPB's Semi-Annual Report to Congress issued on April 6, 2022 is this interesting item under the headings "Research, Markets, and Regulations: Significant Initiatives: Upcoming research and data gathering"
Consumer Credit Panel and Alternative Data. The Office of Research issued a request for proposal to solicit bids for a nationally representative panel of deidentified credit record information from the national credit reporting agencies. In 2022, the CFPB is also updating its Consumer Credit Panel (CCP) data with race/ethnicity probabilities using the CFPB’s Bayesian Improved First Name Surname Geocoding (BIFSG) proxy methodology [emphasis added] and gender probabilities using the CFPB’s Gender proxy methodology.
Despite this statement, in my recent discussions with industry colleagues, there is a general consensus that the CFPB's use of the BIFSG proxy for fair lending supervision and enforcement has yet to be observed. Nevertheless, the CFPB's reference to BIFSG - even in a research context - raises the possibility that we may see this new proxy in future fair lending activity. If so, what do we know about its properties? And are there any important risks or limitations we should be evaluating now in anticipation of its potential adoption?
In this more extended blog post, I address these questions by comparing key performance metrics of the BIFSG and BISG proxies using the following publicly-available datasets:
• A randomly-selected sample of 1 million 2021 Florida voter records with self-reported race/ethnicity, gender, and birth date.[1]
• A randomly-selected sample of 1 million 2022 North Carolina voter records with self-reported race/ethnicity, gender, and birth date.[2]
• A commonly-used, publicly-available database containing the demographic distribution of 4,250 first names compiled from certain lenders' 2007 and 2010 HMDA data (the "Tzioumis first name database").[3]
I then used the CFPB's official methodology to calculate the BISG race/ethnicity proxy probabilities for these two samples. For those records for which a match was found in the Tzioumis first name database, I also calculated the corresponding BIFSG race/ethnicity probabilities consistent with the approach of OCC economist Ioan Voicu.[4] Using these two race/ethnicity proxy estimates, I then evaluated various dimensions of predictive accuracy - at the aggregate and individual levels - using the approach detailed in my 2021 BISG study.
Before presenting the results of this assessment, I note the following considerations and limitations:
• The Florida and North Carolina voter databases are, by definition, state-specific. Accordingly, as noted in my 2021 BISG study, proxy methodologies based on national-level demographic data will contain predictive biases to the extent that state-specific demographics vary from national-level demographics. However, given that our focus in the present analysis is to estimate the change in BISG predictive accuracy by incorporating first name demographics, these biases should net out in our results.
• Notwithstanding the above consideration, the state-specific BISG predictive accuracy results - on their own - do provide additional insight to the potential magnitudes of predictive biases when national-level race/ethnicity proxies - such as BISG - are applied to sub-national data samples for fair lending analysis - which would occur when analyzing lending patterns of regional or community banks, or lending patterns within specific localized markets (such as states, MSAs, or counties).
• Given that these results are limited to two states, they are not necessarily representative of the potential impacts that BIFSG adoption would have at a broader national level. Nevertheless, these results highlight areas of potential concern that should be further explored before applying the BIFSG proxy to high-stakes uses - such as fair lending disparity testing.
The following sections summarize the main insights from these analyses.
Insight 1: The Tzioumis First Name Demographic Database Has a Relatively High Overall Coverage Rate - But Excludes Disproportionate Shares of First Names Associated With: (1) Non-White, (2) Female, and (3) Younger Individuals
Using the Tzioumis first name database, I was able to match 87% of first names from both the NC and FL voter samples - indicating that BIFSG has the potential to improve race/ethnicity proxy probabilities on a large percentage of the general adult population. However, when I drilled down into individual race/ethnicity groups, I found that the 13% of records with unmatched first names fell disproportionately in the Non-White segments of the voter samples as illustrated in Figures 1A and 1B below.
In fact, as Table 1 below shows, 34-40% of API sample voters, 31-35% of Black sample voters, and 18-19% of Hispanic sample voters have unmatched first names versus only 6% of White sample voters - indicating that the potential benefits of BIFSG are disproportionately attenuated for Non-White individuals given current commonly-used first name demographic databases - particularly for APIs and Blacks.
With respect to gender, my analysis indicates that females are almost twice as likely as males to have unmatched first names (17% vs. 9%), while my analysis of age shows a dramatic difference in birth year distributions for individuals with matched vs. unmatched first names as shown below in Figures 2A and 2B:
As these charts clearly show, individuals with matched first names have much older birth years than those individuals with unmatched first names. In fact, the median birth year for those with matched first names is 1967-1969 while the median birth year for those with unmatched first names is 1981-1986 - about 14-17 years earlier.
Taken as a whole, these analyses of the Tzioumis first name database - as applied to recent NC and FL voter registration data - indicate some important limitations of the BIFSG methodology using current first name demographic data. Specifically, given that the Tzioumis database is compiled from Home Mortgage Disclosure Act ("HMDA") data, the pool of individuals on which the first name demographics are based is not representative of the broader U.S. adult population; rather, it largely reflects the inherent demographics of residential mortgage applicants who tend to be disproportionately older, white, and male individuals.[5] While the primary use of BIFSG is to evaluate lending disparities by race/ethnicity, the first name coverage biases for gender and age could become relevant for the analysis of certain products or certain demographic intersectionalities where age or gender are important (e.g., student loans, student credit cards, and lending disparities for Black females - as just some examples).
In the next sections, I explore the comparative predictive accuracy of BIFSG for race/ethnicity relative to the current BISG proxy approach.[6]
Insight 2: For Aggregate Predictive Accuracy, BIFSG Provides Only Incremental Improvement Over BISG For Sample Members With First Name Matches
To evaluate BIFSG's potential impact to the accuracy of aggregate race/ethnicity estimation, I compared the following two actual vs. predicted accuracy metrics for the subsets of individuals in each state voter sample for which I had a matched first name:
Actual vs. Predicted Race/Ethnicity Distributions (BISG vs. BIFSG)
Pearson Correlation Coefficients Between Race/Ethnicity Proxy Probabilities and Actual Race/Ethnicity Membership (BISG vs. BIFSG)
Actual vs. Predicted Race/Ethnicity Distributions (BISG vs. BIFSG)
Figures 3A and 3B below compare the aggregate actual vs. predicted race/ethnicity distributions for the two voter samples for which matched first names were available ("matched samples").[7] For matched samples in both states, we see that the addition of first name demographics leads to improvements in aggregate predictive accuracy for all race/ethnicity groups.
However, as detailed in Tables 2A and 2B below, while we observe fairly large reductions in relative prediction errors using BIFSG (in some cases, a nearly 60% reduction), the aggregate prediction improvements are more incremental from an absolute perspective (generally no more than one percentage point for Non-Whites).
Correlations of BISG/BIFSG Proxy Probabilities With Actual Race/Ethnicity
As an alternative measure of aggregate predictive accuracy for the matched voter samples, Table 3 below displays the Pearson correlation coefficients between each set of proxy probabilities and the samples' self-reported race/ethnicity designations.
As you can see, the addition of first name demographics yields a modest, but consistent, improvement to the correlations of the BIFSG proxy probabilities with the actual race/ethnicity of sample members.
Insight 3: BIFSG Does NOT Guarantee Improved Aggregate Predictive Accuracy on the Entire Sample
This result is both very interesting and counter-intuitive.
As the results in the prior section illustrate, BIFSG - when available based on matches to the first name demographic database - provides consistent incremental improvement to the aggregate predictive accuracy of actual race/ethnicity for all demographic groups. Accordingly, one would also expect a similar - but smaller - improvement to aggregate predictive accuracy at the overall sample level (i.e., when these records are combined with the BISG probabilities associated with sample members for whom first name matches were not available).
Indeed, this is what is observed with the full North Carolina voter sample as illustrated in Figure 4A below.
Here you can see that the use of BIFSG/BISG proxy probabilities on the full NC voter sample leads to consistent aggregate predictive accuracy improvements for all demographic groups. Table 4 below provides the underlying details showing that the full NC voter sample exhibits lower absolute aggregate prediction errors by between -0.2% and -1.6% for the various race/ethnicity groups - improvements that are at, or below, the corresponding levels observed for the matched NC voter sample in Table 2B.
For the full Florida voter sample, however, I obtained a different outcome as illustrated in Figure 4B below.
Here we see that unlike the full NC voter sample where the use of BIFSG proxy probabilities led to consistent improvements in aggregate predictive accuracy across all race/ethnicity groups, the full Florida voter sample shows improvement only for Hispanics and Whites. Counterintuitively, absolute prediction errors actually increase for APIs and Blacks.
What explains this?
Let's take a closer look at the aggregate prediction improvement for Blacks in the full North Carolina voter sample. Table 5 below disaggregates the actual and predicted Blacks by whether a first name match was found or not. What we see here is that the BISG proxy underpredicts the unmatched segment of the full NC Black voter sample by -15,263 and overpredicts the matched segment by +29,554 for a net overall BISG error of +14,291. This translates into a 1.7% BISG proxy error rate (14,291 / 853,840) as shown in Table 4.
The introduction of BIFSG proxy probabilities only impacts the Matched segment and leads - as expected - to a reduction in aggregate prediction error (+21,816 vs. +29,554). Given that the net BISG aggregate prediction error is positive, this improvement in the matched segment leads to an overall improvement in the net BIFSG aggregate prediction error as well (+6,552 vs. + 14,291). This translates into a smaller 0.8% BIFSG proxy error rate (6,552 / 853,840) as shown in Table 4.
However, when I perform the same analysis for the full Florida Black voter sample, we see something different - as illustrated in Table 6.
Here we see a similar pattern of aggregate underprediction for the unmatched Black segment and overprediction for the matched Black segment. However, in this case, the net aggregate prediction error is negative rather than positive - translating into a -0.4% BISG proxy error rate (-2,807 / 781,836) as shown in Table 4. Under these conditions, the introduction of BIFSG proxy probabilities still yields a reduction in aggregate prediction error for the matched segment (+4,090 vs. + 7,718), but the net aggregate prediction error actually increases (-6,435 vs. -2,807) due to the more dominant underprediction of the unmatched segment being offset by a smaller degree of overprediction from the matched segment. This yields a larger -0.8% BIFSG proxy error rate (-6,435 / 781,836) as shown in Table 4.
Overall, while the use of BIFSG proxy probabilities does appear to improve the aggregate predictive accuracy of race/ethnicity distributions of sample segments for which matched first name demographics are available, the combination of BISG & BIFSG proxy probabilities for the entire sample (i.e., for both matched and unmatched first name segments) may - in some cases - actually lead to less aggregate predictive accuracy as illustrated by the analysis of Florida Black and API voter samples.
Insight 4: BIFSG Has a Limited Impact on Individual Race/Ethnicity Prediction Accuracy Based on the BISG Max Classification Rule - With Only Incremental Improvements Exhibited Primarily For Blacks and APIs
I evaluate the individual predictive accuracy of the BISG/BIFSG race/ethnicity proxies using the framework described in my research study Modern Fair Lending Analysis: The Hidden Biases in BISG Proxy-Based Disparity Estimates. This framework - shown below in Figure 5 - focuses on two measures of individual predictive accuracy:
The percentage of actual group members that are predicted accurately by the proxy - referred to as "Recall" accuracy.
The percentage of proxy group members that are correct - referred to as "Precision" accuracy.
I briefly summarize this framework below for Black individuals. For further details, please see the study referenced above.
The first row of this table represents the Recall measure of individual predictive accuracy where we divide the True Positive ("TP") predictions by the total number of actual group members (Black, in this example). Along this dimension, the prediction error is driven by the False Negatives ("FNs") - that is, those actual members of the group that are incorrectly predicted by the proxy.
Alternatively, the first column of this table represents the Precision measure of individual predictive accuracy where we divide the amount of True Positive predictions by the total number of predicted group members (Black, in this example). Along this dimension, the prediction error is driven by the False Positives ("FPs") - that is, those proxy-based predicted group members that are incorrect.
Both Recall and Precision Accuracy measures are considered equally important in the fair lending context, as we care about: (1) how many actual group members our proxy is capturing, and (2) the fidelity of the proxy-based predicted group members. Accordingly, it is common to combine both accuracy measures into what is called an “F1 Score” which is a single blended “average” of the two accuracy measures.
Individual Predictive Accuracy: Matched Voter Samples
Table 7A below summarizes these individual predictive accuracy metrics for the matched Florida voter sample. Classification of individuals into a single race/ethnicity group is performed using the BISG/BIFSG Max classification rule in which group membership is determined by the highest proxy probability value.[8]
The table on the left summarizes the Precision accuracy, Recall accuracy, and F1 Scores based on: (1) BISG proxy probabilities (top 3 rows) and (2) BIFSG proxy probabilities (middle three rows). The incremental accuracy improvement of the BIFSG proxy is summarized in the bottom three rows. Additionally, the table on the right provides a more disaggregated view of individual predictive accuracy using the foundational components (TP, FN, and FP) described in Figure 5.
As these two tables show, the BIFSG proxy approach has relatively minor impacts on individual predictive accuracy for the matched White and Hispanic FL voters but somewhat larger (but still incremental) impacts for matched API and Black FL voters. In particular, I note:
• BIFSG improves API Precision accuracy by 0.07 points by reducing False Positives by -6.5% (from 36.0% to 29.5%); however, API Recall accuracy actually falls slightly due to an increase in False Negatives by 0.8% (from 43.0% to 43.7%) - thereby yielding only a slight change in the API F1 Score.
• BIFSG improves Black Precision accuracy by 0.03 points by reducing False Positives by -3.1% (from 30.1% to 27.0%). Black Recall accuracy is also improved by 0.05 points by reducing False Negatives by -5.2% (from 42.0% to 36.9%). Together, this yields a small 0.04 increase in the Black F1 Score.
• BIFSG has rather de minimis impacts on Precision and Recall accuracy for matched White and Hispanic FL voters - reducing FNs and FPs by only about 1% - thereby yielding relatively de minimis changes to their F1 Scores.
The results for the matched North Carolina voter sample are qualitatively similar - although matched NC Hispanic voters experience a somewhat larger improvement in False Positives (-4.0%).[9]
Individual Predictive Accuracy: Full Voter Samples
Table 7B shows the results for the full voter samples in which the BIFSG proxy is applied to those individuals with a match to the first name database, and the BISG proxy is applied to all other individuals.
Comparing Tables 7A and 7B reveals that, taking into account the 87% coverage rate of the Tzioumis first name database, the practical impacts of the BIFSG proxy are even more muted than those discussed in Table 7A. While, qualitatively, the pattern of individual accuracy improvements are the same, the magnitudes of such improvement are even lower.
Insight 5: BIFSG Offers More Moderate Improvement to the Individual Race/Ethnicity Prediction Accuracy Based on the BISG 80% Threshold Classification Rule - Even at the Full Sample Level
Individual Predictive Accuracy: Matched Voter Samples
Table 8A summarizes these individual predictive accuracy metrics for the matched North Carolina voter sample. Classification of individuals into a single race/ethnicity group is performed using the BISG 80% threshold classification rule in which group membership is determined by the proxy probability value that is at least 80%. If no proxy probability value is at least 80%, then the individual's predicted race/ethnicity is designated as "Unknown". The accuracy metrics below include these "Unknown" predictions as FNs to provide better comparability to the BISG Max classification results.
As these two tables show, the BIFSG proxy approach has larger impacts on individual predictive accuracy under the BISG 80% threshold classification rule for matched Black, Hispanic, and White NC voters. In particular, we note:
• BIFSG improves Black Recall accuracy by 0.11 points by reducing False Negatives by -11.3% (from 70.6% to 59.3%); however, Black Precision accuracy actually falls slightly due to an increase in False Positives by 0.7% (from 16.1% to 16.7%). Together, this yields a 0.11 increase in the Black F1 Score.
• BIFSG improves Hispanic Recall accuracy by 0.08 points by reducing False Negatives by -8.1% (from 48.9% to 40.8%). Hispanic Precision accuracy is also improved by 0.05 points by reducing False Positives by -5.4% (from 26.5% to 21.1%). Together, this yields a 0.07 increase in the Hispanic F1 Score.
• BIFSG improves White Recall accuracy by 0.07 points by reducing False Negatives by -7.1% (from 30.2% to 23.1%); however, White Precision accuracy actually falls very slightly due to an increase in False Positives by 0.3% (from 4.9% to 5.2%). Together, this yields a 0.04 increase in the White F1 Score.
• BIFSG has rather de minimis impacts on Precision and Recall accuracy for matched API NC voters - reducing FNs and FPs by only about 1.1-1.4%, and changing the API F1 Score by only 0.01.
The results for the matched Florida voter sample are qualitatively similar - although Recall accuracy improvements are consistently lower (due to smaller reductions in False Negatives) while Precision accuracy changes are small and mixed.[10]
Individual Predictive Accuracy: Full Voter Samples
Table 8B shows the results for the full voter samples in which the BIFSG proxy is applied to those individuals with a match to the first name database, and the BISG proxy is applied to all other individuals.
Comparing Tables 8A and 8B reveals that, taking into account the 87% coverage rate of the Tzioumis first name database, the practical impacts of the BIFSG proxy are more muted than those discussed in Table 8A - although, unlike the BISG/BIFSG Max classification results, the net impacts of BIFSG are more meaningful on the full BISG 80% threshold classification results.
Insight 6: The BIFSG Proxy Has Only Incremental Impacts to Disparate Treatment Disparity Levels - Whether the Proxy is Used in a Continuous Manner or via the Classification Approach.
For my final analysis, I use each state's full voter sample to simulate a 100 bps disparate treatment price disparity for actual Black, Hispanic, and API individuals, separately. Then, I use the BISG and BIFSG proxies to estimate these known disparity levels to identify: (1) potential biases in the proxy approach's estimates of ground-truth disparities, and (2) how the BIFSG proxy approach impacts these estimated biases. For further details on this analysis, I refer you to my 2021 BISG study.
Tables 9A and 9B below present the results of this analysis for the North Carolina and Florida voter samples, respectively.
Within each table, the first set of rows corresponds to the disparate treatment estimation results using the BISG proxy methodology, the second set of rows corresponds to the results using the BIFSG proxy methodology, and the third set of rows calculates the differences between the two sets of results. Within each set of rows, the first row corresponds to the scenario in which actual Black individuals have the 100 bps price disparity, the second row is the scenario where Hispanic individuals have the 100 bps price disparity, and the third row is where the API individuals have the 100 bps disparity.
The columns labeled "BISG Continuous"/"BIFSG Continuous" correspond to estimated disparity levels when the proxy probabilities are used in their continuous form, while the "BISG 80%"/"BIFSG 80%" columns correspond to estimated disparity levels using the 80% classification threshold, and the "BISG Max"/"BIFSG Max" columns correspond to estimated disparity levels using the BISG Max classification approach.
I note the following observations regarding these estimates:
• Unlike the national-level results in my 2021 BISG study, the estimated disparities under the "BISG Continuous"/"BIFSG Continuous" columns - particularly for the Hispanic and API scenarios - are not close to 100 bps. As noted in that study, this is reflective of the biases that arise when a proxy methodology based on national-level demographic data is applied to sub-national samples.
• The BISG/BIFSG Continuous disparity estimates indicate that BIFSG may cause further underprediction of ground-truth disparate treatment disparities for Black and Hispanic individuals, and smaller underprediction of such disparities for API individuals; however, the magnitudes are only 1-2 bps.
• For the two classification-based approaches, the use of BIFSG proxy probabilities generally reduces the underprediction of disparities by up to 6 bps - although in both state samples the underprediction of Black disparity estimates increases very slightly. Overall, though, the impact of BIFSG is fairly limited.
So what's the primary takeaways from these analyses?
The incorporation of first name demographics into the BISG proxy approach has the potential to offer what appears to be a small degree of aggregate and individual predictive accuracy improvement. However,
• A commonly-used first name demographic database based on HMDA filings excludes about 13% of first names (based on NC and FL voter samples) and disproportionately excludes the first names of Non-White, female, and younger individuals due to the non-representativeness of the HMDA data relative to the adult U.S. population. Accordingly, such limitations should be considered each time BIFSG is proposed to analyze a particular fair lending use case.
• The non-random nature of first name exclusions may result in the counter-intuitive decrease in BIFSG/BISG aggregate predictive accuracy for one or more groups.
• For disparate treatment disparity estimation, a BIFSG/BISG proxy approach may counter-intuitively expand underprediction biases in disparate treatment scenarios by relatively small amounts.
Accordingly, companies should ensure they are performing appropriate due diligence of the potential risks and limitations of a new race/ethnicity proxy approach - particularly based on the specific characteristics of its own customers. To the extent that the costs, benefits, risks, and limitations of first name demographics are favorable, on net, then further efforts are recommended to produce or procure a first name demographic database that has appropriate coverage rates, as well as an underlying aggregate demographic distribution that is sufficiently aligned to your customer base.
* * *
ENDNOTES:
[1] Source: https://www.dos.myflorida.com/elections/data-statistics/voter-registration-statistics/voter-extract-disk-request/
For the 1 million FL voter records, I was unable to geocode successfully 144,362 records using the U.S. Census Batch Geocoder system. Of those that were successfully geocoded, 981 records could not be matched to the 2010 U.S. Census SF1 demographic data and 72,821 records could not be matched to the 2010 U.S. Census Surname file. This yielded 781,836 records for the BISG analysis. With respect to the BIFSG analysis, I was unable to match 116,850 first names to the Tzioumis (2018) first name database.
For the 1 million NC voter records, I was unable to geocode successfully 102,005 records using the U.S. Census Batch Geocoder system. Of those that were successfully geocoded, 151 records could not be matched to the 2010 U.S. Census SF1 demographic data and 44,003 records could not be matched to the 2010 U.S. Census Surname file. This yielded 853,841 records for the BISG analysis. With respect to the BIFSG analysis, I was unable to match 115,367 first names to the Tzioumis (2018) first name database.
[3] See Tzioumis, Konstantinos (2018) Demographic aspects of first names, Scientific Data, 5:180025 [dx.doi.org/10.1038/sdata.2018.25].
[4] Voicu, Ioan. (2018). Using First Name Information to Improve Race and Ethnicity Classification. Statistics and Public Policy. 5. 10.1080/2330443X.2018.1427012. I note that the modification of the BISG probability formula to include first name demographic information is a straight-foward computation.
[5] I also note that Tzioumis excludes from his dataset individuals for whom race/ethnicity are not reported. To the extent that such individuals are not a random subset of the overall HMDA dataset, this methodological feature may lead to additional biases in the final first name demographics.
[6] In the remainder of this article, I focus solely on the use of BIFSG for overall race/ethnicity prediction. Deeper dives into the predictive accuracy at more disaggregated segments that reflect intersectionality with gender and/or age are outside my scope, but a valid extension for further research.
[7] For each race/ethnicity group, the predicted % is equal to the sum of that's group's proxy probabilities (across all sample members) divided by the total number of sample members.
[8] The BISG Max classification rule assigns an individual race/ethnicity proxy to every sample member. This differs from the threshold-based classification rules (presented further below in this section) in which individuals whose proxy probabilities do not meet or exceed the threshold are assigned an "Unknown" race/ethnicity designation.
[9] The following table presents the same summary of results for the Matched NC voter sample.
[10] The following table presents the same summary of results for the Matched FL voter sample.
© Pace Analytics Consulting LLC, 2023
Comments