Validation against the Survey of Personal Incomes#

To ensure accuracy and precision, PolicyEngine’s free and open-source simulation and microsimulation model is periodically validated against existing external datasets. This process involves taking external datasets meant to represent a particular slice of British society and loading their inputs, such as total individual employment income, into our data model. We then compare our calculated outputs with theirs in an effort to find discrepancies between the two. In this post, we’ll describe our efforts to validate our model against the newly released Survey of Personal Incomes (SPI) 2020/21 dataset.

The UK Survey of Personal Incomes#

The Survey of Personal Incomes (SPI) is a comprehensive dataset produced by HM Revenue and Customs (HMRC). It provides detailed information on individuals’ income, tax, and other financial data based on a sample of HMRC records. The SPI is among the most comprehensive UK individual-level financial datasets, and is used by HMRC, HM Treasury, and individual Members of Parliament, among others.[1]

Validation results#

We chose to validate PolicyEngine against the SPI’s total individual income, using the SPI’s input variables to compare PolicyEngine UK’s calculated total income tax against the SPI’s reported amount.

To do so, we first remove the SPI’s “composite records,” which are used to represent smaller population groups as a collective. This group of records is only meant to be employed within society-wide modelling, and its outputs cannot be correctly computed using its inputs. This group of records contains 1,770 observations, representing around 0.2% of total records.

We then treat relevant variables from the SPI 2020/21 dataset[2] as input variables into the PolicyEngine UK model. Using this data, we run household-level simulations for each input and categorise the outputs logarithmically based on how close PolicyEngine’s calculated 2020/21 total income tax is to that of the SPI. We treat a result within £10 as a match due to differences in rounding. Our final match statistics are displayed in the table below.

Total error (£)

Percentage of all records

<=10

92.7%

<100

93.5%

<1,000

98.3%

>=1,000

1.7%

For 92.7% of individuals included within the SPI, PolicyEngine UK’s calculated total income tax is within £10 of the SPI’s calculated amount. We view this as a robust demonstration of PolicyEngine’s accuracy and precision.

Analysing model discrepancies#

For the remaining records where our calculations do not closely match the SPI, we believe the differences are attributable to two main factors:

  1. Differences in input information: The SPI sometimes uses additional input data that is not available to PolicyEngine, leading to differences in calculations.

  2. Structural differences: At times, the SPI model’s structure differs from that of PolicyEngine in a way that complicates further model validation.

Four relevant error clusters appeared among the records where the SPI and PolicyEngine calculate differing outputs:

  1. Cases where an individual appears to transfer a portion of their Personal Allowance to their spouse as part of the Marriage Allowance. This is because the SPI removes the transferred amount from the individual’s Personal Allowance, but does not indicate that the subsequent lower amount is driven by the Marriage Allowance. This may represent as much as 37,998 records (4.8% of total).

  2. Cases where an individual receives less than £100,000 in annual income and is non-resident for taxation purposes. In this situation, the individual receives £0 Personal Allowance, even though our model would project a full £12,500 allowance. The SPI uses a variable that is not publicly available to calculate this status. This represents 2,054 records (0.3% of total).

  3. Cases where both the SPI and PolicyEngine calculate the same amount of taxable savings income for an individual receiving any form of savings income classified by the SPI as “Other Investment Income” (e.g., interest on securities), but the SPI finds that they owe around £1,000 more in income tax than we do. This represents 1,276 records (0.2% of total).

  4. Cases where an individual near the Personal Allowance taper value who claims Gift Aid receives a full Personal Allowance of £12,500 within the SPI model. The SPI model’s Gift Aid variable signifies the amount an individual deducts due to Gift Aid donations, while PolicyEngine’s represents an individual’s total Gift Aid-eligible donations. We then calculate the individual’s deduction amount from this value. This error represents 738 records (0.1% of total).

Conclusion#

PolicyEngine’s UK model shows a high level of accordance with the SPI’s 2020/21 data, with almost 93% of records with the same inputs returning the same total income tax calculation. While some discrepancies exist, they are limited in scope and often attributable to either testability limitations within the SPI’s structure or to inputs that the SPI has not made publicly available. This validation exercise builds upon our previous validations, such as our Autumn 2023 model calibration. We welcome questions and comments on our findings or methodology - please feel free to get in touch.