Never violate the golden rule! Excess deaths and Simpsons Paradox

In a recent post, I showed how a Professor who was making a point about vaccination rates and excess deaths was using the wrong denominator. By dividing excess deaths by population, rather than by expected deaths, he was necessarily biasing the results. A state with a high death rate, would always look to have a higher % of deaths compared to population than a state with a lower death rate. Please read this post if this is not clear.

Riffing on El Gato's Latest

This past weekend I came out of my (mostly) retirement on Twitter to comment on some posts that seemingly showed a high correlation between excess death % and vaccination rates among the states. I pointed to the post above to to show that I found the correlation to be much stronger in Jun 2020-March 2021 (when vax was far less prevalent) compared to Jun 2021-March 2022 when vax rates were much higher.

It occurred to me this morning, however, that there is even more wrong with the way the data is being presented. I had casually thought that simply taking excess deaths and dividing by expected deaths was a fair way to present the excess death %, but I’m not sure that it is. I had forgotten my previously stated Golden Rule of COVID analysis…. you must attempt to consider age in all cases. Why is it not ok to simply take the excess deaths and divide by expected deaths without considering age? Doesn’t the expected deaths take into account the different age characteristics of different states? It does in a way, and probably works fine in “normal” years, but I will argue not during an event that increases deaths generally.

Example

Let’s create two fictional states.

• “Party Island” contains a population that likes to smoke, drink, eat whatever they want, and is not a fan of exercise. Divided into three age groups:

  • Old folks: Population 100, 10 people die in an average year

  • Middle aged folks: Population 100, 3 people die in an average year

  • Young folks: Population 100, 1 person dies in an average year

• “Triathlonville”: contains a population that never smokes, drinks only a single glass of red wine periodically, tracks their macro’s consistently, and is exercises religiously. As unhealthy living tends to catch up with you more as you age, old folks in “Triathlonville” die at only half the rate of Party Island, middle aged folks at 2/3 the rate and young people die at the same rate (young people deaths are sufficiently rare, regardless of overall health of population) . Divided into three age groups:

  • Old folks: Population 100, 5 people die in an average year

  • Middle aged folks: Population 100, 2 people die in an average year

  • Young folks: Population 100, 1 person dies in an average year

Now, a global pandemic comes around that almost exclusively causes serious illness in the elderly. Party Island experiences 20 deaths among old folks, while Triathlonville has 10 deaths among the old. Fortunately, this pandemic causes mostly mild symptoms among the middle aged and young, and therefore both states experience their normal number of deaths in the middle and young age groups.

Looking at the excess data by age group, we conclude that the pandemic affected both states equally. That is, both states had 100% excess deaths among their old folks, and zero excess deaths among the middle and young aged folks. However, if we looks at the data across the age groups, we would say that Party Island had (10 excess deaths/ 14 expected deaths)= 71% expected deaths, while Triathlonville had (5 excess deaths/8 expected deaths) = 62.5% expected deaths.

This is another example of a possible Simpson’s Paradox.

The technical detail here is that if the relative difference between states’ death rates differ by age (certainly plausible) and COVID disproportionately causes deaths ins certain age groups (I think all but certain), then aggregating all age groups together in calculating excess death as a percentage could be misleading.

I will look to have a follow up post where I break the excess death data into age groups and see how much, if any, this may be having on analysis of excess deaths when looking across all age groups.