Social Distancing Scoreboard is Utterly Flawed, Still Gets Headlines

Several news organizations, both in Oregon and nationally, have reported on a Social Distancing Scoreboard offered by the website Uncast – the resultant headlines have been grabby, but also inaccurate, given the Uncast’s methodology. 

Sources around Oregon touted the site’s findings with headlines pegging Oregon in the top five worst states for social distancing – which makes for grabby clickbait, but it also denies the widespread efforts individuals are making throughout the state. It is also egregiously flawed journalism just as the public most needs reliable information.  

These numbers are in flux: Oregon is not in the top five on the Scoreboard at the time of writing this. It is somewhat unclear if at the time of the news outlets publishing articles about this topic if Unacast’s had Oregon in the bottom five. Of all the news outlets that reported on this topic not one has examined the veracity of Unacast’s findings, or even issued corrections to reflect the potential change in data.  

However, news outlets all over country have reported on this without any question surrounding the validity of the data being presented.  

What is Uncast: Amid the social distancing measures that have been enacted to flatten the curve of COVID-19 cases in the United States a company called Unacast has created a pro bono COVID-19 Toolkit. This toolkit is a scorecard for social distancing state to state, the grades for this are based on the changes in distance travelled pre and post COVID-19 outbreaks, and according to Unacast  “(They) juxtaposed the Social Distancing score with the number of reported cases, sourced from the Corona Data Scraper, to show correlation with changes in behavior over time.”    

The company was founded by Thomas Walle and Kjartan Slette after creating a music streaming service known as TIDAL. After it was acquired by Jay-Z in 2014 they focused on providing insights to better understand where users went to concerts in order to create better playlists and recommendations. They set out to build what they call the Real World Graph to provide these insights on how people move around the world to a variety of companies and industries. Unacast claims to cover over 5 million Points of interest, over 127 million active monthly users and 4,500 brands using publicly available maps.    

The Social Distancing Scoreboard: The scoreboard gives each state a grade A-F based on changes in average distances travelled, which seems to be the only factor that truly determines a state’s final grade. Oregon received a C from Unacast with a 29% drop in average travel distance, just missing the cutoff for a B. The bottom 5 states include New Mexico with 23% drop giving them a C, followed be Idaho, Hawaii, Montana with D’s, and lastly Wyoming received an F showing a 6% increase according to Unacast. For additional reference Louisiana, which has cases spreading faster than anywhere else in the world, received a B showing a 39% drop in distance travelled. While the state of New York has received an A, with a reported 49% decrease, just missing out on a top 5 spot. Florida has also received an A grade. 

Flaws in Methodology:  According to Unacast they are “Using the change in distance traveled from pre-COVID-19 days as a proxy, we determined a “Social Distancing” score for each county.” Implying that their data is solely based on changes in travel distances and then “juxtaposed the Social Distancing score with the number of reported cases.” The CEO of Unacast goes into more detail about their methodology in a post on the company site detailing and acknowledging the limitations of the scoreboard. In essence the CEO of Unacast is stating that they have had difficulties with the accuracy of their data points because, according to Walle, “The models, which we had built and optimized towards a non-COVID-19 world, now need revisiting.”  

There is no strong indicator in this post that increases in COVID-19 cases and other factors such as when states enacted social distancing measures had any direct factor in Unacast’s state scores. The implication is that the data being presented in the scoreboard serves as a correlate to increase in cases around the nation to show the importance of social distancing measures. Moreover, the data compiled does not seem to factor in travel in rural vs urban areas, commute time variability in different regions – and the list of missing variables could go on.  

Relevance of Scorecard: The stated goal of this scoreboard is “to help public health experts, policy makers, academics, community leaders, and businesses in retail and real estate.” And “to help raise awareness of and reinforce the importance of social distancing. We also believe it will not only help make sense of what’s happening now but unearth trends that will help project scenarios in the short- and mid-term future.”  

As of today, the Social Distancing Scoreboard created by Unacast provides little utility and could easily be considered an erroneous misrepresentation into social distancing practices. Unacast has offered some level of acknowledgment of this stating “we are in the process of understanding the best way to add layers that capture more of the complexity of social distancing.”  

To Uncast’s credit, they have acknowledged some of their own shortcomings, which is why it so surprising that journalists throughout the country have not done the same.  

Finally, it is worth noting that among the more tawdry statistical models being tossed around for clickbait right about now, that there are plenty of well reputed organizations offering methodically excellent work about the Covid-19 outbreak. And, it is journalists that are supposed be the professional skeptics separating what is a worth a read, versus what is not.  

Analysis by Sam Schultz