DS4A Capstone Project Spotlight: Can Scraping Twitter Aid in Helping Local Governments Better Predict Covid Outbreaks?

Blessing Ogunyemi is a graduate of Johns Hopkins University where she studied Public Health. She chose to join DS4A Empowerment because she realized that it would be a great opportunity to learn and network with amazing individuals while diving deeper into the world of data science and improving her skills. Her goals post DS4A include learning more about Machine Learning in the health field as well as bias reduction in AI. She is also on a journey to find the perfect intersection where data science, health, and social justice collide.

Desi Warren II is a third year undergraduate student majoring in International Affairs with a concentration in Comparative Polticial, Economic, and Social Systems at The George Washington University. He chose to join DS4A to improve his data science skills and join a like-minded community. His goal post-program is to complete his undergraduate degree, travel the world with friends, and continue to work on his data science skills.

For the Twitter data we had to approximate a centroid and a radius for each county since there is currently no way to query tweets at the county level. Furthermore, we wrote a script to gather the tweets by county from February to November 2020 but initial benchmarks showed that it would take us 9 days to gather the tweets for all the counties. We were thinking of reducing our time range to speed up the collection of tweets but then we had an idea. We came up with the idea of splitting the dates for data collection into three and running three processes simultaneously each gathering the data for a separate slice of time. With this change to the script, our time for data collection was reduced from 9 to 3 days.

Data collection presented challenges. One was combining the different data
sources (economic data, unemployment rates, demographic & policy data) we had based on date and county into a single dataset. Once we had the data we weren’t sure of the best way to visualize it. We tried making plots by county but the graphs would end up too convoluted (NY state has 62 counties) and detract away from clear, easily identifiable patterns. To fix this issue, we used k-means clustering to group counties based on demographic and economic data. We ended up with 3 clusters (small counties, medium counties, NYC+Long Island) that grouped the counties primarily by population.

Our mentors were Peggy Sah, lead ML engineer at athenahealth, and David Rimshnick, Senior Director of Data Science at Aetna Analytics. They helped us focus the scope of our project and to take decisions such as the features we should use for k-means clustering.