"Contagion and #BlackLivesMatter: A Network Analysis"

This is the Github repo where I provide the code I used for collecting data on and analyzing the network of Twitter users who tweeted #BlackLivesMatter in the week after George Floyd's death.

The Jupyter Notebooks

The heart of the analysis is a series of 4 Jupyter notebooks in the root of this repo. They are intended to be run in order, as some of the dataframes used in later notebooks are generated mid-analysis. The notebooks rely on functions in the utils/data_analysis/load_..py files, which convert the data stored in the .csv and .sqlite files produced during data collection into pandas dataframes.

1_group_geo_users_in_clusters.ipynb

Groups geotagged #BLM adopters according to salient geographic characteristics of their county subdivisions (racial makeup, class, rates of police violence, etc.). This notebook:

• Loads county subdivision boundaries and merges them with the American Community Survey (ACS) and Mapping Police Violence (MPV) datasets.
• Assigns each geotagged user to a county subdivision based on their computed location.
• Runs Lasso-regularized linear regressions (on per-capita #BLM adoption and mean friend count) to identify the most salient neighborhood characteristics.
• Uses K-Means clustering (with the "elbow" method to choose K) to group users along the six most salient characteristics.

This notebook is provided "first" because it connects geotagged users with their neighborhood characteristics, which is necessary for the succeeding analyses.

2_examine_population_characteristics.ipynb

A birds-eye view of the adopter population, comparing geotagged users to their peers. This notebook:

• Examines what percentage of recorded tweets are geotagged, and the kinds of places users geotag.
• Computes each user's "location" (their modal geotagged place, validated against their mean coordinates).
• Examines geotagging behavior by account age.
• Charts cumulative adoption over time — overall, among geotagged users, and among Minnesota users — as well as follower/following counts by adoption time.

3_examine_network_characteristics.ipynb

Examines the structure of the follower network, reproducing several methods from Fink et al. (2015), "Complex Contagions and the Diffusion of Popular Twitter Hashtags in Nigeria." This notebook:

• Converts the edges data (from friends.sqlite) into a GEPHI-friendly format for network visualization.
• Labels and characterizes the clusters produced in notebook 1, with 3D visualizations of their distributions.
• Examines adoptions by state and US region.
• Calculates within/cross-cluster tie ratios, rates of "instigators" (adopters with no prior network exposure) versus adopters with 2+ adopting peers, tie ratios among early adopters, and daily overlap.

4_simulations.ipynb

Compares the actual rate of #BlackLivesMatter adoption to models of simple contagion, complex contagion, and a broadcast (null) model I developed for my research — what we would have expected to happen in the absence of online social contagion. This notebook:

• Builds per-user activity rates and hourly exposure counts.
• Constructs random log-in schedules for simulated users.
• Runs grid-searches to fit the broadcast, simple contagion, and complex contagion models, and compares their error against observed adoption — over the full study period, the first 48 hours, and by cluster.

Trained parameters from these simulations are saved in ./sim_results.

A note on reproducibility: I use multiple non-deterministic algorithms, such as K-Means clustering, so the latest output of the notebooks may differ from the data cited in my research.

Reproduction Steps

Reproducing this work involves two phases: (1) collecting the data from Twitter, and (2) running the 4 Jupyter notebooks over the collected data.

An important caveat: since I collected this data, Twitter has announced it is deprecating the Academic API, so future data collection in this vein may be stifled — but this code is provided for transparency nonetheless.

Phase 1: Data Collection

Over several months Fall 2021 / Winter 2022, I downloaded the following data from Twitter:

• All publicly available tweets containing the hashtag #BlackLivesMatter or #BLM from May 25th @ 12am ET to June 2nd 8pm, 2020.
• For each public hashtag adopter, I collected:
  • Account age
  • number of followers and friends (users they follow)
• For each adopter with at least 1 geotagged #BLM tweet during study period, I collected:
  • All geotagged tweets sent by user during study period
  • List of friends and followers as of Winter 2022
  • Tweet rate per hour from May 2nd to June 2nd

I initially wrote this code to be somewhat extensible, such that it could be easily modified to conduct data collection for a myriad of Twitter network research projects.

Setup

The download scripts assume Python 3.6+.

Run the following terminal commands:

pip3 install pynput
pip3 install requests
pip3 install requests-oauthlib
pip install sqlitedict
conda install -c conda-forge pygeos

Create a .env file in the data-collection directory to include your log-in information: TWIT_BEARER_TOKEN="{Your Token here.}"

Steps

Each make command below imports and runs a python script in utils/data_collection — by running these scripts from the root directory, we can ensure they can access all the utils/common resources.

1. Download tweets using the command make tweets (calls download_tweets.py).

• Twitter breaks down large query results into a paginated list of results; this code repeatedly queries the results, stepping through each page, until data collection is complete. This code was designed to be run on a personal computer; for big datasets, you may not be able to fetch all in one sitting. By typing any key and then enter, the program will save its progress safely and can be restarted to continue at the current "page" of tweets.
• If the program CRASHES during download, it will attempt to save your current data, but check the last page of tweets to be sure.

2. Download user information about those tweeters using make users (calls download_users.py).

• download_users.py downloads 100 users at a time, so each 100 "runs" (API calls) = 100,000 user objects populated.
• If you have a large number of users, I suggest feeding a pre-processed version of the tweets CSV with only unique users (i.e. if a user posts multiple times, only include one of their tweets).

3. Download additional information using make prior-adopters|geos|friends|activity

• To retrieve just geotagged users, I used a grep command to filter for #BLM tweets which contain a geotag.
• I only retrieved detailed information about users who had a geotagged #BLM tweet to save time. These scripts, however, could be run on the whole population with enough time. Additionally, since make geos fetches geotags for all of a user's tweets, not just their #BLM tweets, the pool of geotagged users can also be further expanded before conducting additional analysis. I ran out of time to do this for this project.

For reference, the full set of make commands:

make tweets
make users

# download metadata about users
make geos
make prior-adopters
make friends
make activity

Phase 2: Data Analysis

Setup

Truthfully, I cannot recall which packages I had to install for this, given that the installation of conda downloads packages such as sklearn by default. I think you at least need to install pyarrow:

conda install -c conda-forge pyarrow

You will also need several external datasets:

• Mapping Police Violence
• American Community Survey
• Social Explorer's 2019 Census-defined County Subdivisions Excluded from this repo as it is proprietary data. You should be able to find equivalent data on Census website for free; this copy is just formatted to be easy to work with.

Finally, I estimated the number of news articles referencing #BlackLivesMatter by day using Nexis Uni

Steps

Run the 4 Jupyter notebooks in order (see The Jupyter Notebooks above for what each one does):

1. 1_group_geo_users_in_clusters.ipynb
2. 2_examine_population_characteristics.ipynb
3. 3_examine_network_characteristics.ipynb
4. 4_simulations.ipynb

Later notebooks depend on dataframes generated (and saved as feather files) mid-analysis in earlier notebooks, so skipping ahead will fail.

File Structure

.                     - root contains Makefile, Jupyter notebooks
./gephi_data          - network visualization data, excluded due to PII
./sim_results         - saving trained params for contagion models
./utils
  /common             - helper functions and User, Place object definitions
  /data_analysis      - scripts for loading dataframes and datasets
  /data_collection    - scripts for retrieving data from Twitter

Implementation Notes

Data Collection

make tweets saves the list of tweets to a CSV, and stores the pagination token (how far in results we've saved) to the .env file.

I originally downloaded information about users using SqliteDict, which stores a simple dictionary userID->userObject in an SQL table. The structure of the User and Place (Twitter-defined location) objects is defined in utils/common/user.py

Data Analysis

In the process of data analysis, I discovered that SqliteDict was too slow to easily handle the amount of data I was working with. I shifted towards using pandas for most all data analysis, saving offline copies of my dataframes using pyarrow.feather (.ftr is an efficient file format for storing dataframes).

Attributions

The twitter_api_connect.py script's connect_to_endpoint() and bearer_oauth() functions were copied from Twitter's example code.