Political Polarization by Policy Area

In one paper from my dissertation, I assess the effects of political polarization on the growth of interest groups in the U.S. since the 1970s. The most common measure of polarization is DW-NOMINATE which, based on roll call votes, is a measure of the ideological positions of Senators and Representatives in the House.

Because I am interested in the growth of interest groups that focus on a particular policy area (say, healthcare or the environment), I wanted to measure polarization at that level. To do so, I merged roll call data from Voteview with roll call votes broken down by policy area from the Comparative Agendas Project. I then measure how far apart the two major parties' voting behavior is based on roll call votes by policy area (see formula below).

# import libraries
import pandas as pd
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
### import data
# import Comparative Agendas Project Roll Call Data
pap_roll = pd.read_csv('../../../../Research/Dissertation/Data/Policy Agendas Project/US-Legislative-roll_call_votes_18.2.csv', low_memory=False)

# import Voteview Roll Call Data
vv_house_roll_call = pd.read_excel('../../../../Research/Dissertation/Data/hdemrep35_113.xlsx')
vv_senate_roll_call = pd.read_excel('../../../../Research/Dissertation/Data/sdemrep35_113.xlsx')

Polarization Measure

The formula for calculating how partisan each roll call vote is:

\[\left|\frac{R_{y}}{R_{y}+R_{n}}-\frac{D_{y}}{D_{y}+D_{n}}\right|\]

where ${R_{y}}$ and ${D_{y}}$ are the number of Republicans and Democrats, respectively, that vote yes on a bill and ${R_{n}}$ and ${D_{n}}$ are the number of Republicans and Democrats, respectively, that vote no on a bill.

# create polarization measure for each bill
vv_house_roll_call['house_partisan'] = np.abs((vv_house_roll_call['r_yeas'] / (vv_house_roll_call['r_yeas'] + vv_house_roll_call['r_nays'])) - (vv_house_roll_call['d_yeas'] / (vv_house_roll_call['d_yeas'] + vv_house_roll_call['d_nays'])))

vv_senate_roll_call['senate_partisan'] = np.abs((vv_senate_roll_call['r_yeas'] / (vv_senate_roll_call['r_yeas'] + vv_senate_roll_call['r_nays'])) - (vv_senate_roll_call['d_yeas'] / (vv_senate_roll_call['d_yeas'] + vv_senate_roll_call['d_nays'])))
# filter to only House votes from 1965-2015 in CAP
pap_roll_house = pap_roll[pap_roll['filter_House'] == 1]
pap_roll_house_1965_2015 = pap_roll_house[(pap_roll_house['year'] > 1964) & (pap_roll_house['year'] < 2016)]

# filter to only Senate votes from 1965-2015 in CAP
pap_roll_senate = pap_roll[pap_roll['filter_Senate'] == 1]
pap_roll_senate_1965_2015 = pap_roll_senate[(pap_roll_senate['year'] > 1964) & (pap_roll_senate['year'] < 2016)]

# filter Voteview data to same years as CAP years above
vv_house_roll_call_1965_2015 = vv_house_roll_call[(vv_house_roll_call['year'] > 1964) & (vv_house_roll_call['year'] < 2016)]
vv_senate_roll_call_1965_2015 = vv_senate_roll_call[(vv_senate_roll_call['year'] > 1964) & (vv_senate_roll_call['year'] < 2016)]

# print (vv_house_roll_call_1965_2015.shape)
# print (pap_roll_house_1965_2015.shape)

merged_house_roll = pd.merge(vv_house_roll_call_1965_2015, pap_roll_house_1965_2015, on=['year', 'cong', 'rc_count']).rename(columns={'year': 'Year', 'pap_majortopic' : 'MajorTopic', 'rc_count' : 'num_house_bills'})

merged_senate_roll = pd.merge(vv_senate_roll_call_1965_2015, pap_roll_senate_1965_2015, on=['year', 'cong', 'rc_count']).rename(columns={'year': 'Year', 'pap_majortopic' : 'MajorTopic', 'rc_count' : 'num_senate_bills'})

# print (merged_house_roll.shape)

print ('Data merge kept', str(round(merged_house_roll.shape[0] / vv_house_roll_call_1965_2015.shape[0] * 100, 2)), 'percent of total roll call House votes in Voteview data.')
print ('Data merge kept', str(round(merged_senate_roll.shape[0] / vv_senate_roll_call_1965_2015.shape[0] * 100, 2)), 'percent of total roll call Senate votes in Voteview data.')
Data merge kept 99.77 percent of total roll call House votes in Voteview data.
Data merge kept 98.05 percent of total roll call Senate votes in Voteview data.
# keep just Major Policy Domains from CAP
merged_house_roll = merged_house_roll[(merged_house_roll['MajorTopic'] > 0) & (merged_house_roll['MajorTopic'] < 22)]
merged_senate_roll = merged_senate_roll[(merged_senate_roll['MajorTopic'] > 0) & (merged_senate_roll['MajorTopic'] < 22)]
# rename Policy Domains codes
di = {1 : 'Macroeconomics', 
      2 : 'Civ. Rights, Minority Issues, & Civ. Lib.',
      3 : 'Health',
      4 : 'Agriculture',
      5 : 'Labor, Employment, and Immig.',
      6 : 'Education',
      7 : 'Environment',
      8 : 'Energy',
      9 : 'Immigration',
      10 : 'Transportation',
      12 : 'Law, Crime, and Family',
      13 : 'Social Welfare',
      14 : 'Community Dev. and Housing',
      15 : 'Banking, Finance, & Dom. Comm.',
      16 : 'Defense',
      17 : 'Space, Science, Tech., and Comms.',
      18 : 'Foreign Trade',
      19 : 'Int\'l Affairs and Foreign Aid',
      20 : 'Government Operations',
      21 : 'Public Lands and Water Mgmt'}

merged_house_roll = merged_house_roll.replace({'MajorTopic' : di})
merged_senate_roll = merged_senate_roll.replace({'MajorTopic' : di})

Ten Most Polarized Policy Areas (1965-2015)

# Get average polarization by policy area and sort
merged_house_roll.groupby('MajorTopic')['house_partisan'].mean().reset_index().sort_values(by='house_partisan', ascending=False).head(10)
MajorTopichouse_partisan
15Macroeconomics0.531904
13Labor, Employment, and Immig.0.496616
6Energy0.475584
3Community Dev. and Housing0.475444
9Government Operations0.465232
2Civ. Rights, Minority Issues, & Civ. Lib.0.447487
5Education0.434153
1Banking, Finance, & Dom. Comm.0.423594
7Environment0.418144
10Health0.411548

Reshape Data to Policy Area-Year

house_roll_by_year = merged_house_roll.groupby(['Year', 'MajorTopic']).agg({'house_partisan' : 'mean', 'num_house_bills' : 'size'}).reset_index()
senate_roll_by_year = merged_senate_roll.groupby(['Year', 'MajorTopic']).agg({'senate_partisan' : 'mean', 'num_senate_bills' : 'size'}).reset_index()
# merge House and Senate votes
# merge 'outer' to keep all
congress_roll_by_year = pd.merge(house_roll_by_year, senate_roll_by_year, on=['Year', 'MajorTopic'], how='outer')
# Get total number of bills by policy area-year
congress_roll_by_year['total_bills'] = congress_roll_by_year['num_house_bills'] + congress_roll_by_year['num_senate_bills']

Calculate Weighted Polarization Measure

# Get weighted average of House and Senate polarization by policy area-year
congress_roll_by_year['Weighted Polarization'] = ((congress_roll_by_year['house_partisan'] * congress_roll_by_year['num_house_bills']) \
                                                  + (congress_roll_by_year['senate_partisan'] * congress_roll_by_year['num_senate_bills'])) \
/ congress_roll_by_year['total_bills']
# summary stats for total bills column
congress_roll_by_year['total_bills'].describe()
count    940.000000
mean      47.595745
std       49.548122
min        2.000000
25%       18.000000
50%       32.000000
75%       57.000000
max      376.000000
Name: total_bills, dtype: float64

Ten Most Polarized Policy Area-Years

# sort by weighted polarization measure for policy area-years with at least 20 total bills
congress_roll_by_year[congress_roll_by_year['total_bills'] > 20].sort_values(by='Weighted Polarization', ascending=False).head(10)
YearMajorTopichouse_partisannum_house_billssenate_partisannum_senate_billstotal_billsWeighted Polarization
5701993Macroeconomics0.79196028.00.84134735.063.00.819397
9592013Energy0.76735083.00.6442312.085.00.764453
9602013Environment0.78138529.00.6892397.036.00.763468
6891999Macroeconomics0.69525521.00.80607627.048.00.757592
7672003Labor, Employment, and Immig.0.70024820.00.85936311.031.00.756708
9232011Health0.76038256.00.6616344.060.00.753799
7692003Macroeconomics0.69275547.00.81156340.087.00.747379
9262011Labor, Employment, and Immig.0.72205525.00.75469312.037.00.732640
6101995Macroeconomics0.75472074.00.69724567.0141.00.727409
9392012Energy0.72730979.00.71814110.089.00.726279

Polarization by Policy Area by Year

topic_plot = sns.FacetGrid(congress_roll_by_year, col="MajorTopic", col_wrap=3, height=2.5, aspect=1.5, margin_titles=False)
topic_plot = topic_plot.map(plt.plot, "Year", "Weighted Polarization")

png