Party Divisions

[jetilton]

I am beginner/intermediate python user and have never gone through a pull request, so please correct me if needed. I didn't see any data with regards to the party divisions of the House or Senate. I wanted that data and thought I would try and add to this ecosystem if it is wanted.

This is the party divisions from the house with leadership roles by Congress. Two files have been added, a scraper for all congress links found http://history.house.gov/Institution/Party-Divisions/Party-Divisions/
and the resultant data in a yaml file.

[JoshData]

Hi there. Thanks for your interest in improving this repository!

There's a lot of information in this pull request so there's a lot to comment on.

• It might be best to split up the different sorts of data into multiple files: party divisions (+ total membership), party leadership (i.e. leadership positions held by members of the House), and officers (i.e. non-members like clerk and so on).
• Party leadership might be best not in a separate file but instead merged into the leadership_roles field in the legislator files. This would require mapping the names to the existing records for legislators, and then mapping the Congress numbers to the right entry in the legislator's terms. Wherever the data goes, mapping the names of legislators to their IDs would be important and perhaps a requirement before merging this in. Because this is the hardest thing to do, it might be useful to open a pull request that only adds the party division data and leaves the rest for a separate PR.
• The Congress numbers (e.g. 100th Congress (1987–1989)) should be turned into integers (100) and the Congresses should be in forward chronological order (to match the order of the legislator files).
• The party divisions and total membership values should be integers instead of strings.
• There are some white space issues ('Democrats ': '188').
• Instead of naming the file party-divisions-house.yaml, it might be better to have a single file that will eventually hold information for both the House and Senate.
• The bottom of the table on house.gov has some footnotes so we'd want to make sure that information is captured somewhere.
• Before merging, some documentation should be added to the README.

[jetilton]

Understood, thanks!

[jetilton]

How are you envisioning a single file for house and senate?
My preference is to keep it {house:{{1:data},{2:data},{3:data}}, senate:{{1:data},{2:data},{3:data}}}
That way I can have a house scraper and senate scraper since they are in different locations and combine them in a single dictionary afterwards and do one yaml.dump . It seems easier to deal with if one breaks and makes for smaller scripts.

Thoughts?

[JoshData]

It would be a lot clearer from a user's point of view to do it by Congress and then by chamber within each Congress, but it's not hugely important.

[jetilton]

Hi Josh, I made the majority of the revisions you requested, but I have not tried to merge the party leadership.

Some things I would appreciate you taking a look at are:

• congress 107, the senate has some funny dates going on. It isn't very clean right now, but I want to know how you would like it before I try cleaning it up.
• In some of the congresses it lists which major party the third party caucused with. You can see how I handled this in congress 92 and others. Is this acceptable?

Thanks,
Jeff

[JoshData]

Hey, I'm sorry I dropped off, I just don't have the time right now to help on this. :(

[konklone]

I'll try and take a look at this this weekend.