Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update summary file layout and tests #4

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Update summary file layout and tests #4

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c515b52
Update summary file layout and tests
hancush Feb 22, 2023
6cde191
Add reporting unit name and total ballots cast to Race object
hancush Feb 23, 2023
44ed7e1
Load response as UTF-8
hancush Feb 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
*.egg-info
*.pyc
*.swp
.DS_Store
55 changes: 25 additions & 30 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,55 +3,48 @@ chi-elections

[![Build Status](https://travis-ci.org/datamade/python-chicago-elections.svg?branch=master)](https://travis-ci.org/datamade/python-chicago-elections)

chi-elections is a Python package for loading and parsing election results from the [Chicago Board of Elections](http://www.chicagoelections.com/).
chi-elections is a Python package for loading and parsing election results from the [Chicago Board of Elections](https://www.chicagoelections.gov/).

Summary Results
---------------

The Board of Elections provides election-night results at a racewide level. The file lives at

http://www.chicagoelections.com/results/ap/summary.txt
https://chicagoelections.gov/results/ap/

before election day, for testing.

It lives at

http://www.chicagoelections.com/ap/summary.txt
https://chicagoelections.gov/ap

on election night.

Per the Chicago Board of Elections, the results file will contain candidate and race names on election night and be kept updated until all votes are counted.

### Text layout

From http://www.chicagoelections.com/results/ap/text_layout.txt:
From https://chicagoelections.gov/results/ap/SummaryExportFormat.xls:

```
Summary Export File Format Length Column Position
Contest Code 4 1-4
Candidate Number 3 5-7
Num. Eligible Precincts 4 8-11
Votes 7 12-18
Num. Completed precincts 4 19-22
Party Abbreviation 3 23-25
Political Subdivision Abbrev 7 26-32
Contest name 56 33-88
Candidate Name 38 89-126
Political subdivision name 25 127-151
Vote For 3 152-154
Summary Export File Format Length Column Position
Record type 1 1
Global contest order 5 2-6
Global choice order 5 7-11
# Completed precincts 5 12-16
Votes 7 17-23
Contest Total registration 7 24-30
Contest Total ballots cast 7 31-37
Contest Name 70 38-107
Choice Name 50 108-157
Choice Party Name 50 158-207
Choice Party Abbreviation 3 208-210
District Type Name 50 211-260
District Type Global Order 5 261-265
# of Eligible Precincts 5 266-270
Vote For 2 271-272
```

### Gotchas

Prior to election night, the test file will include all fields. At some point on election night, the file will only contain the numeric values in the first 22 columns.

This means that you need to:

* Make sure you save candidate names in some way, like a database, before election night
* Make sure you store ballot order (Candidate Number in the text layout above) with the candidate. You'll need to use this, in combination with Contest Code, to look up the cached candidates.

At some point at the end of election night, the results file will no longer be available at http://www.chicagoelections.com/ap/summary.txt and will be available at http://www.chicagoelections.com/results/ap/summary.txt
. ~~However, it will not be updated. You'll need to scrape, enter or load results in some other way if you need updates after election night.~~ As of the April 2, 2019 municipal election, the Chicago Board of Elections says it will keep summary.txt updated until all votes are counted.


### Results client

To access the results:
Expand All @@ -78,15 +71,17 @@ client = SummaryClient(url='http://www.chicagoelections.com/results/ap/summary.t
Precinct Results
----------------

**N.b., The format of precinct results has changed and needs to be updated.**

After election night, precinct-level results are published to https://chicagoelections.com/en/election-results.html. The results are HTML files, so we have to scrape the results from HTML tables.

### Results client

To access the results:

```python
from chi_elections import elections

```python
muni_elections = [election for name, election in
elections().items() if 'municipal' in name.lower()]

Expand Down
88 changes: 56 additions & 32 deletions chi_elections/summary.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,15 +23,18 @@
from .constants import SUMMARY_URL
from .transforms import replace_single_quotes


class FixedWidthField(object):
def __init__(self, index, length, transform=None):
self.index = index
self.index = self._index(index)
self.length = length
self.transform = transform
self.name = None

def parse(self, s):
def _index(self, index):
return index

def parse(self, s):
val = s[self.index:self.index + self.length]
val = val.strip()
if self.transform is None:
Expand All @@ -43,6 +46,12 @@ def parse(self, s):
return None


class OneIndexedFixedWidthField(FixedWidthField):

def _index(self, index):
return index - 1


class FixedWidthParserMeta(type):
def __new__(cls, name, parents, dct):
dct['_fields'] = []
Expand All @@ -67,34 +76,46 @@ def parse_line(self, line):


class ResultParser(FixedWidthParser):
# Summary Export File Format Length Column Position
# Contest Code 4 1-4
# Candidate Number 3 5-7
# # of Eligible Precincts 4 8-11
# Votes 7 12-18
# # Completed precincts 4 19-22
# Party Abbreviation 3 23-25
# Political Subdivision Abbreviation 7 26-32
# Contest name 56 33-88
# Candidate Name 38 89-126
# Political subdivision name 25 127-151
# Vote For 3 152-154
contest_code = FixedWidthField(0, 4, transform=int)
candidate_number = FixedWidthField(4, 3, transform=int)
precincts_total = FixedWidthField(7, 4, transform=int)
vote_total = FixedWidthField(11, 7, transform=int)
precincts_reporting = FixedWidthField(18, 4, transform=int)
party = FixedWidthField(22, 3)
reporting_unit_name = FixedWidthField(25, 7)
race_name = FixedWidthField(32, 56)
candidate_name = FixedWidthField(88, 38, transform=replace_single_quotes)
reporting_unit_name = FixedWidthField(126, 25)
vote_for = FixedWidthField(151, 3, transform=int)
"""
Summary Export File Format Length Column Position

Record type 1 1
Global contest order 5 2-6
Global choice order 5 7-11
# Completed precincts 5 12-16
Votes 7 17-23
Contest Total registration 7 24-30
Contest Total ballots cast 7 31-37
Contest Name 70 38-107
Choice Name 50 108-157
Choice Party Name 50 158-207
Choice Party Abbreviation 3 208-210
District Type Name 50 211-260
District Type Global Order 5 261-265
# of Eligible Precincts 5 266-270
Vote For 2 271-272

Source: https://chicagoelections.gov/results/ap/SummaryExportFormat.xls
"""
record_type = OneIndexedFixedWidthField(1, 1, transform=int)
contest_code = OneIndexedFixedWidthField(2, 5, transform=int)
candidate_number = OneIndexedFixedWidthField(7, 5, transform=int)
precincts_reporting = OneIndexedFixedWidthField(12, 5, transform=int)
vote_total = OneIndexedFixedWidthField(17, 7, transform=int)
race_total_registration = OneIndexedFixedWidthField(24, 7, transform=int)
race_total_ballots_cast = OneIndexedFixedWidthField(31, 7, transform=int)
race_name = OneIndexedFixedWidthField(38, 70)
candidate_name = OneIndexedFixedWidthField(108, 50, transform=replace_single_quotes)
party = OneIndexedFixedWidthField(158, 50)
party_abbreviation = OneIndexedFixedWidthField(208, 3)
reporting_unit_name = OneIndexedFixedWidthField(211, 50)
reporting_unit_code = OneIndexedFixedWidthField(261, 5, transform=int)
precincts_total = OneIndexedFixedWidthField(266, 5, transform=int)
vote_for = OneIndexedFixedWidthField(271, 2, transform=int)


class Result(object):
def __init__(self, candidate_number, full_name, party, race, vote_total,
reporting_unit_name):
def __init__(self, candidate_number, full_name, party, race, vote_total):
self.candidate_number = candidate_number
self.full_name = full_name
self.party = party
Expand All @@ -114,10 +135,12 @@ def serialize(self):


class Race(object):
def __init__(self, contest_code, name, precincts_total=0,
precincts_reporting=0, vote_for=1):
def __init__(self, contest_code, name, reporting_unit_name, total_ballots_cast,
precincts_total=0, precincts_reporting=0, vote_for=1):
self.contest_code = contest_code
self.name = name
self.reporting_unit_name = reporting_unit_name
self.total_ballots_cast = total_ballots_cast
self.candidates = []
self.precincts_total = precincts_total
self.precincts_reporting = precincts_reporting
Expand Down Expand Up @@ -153,7 +176,6 @@ def parse(self, s):
party=parsed['party'],
race=race,
full_name=parsed['candidate_name'],
reporting_unit_name=parsed['reporting_unit_name'],
)
race.candidates.append(result)

Expand All @@ -164,6 +186,8 @@ def get_or_create_race(self, attrs):
race = Race(
contest_code=attrs['contest_code'],
name=attrs['race_name'],
reporting_unit_name=attrs['reporting_unit_name'],
total_ballots_cast=attrs['race_total_ballots_cast'],
precincts_total=attrs['precincts_total'],
precincts_reporting=attrs['precincts_reporting'],
vote_for=attrs['vote_for'],
Expand All @@ -189,8 +213,8 @@ def get_url(self):

def fetch(self):
url = self.get_url()
r = requests.get(url)
self._parser.parse(r.text)
response = requests.get(url)
self._parser.parse(response.content.decode("utf-8"))

@property
def races(self):
Expand Down
Loading