Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Datainput #44

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Datainput #44

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1023f57
Update readme file
HyperionZhou Feb 21, 2019
b731048
can directly feed data to load_benchmark
HyperionZhou Feb 21, 2019
eaaf87b
Update README.md
Feb 21, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 9 additions & 4 deletions deepmoji/finetuning.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@
from attlayer import AttentionWeightedAverage


def load_benchmark(path, vocab, extend_with=0):
def load_benchmark(pathOrData, vocab, extend_with=0):
""" Loads the given benchmark dataset.

Tokenizes the texts using the provided vocabulary, extending it with
Expand All @@ -38,7 +38,9 @@ def load_benchmark(path, vocab, extend_with=0):
suggested batch_size.

# Arguments:
path: Path to the dataset to be loaded.
pathOrData: Path to the dataset to be loaded. If instead of being passed
a file path, the data (python dict) will be passed. Then will directly
use the data.
vocab: Vocabulary to be used for tokenizing texts.
extend_with: If > 0, the vocabulary will be extended with up to
extend_with tokens from the training set before tokenizing.
Expand All @@ -54,8 +56,11 @@ def load_benchmark(path, vocab, extend_with=0):
maxlen: Maximum length of an input.
"""
# Pre-processing dataset
with open(path) as dataset:
data = pickle.load(dataset)
if not isinstance(pathOrData, dict):
with open(pathOrData) as dataset:
data = pickle.load(dataset)
else:
data = pathOrData

# Decode data
try:
Expand Down