Skip to content

Instantly share code, notes, and snippets.

@chssch

chssch/spac_pipe.py

Last active April 24, 2017 21:23
Show Gist options
  • Save chssch/25c4efb2a165607c88328b788ab1a85c to your computer and use it in GitHub Desktop.
Save chssch/25c4efb2a165607c88328b788ab1a85c to your computer and use it in GitHub Desktop.
Download ZIP
Raw
spac_pipe.py
import spacy
nlp = spacy.load('en', parser=None, entity=None)
def get_data():
for index, row in df.iterrows():
yield unicode(row['question1'])
yield unicode(row['question2'])
res = []
temp = None
j = 0
i = 0
pairs = []
for e in nlp.pipe(get_data(), n_threads=6, batch_size=10000):
if i % 2 == 1:
pairs.append((e, sim))
j += 1
else:
temp = e
i += 1
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment