Skip to content

Instantly share code, notes, and snippets.

@ResidentMario

ResidentMario/combine.py

Created October 13, 2020 18:04
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save ResidentMario/da73e4650cf5f6280ba6a31d998ed19a to your computer and use it in GitHub Desktop.
Save ResidentMario/da73e4650cf5f6280ba6a31d998ed19a to your computer and use it in GitHub Desktop.
Download ZIP
Raw
combine.py
if __name__ == "__main__":
from distributed import Client, LocalCluster
import dask.dataframe as df
import dask.array as da
cluster = LocalCluster()
client = Client(cluster)
matches = da.from_npy_stack("/spell/data/")
matches = df.from_array(matches)
# IMPORTANT: note that this repartition is optional, if you want a partitioned write
matches.repartition(npartitions=1).to_csv("predictions.csv")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment