Skip to content

Instantly share code, notes, and snippets.

@tuulos

tuulos/dump_data.py

Created May 12, 2023 18:56
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save tuulos/dd80913552f6cfe87166ace1faa672a9 to your computer and use it in GitHub Desktop.
Save tuulos/dd80913552f6cfe87166ace1faa672a9 to your computer and use it in GitHub Desktop.
Download ZIP
export Metaflow tasks in a CSV
Raw
dump_data.py
from metaflow import namespace, Metaflow, Run
​
def fmt(t):
return t.strftime('%Y-%m-%dT%H:%M:%SZ')
​
print('flow,run,step,task,created,finished,user,runtime,pod_id,pod_name')
namespace(None)
for flow in Metaflow():
for run in flow:
if run.successful:
for step in run:
for task in step:
created = fmt(task.created_at)
finished = fmt(task.finished_at)
meta = task.metadata_dict
user = meta.get('user', '')
runtime = meta['runtime']
pod_id = meta.get('kubernetes-pod-id', '')
pod_name = meta.get('kubernetes-pod-name', '')
fields = [
flow.id,
run.id,
step.id,
task.id,
created,
finished,
user,
runtime,
pod_id,
pod_name]
print(','.join(fields))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment