Skip to content

Instantly share code, notes, and snippets.

@tfayyaz

tfayyaz/dataproc-history-workflow-create-cluster.sh

Last active April 6, 2020 16:29
Show Gist options
  • Save tfayyaz/ed74bedb61d6c445cac89de0778dea19 to your computer and use it in GitHub Desktop.
Save tfayyaz/ed74bedb61d6c445cac89de0778dea19 to your computer and use it in GitHub Desktop.
Download ZIP
Raw
dataproc-history-workflow-create-cluster.sh
gcloud beta dataproc workflow-templates set-managed-cluster \
$TEMPLATE_ID \
--region $REGION \
--zone $ZONE \
--cluster-name $CLUSTER \
--master-machine-type n1-standard-4 \
--master-boot-disk-size 500 \
--worker-machine-type n1-standard-4 \
--worker-boot-disk-size 500 \
--num-workers 20 \
--image-version 1.5 \
--scopes=cloud-platform \
--bucket=${BUCKET_NAME} \
--properties=dataproc:job.history.to-gcs.enabled=true,spark:spark.eventLog.dir=gs://${BUCKET_NAME}/history/spark/eventlog,spark:spark.history.fs.logDirectory=gs://${BUCKET_NAME}/history/spark/eventlog
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment