Skip to content

Instantly share code, notes, and snippets.

@sshleifer
Created October 4, 2021 18:18
Show Gist options
  • Save sshleifer/744f6fba88d76b139bee72aa8506079b to your computer and use it in GitHub Desktop.
Save sshleifer/744f6fba88d76b139bee72aa8506079b to your computer and use it in GitHub Desktop.
git fetch
git checkout paper-v2
export SD=/data/users/sshleifer/fairseq-py/roberta_azure
train_roberta_base () {
export NCCL_DEBUG="warn"
./fb_sweep/bmr.py -g 8 -t 1 -n 8 --dl 12 --embed-dim 768 \
--bs 32 --li 50 --epg 0 --mu 2000000 --ebs 2048 --arch prenorm \
--resume-failed --nw 0 -p bl \
--opt adam --local-checkpoints-dir $SD --checkpoints-dir $SD --use-fused-softmax \
--ddp fully_sharded "$@"
}
train_roberta_base_aws --arch prenorm --lr 1e-3 --wu 2000 --mu 500000 --scale-resids --scale-heads --lfc --lat -n 8 -p ln_sr
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment