Created
February 22, 2020 13:41
-
-
Save aditya-malte/2d4f896f471be9c38eb4d723a710768b to your computer and use it in GitHub Desktop.
smallBERTa_Pretraining.ipynb
Hi,
That’s great to hear! Also, thanks a lot for making an updated training script. I’ve been busy lately (earlier with work and now my Master’s), so your updated script is much appreciated.
I had to update step #26 from tokenizer.save to tokenizer.save_model. FYI
tokenizer.save_model("/content/models/smallBERTa", "smallBERTa")
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I ran into issues while following the directions from the 2020 blog post https://huggingface.co/blog/how-to-train. This gist was more helpful. Thank you 👍
For anyone interested in running through training with an updated transformers: I have a write-up here of a complete example on training from scratch using
transformers 4.10and the updatedrun_language_modeling.pyscript (https://github.com/huggingface/transformers/blob/4a872caef4e70595202c64687a074f99772d8e92/examples/legacy/run_language_modeling.py) committed on Jun 25, 2021.https://github.com/sv-v5/train-roberta-ua
Python package verisions are locked with pipenv so the example remains reproducible. Tested on Linux and Windows on GPU and CPU.
Happy training