This is a fork of gpt2-ml, gpt2-ml is a wonderful project which is not maintained anymore. Hope @imcaspar is all good. This fork fixed some download link and made the pre-trained sustainable which means you don't need to download pre-trained file every time...

Try it now:
If it runs failed, check the dependence:

中文说明 | English

• Simplifed GPT2 train scripts（based on Grover, supporting TPUs）
• Ported bert tokenizer, multilingual corpus compatible
• 1.5B GPT2 pretrained Chinese model ( ~15G corpus, 10w steps )
• Batteries-included Colab demo #
• 1.5B GPT2 pretrained Chinese model ( ~30G corpus, 22w steps )

Corpus from THUCNews and nlp_chinese_corpus

Using Cloud TPU Pod v3-256 to train 22w steps

Due to the reason of colab (google reduced free gpu performance), the colab demo might be stale or no response unless you have a paid account (paid account is charged by google company, I have nothing to do with that. me and/or all the contributors won't get any money from it since this is a completely free and completely open sources project).

With just 2 clicks (not including Colab auth process), the 1.5B pretrained Chinese model demo is ready to go:

[Colab Notebook]

The contents in this repository are for academic research purpose, and we do not provide any conclusive remarks.

@misc{GPT2-ML,
  author = {Zhibo Zhang},{zxkmm}
  title = {GPT2-ML: GPT-2 for Multiple Languages},
  year = {2019},{2022}
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/imcaspar/gpt2-ml}},
}

https://github.com/google-research/bert

https://github.com/rowanz/grover

Research supported with Cloud TPUs from Google's TensorFlow Research Cloud (TFRC)

[机器之心] 只需单击三次，让中文GPT-2为你生成定制故事

[科学空间] 现在可以用Keras玩中文GPT2了