bug: text generation result is different for different operating systems (macOS, windows, and Linux)

[xdotli]

Repo of the project where the bug is spotted:

https://github.com/l1xiangyi/GPT2/tree/main

What is the problem

The problem occurs in main.py line 89 where I try to generate texts based on a NeruIPS dataset. This line calls the sample function that generates subsequent sentences based on the start_text. However the texts generated on macOS, Windows, and Linux are very different even though the seed is fixed in the config.yml.

• macOS result: sumbit/samples.txt
• Windows result: sumbit/sample_windows.txt
• Windows result: sumbit/sample_linux.txt

Strange words like "Zombie" pop up a lot in the Windows and Linux generated texts.

How to reproduce this problem

Linux

The simplest way is go create a new Colab notebook and run the following commands:

!git clone https://github.com/l1xiangyi/gpt2/
%cd gpt2
!python -m pip install -r requirements.txt
!python main.py

Then check the samples.txt in the submit folder.

Windows

Similar steps to the Linux version. Just drop the special sign before each commands:

git clone https://github.com/l1xiangyi/gpt2/
cd gpt2
python -m pip install -r requirements.txt
python main.py