Using a KG-Copy Network for Non-Goal Oriented Dialogues

    

Requirements

• python 3.6
• pytorch 1.2.0
• Run pip install -r requirements.txt

**NOTE: The following pre-processing step is not required if you just want to train the system on our processed data (since all the required pre-processed data are included in the project directory).

Pre-processing:

Building Knowledge Graph:

Running the following code will download information from wikipedia and will create a Knowledge Graphs for clubs and national teams respectively. Names of the selected clubs and national teams are currently hard-coded into the 'build_KG_clubs.py','build_KG_national_teams.py' files:

python kg_build/build_KG_clubs.py
python kg_build/build_KG_national_teams.py

python kg_build/build_incar_data.py
python kg_build/build_dataset_KVR.py

Building vocabulary:

In order to build a vocabulary for the system, run the following command. Running the commands will create vocabulary for the system for the given KGs (which we have already built in the previous step) :

wget https://dl.fbaipublicfiles.com/fasttext/vectors-wiki/wiki.en.vec
mv wiki.en.vec vocab/
python create_vocab_kb.py

Running the commands will generate 'glove300.npy','vocab.npy','w2i.npy' files inside 'vocab/' directory

Generating train/test/dev data from AMT data (soccer conversations)

To create and preprocess train-test-dev data, run the following command (Train, test, validation data are already pre-processed and generated inside \preproc_files directory). No need to preprocess again if you just want to train/test the model.

python -m spacy download en_core_web_sm
python -m spacy download en
python -m spacy download en_core_web_lg

python preprocess_kb_2.py --data_dir soccer_conversations/
python preprocess_kb_incar.py --data_dir preproc_files/incar/ --stoi vocab/w2i_incar.npy --vocab_glove vocab/glove300_incar.npy
python utils/generate_entities_soccer.py

Train & Test

Pre-processing is not required if you just want to train/test the model at this point. To train the system run the following command:

For Soccer Domain:

python -u ./train_kg_copy.py --batch_size 32 --hidden_size 128 --teacher_forcing 12 --resp_len 10 --lr 0.001 --num_layer 1 --gpu 1 --epochs 150 --data_dir preproc_files/soccer/

python train_mem2seq_soccer.py -lr=0.001 -layer=1 -hdd=128 -dr=0.2 -dec=Mem2Seq -bsz=8 -ds=kvr -t=

python train_vanilla_soccer.py -lr=0.001 -layer=1 -hdd=128 -dr=0.2 -dec=VanillaSeqToSeq -bsz=8 -ds=kvr -t=

For incar settings:

python -u ./train_kg_copy_incar.py --batch_size 64 --hidden_size 512 --teacher_forcing 12 --resp_len 20 --lr 0.0001 --num_layer 1 --gpu 1 --epochs 300 --data_dir preproc_files/incar/ --stoi vocab/w2i_incar.npy --vocab_glove vocab/glove300_incar.npy

In each epochs the best trained model so far will be saved inside '/models' directory with a file name 'Sentient_model2.bin'. The saved model can later be used for testing purpose on new data. After completing the training the command will also generate a file 'test_predicted_kg_attn2.csv' where we can check predicted output along with given input test data.