# GPT2 for QA Pair Generation **URL:** \ **Category:** Research\ **Created:** [August 18, 2020, 9:59pm UTC](https://discuss.huggingface.co/t/gpt2-for-qa-pair-generation/759 "2020-08-18T21:59:56Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![aclifton314](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aclifton314/32/55465_2.png) [@aclifton314](https://discuss.huggingface.co/u/aclifton314)\ **Post date:** [August 18, 2020, 9:59pm UTC](https://discuss.huggingface.co/t/gpt2-for-qa-pair-generation/759/1 "2020-08-18T21:59:56Z")
I was wondering if it were possible to somehow train GPT2 to generate question-answer pairs in a particular domain? ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [August 19, 2020, 9:08am UTC](https://discuss.huggingface.co/t/gpt2-for-qa-pair-generation/759/2 "2020-08-19T09:08:31Z")
I’ve tried this with seq2seq models. I have worked on qa pair generation (separately) using T5 with descent results. You can find it [here](https://github.com/patil-suraj/question_generation). One way we can do this with GPT-2 is prepare our input like this Our context is `42 is the answer to life, the universe and everything` , answer is `42` and target question is `What is the answer to life, universe and everything ?` Then input text: `context: 42 is the answer to life, the universe and everything. question: What is the answer to life, universe and everything ? answer: 42 ` and prepare the attention mask such that, there will be no attention from `question: ...` part, so model won’t look into future tokens and calculate loss only on the `question: ...` part. And it inference time we will feed only the context part and ask the model to generate the question. This just one one way I can think of the of my mind. Feel free to correct me if this is wrong. ---
**Author:** ![aclifton314](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aclifton314/32/55465_2.png) [@aclifton314](https://discuss.huggingface.co/u/aclifton314)\ **Post date:** [August 19, 2020, 6:37pm UTC](https://discuss.huggingface.co/t/gpt2-for-qa-pair-generation/759/3 "2020-08-19T18:37:53Z")
@valhalla Thanks for your response. That’s an interesting approach! Does that still require humans to create training “context” strings for gpt2? ---
**Author:** ![aclifton314](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aclifton314/32/55465_2.png) [@aclifton314](https://discuss.huggingface.co/u/aclifton314)\ **Post date:** [October 12, 2020, 7:51pm UTC](https://discuss.huggingface.co/t/gpt2-for-qa-pair-generation/759/4 "2020-10-12T19:51:20Z")
@valhalla If I understand this correctly: 1. The input text will look like `context: 42 is the answer to life, the universe and everything. question: What is the answer to life, universe and everything ? answer: 42 ` 2. Mask out the `question` part so the new text will look like `context: 42 is the answer to life, the universe and everything. answer: 42 ` 3. That is what gets fed as input text into the GPT2 model Does this mean I define the `labels` into the model as the text that is masked? ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [October 13, 2020, 7:25am UTC](https://discuss.huggingface.co/t/gpt2-for-qa-pair-generation/759/5 "2020-10-13T07:25:40Z")
By mask, I meant `attention_mask` , the `attention_mask` should be zero on the text you want to predict, so the model won’t peek into future. So if you want to generate question and answer, then the question and answer tokens should have 0 in attention mask. ---
**Author:** ![aclifton314](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aclifton314/32/55465_2.png) [@aclifton314](https://discuss.huggingface.co/u/aclifton314)\ **Post date:** [October 13, 2020, 11:19am UTC](https://discuss.huggingface.co/t/gpt2-for-qa-pair-generation/759/6 "2020-10-13T11:19:41Z")
Ah yes, sorry for my misunderstanding. So we mask out the parts we want to predict by setting the `attention_mask` of those tokens to 0. With these tokens masked in `attention_mask`, do we then pass it and the input string to GPT2 and train it with the language model head with no labels? ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [October 13, 2020, 3:06pm UTC](https://discuss.huggingface.co/t/gpt2-for-qa-pair-generation/759/7 "2020-10-13T15:06:14Z")
You’ll still need to pass `labels` for training. Training will be same as training any GPT-2 model, only difference is the `attention_mask` ---
**Author:** ![aclifton314](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aclifton314/32/55465_2.png) [@aclifton314](https://discuss.huggingface.co/u/aclifton314)\ **Post date:** [October 13, 2020, 3:53pm UTC](https://discuss.huggingface.co/t/gpt2-for-qa-pair-generation/759/8 "2020-10-13T15:53:54Z")
If I only wanted to generate questions, would I set the `attention_mask` for those tokens to 0 and use their text as the `labels`? Something like: ```python from transformers import GPT2Tokenizer tokenizer = GPT2Tokenizer.from_pretrained('gpt2') def my_data_collator(text_str): encoded_results = tokenizer(text_str, padding=True, truncation=True, return_tensors='pt', return_attention_mask=True) enncoded_results['attention_mask'] = set_my_attention_mask(encoded_results) #function to set attention mask to 0 on tokens in the question:... part of text_str label_ids = get_my_label_str(encoded_results['input_ids']) #function to return list of token ids for question:... part of text_str batch = {} batch['input_ids'] = encoded_results['input_ids'] batch['past'] = None batch['attention_mask'] = encoded_results['attention_mask'] batch['position_ids'] = None batch['head_mask'] = None batch['inputs_embeds'] = None batch['labels'] = label_ids batch['use_cache'] = True return batch text_str = 'context: 42 is the answer to life, the universe and everything. question: What is the answer to life, universe and everything ? answer: 42' ``` And `batch` would get passed to a `GPT2LMHeadModel`? ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [October 13, 2020, 4:09pm UTC](https://discuss.huggingface.co/t/gpt2-for-qa-pair-generation/759/9 "2020-10-13T16:09:11Z")
This seems correct. One more thing to add, you can calculate loss only on the `question: ...` part. To do this set `labels` to -100 for tokens before the `question:` part, so cross entropy will ignore it. Also you won’t need to explicitly set some arguments (`position_ids`, `head_mask` etc) to `None`. They are by default `None` so it’s okay if don’t pass them. Will make the code more cleaner. ---
**Author:** ![bilalghanem](https://avatars.discourse-cdn.com/v4/letter/b/b2d939/32.png) [@bilalghanem](https://discuss.huggingface.co/u/bilalghanem)\ **Post date:** [March 23, 2022, 5:27pm UTC](https://discuss.huggingface.co/t/gpt2-for-qa-pair-generation/759/10 "2022-03-23T17:27:21Z")
@valhalla if we set the context labels to -100, this will make the model ignore the context while training. In other words, the generation of the questions won’t be based context-based. Am I right?