# BART\_LM: Odd Beam Search Output

**URL:** <https://discuss.huggingface.co/t/bart-lm-odd-beam-search-output/618>\
**Category:** Intermediate\
**Created:** [August 6, 2020, 4:47pm UTC](https://discuss.huggingface.co/t/bart-lm-odd-beam-search-output/618 "2020-08-06T16:47:31Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![chrisdoyleIE](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/chrisdoyleie/32/2546_2.png) [@chrisdoyleIE](https://discuss.huggingface.co/u/chrisdoyleIE)\
**Post date:** [August 6, 2020, 4:47pm UTC](https://discuss.huggingface.co/t/bart-lm-odd-beam-search-output/618/1 "2020-08-06T16:47:32Z")

</div>

Hi folks,

Problem: fine-tuned model adopts peculiar behaviour with beam search.

Specifically, beam search outputs include 2 bos tokens and exclude the first word token. I have double checked my data feed and the inputs look fine, with one bos and the first word token is present. Even more strange, there are no such problems with nucleus sampling, which suggests that this is particular to beam search.

for example, here are two outputs from the same fine-tuned model;

```auto
beam search output: <s><s>rek likes to play football. </s>
nuleus sampling output: <s> Derek likes to go to the football pitch. </s>

```

Any clues?

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [August 6, 2020, 6:21pm UTC](https://discuss.huggingface.co/t/bart-lm-odd-beam-search-output/618/2 "2020-08-06T18:21:55Z")

</div>

Hi,how did you prepare `labels` and `decoder_input_ids` ?

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [August 6, 2020, 6:25pm UTC](https://discuss.huggingface.co/t/bart-lm-odd-beam-search-output/618/3 "2020-08-06T18:25:03Z")

</div>

As the model is generating bos token my guess is you didn’t shift labels. For BART we need to prepare `labels` and `deocder_input_ids` like this

```auto
target_ids = tokenizer("...")
decoder_input_ids = target_ids[:, :-1].contiguous()
labels = target_ids[:, 1:].clone()

```

---

<div class="post-metadata">

**Author:** ![chrisdoyleIE](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/chrisdoyleie/32/2546_2.png) [@chrisdoyleIE](https://discuss.huggingface.co/u/chrisdoyleIE)\
**Post date:** [August 6, 2020, 6:56pm UTC](https://discuss.huggingface.co/t/bart-lm-odd-beam-search-output/618/4 "2020-08-06T18:56:51Z")

</div>

Hi @valhalla, double checked but there might be a hole - here is my training\_step function (I use lightning) :

```python
    def training_step(self, batch, batch_id):
        """see lighting docs. Need to make sure that first token is not ignored"""

        decoder_inputs = batch["target_ids"][:, :-1].contiguous()
        decoder_labels = batch["target_ids"][:, 1:].clone()
        decoder_labels[batch["target_ids"][:, 1:] == self.tokenizer.pad_token_id] = -100

        loss = self(source_ids=batch["source_ids"],
                    padding_mask=batch["padding_mask"],
                    decoder_inputs=decoder_inputs,
                    decoder_labels=decoder_labels)[0]

        return {"loss": loss}

```

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [August 6, 2020, 7:01pm UTC](https://discuss.huggingface.co/t/bart-lm-odd-beam-search-output/618/5 "2020-08-06T19:01:43Z")

</div>

Pinging @sshleifer

---

<div class="post-metadata">

**Author:** ![chrisdoyleIE](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/chrisdoyleie/32/2546_2.png) [@chrisdoyleIE](https://discuss.huggingface.co/u/chrisdoyleIE)\
**Post date:** [August 6, 2020, 7:06pm UTC](https://discuss.huggingface.co/t/bart-lm-odd-beam-search-output/618/6 "2020-08-06T19:06:55Z")

</div>

here’s my dataset **getitem** for some extra info:

```python
    def __getitem__ (self, index) -> dict:
        """Generates ONE sample of data, DataLoader takes care of batches. We return everything here to sacrifice speed
        for ease of coding / reading. Can make faster in time but not needed as of yet.
        :param index: ID of single file we want to load data from
        :return item: dictionary containing all possible permutations of data for all possible objectives, such that X
        is the dialogue, y is the summary, and s is the sep_token " TL;DR: ".
        """

        SAMsumID = self.ids[index - 1]

        # load source sample string
        # space added to beginning for tokenization purposes ie tok("name") !+ tok(" name")
        source_file_path = f"{self.root}/source/source_{SAMsumID}.txt"
        with open(source_file_path, "r") as source_file:
            dialogue = source_file.readline()
            if "t5" in self.arch: dialogue = "summarize: " + dialogue

        # load target sample string
        # space added to beginning for tokenization purposes ie tok("name") !+ tok(" name")
        target_file_path = f"{self.root}/target/target_{SAMsumID}.txt"
        with open(target_file_path, "r") as target_file:
            summary = target_file.readline()

        # we will return a dictionary for each getitem() call that is collated by dataloader
        item = {}
        input_tokens = self.tokenizer(dialogue, padding="max_length", truncation=True, add_special_tokens=True)
        target_tokens = self.tokenizer(summary, padding="max_length", truncation=True, add_special_tokens=True)

        # return necessary batch tensors
        item["SAMsumID"] = SAMsumID
        item["source_ids"] = torch.tensor(input_tokens["input_ids"])
        item["padding_mask"] = torch.tensor(input_tokens["attention_mask"])
        item["target_ids"] = torch.tensor(target_tokens["input_ids"])

        return item

```

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\
**Post date:** [August 6, 2020, 7:39pm UTC](https://discuss.huggingface.co/t/bart-lm-odd-beam-search-output/618/7 "2020-08-06T19:39:48Z")

</div>

can you show an example where you call `generate` twice with the same input\_ids and model, but different kwargs, and get wierd beam search output?

I suspect the issue is in generate kwargs/`decoder_start_token_id` if same model -\> different outputs.

---

<div class="post-metadata">

**Author:** ![chrisdoyleIE](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/chrisdoyleie/32/2546_2.png) [@chrisdoyleIE](https://discuss.huggingface.co/u/chrisdoyleIE)\
**Post date:** [August 7, 2020, 8:25am UTC](https://discuss.huggingface.co/t/bart-lm-odd-beam-search-output/618/8 "2020-08-07T08:25:55Z")

</div>

Both models have the same, default `decoder_start_token_id`, and I when I manually override this to any arbitrary token `<a>`, the beam output looks something like `<a><s>rek likes to play football. </s>.`

EDIT: The below code has decoder `skip_special_tokens=True`, but this flag is false for above examples.

```python
    def test_step(self, batch, batch_id):
        """Generate a summary and write it to a file while calculating rouge"""

        # We store the file id to help with consistent results across several GPUs
        SAMsumIDs = batch["SAMsumID"]

        # generate y_preds given seed_ids, placing kwargs in dict is handy for debugging
        if self.sampling == "beam":
            generate_kwargs = {"input_ids": batch["source_ids"],
                               "early_stopping": self.generation_early_stopping,
                               "no_repeat_ngram_size": self.no_repeat_ngrams,
                               "max_length": self.tokenizer.max_len,
                               "top_k": 50,
                               "num_beams": self.beam_size}

        elif self.sampling == "nucleus":
            generate_kwargs = {"input_ids": batch["source_ids"],
                               "max_length": self.tokenizer.max_len,
                               "top_k": 50,
                               "do_sample": True,
                               "top_p": 0.95}

        y_preds = self.model.generate(**generate_kwargs)
        # save generated hypotheses to hypo dest with the same name as the original file, but with a 0 prepended
        for file_id, hypo_token_ids in zip(SAMsumIDs, y_preds):
            prefix = "source_0" if self.anti else "hypo_"
            filename = f"{self.hypo_dest}/{prefix}{file_id}.txt"
            with open(filename, "w") as handle:
                hypo = self.tokenizer.decode(hypo_token_ids, skip_special_tokens=True)
                handle.write(f"{hypo}\n")

        return {"hypo_dialogues": None}

```

---

<div class="post-metadata">

**Author:** ![chrisdoyleIE](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/chrisdoyleie/32/2546_2.png) [@chrisdoyleIE](https://discuss.huggingface.co/u/chrisdoyleIE)\
**Post date:** [August 7, 2020, 2:32pm UTC](https://discuss.huggingface.co/t/bart-lm-odd-beam-search-output/618/9 "2020-08-07T14:32:12Z")

</div>

Another update: everything runs smoothly for T5 with all other code unchanged

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\
**Post date:** [August 7, 2020, 2:44pm UTC](https://discuss.huggingface.co/t/bart-lm-odd-beam-search-output/618/10 "2020-08-07T14:44:48Z")

</div>

Maybe this line is causing an issue? [https://github.com/huggingface/transformers/blob/3be2d048848df43e2dacb9127d2278443157ba78/src/transformers/modeling\_bart.py#L1076](https://github.com/huggingface/transformers/blob/3be2d048848df43e2dacb9127d2278443157ba78/src/transformers/modeling_bart.py#L1076)

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [August 7, 2020, 2:57pm UTC](https://discuss.huggingface.co/t/bart-lm-odd-beam-search-output/618/11 "2020-08-07T14:57:06Z")

</div>

@sshleifer, why do we need to manually set the `bos_token_id` here, I think the `generate` function handles that here already [https://github.com/huggingface/transformers/blob/master/src/transformers/generation\_utils.py#L402](https://github.com/huggingface/transformers/blob/master/src/transformers/generation_utils.py#L402)

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\
**Post date:** [August 7, 2020, 7:23pm UTC](https://discuss.huggingface.co/t/bart-lm-odd-beam-search-output/618/12 "2020-08-07T19:23:32Z")

</div>

It’s copied from fairseq, but the basic idea is that you want to know the real probability of BOS, not just force it with probability 1. If I recall (open to being disproved), removing it breaks

```auto
RUN_SLOW=1 pytest tests/test_modeling_bart.py

```

---

<div class="post-metadata">

**Author:** ![chrisdoyleIE](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/chrisdoyleie/32/2546_2.png) [@chrisdoyleIE](https://discuss.huggingface.co/u/chrisdoyleIE)\
**Post date:** [August 8, 2020, 6:32pm UTC](https://discuss.huggingface.co/t/bart-lm-odd-beam-search-output/618/13 "2020-08-08T18:32:30Z")

</div>

I look forward to taking a closer look at this breakage… eventually. Thank you both for the help!

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [August 10, 2020, 12:26pm UTC](https://discuss.huggingface.co/t/bart-lm-odd-beam-search-output/618/14 "2020-08-10T12:26:27Z")

</div>

@chrisdoyleIE,  
so the reason for two `bos` tokens in beam search is that, [here](https://github.com/huggingface/transformers/blob/master/src/transformers/generation_utils.py#L402) the generate function sets the `decoder_start_token_id` (if not defined then `bos_token_id` , which `<s>` for BART) as the prefix token,  
and [this](https://github.com/huggingface/transformers/blob/3be2d048848df43e2dacb9127d2278443157ba78/src/transformers/modeling_bart.py#L1076) forces the generation of `bos_token` (`<s>`) when the current length is one.  
So I also faced the same issue for my bart model fine-tuned for question generation when using beam search. This is what it produced for one of the inputs.  
`['<s><s> created Python in 1991?']`

So when I just returned logits as it is from `adjust_logits_during_generation` instead of forcing another `bos_token` it generated this  
`['<s>Who created Python?']`  
Which is what I expected.

But as @sshleifer said, when I modified `adjust_logits_during_generation` as said above it breaks the  
`RUN_SLOW=1 pytest tests/test_modeling_bart.py` because the generated summaries don’t match the fairseq ones.

So when I looked at the configs of the [bart-large-cnn](https://s3.amazonaws.com/models.huggingface.co/bert/facebook/bart-large-cnn/config.json) and [bart-large-xsum](https://s3.amazonaws.com/models.huggingface.co/bert/facebook/bart-large-xsum/config.json), it used `eos_token` (`</s>`) as the `decoder_start_token_id` and test only pass when the first two token are `</s><s>` (`</s>` is set as first token by `.generate` and `<s>` is set by `adjust_logits_during_generation`.

So I suspected that the weird beam search output for my model is because of using `<s>` as `decoder_start_token_id`, when I tried using `</s>` as the start token like the summerization models it again gave weird results.  
`['</s><s> created Python in 1991?']`

So to me it seems that `adjust_logits_during_generation` and `</s>` as the `decoder_start_token` is only working for the pre-trained summrization models and not for other bart-large or bart-base finetuned models.

@chrisdoyleIE can you also confirm this with your model

1. first do beam search using `eos_token_id` as the `decoder_start_token_id`
2. use the bos\_token as start token (as usual) and return logits as it is from `adjust_logits_during_generation` instead of forcing `<s>`. You can do it using

```auto
def adjust_logits(logits, **kwargs):
    return logits
model.adjust_logits_during_generation = adjust_logits

```

@sshleifer, if `</s>` is the `decoder_start_token` then why don’t we set it when fine-tuning the model ? Also in fairseq they seem to use the bos token `<s>` as the first prefix. [Here](https://github.com/pytorch/fairseq/blob/master/fairseq/models/bart/hub_interface.py#L123)

Sorry for this long issue, but no other way to explain it.

---

<div class="post-metadata">

**Author:** ![chrisdoyleIE](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/chrisdoyleie/32/2546_2.png) [@chrisdoyleIE](https://discuss.huggingface.co/u/chrisdoyleIE)\
**Post date:** [August 11, 2020, 10:31am UTC](https://discuss.huggingface.co/t/bart-lm-odd-beam-search-output/618/15 "2020-08-11T10:31:04Z")

</div>

@valhalla this is a stellar investigation, thank you very much! I’ll need about 2 weeks to submit my thesis then look forward to taking a look.

🚀

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [August 11, 2020, 12:35pm UTC](https://discuss.huggingface.co/t/bart-lm-odd-beam-search-output/618/16 "2020-08-11T12:35:45Z")

</div>

@sshleifer, @chrisdoyleIE  
After modifying the `adjust_logits_during_generation` BLUE-4 score for my model went from 13.09 to 19.14. So this indeed is an issue.

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\
**Post date:** [August 11, 2020, 7:09pm UTC](https://discuss.huggingface.co/t/bart-lm-odd-beam-search-output/618/17 "2020-08-11T19:09:35Z")

</div>

Can you PR a fix/ or update the docs with the suggested behavior? I am a little busy at the moment.

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\
**Post date:** [August 16, 2020, 7:08pm UTC](https://discuss.huggingface.co/t/bart-lm-odd-beam-search-output/618/18 "2020-08-16T19:08:03Z")

</div>

> [@valhalla](#):
>
> After modifying the `adjust_logits_during_generation` BLUE-4 score for my model went from 13.09 to 19.14. So this indeed is an issue.

In the config of your finetuned model, what was decoder\_start\_token\_id?  
In your training code, what did a batch “look like”?

---

<div class="post-metadata">

**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\
**Post date:** [August 17, 2020, 4:50am UTC](https://discuss.huggingface.co/t/bart-lm-odd-beam-search-output/618/19 "2020-08-17T04:50:18Z")

</div>

`decoder_start_token_id` was not defined in the `config`. So `generate` used `bos` when it’s not defined in the `config`. And the batches were simply prepared like this

```auto
src_text = "Python was created by <hl> Guido van Rossum <hl> in 1991."
target _text = "Who created python ?"
input_ids = tokenizer(src_text, return_tensors="pt")["input_ids"]
target_ids = tokenizer(target , return_tensors="pt")["input_ids"]
decoder_input_ids = target_ids[:, :-1].contiguous()
labels = target_ids[:, 1:].clone()
```