# Decoding Modified Sentence Embeddings

**URL:** <https://discuss.huggingface.co/t/decoding-modified-sentence-embeddings/42455>\
**Category:** 🤗Transformers\
**Created:** [June 7, 2023, 6:53pm UTC](https://discuss.huggingface.co/t/decoding-modified-sentence-embeddings/42455 "2023-06-07T18:53:08Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![jmindel](https://avatars.discourse-cdn.com/v4/letter/j/c4cdca/32.png) [@jmindel](https://discuss.huggingface.co/u/jmindel)\
**Post date:** [June 7, 2023, 6:53pm UTC](https://discuss.huggingface.co/t/decoding-modified-sentence-embeddings/42455/1 "2023-06-07T18:53:08Z")

</div>

Hi all! I’m attempting to perform operations on the semantic meaning of a sentence by transforming hidden layer embeddings that I’m retrieving from a pretrained T5 model. However, I’m having difficulty decoding these embeddings once they’ve been modified. I’ve tried to write code that treats T5’s encoder and decoder as decoupled, but I haven’t yet been able to figure out **how to generate text with the decoder that is conditioned on the modified embedding**.

From my understanding, I’d need to re-implement beam search or top-k filtering in order to decode a sentence embedding, based on [this post](https://discuss.huggingface.co/t/control-encoderdecodermodel-to-generate-tokens-step-by-step/1756). Is that the case? How would I go about doing so?

Here’s some pseudocode I’ve tried to write so far, based on the previously linked post:

```auto
t5_model = transformers.T5ForConditionalGeneration.from_pretrained("t5-large")
t5_tok = transformers.T5Tokenizer.from_pretrained("t5-large")

text = "This is some example text."
tokenized = t5_tok(text, return_tensors="pt")
input_ids = tokenized.input_ids
attn_mask = tokenized.attention_mask
encoder_output_vectors = t5_model.encoder(input_ids, return_dict=True, attention_mask=attn_mask).last_hidden_state

encoder_output_vectors = some_vector_operation(encoder_output_vectors)

start_token = "<pad>"
eos_token = "</s>"
generated_so_far = [start_token]
while eos_token not in generated_so_far:
    outputs = t5_model.decoder(decoder_input_ids=generated_so_far, encoder_hidden_states=encoder_output_vectors)
    next_token_logits = outputs.last_hidden_state # outputs[0][0, -1, :] 
    filtered_logits = top_k_top_p_filtering(next_token_logits, top_k=top_k, top_p=top_p)
    next_token = torch.multinomial(torch.nn.functional.softmax(filtered_logits, dim=-1), num_samples=1) 
    generated_so_far = torch.cat((generated_so_far, next_token.unsqueeze(0)), dim=1)

```

The above should be using `t5_tok.decode` somewhere as well, but I’m not sure exactly where.

Some examples of what I mean by “operations” on sentence embeddings: interpolating between two embeddings, finding an orthogonal sentence embedding, selecting from a random ball around an embedding, etc. The model doesn’t need to be T5; any pretrained encoder-decoder language model will do.

Many thanks for your support!