# Exploring Accent “Delta” Vectors Using Speaker Embeddings (Accent → Native → Accent Control)

**URL:** <https://discuss.huggingface.co/t/exploring-accent-delta-vectors-using-speaker-embeddings-accent-native-accent-control/170465>\
**Category:** Research\
**Created:** [November 14, 2025, 8:08pm UTC](https://discuss.huggingface.co/t/exploring-accent-delta-vectors-using-speaker-embeddings-accent-native-accent-control/170465 "2025-11-14T20:08:39Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![mquiroga2912](https://avatars.discourse-cdn.com/v4/letter/m/ed8c4c/32.png) [@mquiroga2912](https://discuss.huggingface.co/u/mquiroga2912)\
**Post date:** [November 14, 2025, 8:08pm UTC](https://discuss.huggingface.co/t/exploring-accent-delta-vectors-using-speaker-embeddings-accent-native-accent-control/170465/1 "2025-11-14T20:08:39Z")

</div>

There is this idea that has haunted me for weeks and I can’t get it out of my mind.

I’ve been experimenting with voice cloning and multilingual TTS, and I’m trying to understand if someone has already explored this path or something close to it.

I’m a native Spanish speaker (Mexico). When I speak English, I have a noticeable accent. I cloned my voice in ElevenLabs and generated two recordings of the same sentences:

1. Me speaking English with my real accent

2. My cloned voice speaking the same sentences with a native English accent

The starting point looks similar to this post:  
[https://discuss.huggingface.co/t/voice-cloning-and-accent-fine-tuning/162306](https://discuss.huggingface.co/t/voice-cloning-and-accent-fine-tuning/162306?utm_source=chatgpt.com)  
The difference is that ElevenLabs already gives me a “good enough” native-accent output of my own voice, so I don’t need to train anything for that.

The idea:  
If I extract speaker embeddings for both versions, I can treat them like vectors:

- A = embedding of my accented English

- B = embedding of the native-accent version

- C = B − A (a kind of “accent delta”)

If the embeddings are even somewhat linear, C should encode the difference between my accented English and the native version.

Then a few things become possible:

- **Native + C → hyper-native version of myself** , which I can imitate to get closer to the real pronunciation.

- **Spanish + C → how an American speaker might “perceive” me speaking Spanish** , since I’d be adding the “American accent vector” to Spanish.

- **Native + k\*C → accent strength control** , continuous from “barely accented” to “exaggerated native.”

ElevenLabs doesn’t expose embeddings, so I can’t run this experiment there. I know I need open-source tools for this. My initial plan is to try:

- Resemblyzer or SpeechBrain ECAPA for embeddings

- XTTS-v2 for synthesis, since it accepts external speaker embeddings

Before going deeper, I want to know:

- Has anyone tried this kind of accent-vector subtraction/addition?

- Are accent and timbre separable enough in these embedding spaces to make this useful?

- Any existing pipelines that support this sort of embedding manipulation without collapsing?

- Any prior work on accent disentanglement or conversion that does something similar?

If no one has done it, I’ll prototype the full pipeline (ECAPA embeddings + XTTS-v2 synthesis) and see how far the linearity goes.

Any pointers, warnings, or references would help.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [November 14, 2025, 9:16pm UTC](https://discuss.huggingface.co/t/exploring-accent-delta-vectors-using-speaker-embeddings-accent-native-accent-control/170465/2 "2025-11-14T21:16:23Z")

</div>

Conceptually similar resources seem to exist.

* * *

Below is a focused, structured answer to your four questions, tailored to your concrete idea:

- A = you, L2 English (Mexican accent)
- B = “native” English version of you (e.g., ElevenLabs)
- C = B − A (accent delta)
- Then use A + k·C, B + k·C, Spanish + k·C, etc., inside a TTS/VC system such as XTTS-v2.

I will first answer the four questions concisely, then give more context, and finally a practical “how to prototype this” section.

* * *

## 0. Short answers to your 4 questions

1. **Has anyone tried “accent delta vector” subtraction/addition?**

2. **Are accent and timbre separable enough in these spaces?**

3. **Existing pipelines that can handle embedding manipulation without collapsing?**

4. **Prior work on accent disentanglement / conversion that does something similar?**

In other words: your idea is very much aligned with current research, but you are proposing the **most minimal version** using just speaker embeddings and one personal pair A/B.

* * *

## 1. What your C vector is really doing (conceptual background)

### 1.1 Speaker embeddings: what they encode

Speaker encoders (x-vector, ECAPA-TDNN, Resemblyzer, XTTS speaker encoder) are trained for **speaker recognition** :

- Given utterances, they learn vectors that:

Large reviews of deep speaker embeddings show they capture not just identity but also:

- gender, age,
- accent, language,
- speaking style, channel, and sometimes emotion. ([Frontiers](https://www.frontiersin.org/journals/signal-processing/articles/10.3389/frsip.2024.1339159/full "Reimagining speech: a scoping review of deep learning- ..."))

So if you embed:

- A = you, Mexican-accent English
- B = you, synthetic native English (ElevenLabs)

then C = B − A is not a “pure accent” vector. It encodes:

- accent changes,
- ElevenLabs’ prosody/style vs your own,
- any channel differences,
- plus normal noise in the embedding.

But if accent is a **major systematic difference** between A and B, then C will still have a strong accent component. This is why your idea is plausible.

### 1.2 Where accent is modeled in modern systems

Modern FAC and accented-TTS systems rarely rely purely on speaker embeddings for accent control. Instead they factor speech into three latent spaces:

1. **Content / linguistic units**

2. **Speaker / timbre embeddings**

3. **Accent / style latents**

Your C lives in space (2) only, but real systems often control accent in (3), and sometimes also tweak the content representation (1). This mismatch is why C will be “accent + style” rather than perfectly clean accent.

* * *

## 2. Q1 – Who has done something close to “Accent + Delta Vector”?

No one (so far) publishes exactly “take my L2 voice and my synthetic native clone, do B − A, then add C elsewhere”. But three strands of work are extremely similar.

### 2.1 Principal directions in speaker embedding space

Lux et al. (Interspeech 2023) train a generator of **artificial speaker embeddings** , then:

- Discover **principal directions** in embedding space that correspond to attributes (brightness, breathiness, age, etc.).
- Move embeddings along those directions to control these attributes in a TTS system. ([arXiv](https://arxiv.org/abs/2310.17502 "Controllable Generation of Artificial Speaker Embeddings through Discovery of Principal Directions"))

Mathematically this is exactly what you want:

- C\_attr is a direction;
- S’(k) = S + k·C\_attr controls that attribute;
- They also show how to stay on the speaker manifold when doing this (important for stability).

Accent is just another attribute; there is nothing stopping you from learning an “accent direction” in the same way.

### 2.2 Editing speaker embeddings to change style / Lombard effect

The Interspeech 2025 program includes work on **gradual modeling of the Lombard effect by modifying speaker embeddings** in a TTS model. ([ISCA Archive](https://www.isca-archive.org/interspeech_2025/ "Interspeech 2025"))

- They keep text and speaker ID fixed,
- Modify the speaker embedding by adding a “Lombard direction”,
- Then the model produces more Lombard-style speech (speaking in noise) for the same voice.

Again, same pattern: find a direction in embedding space that corresponds to a style and add it.

### 2.3 Accent embeddings and FAC systems

FAC and accented-TTS models like SpeechAccentLLM, Multi-Scale Accent Modeling, and DART explicitly introduce an **accent latent** :

- SpeechAccentLLM uses discrete speech codes and an LLM-style model that is conditioned on accent for both TTS and FAC. ([arXiv](https://arxiv.org/abs/2507.01348 "A Unified Framework for Foreign Accent Conversion and ..."))
- Multi-Scale Accent Modeling uses **global and local accent embeddings** , separate from speaker embeddings, and shows that you can vary accent while keeping speaker identity largely fixed. ([arXiv](https://arxiv.org/abs/2406.10844 "Multi-Scale Accent Modeling and Disentangling for Multi-Speaker Multi-Accent Text-to-Speech Synthesis"))
- DART uses multi-level VAEs and vector quantization to explicitly **disentangle** accent and speaker in a multi-speaker TTS model, so you can pick any combination “speaker X + accent Y”. ([arXiv](https://arxiv.org/abs/2410.13342 "DART: Disentanglement of Accent and Speaker Representation in Multispeaker Text-to-Speech"))

They do not literally take B − A, but at the conceptual level they:

- Represent accent as a vector/latent;
- Interpolate or swap accent latents;
- Sometimes scale intensity (k) to control accent strength.

So your B − A = C idea is like a **single-speaker, hand-crafted version** of what these systems learn across large populations.

* * *

## 3. Q2 – How separable are accent and timbre in real embedding spaces?

### 3.1 Evidence for entanglement

Several lines of work show that standard speaker embeddings are heavily entangled:

- A large scoping review on deep learning speech systems notes that speaker embeddings (x-vectors, ECAPA) encode language, accent, and channel alongside identity, because the models optimize for speaker discrimination, not factorization. ([Frontiers](https://www.frontiersin.org/journals/signal-processing/articles/10.3389/frsip.2024.1339159/full "Reimagining speech: a scoping review of deep learning- ..."))
- Multi-accent ASR adaptation papers train x-vectors on mixed accent data; these embeddings are used as “accent vectors” to adapt ASR, but they also carry speaker information, which can hurt generalization if not handled carefully. ([AudioCC Lab](https://audiocc.sjtu.edu.cn/user/pages/03.research/02.publications/2022/Layer-Wise%40Fast%40Adaptation%40for%40End-to-End%40Multi-Accent%40Speech%40Recognition/paper.pdf "Layer-Wise Fast Adaptation for End-to-End Multi-Accent ..."))
- A 2025 Interspeech paper on flow of speech for speaker recognition explicitly evaluates ECAPA-TDNN and Resemblyzer, showing that these embeddings are sensitive to speaking style and speech rate, not just identity. ([ISCA Archive](https://www.isca-archive.org/interspeech_2025/martinek25_interspeech.pdf "flow of speech effect on speaker recognition systems"))

Conclusion: **Accent is not orthogonal to timbre** in these spaces. Changing accent will inevitably tug on other attributes.

### 3.2 Evidence that accent and speaker can be separated with structure

The fact that Multi-Scale Accent Modeling and DART _can_ disentangle accent and speaker with explicit design is strong evidence that the information is separable in principle:

- Multi-Scale Accent Modeling:

- DART:

These models only achieve clean accent control because they **force** accent → one latent and speaker → another. That tells you:

- If you stay in plain ECAPA or XTTS speaker embedding space, a single C is at best a useful **approximation** to an accent direction, not a mathematically pure one.

For your purposes, that is acceptable; you are exploring “how linear is this space”, not building production-grade FAC.

* * *

## 4. Q3 – Pipelines that support your kind of embedding manipulation

You already identified the key components correctly. Here is how they fit together, with what we know from docs and papers.

### 4.1 XTTS-v2 + external speaker encoder

XTTS (and XTTS-v2) are multilingual zero-shot TTS models that:

- Use a **pre-trained speaker encoder** to extract a speaker embedding from a reference audio. ([arXiv](https://arxiv.org/html/2406.04904v1 "XTTS: a Massively Multilingual Zero-Shot Text-to-Speech ..."))
- Feed that embedding into the TTS backbone as conditioning, enabling cross-lingual voice cloning from a few seconds of audio. ([arXiv](https://arxiv.org/html/2406.04904v1 "XTTS: a Massively Multilingual Zero-Shot Text-to-Speech ..."))

The Coqui docs and related work highlight that you can:

- Pre-compute d\_vectors with `compute_embeddings.py` and supply them as external speaker embeddings (`d_vector_file`), instead of always using the internal encoder on the fly. ([GitHub](https://github.com/coqui-ai/TTS/discussions/1480 "Speaker Encoder questions (for VITS voice conversion)"))

This is exactly what you need:

- Compute A\_i and B\_i embeddings with the **same** encoder XTTS uses (ideal case).
- Form C = B\_avg − A\_avg.
- Construct S’(k) = normalize(S + k·C).
- Supply S’(k) as the d-vector to XTTS-v2.

Because the encoder is the one XTTS expects, you avoid dimension mismatch and keep edits relatively on-manifold.

### 4.2 Voice conversion frameworks with editable speaker embeddings

Pre-training Approaches for Voice Conversion (Unilight thesis) and related VC work show that:

- You can factor speech into content units + speaker embeddings + sometimes style.
- Speaker embeddings can be modified while leaving content units fixed, giving voice conversion and style control. ([Unilight](https://unilight.github.io/assets/pdf/phd-thesis.pdf "Pre-training Approaches for Voice Conversion to Address ..."))

Non-autoregressive real-time accent conversion models with voice cloning also:

- Use a chain of ASR + content representation + TTS;
- Allow swapping accent while preserving identity. ([arXiv](https://arxiv.org/pdf/2405.13162 "Non-autoregressive real-time Accent Conversion model ..."))

If you plug your S + k·C into such a VC system’s speaker branch, you can test your idea in **speech-to-speech** mode rather than TTS mode.

### 4.3 Keeping embeddings from “collapsing” (stability)

Lux et al. explicitly study how to make embedding edits that:

- Change attributes gradually,
- Stay within the distribution of realistic speakers by generating embeddings from a learned model and moving along principal directions. ([arXiv](https://arxiv.org/abs/2310.17502 "Controllable Generation of Artificial Speaker Embeddings through Discovery of Principal Directions"))

Your simpler approach can borrow some of their heuristics:

- Always renormalize the norm of S + k·C to match typical embeddings.
- Use small |k| (e.g., ≤0.5) to avoid out-of-distribution points.
- Optionally fit a small PCA around your natural embeddings and project edits back onto the top principal components.

In summary: XTTS-v2 plus its speaker encoder, or a VC model with editable speaker embeddings, is a realistic pipeline for doing exactly what you propose without the model instantly collapsing.

* * *

## 5. Q4 – Prior work on accent disentanglement / conversion

You are essentially reinventing, in a personal way, what FAC and accent-TTS papers formalize. It is useful to see how they solve the same problem.

### 5.1 Discrete unit + TTS accent conversion

“Accent conversion using discrete units with parallel data synthesized from controllable accented TTS” (Nguyen et al. 2024) does the following: ([arXiv](https://arxiv.org/abs/2410.03734 "Accent conversion using discrete units with parallel data synthesized from controllable accented TTS"))

- Cluster **self-supervised discrete units** from native speech to get accent-neutral content tokens.
- Use controllable accented TTS to generate synthetic parallel data.
- Train an accent conversion model that maps L2 → “native units” and then back to waveform while preserving speaker identity.

This is very similar to your aim (L2 → L1 keeping speaker) but works primarily in **content + accent latent spaces** , not just speaker space.

### 5.2 Multi-scale accent modeling and DART

- Multi-Scale Accent Modeling:

- DART:

Both show that **accent is a robust latent that can be split from speaker identity** if you design the model that way. Your C is an implicit version of that accent latent, baked into speaker space.

### 5.3 LLM-style unified frameworks: SpeechAccentLLM

SpeechAccentLLM (2025) proposes a **unified framework** for FAC and TTS using:

- SpeechCodeVAE to get discrete speech codes with CTC.
- An LLM-style FAC & TTS model conditioned on accent.
- The same model can perform FAC (L2→L1) or TTS with chosen accent. ([arXiv](https://arxiv.org/abs/2507.01348 "A Unified Framework for Foreign Accent Conversion and ..."))

Compared to your approach:

- They have a **dedicated accent latent** and plenty of data.
- You have **one or a few speakers and a direct manipulation in speaker space** , which is much lighter but rougher.

### 5.4 Older FAC / voice morphing work

Earlier work (voice morphing, foreign accent conversion by voice morphing, etc.) treats FAC as a special case of voice conversion and uses:

- Spectral mapping (GMMs, trajectory-ML, etc.) to morph L2 → L1 while preserving some speaker characteristics. ([Semantic Scholar](https://www.semanticscholar.org/paper/Accent-and-Speaker-Disentanglement-in-Many-to-many-Wang-Ge/9e7f9301ef7e570211a3cb1a6d335df85498bd05 "[PDF] Accent and Speaker Disentanglement in Many-to- ..."))

These provide metrics and listening test protocols that you can reuse to evaluate your own system (see next).

* * *

## 6. How to prototype your idea concretely (A, B, C, XTTS)

Here is a practical experimental plan that matches your mental picture and the current tools.

### 6.1 Data: build good A and B

- Record **many** English sentences (20–50+ distinct sentences) with your real accent.
- Generate corresponding **native-accent versions** with ElevenLabs or another TTS using your cloned voice and the **same text**.
- Keep conditions similar (sample rate, levels, trimming) so that the main systematic difference really is accent & style.

### 6.2 Use the same speaker encoder as XTTS

- Use the **XTTS speaker encoder** rather than a random ECAPA, so your embeddings are in exactly the space the TTS expects. ([arXiv](https://arxiv.org/html/2406.04904v1 "XTTS: a Massively Multilingual Zero-Shot Text-to-Speech ..."))

Embed each utterance:

- A\_i for real accent.
- B\_i for “native” version.

Compute:

- A\_avg = mean\_i(A\_i)
- B\_avg = mean\_i(B\_i)
- C\_raw = B\_avg − A\_avg
- C = C\_raw / ||C\_raw||

Optionally, also average per-pair differences D\_i = B\_i − A\_i; then C = mean\_i(D\_i) normalized, which reduces some pair-specific noise.

### 6.3 Test if C really changes “accent” in embedding space

Before synthesis, test with **simple classifiers** :

1. Collect some **native American English** and **L2 English** samples from public corpora (e.g., L2-ARCTIC, Westbrook English Accent Dataset). ([Unilight](https://unilight.github.io/assets/pdf/phd-thesis.pdf "Pre-training Approaches for Voice Conversion to Address ..."))

2. Embed them with the same encoder and train a small classifier f(x) that predicts “native vs non-native” or “American vs non-American”.

3. For some of your A samples (L2 accent) compute:

4. Evaluate f(x(k)).

If P(native) consistently increases with k, you have evidence that C aligns with a **global accent direction** , not just random noise.

You can even use f(x(k)) to **calibrate k** , e.g.:

- pick k\_native so that P(native) ≈ 0.5–0.7;
- use k beyond that for “hyper-native” and below for “slightly accented”.

### 6.4 Feeding S + k·C to XTTS-v2

Once you are satisfied that C is doing something accent-like at the embedding level:

1. Choose a base speaker embedding S:

2. For each k (e.g., −0.5, 0, 0.25, 0.5, 0.75):

3. Supply S’(k) to XTTS-v2 as the speaker embedding (d-vector).

4. Synthesize:

You will likely observe:

- Small k: subtle accent shifts, some timbre/prosody changes.
- Large k: more drastic accent changes, but speaker identity may drift and artifacts may appear.

### 6.5 Evaluation

Use multiple axes:

- **Speaker similarity** : cosine similarity of embeddings between original you and S’(k) speech (should stay high for reasonable k).
- **Accent classifier output** : P(native) vs k (should increase with k up to a point).
- **ASR WER** : use a strong ASR model, ideally trained on native English, on your synthetic audio; FAC papers use WER reduction as a proxy for “more canonical pronunication”. ([Unilight](https://unilight.github.io/assets/pdf/phd-thesis.pdf "Pre-training Approaches for Voice Conversion to Address ..."))
- **Human listening tests** (even informal with friends): “Which of these sounds more native, and which sounds most like the same person?”

This is exactly how recent FAC work evaluates their methods.

* * *

## 7. Warnings and practical pitfalls

1. **Synthetic B is not ground truth**

2. **Accent is cross-lingual but not language-agnostic**

3. **Large |k| leaves the manifold**

4. **Accent vs intelligibility**

* * *

## 8. Bottom line

- Conceptually, you are in the same space as:

- People have not, as far as the literature shows, published exactly “personal accent delta = synthetic-native minus real-accent” as a reusable vector, but your proposal is a natural, lighter-weight variant of existing methods.

- With XTTS-v2 and its speaker encoder, plus a bit of embedding arithmetic and sanity-check classification, you can absolutely run the experiment you imagined and empirically test how linear and reusable accent vectors really are for your own voice.