# Project UCTF: An Open Research Program on Machine-Native AI Training Representations **URL:** \ **Category:** Research\ **Created:** [June 29, 2026, 2:15pm UTC](https://discuss.huggingface.co/t/project-uctf-an-open-research-program-on-machine-native-ai-training-representations/177240 "2026-06-29T14:15:19Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![k7007research](https://avatars.discourse-cdn.com/v4/letter/k/6bbea6/32.png) [@k7007research](https://discuss.huggingface.co/u/k7007research)\ **Post date:** [June 29, 2026, 2:15pm UTC](https://discuss.huggingface.co/t/project-uctf-an-open-research-program-on-machine-native-ai-training-representations/177240/1 "2026-06-29T14:15:19Z")
Project UCTF: An Open Research Program on Machine-Native AI Training Representations This post is a follow-up to my earlier concept proposal on the Universal Compressed Training Format (UCTF). The discussion that followed—especially the detailed technical feedback from @John6666—made me realize that the original concept combined several different research questions into a single proposal. Rather than trying to solve everything at once, I’ve decided to restructure the project into an open, hypothesis-driven research program. This project is open to feedback, criticism, collaboration, and independent replication. Every stage will be documented publicly, regardless of whether the results support or challenge the original idea. -– Core Research Question Can a machine-native intermediate representation serve as an alternative training representation for multilingual AI models, reducing cross-lingual semantic redundancy while preserving the information required for downstream generation? This is a hypothesis—not a conclusion. The purpose of Project UCTF is to investigate this question through a sequence of independent, measurable experiments. -– Research Philosophy Instead of attempting to prove UCTF from the beginning, each stage will investigate one specific research question. The goal is to let evidence shape the direction of the project. Positive results, negative results, and unexpected findings will all be published. If later experiments show that the hypothesis is incorrect or only partially correct, those results will still be considered valuable contributions. -– Why a Multi-Paper Research Program? The original concept covered multiple independent problems: - multilingual semantic representations - redundancy in multilingual corpora - machine-native representations - language-independent training - efficient AI training These deserve to be investigated separately. By splitting the work into independent papers: - Each paper has standalone value. - Earlier work remains useful even if later stages fail. - Community feedback can improve each stage before moving to the next. - The project evolves based on evidence rather than assumptions. -– Planned Research Roadmap Paper 1 Measuring Semantic Redundancy in Multilingual Training Corpora Research Question How much semantic redundancy actually exists across multilingual training datasets? Objective Measure rather than assume the problem. Possible initial evaluation resources may include multilingual benchmark datasets such as FLORES+, together with multilingual embedding baselines. The exact datasets and models will be selected after further literature review and community feedback. Deliverables - Redundancy measurements - Dataset statistics - Visualizations - Manual inspection of representative examples - Open notebook and reproducible analysis -– Paper 2 Characterizing Universal and Language-Specific Knowledge Research Question Which information is universal across languages, and which information is inherently language-dependent? Objective Develop an empirical taxonomy separating: - universal factual knowledge - mathematical knowledge - scientific knowledge - language-dependent structure - culture-dependent expressions - idioms - conversational nuances This paper defines what any machine-native representation must preserve. -– Paper 3 Design Requirements for a Machine-Native Intermediate Representation Research Question Given the findings from Papers 1 and 2, what should a UCTF-like representation contain? Objective Produce a formal design specification. This stage focuses on defining requirements—not implementing them. -– Paper 4 Prototype Machine-Native Representation Research Question Can a prototype intermediate representation satisfy the requirements identified in earlier stages? Objective Develop an experimental prototype and evaluate: - semantic preservation - compression behavior - reconstruction quality - degradation across multiple settings The implementation approach will be chosen based on the evidence gathered during Papers 1–3 rather than fixed in advance. -– Paper 5 Initial Training Validation Research Question Can a model trained on a machine-native representation achieve comparable downstream performance to conventional multilingual training? Objective Perform the first controlled comparison between: - conventional multilingual training - machine-native representation training Measurements may include: - dataset size - representation efficiency - training time - computational cost - downstream performance - observed limitations Regardless of the outcome, all results will be published. -– Open Research Principles This project follows several principles: - Every experiment should answer one question. - Conclusions should follow evidence. - Negative results are valuable. - All code, notebooks, and intermediate findings will be released publicly whenever practical. - Community feedback will be incorporated before major milestones. - Credit will always be given to contributors whose feedback shapes the project. -– How You Can Contribute Contributions are welcome at every stage. Examples include: - multilingual NLP expertise - literature recommendations - dataset suggestions - methodology review - experiment replication - constructive criticism - identifying prior work - identifying flaws before implementation Finding weaknesses early is just as valuable as proposing improvements. -– Current Status The project is currently entering Paper 1. The immediate objective is not to build UCTF, but to investigate whether multilingual semantic redundancy exists at a scale that justifies further research. Community suggestions regarding evaluation datasets, multilingual benchmarks, encoder baselines, and experimental methodology are highly appreciated. -– Acknowledgment Special thanks to @John6666 for providing detailed technical feedback on the original UCTF proposal. Their comments significantly influenced the restructuring of this work into a staged, hypothesis-driven research program. -– This is an open research effort. The goal is not to prove UCTF right. The goal is to investigate whether the underlying hypothesis is worth pursuing. — K7007 #UCTF #MachineLearning #NLP #MultilingualAI #OpenResearch #RepresentationLearning #Research ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [June 30, 2026, 12:18pm UTC](https://discuss.huggingface.co/t/project-uctf-an-open-research-program-on-machine-native-ai-training-representations/177240/2 "2026-06-30T12:18:35Z")
For now, here are some general things that might be useful for Paper 1: * * * The staged roadmap makes the project much easier to reason about. For **Paper 1** , I would try to make the first artifact a reusable **measurement map** , not a proof of UCTF and not a representation design yet. My direct suggestion would be: > Separate the measurement paths. Use clean aligned data for sanity checks, bitext/mined-bitext resources for translation-equivalent redundancy, corpus-like web data for messy training-data behavior, multiple encoders for agreement/disagreement, and a small manual audit set to calibrate what the scores actually mean. A useful first artifact could be intentionally modest: > **FLORES+ sanity check + one tiny corpus-like streaming sample + exact duplicate baseline + one lightweight multilingual encoder + threshold sweep + `candidate_pairs.csv` + `manual_audit_template.csv`.** Then add larger corpora, stronger encoders such as SONAR, and scalable dedup tools only after the audit protocol is stable. * * * ## 1. A decision tree may be more useful than one more list of open questions Instead of trying to pick one universal dataset or one universal encoder, I would split the work by what each branch is meant to answer. | If the goal is… | Start with… | What this answers | | --- | --- | --- | | Pipeline sanity check | [FLORES+](https://huggingface.co/datasets/openlanguagedata/flores_plus) | Can the pipeline recover known cross-lingual equivalents? | | Translation-equivalent redundancy | [OPUS-100](https://opus.nlpl.eu/OPUS-100.php), [WikiMatrix](https://arxiv.org/abs/1907.05791), [CCMatrix](https://arxiv.org/abs/1911.04944) | Can translation-equivalent pairs be recovered or mined? | | Messy corpus-like redundancy | [FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2), [CulturaX](https://huggingface.co/datasets/uonlp/CulturaX), [OSCAR](https://huggingface.co/datasets/oscar-corpus/OSCAR-2201), [GlotCC](https://huggingface.co/datasets/cis-lmu/GlotCC-V1) | What happens in web/pretraining-like multilingual data? | | Non-semantic baseline | exact duplicate count, fuzzy/MinHash near-dedup, [DataTrove](https://github.com/huggingface/datatrove) | How much is already explained by textual duplication? | | Semantic candidate mining | [SemDeDup](https://arxiv.org/abs/2303.09540), [NeMo Curator semantic dedup](https://docs.nvidia.com/nemo/curator/curate-text/process-data/deduplication/semdedup), [Sentence Transformers paraphrase mining](https://sbert.net/examples/sentence_transformer/applications/paraphrase-mining/README.html) | Which pairs/clusters are meaning-similar candidates? | | Encoder reliability | [SONAR](https://github.com/facebookresearch/SONAR), LaBSE, multilingual-E5, LASER/Sentence Transformers | Where do encoders agree or disagree? | | Low-resource analysis | language-ID audit + corpus-quality audit | Are the scores interpretable for low-resource slices? | The main distinction I would keep is: > Embedding similarity is a **candidate generator** , not the ground truth for semantic redundancy. > **More detailed decision tree** > > A more explicit branch structure could look like this: > > | Branch | Useful first move | Notes | > | --- | --- | --- | > | Controlled aligned sanity check | Use FLORES+ dev/devtest for a few languages | Good for checking that known translations are close | > | Translation-equivalent branch | Use OPUS-100 / WikiMatrix / CCMatrix-like references | Keep this separate from paraphrase or topic similarity | > | Corpus-like branch | Take a tiny streaming sample from FineWeb2 / CulturaX / OSCAR / GlotCC-like data | Record dataset version, filtering, dedup status, and source metadata | > | Exact duplicate baseline | Hash normalized text | Prevent exact duplicates from being counted as semantic discoveries | > | Near-duplicate baseline | Add MinHash/fuzzy dedup later | Useful, but not required for the first small notebook | > | Semantic candidate mining | Use one lightweight multilingual encoder first | Add stronger encoders after the pipeline is inspectable | > | Encoder agreement | Compare 2–4 encoders later | Disagreement is a result, not only noise | > | Manual audit | Export candidate pairs and label match type | This is what makes the numbers interpretable | > | Low-resource slice | Check language-ID and basic text quality | A low score may mean weak measurement, not low redundancy | > > This keeps the reply from becoming a list of demands. It gives the project author routes they can choose from. * * * ## 2. Dataset roles: sanity check, bitext reference, corpus-like sample I would avoid using one dataset for every question. **FLORES+** is useful as a controlled sanity check, but I would not use it alone to estimate training-corpus redundancy. **OPUS / WikiMatrix / CCMatrix** are useful for the translation-equivalent branch. **FineWeb2 / CulturaX / OSCAR / GlotCC** are closer to messy corpus-like conditions, but their filtering and deduplication history should be reported. > **Dataset role map** > > | Resource type | Examples | Good for | Not enough for | > | --- | --- | --- | --- | > | Controlled aligned evaluation | [FLORES+](https://huggingface.co/datasets/openlanguagedata/flores_plus) | sanity check, known translation pairs, threshold calibration on clean aligned data | messy training-corpus redundancy | > | Parallel / mined bitext | [OPUS-100](https://opus.nlpl.eu/OPUS-100.php), [WikiMatrix](https://arxiv.org/abs/1907.05791), [CCMatrix](https://arxiv.org/abs/1911.04944) | translation-equivalent redundancy | paraphrase, boilerplate, same-topic-only similarity | > | Corpus-like multilingual web data | [FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2), [CulturaX](https://huggingface.co/datasets/uonlp/CulturaX), [OSCAR](https://huggingface.co/datasets/oscar-corpus/OSCAR-2201), [GlotCC](https://huggingface.co/datasets/cis-lmu/GlotCC-V1) | messy pretraining-like samples | raw-web claims unless processing history is included | > | Manual audit sample | exported candidate pairs | threshold calibration, false positives, label quality | corpus-scale estimates by itself | > > ### FLORES+ > > [FLORES+](https://huggingface.co/datasets/openlanguagedata/flores_plus) is useful because it is small, multilingual, aligned, and designed for multilingual NLP / MT evaluation. I would use it for: > > - known aligned sentence pairs > - positive-pair vs random-negative score distributions > - checking whether encoders place translations near each other > - threshold sanity checks > > But I would keep this result separate from corpus-like redundancy. > > ### OPUS-100 / WikiMatrix / CCMatrix > > These are useful for **translation-equivalent redundancy**. > > - [OPUS-100](https://opus.nlpl.eu/OPUS-100.php) is useful as an English-centric parallel-data reference. Since it is English-centric, I would not use it alone to characterize non-English ↔ non-English redundancy. > - [WikiMatrix](https://arxiv.org/abs/1907.05791) is useful as a multilingual sentence-embedding / mined-bitext reference. > - [CCMatrix](https://arxiv.org/abs/1911.04944) is useful as a large-scale margin-based bitext-mining reference. > > I would report this branch separately from broader semantic redundancy. > > ### FineWeb2 / CulturaX / OSCAR / GlotCC > > These are better for messy corpus-like behavior. > > Useful references: > > - [FineWeb2 dataset card](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2) > - [FineWeb2 paper](https://arxiv.org/abs/2506.20920) > - [FineWeb2 GitHub](https://github.com/huggingface/fineweb-2) > - [CulturaX dataset card](https://huggingface.co/datasets/uonlp/CulturaX) > - [CulturaX paper](https://arxiv.org/abs/2309.09400) > - [OSCAR dataset card](https://huggingface.co/datasets/oscar-corpus/OSCAR-2201) > - [GlotCC dataset card](https://huggingface.co/datasets/cis-lmu/GlotCC-V1) > - [GlotCC paper](https://arxiv.org/abs/2410.23825) > - [GlotCC GitHub](https://github.com/cisnlp/GlotCC) > > FineWeb2 is especially interesting because its pipeline is documented, and the [FineWeb2 repository](https://github.com/huggingface/fineweb-2) describes per-language global deduplication and `minhash_cluster_size` metadata. That metadata may be useful as a non-semantic near-dedup signal. > > But this also means the measurement should be framed carefully: > > > If using FineWeb2-like data, the measurement is on **remaining redundancy after an existing filtering/deduplication pipeline** , not raw web redundancy. > > For corpus-like samples, I would record: > > - dataset name > - dataset version > - source field if available > - language label > - whether the data is raw / filtered / deduplicated > - language-ID method if documented > - existing dedup metadata if available > - raw text > - cleaned text > - stable unit ID * * * ## 3. Redundancy types should be separated before measuring “Semantic redundancy” can mean several different things. I would not collapse them too early. A small manual label set could make the measurement much easier to interpret. | Label | Meaning | | --- | --- | | `exact_duplicate` | identical or normalized-identical text | | `near_duplicate` | formatting, template, minor edits | | `translation_equivalent` | same content across languages | | `paraphrase_equivalent` | same meaning, different wording | | `same_fact_different_context` | same factual content embedded differently | | `boilerplate_template` | repeated legal text, navigation, footer, SEO template | | `same_topic_only` | same entities/topic, different claim | | `entity_number_date_mismatch` | high similarity but changed entity, number, date, relation | | `contradiction` | same entities but opposite claim | | `uncertain` | not enough context | > **Why this matters** > > High embedding similarity may mean: > > - real translation equivalence > - paraphrase > - boilerplate > - same topic only > - near duplicate > - template reuse > - copied structure with changed facts > - false positive > - contradiction > > The useful artifact is not only “how many pairs are high similarity,” but **what kind** of high similarity they are. > > Nearby references: > > - [SemDeDup](https://arxiv.org/abs/2303.09540) is useful because it treats semantic duplicates as measurable objects. > - [NeMo Curator semantic deduplication](https://docs.nvidia.com/nemo/curator/curate-text/process-data/deduplication/semdedup) gives a practical embedding/clustering/thresholding shape. > - [Sentence Transformers paraphrase mining](https://sbert.net/examples/sentence_transformer/applications/paraphrase-mining/README.html) is useful for small candidate-mining experiments. > - [Sentence Transformers hard-negative utilities](https://www.sbert.net/docs/package_reference/util/hard_negatives.html) are a reminder that “similar but not positive” examples exist. For Paper 1, those become high-score false positives worth auditing. * * * ## 4. Baseline ladder: exact → near → semantic Before embedding-based mining, I would keep the baseline ladder simple. 1. **Exact duplicate count** 2. **Near duplicate / MinHash / fuzzy dedup** 3. **Embedding-based semantic candidate mining** 4. **Encoder agreement / disagreement** 5. **Manual audit** This makes the final measurement easier to interpret. > **Baseline ladder details** > > ### Level 1: exact duplicate baseline > > Start with a very small baseline: > > - normalize whitespace > - optionally lowercase where appropriate > - hash normalized text > - group by hash > - count exact duplicates > > This is not enough, but it prevents exact duplicates from being reported as semantic discoveries. > > ### Level 2: near-duplicate baseline > > For larger runs, fuzzy / MinHash near-dedup becomes useful. > > Practical references: > > - [DataTrove](https://github.com/huggingface/datatrove) > - [DataTrove MinHash duplicate IDs issue](https://github.com/huggingface/datatrove/issues/213) > - [DataTrove MinHash example](https://github.com/huggingface/datatrove/blob/main/examples/minhash_deduplication.py) > > The DataTrove duplicate-ID issue is useful because it shows a real audit-trail problem: if a pipeline removes documents or forms clusters, later readers need IDs and cluster membership, not only aggregate counts. > > ### Level 3: semantic candidate mining > > Only then add embedding-based mining. > > References: > > - [SemDeDup paper](https://arxiv.org/abs/2303.09540) > - [SemDeDup GitHub](https://github.com/facebookresearch/SemDeDup) > - [NeMo Curator semantic deduplication docs](https://docs.nvidia.com/nemo/curator/curate-text/process-data/deduplication/semdedup) > - [Sentence Transformers paraphrase mining](https://sbert.net/examples/sentence_transformer/applications/paraphrase-mining/README.html) > > ### Level 4: encoder agreement / disagreement > > If multiple encoders agree, that is a stronger candidate. > > If only one encoder fires, that is still useful information. > > ### Level 5: manual audit > > Manual audit is the calibration layer. It turns “embedding neighbors” into interpretable redundancy categories. * * * ## 5. Encoder baselines: useful instruments, not ground truth I would not let one multilingual encoder define semantic redundancy. For the first notebook, a lightweight Sentence Transformers model may be enough to validate the pipeline. SONAR can be added later as a stronger cross-lingual sentence-level baseline. > **Encoder baseline details** > > Possible baseline set: > > | Encoder / family | Why useful | Caveat | > | --- | --- | --- | > | lightweight Sentence Transformers multilingual model | easy first notebook baseline | not necessarily strongest cross-lingual model | > | multilingual-E5-small / base / large | useful multilingual embedding baseline | prefix / preprocessing rules matter | > | LaBSE | classic multilingual sentence embedding / bitext retrieval baseline | may not be ideal for every semantic-similarity subtype | > | SONAR | strong cross-lingual sentence-level baseline | heavier dependency; should be one instrument, not ground truth | > | LASER / LASER-like | useful historical bitext-mining baseline | check current implementation/model coverage | > > Useful references: > > - [SONAR GitHub](https://github.com/facebookresearch/SONAR) > - [SONAR paper](https://arxiv.org/abs/2308.11466) > - [SONAR on Hugging Face](https://huggingface.co/facebook/SONAR) > - [LaBSE model card](https://huggingface.co/sentence-transformers/LaBSE) > - [multilingual-e5-small model card](https://huggingface.co/intfloat/multilingual-e5-small) > - [MTEB paper](https://arxiv.org/abs/2210.07316) > - [MTEB on Hugging Face](https://huggingface.co/mteb) > - [MMTEB paper](https://arxiv.org/abs/2502.13595) > - [Sentence Transformers docs](https://sbert.net/) > > For every encoder, I would save: > > - exact model name > - model revision if possible > - pooling method > - normalization method > - input prefix / prompt rule > - similarity function > - language coverage notes > - whether the unit is sentence, paragraph, or document > > This is especially important for E5-style models, where prefix / prompt conventions can affect reproducibility. * * * ## 6. A small first notebook could optimize for inspectability, not scale The first notebook does not need to be a full web-scale experiment. It can be a measurement rehearsal. A useful v0 could be: | Step | Action | Output | | --- | --- | --- | | 1 | State purpose and non-goals | readers know this is not a global estimate | | 2 | Load FLORES+ for aligned sanity check | aligned sentence table | | 3 | Load one tiny corpus-like sample with `streaming=True` | corpus sample table | | 4 | Keep `raw_text`, `cleaned_text`, `source`, `language`, `unit_id` | audit trail | | 5 | Run exact duplicate baseline | `exact_duplicate_summary.csv` | | 6 | Encode with one lightweight multilingual model | embedding metadata | | 7 | Mine candidate pairs | `candidate_pairs.csv` | | 8 | Sweep thresholds | `threshold_summary.csv` | | 9 | Export manual audit template | `manual_audit_template.csv` | | 10 | Write observations | `observations.md` | > **Notebook skeleton details** > > Useful implementation references: > > - [HF Datasets streaming docs](https://huggingface.co/docs/datasets/en/stream) > - [Map-style vs IterableDataset docs](https://huggingface.co/docs/datasets/en/about_mapstyle_vs_iterable) > - [Sentence Transformers semantic search](https://sbert.net/examples/sentence_transformer/applications/semantic-search/README.html) > - [Sentence Transformers paraphrase mining](https://sbert.net/examples/sentence_transformer/applications/paraphrase-mining/README.html) > > A simple mining branch: > > | If sample size is… | Candidate mining method | > | --- | --- | > | very small | all-pairs cosine is transparent and easy to audit | > | moderate | chunked `semantic_search` / `paraphrase_mining` | > | large | FAISS / DataTrove / NeMo-style scaling later | > > I would not start with the biggest possible corpus. Large dedup quickly becomes an engineering benchmark: tokenization cost, RAM, I/O, distributed execution, file handles, and cluster bookkeeping. > > Useful engineering reminders: > > - [DataTrove duplicate IDs / clusters issue](https://github.com/huggingface/datatrove/issues/213) > - [DataTrove local machine hanging issue](https://github.com/huggingface/datatrove/issues/222) > - [DataTrove too many open files issue](https://github.com/huggingface/datatrove/issues/335) > > For Paper 1, I would first stabilize the measurement and audit protocol, then scale. * * * ## 7. Manual audit: export candidate pairs, not only aggregate counts The most useful v0 artifact may be `candidate_pairs.csv` plus `manual_audit_template.csv`. Suggested columns: | Column | Purpose | | --- | --- | | `pair_id` | stable audit unit | | `unit_id_a`, `unit_id_b` | traceability | | `source_a`, `source_b` | corpus/source analysis | | `lang_a`, `lang_b` | language-pair matrix | | `raw_text_a`, `raw_text_b` | auditability | | `cleaned_text_a`, `cleaned_text_b` | reproducibility | | `encoder` | model comparison | | `score` | threshold analysis | | `threshold_band` | calibration | | `candidate_type` | exact / near / semantic candidate | | `match_label` | human-filled label | | `notes` | error notes | > **Manual audit sampling** > > Audit samples should include more than top-scoring pairs: > > - high-score / all-encoder agreement > - high-score / one-encoder-only > - near-threshold cases > - same-script language pairs > - cross-script language pairs > - low-resource slices > - suspected boilerplate/template clusters > - same-topic false positives > - entity/date/number mismatches > - contradictions > > This is where STS-style work is useful as background. [SemEval-2017 Task 1](https://arxiv.org/abs/1708.00055) is not the same task as corpus redundancy measurement, but it is a useful reference for multilingual / cross-lingual semantic similarity judgments. * * * ## 8. Low-resource slices need LangID and corpus-quality checks For low-resource languages, I would be cautious. A low redundancy score may mean: - actual low redundancy - language-ID noise - weak corpus quality - domain mismatch - script / orthography issue - closely related language confusion - macrolanguage vs variety confusion - weak encoder coverage - threshold calibrated on high-resource languages So I would treat language-ID and basic corpus quality as part of Paper 1’s measurement, not just preprocessing. > **Low-resource and corpus-quality references** > > Useful references: > > - [Language ID in the Wild](https://aclanthology.org/2020.coling-main.579/) shows that LangID can look strong on held-out test data while failing badly on web-crawl corpora for lower-resource languages. > - [Quality at a Glance](https://aclanthology.org/2022.tacl-1.4/) audits many language-specific corpora and reports systematic quality issues, especially in lower-resource settings. > - [GlotLID paper](https://arxiv.org/abs/2310.16248) discusses low-resource language-ID challenges such as incorrect metadata, leakage from high-resource languages, closely related languages, macrolanguages/varieties, and noisy data. > - [GlotLID GitHub](https://github.com/cisnlp/GlotLID) > - [GlotCC paper](https://arxiv.org/abs/2410.23825) > - [GlotCC GitHub](https://github.com/cisnlp/GlotCC) > > A small low-resource audit could check: > > - Is the language label plausible? > - Is the text usable natural language? > - Is it boilerplate, OCR noise, navigation text, or code-mixed text? > - Is it actually another closely related language? > - Do different encoders agree? > - Does the threshold seem calibrated for that language/script? > > Even a small manual slice can prevent overinterpreting weak scores. * * * ## 9. Reporting format: a redundancy map, not only one percentage I would avoid reporting only: > “X% semantic redundancy” A single number hides too many choices: dataset, text unit, encoder, threshold, language pair, processing history, duplicate type, and audit precision. More reusable outputs: | Output | Why useful | | --- | --- | | threshold curves | shows sensitivity to cutoff | | language-pair matrix | shows cross-lingual structure | | encoder agreement table | shows measurement stability | | cluster-size distribution | shows whether redundancy is concentrated | | dataset/source breakdown | separates benchmark vs corpus-like behavior | | exact/near/semantic split | prevents conflating duplicate types | | manual-audit label counts | calibrates score interpretation | | true-positive examples | helps future readers understand success cases | | false-positive examples | helps future readers understand failure modes | | false-negative examples | important if using controlled aligned data | | `observations.md` | preserves qualitative lessons | > **How to interpret possible outcomes** > > | Observation | Possible interpretation | > | --- | --- | > | FLORES+ works, corpus-like data fails | pipeline can recover clean aligned pairs, but messy corpus measurement is harder | > | exact/near duplicates dominate | semantic redundancy may be smaller than expected after textual dedup | > | boilerplate dominates | web-template redundancy should be separated from semantic content redundancy | > | encoders disagree strongly | encoder choice is part of the measurement | > | low-resource scores are unstable | LangID / quality / encoder coverage need audit | > | translation-equivalent branch works but paraphrase branch is weak | later UCTF experiments should not assume all redundancy types are equally compressible | > > This makes Paper 1 useful even if the result is negative or mixed. * * * ## 10. Minimal roadmap One possible route: | Stage | Goal | Main artifact | | --- | --- | --- | | v0 | Measurement sanity notebook | FLORES+ + tiny corpus sample + candidate pairs + audit CSV | | v1 | Add corpus realism | FineWeb2 / CulturaX / OSCAR / GlotCC-like sample | | v2 | Add encoder comparison | lightweight ST model + E5 + LaBSE + SONAR | | v3 | Add low-resource audit slice | LangID / corpus-quality checks | | v4 | Scale only after audit protocol is stable | DataTrove / NeMo / FAISS-style scaling | The important part is that each stage remains useful even if later stages change direction. > **Why this helps later UCTF work** > > This would show where multilingual redundancy is: > > - exact > - near-duplicate > - translation-equivalent > - paraphrastic > - same-fact but context-dependent > - boilerplate/template-driven > - same-topic-only > - encoder-dependent > - unreliable under current measurement tools > > That gives later UCTF experiments a clearer target. > > If there is a strong redundancy signal, later papers can test representation bottlenecks. > If there is not, that is also useful: it means the project should narrow the claim or change the experimental target. > If the result depends heavily on language, domain, dataset, or encoder, that becomes part of the map. ---
**Author:** ![k7007research](https://avatars.discourse-cdn.com/v4/letter/k/6bbea6/32.png) [@k7007research](https://discuss.huggingface.co/u/k7007research)\ **Post date:** [July 1, 2026, 12:04am UTC](https://discuss.huggingface.co/t/project-uctf-an-open-research-program-on-machine-native-ai-training-representations/177240/3 "2026-07-01T00:04:21Z")
Thank you, @John6666. This is more structured guidance than I expected, and I want to use it carefully rather than rush past it. I’ll follow the staged approach you outlined: v0 first — FLORES+ sanity check, one small corpus-like sample, exact duplicate baseline, one lightweight encoder, threshold sweep, and candidate\_pairs.csv with manual\_audit\_template.csv. I’ll separate the redundancy types before measuring rather than collapsing them into a single score, and I’ll treat embedding similarity as a candidate generator — not ground truth. The decision tree is especially helpful. I’ll use it to keep each measurement branch answering one specific question. I’ll publish the v0 notebook here when it’s ready for review. Thank you for treating this as a serious research effort. — K7007 ---
**Author:** ![k7007research](https://avatars.discourse-cdn.com/v4/letter/k/6bbea6/32.png) [@k7007research](https://discuss.huggingface.co/u/k7007research)\ **Post date:** [July 1, 2026, 9:02am UTC](https://discuss.huggingface.co/t/project-uctf-an-open-research-program-on-machine-native-ai-training-representations/177240/5 "2026-07-01T09:02:59Z")
Thank you @davidwarner234t. Agreed completely. The plan for v0 follows exactly this — small auditable pipeline first, embeddings as candidate generator only, manual validation before scaling. That sequence is now the core principle of Paper 1. — K7007 ---
**Author:** ![masonreed11](https://avatars.discourse-cdn.com/v4/letter/m/ccd318/32.png) [@masonreed11](https://discuss.huggingface.co/u/masonreed11)\ **Post date:** [July 1, 2026, 9:51am UTC](https://discuss.huggingface.co/t/project-uctf-an-open-research-program-on-machine-native-ai-training-representations/177240/6 "2026-07-01T09:51:03Z")
The staged approach is a good move. Measuring the problem first before proposing a solution will make the results much stronger. I’d pay extra attention to baselines and evaluation methods so the findings are comparable with existing multilingual NLP research. Even proving where UCTF does **not** work would be useful. ---
**Author:** ![k7007research](https://avatars.discourse-cdn.com/v4/letter/k/6bbea6/32.png) [@k7007research](https://discuss.huggingface.co/u/k7007research)\ **Post date:** [July 18, 2026, 7:21pm UTC](https://discuss.huggingface.co/t/project-uctf-an-open-research-program-on-machine-native-ai-training-representations/177240/7 "2026-07-18T19:21:40Z")
Paper 1 v0 is complete. Code, data, and results are now available on GitHub: [GitHub - kesavanvjp0007/project-uctf · GitHub](http://github.com/kesavanvjp0007/project-uctf) — feedback welcome. ---
**Author:** ![k7007research](https://avatars.discourse-cdn.com/v4/letter/k/6bbea6/32.png) [@k7007research](https://discuss.huggingface.co/u/k7007research)\ **Post date:** [July 18, 2026, 7:33pm UTC](https://discuss.huggingface.co/t/project-uctf-an-open-research-program-on-machine-native-ai-training-representations/177240/8 "2026-07-18T19:33:30Z")
**Paper 1 — v0 Results Summary** **Measuring Semantic Redundancy in Multilingual Training Corpora** * * * The first experiment is complete. Here are the results. * * * **Dataset** 20 semantically aligned sentences across 4 languages: English, Chinese (Simplified), Japanese, Russian 80 total sentence units. Encoder: paraphrase-multilingual-MiniLM-L12-v2 * * * **Finding 1 — Exact Duplicate Baseline** Exact text duplicate rate: 0.0% Text-level deduplication detects zero cross-lingual redundancy by design. The same sentence in English and Chinese shares no character overlap. This confirms that semantic redundancy measurement requires embedding-based approaches, not text matching. * * * **Finding 2 — Cross-Language Semantic Similarity** Sample result for sentence: ‘The sun rises in the east’ English → Chinese: 0.969 English → Japanese: 0.978 English → Russian: 0.985 Chinese → Japanese: 0.984 Chinese → Russian: 0.969 Japanese → Russian: 0.985 Average cross-language similarity across all 20 sentences: Mean: 0.9570 Minimum: 0.8529 Maximum: 0.9933 Four completely different scripts. Zero text overlap. 95.7% semantic similarity on average. * * * **Finding 3 — Candidate Pair Mining** At similarity threshold 0.85: Total candidate pairs found: 116 True positives (same sentence): 116 False positives (different sentence): 0 Precision: 100% * * * **Finding 4 — Threshold Sweep** Threshold | Pairs Found | Precision 0.70 | 120 | 100.0% 0.75 | 120 | 100.0% 0.80 | 119 | 100.0% 0.85 | 116 | 100.0% 0.90 | 103 | 100.0% 0.95 | 85 | 100.0% 0.99 | 7 | 100.0% 100% precision maintained across all tested thresholds. Zero false positives at any threshold level. * * * **Interpretation** These results confirm three things at v0 scale: 1. Text-level deduplication is blind to cross-lingual semantic redundancy. Standard dedup pipelines would miss all 116 semantically equivalent pairs in this dataset. 2. Multilingual sentence encoders can reliably detect cross-lingual semantic equivalence with high precision across English, Chinese, Japanese, and Russian — including across completely different scripts and writing systems. 3. The redundancy is substantial. At 95.7% average semantic similarity, the same information encoded across 4 languages represents significant measurable redundancy in training data. * * * **Limitations of v0** This is a controlled small-scale experiment — 20 sentences, manually curated, perfect translations. Real training corpus behavior will differ: - Larger corpora will contain noisy, imperfect translations - Low-resource languages may show encoder degradation - Web-crawled data introduces domain and quality variation - 20 sentences is not sufficient to generalize — v1 will scale to corpus-like data These limitations are documented. v1 will follow John6666’s recommended decision tree — adding corpus-like streaming samples, additional encoders, and a low-resource language audit slice. * * * **Files available:** candidate\_pairs.csv — 116 semantic candidate pairs threshold\_summary.csv — precision at 7 thresholds flores\_4lang.csv — aligned dataset manual\_audit\_template.csv — ready for human audit uctf\_paper1\_results.png — visualizations GitHub link will be shared once the moderation hold on the previous post is cleared. * * * Feedback on methodology, limitations, and next steps welcome. — K7007 ---
**Author:** ![k7007research](https://avatars.discourse-cdn.com/v4/letter/k/6bbea6/32.png) [@k7007research](https://discuss.huggingface.co/u/k7007research)\ **Post date:** [July 19, 2026, 6:04am UTC](https://discuss.huggingface.co/t/project-uctf-an-open-research-program-on-machine-native-ai-training-representations/177240/9 "2026-07-19T06:04:52Z")
**Paper 1 — v1 Update** **Real Corpus Results: 1500 Parallel Sentence Pairs** * * * v1 is complete. Results differ significantly from v0 — which is expected and informative. * * * **Dataset** Source: OPUS-100 parallel corpus (Helsinki-NLP) 1500 sentence pairs across 3 language pairs: English-Chinese, English-Japanese, English-Russian 3000 total sentence units. Encoder: paraphrase-multilingual-MiniLM-L12-v2 (same as v0) * * * **v0 vs v1 Comparison** Metric | v0 (20 manual) | v1 (1500 real) Mean similarity | 0.9570 | 0.7474 Min similarity | 0.8529 | -0.0447 Max similarity | 0.9933 | 1.0000 Pairs at 0.85 | 116 (100%) | 541 (36.1%) * * * **By Language Pair** English-Chinese: mean=0.7948 English-Russian: mean=0.7886 English-Japanese: mean=0.6589 English-Japanese shows notably lower similarity than the other pairs — likely reflecting structural divergence between Japanese and English grammar rather than semantic difference. * * * **Key Findings** 1. Real corpus data is significantly noisier than controlled translations. Mean similarity dropped from 95.7% to 74.7% — expected but important to quantify. 2. Language pair matters. English-Japanese similarity (65.9%) is substantially lower than English-Chinese (79.5%) and English-Russian (78.9%). Compression approaches cannot assume uniform redundancy across all language pairs. 3. Negative similarity cases exist (-0.0447). Some translation pairs diverge enough that the encoder produces near-zero or negative cosine similarity. These may represent imperfect translations, domain mismatches, or encoder limitations on structurally distant language pairs. 4. Substantial redundancy still exists at scale. Even in noisy real corpus conditions, 67.8% of pairs exceed 0.70 similarity threshold — confirming that cross-lingual semantic redundancy is measurable and significant at corpus scale. * * * **Limitations of v1** - OPUS-100 is a parallel translation corpus — cleaner than real web-crawled training data - Only 3 language pairs tested — no low-resource languages yet - Single encoder used — v2 will compare multiple encoders - Negative similarity cases need manual inspection * * * **GitHub** Code, data, and visualizations for both v0 and v1 now available at: > **[GitHub - kesavanvjp0007/project-uctf](https://github.com/kesavanvjp0007/project-uctf)** > > Contribute to kesavanvjp0007/project-uctf development by creating an account on GitHub. * * * Feedback on the English-Japanese divergence and negative similarity cases especially welcome. — K7007 ---
**Author:** ![k7007research](https://avatars.discourse-cdn.com/v4/letter/k/6bbea6/32.png) [@k7007research](https://discuss.huggingface.co/u/k7007research)\ **Post date:** [July 19, 2026, 7:11am UTC](https://discuss.huggingface.co/t/project-uctf-an-open-research-program-on-machine-native-ai-training-representations/177240/10 "2026-07-19T07:11:50Z")
Paper 1 v2 complete. Compared 4 encoders (MiniLM, mE5, LaBSE, mpnet) on 1500 real parallel pairs using GPU. mE5-small emerged as the strongest baseline — mean similarity 0.8820, std deviation 0.0455, zero negative similarity cases, and near-elimination of the English-Japanese divergence gap observed in v1. ---
**Author:** ![k7007research](https://avatars.discourse-cdn.com/v4/letter/k/6bbea6/32.png) [@k7007research](https://discuss.huggingface.co/u/k7007research)\ **Post date:** [July 19, 2026, 7:21am UTC](https://discuss.huggingface.co/t/project-uctf-an-open-research-program-on-machine-native-ai-training-representations/177240/11 "2026-07-19T07:21:30Z")
mE5-small demonstrated strong robustness to three types of realistic training data noise — character-level errors (96.43% mean similarity to clean), word-order shuffling (97.50%), and code-switching (94.87%). All noise types maintained above 94% similarity to clean text, with 98.7–100% of pairs exceeding the 0.85 threshold. This suggests mE5 is a viable encoder for real-world noisy multilingual training corpora where perfect translations cannot be assumed. ---
**Author:** ![k7007research](https://avatars.discourse-cdn.com/v4/letter/k/6bbea6/32.png) [@k7007research](https://discuss.huggingface.co/u/k7007research)\ **Post date:** [July 19, 2026, 7:28am UTC](https://discuss.huggingface.co/t/project-uctf-an-open-research-program-on-machine-native-ai-training-representations/177240/12 "2026-07-19T07:28:49Z")
mE5-small maintained strong semantic similarity detection for low-resource languages — Sinhala (0.9083), Nepali (0.9040), and Kyrgyz (0.9013) — performing comparably to high-resource language pairs like English-Chinese (0.8694) and English-Japanese (0.8630). This suggests mE5 may be a viable encoder baseline for UCTF even in low-resource settings, though the higher standard deviation for Kyrgyz (0.0582 vs 0.0410 for Sinhala) indicates increased variability that warrants further investigation. ---
**Author:** ![k7007research](https://avatars.discourse-cdn.com/v4/letter/k/6bbea6/32.png) [@k7007research](https://discuss.huggingface.co/u/k7007research)\ **Post date:** [July 19, 2026, 7:56am UTC](https://discuss.huggingface.co/t/project-uctf-an-open-research-program-on-machine-native-ai-training-representations/177240/13 "2026-07-19T07:56:37Z")
At corpus scale (10,000 pairs, 20,000 sentences, 5 languages), mE5-small encoded 20,000 sentences in 13.2 seconds (1,515 sentences/second) using only 458 MB VRAM on a consumer RTX 3050 GPU — demonstrating practical feasibility for resource-constrained environments. Critically, low-resource languages (Sinhala: 91.84%, Nepali: 90.41%) outperformed high-resource languages (Chinese: 87.55%, Japanese: 86.67%) at corpus scale, a counterintuitive finding that warrants further investigation in Paper 2. ---
**Author:** ![k7007research](https://avatars.discourse-cdn.com/v4/letter/k/6bbea6/32.png) [@k7007research](https://discuss.huggingface.co/u/k7007research)\ **Post date:** [July 19, 2026, 10:22am UTC](https://discuss.huggingface.co/t/project-uctf-an-open-research-program-on-machine-native-ai-training-representations/177240/14 "2026-07-19T10:22:53Z")
**Project UCTF — Paper 1 Complete Results** **Measuring Semantic Redundancy in Multilingual Training Corpora** * * * Paper 1 is complete.. Full PDF report and all code, data, and notebooks are available on GitHub: > **[GitHub - kesavanvjp0007/project-uctf](https://github.com/kesavanvjp0007/project-uctf)** > > Contribute to kesavanvjp0007/project-uctf development by creating an account on GitHub. * * * **What we measured** We tested whether cross-lingual semantic redundancy in multilingual AI training data is measurable, substantial, and encoder-dependent — and whether an embedding-based approach can detect it reliably. * * * **Experiment overview** Version | Scale | Setup | Mean similarity v0 | 20 pairs | Manual aligned, 4 languages, MiniLM | 0.9570 v1 | 1,500 pairs | OPUS-100 real corpus, MiniLM | 0.7474 v2 | 1,500 pairs | 4 encoders compared on GPU | 0.8820 (mE5) v3a | 900 sentences | 3 noise types, mE5 | 0.9627 avg v3b | 600 pairs | Nepali, Sinhala, Kyrgyz, mE5 | 0.9045 avg v4 | 10,000 pairs | 5 languages, corpus scale, GPU | 0.8948 Total pairs analyzed: 13,320 Languages tested: English, Chinese, Japanese, Russian, Nepali, Sinhala, Kyrgyz Hardware: NVIDIA RTX 3050 4GB (consumer laptop GPU) * * * **Finding 1 — Text deduplication fails completely** Exact duplicate detection rate: 0.0% Standard text-level deduplication finds zero cross-lingual redundancy by design. The same sentence in English and Chinese shares no character overlap. This confirms that semantic redundancy measurement requires embedding-based approaches — not text matching. * * * **Finding 2 — mE5-small is the strongest encoder baseline** Encoder comparison at 1,500 pairs (v2): Encoder | Eng-Chi | Eng-Jap | Eng-Rus | Overall | Neg. pairs | Std dev MiniLM | 0.7948 | 0.6589 | 0.7886 | 0.7474 | 1 | 0.1878 mE5 | 0.8694 | 0.8630 | 0.9136 | 0.8820 | 0 | 0.0455 LaBSE | 0.7898 | 0.6149 | 0.7855 | 0.7301 | 0 | 0.1990 mpnet | 0.8249 | 0.7044 | 0.8176 | 0.7823 | 0 | 0.1733 mE5-small outperforms all tested encoders on three criteria simultaneously: highest mean similarity (0.8820), lowest standard deviation (0.0455), and zero negative similarity pairs. It also reduces the English-Japanese divergence gap from 0.136 (MiniLM) to 0.006 — a finding that directly addresses the low-resource language concern raised in earlier feedback. * * * **Finding 3 — mE5 is robust to real-world noise (v3a)** Noise type | Mean sim to clean | Above 0.85 threshold Word shuffle (30% swapped) | 0.9750 | 100.0% Char noise (5% OCR errors) | 0.9643 | 100.0% Code switching (20% foreign) | 0.9487 | 98.7% mE5 maintains 94.9–97.5% similarity to clean text under three types of realistic noise. This suggests the encoder is viable for real training corpora where perfect translations cannot be assumed. * * * **Finding 4 — Low resource languages match high resource (v3b + v4)** At corpus scale (2,000 pairs each): Language | Type | Mean similarity | Above 0.85 Sinhala | Low resource | 0.9184 | 93.5% Russian | High resource | 0.9092 | 91.3% Nepali | Low resource | 0.9041 | 89.3% Chinese | High resource | 0.8755 | 77.6% Japanese | High resource | 0.8667 | 67.2% Sinhala (low resource) outperforms Chinese and Japanese (high resource) at corpus scale. This is counterintuitive and we do not have a confident explanation yet. It is flagged as a priority investigation for Paper 2. * * * **Finding 5 — Consumer hardware is sufficient** 20,000 sentences encoded in 13.2 seconds 1,515 sentences per second 458 MB VRAM used out of 4,096 MB available Hardware: NVIDIA RTX 3050 Laptop GPU Corpus-scale multilingual semantic redundancy measurement is practical on consumer hardware. This is relevant for the UCTF democratisation goal. * * * **Limitations** OPUS-100 is a parallel translation corpus — cleaner than real web-crawled training data. Only 7 languages tested — broader coverage needed. Noise simulation is controlled — real corpus noise is more varied. No manual audit of candidate pairs completed yet. Negative similarity cases require manual inspection. * * * **Conclusion** Cross-lingual semantic redundancy is measurable, substantial, and encoder-dependent. mE5-small is a viable baseline for a UCTF compression layer — robust to noise, effective across language families including low-resource settings, and practical on consumer hardware. The core UCTF hypothesis — that a machine-native compressed representation could reduce multilingual training data redundancy — is supported by these preliminary findings. * * * **What comes next — Paper 2** Characterizing which information is universal vs language-specific. Manual audit of candidate pairs. SOSAR and LaBSE comparison at corpus scale. Test on web-crawled corpora (FineWeb2, CulturaX). Investigate the low-resource language outperformance finding. * * * **Resources** Full PDF report: [github.com/kesavanvjp0007/project-uctf/paper1\_final](http://github.com/kesavanvjp0007/project-uctf/paper1_final) All code and notebooks: [github.com/kesavanvjp0007/project-uctf/paper1\_v0](http://github.com/kesavanvjp0007/project-uctf/paper1_v0) through v4 Candidate pairs CSV: 116 semantic pairs with similarity scores Threshold sweep CSV: precision at 7 thresholds Encoder comparison CSV: 6,000 records across 4 encoders Feedback on the low-resource outperformance finding and the noise robustness results especially welcome. Special thanks to @John6666 for the research architecture guidance that shaped this entire program. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [July 20, 2026, 2:49am UTC](https://discuss.huggingface.co/t/project-uctf-an-open-research-program-on-machine-native-ai-training-representations/177240/15 "2026-07-20T02:49:03Z")
Hi. Um, taking everything so far into account, maybe something like this: * * * This has moved well beyond the original v0 sanity check. The v0–v4 sequence now looks like a substantial exploratory measurement package with three useful branches: 1. **translation-equivalent sentence measurement,** 2. **multilingual encoder comparison,** 3. **exploratory noise and low-resource observations.** My short reading is: > The experiments provide useful evidence that translation-equivalent sentences can be detected in multilingual embedding spaces, and that the measured result depends materially on the encoder and dataset setup. I would not rush to add many more large runs immediately. The least disruptive route may be: 1. make the existing package easy to inspect, 2. add one small calibration slice if useful, 3. then let the work branch naturally toward Paper 2, a web-corpus extension, or later representation-preservation tests. None of those branches needs to become an obligation or a blocker for the others. ## Where the work seems to be now | Branch | Earlier stage | Current result | Low-cost next option | | --- | --- | --- | --- | | Translation alignment | FLORES+ controlled sanity check | OPUS-based aligned-pair measurements at larger scale | compare positives with mismatched or hard-negative pairs | | Encoder comparison | one lightweight baseline | four encoders compared; mE5 performed best in this setup | preserve the exact preprocessing and optionally add ranking or margin diagnostics | | Noise | proposed failure categories | character noise, code switching, and word shuffle tested | separate surface-noise invariance from sensitivity to meaning changes | | Low-resource slice | initially an uncertainty | Sinhala and Nepali produced unexpectedly high scores | inspect per-language negative separation, sampling stability, and high/low tails | | Training-corpus generalization | planned branch | experiments remain primarily parallel-corpus based | optional small FineWeb2/CulturaX-like branch | | Reproducibility | public release planned | results and resource links reported in the thread | one stable Paper 1 page linking the artifacts and experiment settings | | Representation/compression | later-paper question | similarity evidence now exists | preservation, reconstruction, compression, and training tests later | | Paper 2 | taxonomy planned | the low-resource result creates an additional question | conceptual work can begin in parallel | ### Directly on the two feedback questions **Low-resource result:** I would not dismiss the Sinhala/Nepali result, but resource level alone probably does not explain it. It may be a real property of this particular **dataset–encoder–sampling combination**. The most informative lightweight check would probably be to compare the positive and mismatched-negative score distributions separately for each language. **Noise result:** character corruption and limited code switching look like useful invariance tests. Word-shuffle similarity is more ambiguous: it may indicate robustness, but it may also indicate that cosine similarity is not very sensitive to word order. That makes it an interesting diagnostic result rather than necessarily a direct simulation of real-corpus robustness. One practical note: when I checked the [public Project UCTF repository](https://github.com/kesavanvjp0007/project-uctf), the public view still showed only the short README, and the report/notebook paths linked in the thread did not resolve from that view. That may simply be upload timing, propagation, or an incomplete public view. Even one stable Paper 1 README linking the report, notebooks, CSVs, environment, and limitations would make the package much easier to inspect. > **What the current experiments appear to establish** > > I would separate the direct observations from the later hypotheses roughly like this: > > | Reported observation | Relatively direct reading | Question left for another branch | > | --- | --- | --- | > | Exact duplicate rate was 0% across translated pairs | exact text matching does not recover differently written cross-lingual translation pairs | how much is detected by fuzzy, semantic, or multilingual-mining methods | > | Aligned OPUS pairs received high similarity | tested encoders frequently place known translations close together | how often arbitrary web-corpus neighbors are actually redundant | > | Encoder results differed substantially | encoder choice is part of the measurement | which encoder is preferable for clustering, mining, reconstruction, or training | > | mE5 produced the strongest scores in v2 | mE5 performed best among the four tested encoders in this OPUS-based setup | whether it is also best for candidate discrimination or representation preservation | > | Synthetic perturbations retained high similarity | the tested embedding was stable under those transformations | whether it distinguishes meaning-preserving changes from meaning-changing changes | > | Sinhala/Nepali scores were high | those OPUS slices aligned strongly under mE5 | whether this comes from language, corpus composition, translation style, sampling, or model calibration | > | 20,000 sentences were embedded quickly on an RTX 3050 | embedding generation at that tested scale was feasible on the reported laptop GPU | full mining, clustering, decoding, compression, and training cost | > | Cross-lingual similarity is measurable | a shared embedding space is a useful measurement instrument | whether redundant content can be merged or compressed without losing required information | > > This is already a useful result. It supports moving from “does a measurable signal exist?” toward “what exactly does that signal represent?” > > I would keep the original paper boundaries because they form a useful separation of questions: > > - **Paper 1:** measure redundancy candidates and their structure, > - **Paper 2:** distinguish potentially universal and language-specific information, > - **Paper 3:** define representation requirements, > - **Paper 4:** test prototype preservation, reconstruction, and compression, > - **Paper 5:** test downstream training behavior. > > Under that division, Paper 1 does not need to carry the full compression-layer claim. It can instead provide a more precise result: > > > Translation-equivalent similarity is measurable under the tested conditions, but its interpretation depends on encoder choice, dataset structure, negative separation, and human audit. > **How I would read the low-resource result** > > The Sinhala/Nepali result is interesting because it runs against a simple expectation that higher-resource languages must always align better. Existing multilingual-retrieval research suggests that low-resource behavior is not determined by “resource level” alone. > > There are examples of strong transfer to low-resource bitext mining, including [OneAligner](https://arxiv.org/abs/2205.08605) and earlier [LASER-based filtering for Nepali–English and Sinhala–English](https://arxiv.org/abs/1906.08885). There are also examples where zero-shot alignment for a lower-resource language is weak until some adaptation is added. The practical pattern seems to be that model training, language pair, domain, and corpus construction all matter. > > So my first interpretation would be: > > > The result may be real for these OPUS slices and this encoder, but “low-resource” is probably not the causal variable by itself. > > [OPUS-100](https://huggingface.co/datasets/Helsinki-NLP/opus-100) is English-centric: its training pairs include English on either the source or target side. Its language subsets may also differ in source corpus, sentence length, translation literalness, domain, formatting, and alignment quality. Any of those could affect the mean score. > > A low-effort diagnostic tree could be: > > | If this happens… | A reasonable interpretation | > | --- | --- | > | the language ordering remains similar across several random samples | probably a stable dataset/model interaction | > | the ordering changes strongly across seeds | sampling composition is important | > | Sinhala/Nepali remain high for all tested encoders | corpus or alignment characteristics may be important | > | the result is mainly present for mE5 | encoder-specific calibration or training coverage may be important | > | positive scores are high but mismatched negatives are also high | absolute cosine is high, but discrimination may not be | > | high scores are concentrated in short or formulaic sentences | corpus composition may explain part of the effect | > | inspection of the high/low tails finds alignment differences | translation or data-quality differences may be contributing | > | the result persists for non-English↔non-English pairs | it is less likely to be only an English-pivot effect | > > If only one check is convenient, I would choose this: > > > For each language, compare the known translation-pair distribution against a same-size mismatched-pair distribution. > > That gives more information than the positive mean alone while remaining small enough to run with the current setup. > > If the result survives resampling and negative comparison, it becomes a useful dataset–encoder interaction to carry into Paper 2. If it changes substantially by encoder or sample, that is also informative: the result can be reported as calibration or corpus sensitivity rather than a general low-resource advantage. > > For a later web-corpus branch, low-resource interpretation becomes more difficult because language identification and corpus quality become part of the measurement. [Language ID in the Wild](https://aclanthology.org/2020.coling-main.579/) found severe LangID failures for many lower-resource web-crawl corpora despite strong held-out test results, and [Quality at a Glance](https://aclanthology.org/2022.tacl-1.4/) found systematic quality and labeling problems across lower-resource web corpora. > > That does not make low-resource analysis impractical; it means language-ID and basic corpus-quality checks should remain visible in the measurement. > **How I would read the noise results** > > The three perturbations appear to test somewhat different properties. > > | Perturbation | What high similarity may indicate | What it does not establish by itself | > | --- | --- | --- | > | character corruption | stability to limited surface/OCR-like damage | robustness to arbitrary extraction or encoding failures | > | limited code switching | stability when some lexical material changes language | preservation under natural code switching with semantic shifts | > | word shuffle | stability to token-order changes | preservation of syntax or proposition-level meaning | > > The word-shuffle result is especially interesting. A sentence encoder can remain close under shuffling because it is robust to noisy ordering, but it can also remain close because its cosine geometry is driven heavily by lexical content. > > [SenTest](https://arxiv.org/abs/2311.17722) studies sentence-encoder behavior under perturbations including word-order changes. [ALIGN-SIM](https://aclanthology.org/2024.findings-emnlp.436/) evaluates sentence embeddings using semantic distinction, synonym replacement, antonym replacement, paraphrasing, and sentence jumbling. These provide useful ways to separate invariance from semantic sensitivity. > > A practical decision tree could be: > > - **If the goal is robustness to surface corruption:** keep character noise and meaning-preserving code switching. > - **If the goal is sensitivity to meaning:** add controlled entity, number, date, antonym, and negation changes. > - **If the goal is realistic web noise:** sample actual boilerplate, truncation, HTML extraction errors, OCR errors, repeated navigation text, and bad alignments. > - **If shuffled sentences remain very close:** report this as a word-order-sensitivity observation, not only as robustness. > > A compact control set could reuse the same base sentences: > > | Pair construction | Desired relative behavior | > | --- | --- | > | faithful translation | high similarity | > | faithful paraphrase | high similarity | > | synonym replacement | high similarity | > | limited character corruption | moderately high similarity | > | meaning-preserving code switching | high similarity | > | changed entity | lower similarity | > | changed number or date | lower similarity | > | antonym replacement | lower similarity | > | negation reversal | lower similarity | > | random mismatched translation | low similarity | > | shuffled text whose meaning becomes unclear or changes | lower similarity if order-sensitive meaning is captured | > > Negation is worth checking because embedding-based metrics can sometimes assign high similarity to statements whose truth conditions have been reversed. [Negation-aware Evaluation of Language Generation Systems](https://aclanthology.org/2023.inlg-main.12/) gives examples of this failure mode. > > This does not need to become a large benchmark. A few deliberately constructed pairs can clarify what the current “noise robustness” score is preserving. > **One small calibration layer, if useful** > > If one modest addition were chosen before moving on, I would probably favor a compact calibration slice rather than another large positive-pair run. > > The exact size is flexible. Even something on the order of 100–200 deliberately selected pairs could connect the similarity distributions to interpretable categories. > > Possible sampling: > > - highest-scoring true translation pairs, > - lowest-scoring true translation pairs, > - pairs near the current threshold, > - random mismatched pairs, > - nearest non-matching candidates, > - same-topic but different-claim pairs, > - changed entity/date/number pairs, > - negation or contradiction pairs, > - cases where encoders disagree, > - selected high/low tails from Sinhala and Nepali. > > Possible labels: > > - `translation_equivalent` > - `paraphrase_equivalent` > - `near_duplicate` > - `same_fact_different_context` > - `boilerplate_template` > - `same_topic_only` > - `entity_number_date_mismatch` > - `contradiction` > - `bad_alignment` > - `uncertain` > > The useful output would not need to be complicated: > > | Output | What it adds | > | --- | --- | > | precision by threshold band | helps interpret the threshold sweep | > | positive vs mismatched-negative distributions | shows discrimination | > | false-positive categories | shows what the embedding confuses | > | false-negative examples | shows what translations it misses | > | encoder-disagreement examples | helps compare the baselines | > | short annotation notes | preserves how ambiguous cases were handled | > > A low-burden version could be selected according to the current interest: > > - **If the low-resource result is the focus:** inspect Sinhala/Nepali high and low tails. > - **If mE5 is the focus:** inspect examples where mE5 disagrees with the other encoders. > - **If thresholding is the focus:** inspect around the threshold and the highest-scoring mismatches. > - **If Paper 2 is the focus:** label whether each candidate preserves universal content, language-specific expression, or both. > > This reconnects the current work with the v0 principle that embedding similarity is a candidate generator rather than ground truth. > **Alignment score versus candidate-mining quality** > > Mean positive-pair cosine answers one useful question: > > > Are known aligned translations placed close together? > > It does not fully answer another question: > > > Can the system distinguish the correct counterpart from nearby incorrect candidates? > > For that second question, useful diagnostics include: > > | Diagnostic | Question answered | > | --- | --- | > | positive mean cosine | are known pairs aligned? | > | positive–negative score gap | are correct pairs separated from mismatches? | > | Recall@1 | is the correct translation ranked first? | > | MRR | how high is the correct translation ranked? | > | local margin score | is the pair unusually close relative to its neighborhoods? | > | encoder agreement | is the candidate stable across measurement instruments? | > > The distinction matters because absolute cosine scales can differ by model, language, and local embedding neighborhood. [Margin-based Parallel Corpus Mining](https://aclanthology.org/P19-1309/) was proposed because a single hard cosine threshold can suffer from scale inconsistency; it compares a candidate pair with nearby alternatives instead. > > A simple branch would be: > > - **If the goal is alignment measurement:** the current positive-pair means remain useful. > - **If the goal is candidate mining:** include mismatched candidates and report retrieval or margin results. > - **If the goal is dedup thresholding:** calibrate threshold behavior by encoder and possibly by language pair. > - **If the goal is representation design:** move beyond similarity to preservation and reconstruction tests. > > This slightly changes how I would describe the mE5 result: > > > mE5 produced the strongest positive-pair alignment in the current comparison. > > If its negative separation and retrieval ranking are also strongest, the broader baseline conclusion becomes stronger naturally. > > mE5 is a plausible strong result here because the E5 family is trained contrastively on large multilingual text-pair collections and uses hard-negative training. The [multilingual-e5-small model card](https://huggingface.co/intfloat/multilingual-e5-small) also documents an important reproduction detail: for symmetric tasks such as semantic similarity, bitext mining, and paraphrase retrieval, both inputs should use the `query: ` prefix. > > It may therefore help to preserve: > > - exact model identifier and revision, > - prefixes or prompts, > - pooling, > - embedding normalization, > - truncation length, > - score function, > - sampling seed. > **Translation-pair branch versus training-corpus-like branch** > > The current OPUS results appear most directly useful as a **translation-equivalent branch** of Paper 1. > > That is already valuable. It establishes a cleaner, controlled form of cross-lingual semantic overlap at larger scale than v0. > > A separate branch is needed only if the desired statement becomes: > > > What proportion and type of semantic redundancy remain in multilingual pretraining corpora? > > Those corpora contain additional phenomena: > > - exact and near duplicates, > - translated pages, > - paraphrases, > - common factual descriptions, > - repeated templates, > - cookie/legal/navigation boilerplate, > - OCR or extraction damage, > - language-ID errors, > - same-topic but non-equivalent text, > - copied text with changed dates, entities, or numbers. > > A decision tree could be: > > - **If the goal is to finish the translation-equivalent branch:** add negatives, ranking/margin evaluation, or a small audit. > - **If the goal is broader training-corpus claims:** add one small web-corpus sample. > - **If the goal is representation compression:** move to preservation/reconstruction rather than enlarging Paper 1 indefinitely. > - **If Paper 2 is already the priority:** begin it while leaving the web-corpus branch explicitly open. > > [FineWeb2](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2) may be a practical corpus branch because it exposes language, script, language score, Common Crawl metadata, and `minhash_cluster_size`. Its [repository](https://github.com/huggingface/fineweb-2) explains that the corpus was deduplicated globally per language and that the retained document stores the duplicate-cluster size. > > That creates a useful comparison: > > | Signal | Possible interpretation | > | --- | --- | > | `minhash_cluster_size` | existing textual/near-duplicate structure | > | semantic nearest neighbors | possible meaning-level structure | > | boilerplate label | template repetition | > | language/source metadata | distribution and data-quality context | > | manual label | actual relation between examples | > > The interpretation should remain: > > > remaining redundancy after FineWeb2’s existing pipeline, > > rather than raw-web redundancy. > > A small streaming sample would be enough for a first comparison; it would not need to become a web-scale dedup project immediately. > **Making the current package easier to inspect** > > The highest-value release artifact may be one stable **Paper 1 research card** , rather than a complicated repository structure. > > Something like: > > ### What was tested > > - v0–v4 experiment summary, > - datasets and subsets, > - languages and pair counts, > - encoders, > - perturbations, > - metrics, > - hardware. > > ### What was not tested > > - large noisy web corpora, > - human precision of semantic candidates, > - representation reconstruction, > - information-loss curves, > - downstream training, > - full pipeline compute cost. > > ### Reproduction information > > - dataset revision/subset/split, > - sampling procedure and random seed, > - exact encoder revision, > - prefixes and preprocessing, > - pooling and normalization, > - thresholds, > - environment or requirements, > - notebook execution order. > > ### Linked artifacts > > - report, > - notebooks or scripts, > - result CSVs, > - candidate pairs, > - hardware log, > - known limitations, > - open follow-up branches. > > The [Hugging Face dataset-card guide](https://huggingface.co/docs/hub/en/datasets-cards) uses a similar general structure: keep the artifact context, language, license, size, intended use, and limitations close to the artifact. A lightweight Paper 1 card can apply the same idea without requiring a large documentation effort. > > At the time of this check, the [Project UCTF repository](https://github.com/kesavanvjp0007/project-uctf) still displayed one commit and the short README in its public tree. Since forum and repository updates can appear out of order, this may simply be a timing or visibility issue. The main practical point is to have one stable landing page once the files are visible. > > Preserving stable IDs is also useful. A [DataTrove MinHash discussion](https://github.com/huggingface/datatrove/issues/213) shows the difficulty of reconstructing duplicate membership from removal and cluster files after the fact. Keeping pair IDs, source IDs, scores, and cluster IDs early avoids that problem later. > **Setup-qualified wording** > > The findings are useful. Setup-qualified wording can keep each conclusion attached to the experiment that produced it. > > | Broad summary | Setup-qualified version | > | --- | --- | > | Text deduplication fails completely | Exact text matching does not recover the tested translation-equivalent cross-lingual pairs | > | Semantic redundancy is substantial in multilingual training data | High translation-equivalent similarity was observed in the tested parallel-corpus settings | > | mE5 is the strongest encoder baseline | mE5 performed best among the four tested encoders in this OPUS-based comparison | > | mE5 is robust to real-world noise | mE5 remained stable under the three tested synthetic perturbations | > | Low-resource languages match or outperform high-resource languages | Sinhala and Nepali produced unexpectedly high scores in the tested OPUS slices | > | Consumer hardware is sufficient | embedding generation for 20,000 tested sentences was feasible on the reported RTX 3050 setup | > | mE5 is viable as a UCTF compression layer | mE5 is a promising measurement encoder and later representation candidate | > | The core UCTF hypothesis is supported | the findings strengthen the motivation for the next UCTF experiments | > > This is not about reducing the result. It keeps measurement, representation, and training claims in their respective experimental branches. > > The encoder qualification is especially useful because [MTEB](https://aclanthology.org/2023.eacl-main.148/) found that no single embedding method dominated across all evaluated tasks. [MMTEB](https://arxiv.org/abs/2502.13595) has since expanded multilingual embedding evaluation to more than 500 tasks across 250+ languages, further emphasizing task, language, and domain dependence. > **Possible parallel routes from here** > > I do not think every Paper 1 extension has to block Paper 2. > > A possible branch map: > > | Current priority | Natural route | > | --- | --- | > | preserve the current research record | publish one stable Paper 1 card and artifact links | > | understand the low-resource result | per-language negatives, resampling, and tail inspection | > | strengthen mE5 baseline selection | ranking/margin evaluation and encoder disagreement analysis | > | understand noise behavior | meaning-preserving vs meaning-changing controls | > | support broader training-corpus claims | small FineWeb2/CulturaX-like branch | > | begin Paper 2 | start the universal vs language-specific taxonomy in parallel | > | test UCTF representation viability | preserve for later reconstruction, bottleneck, and downstream-training experiments | > > A default route with minimal disruption could be: > > 1. publish or stabilize the current Paper 1 artifact, > 2. select one small calibration slice, > 3. begin Paper 2 conceptually, > 4. leave web-corpus and representation branches explicitly open. > > If the low-resource result becomes central to Paper 2, the smallest useful bridge would be: > > > verify that the result survives mismatched negatives and resampling, then treat it as a dataset–encoder interaction to explain rather than a general low-resource advantage. > > For the broader [ShinBay-UCTF framework](https://discuss.huggingface.co/t/shinbay-uctf-a-conceptual-framework-for-privacy-preserving-continuous-learning-episodic-memory-and-compressed-knowledge-representation/177924), I think the present results are enough to motivate using a shared semantic space as a **provisional Layer-1 interface**. > > I would keep “validated compressed representation” for later tests of: > > - information preservation, > - reconstruction, > - controlled forgetting or merging, > - provenance, > - deletion semantics, > - downstream generation or training behavior. > > That keeps the connection between the two threads clear: > > > Paper 1 supplies evidence and measurement tools; later papers determine what the architecture can safely do with the representation. > **References directly relevant to this checkpoint** > > ### Current project > > - [Project UCTF thread](https://discuss.huggingface.co/t/project-uctf-an-open-research-program-on-machine-native-ai-training-representations/177240) > - [Paper 1 v0–v4 result summary](https://discuss.huggingface.co/t/project-uctf-an-open-research-program-on-machine-native-ai-training-representations/177240/14) > - [Project UCTF repository](https://github.com/kesavanvjp0007/project-uctf) > - [ShinBay-UCTF concept thread](https://discuss.huggingface.co/t/shinbay-uctf-a-conceptual-framework-for-privacy-preserving-continuous-learning-episodic-memory-and-compressed-knowledge-representation/177924) > > ### Parallel corpus and candidate mining > > - [OPUS-100 dataset card](https://huggingface.co/datasets/Helsinki-NLP/opus-100) > - [Margin-based Parallel Corpus Mining](https://aclanthology.org/P19-1309/) > - [multilingual-e5-small model card](https://huggingface.co/intfloat/multilingual-e5-small) > > ### Sentence-embedding interpretation > > - [MTEB](https://aclanthology.org/2023.eacl-main.148/) > - [MMTEB](https://arxiv.org/abs/2502.13595) > - [ALIGN-SIM](https://aclanthology.org/2024.findings-emnlp.436/) > - [SenTest](https://arxiv.org/abs/2311.17722) > - [Negation-aware Evaluation](https://aclanthology.org/2023.inlg-main.12/) > > ### Optional web-corpus branch > > - [FineWeb2 dataset card](https://huggingface.co/datasets/HuggingFaceFW/fineweb-2) > - [FineWeb2 repository](https://github.com/huggingface/fineweb-2) > - [Language ID in the Wild](https://aclanthology.org/2020.coling-main.579/) > - [Quality at a Glance](https://aclanthology.org/2022.tacl-1.4/) Overall, I think the current work has already produced a useful Paper 1 checkpoint. The most informative next move does not have to be larger. It may simply be to preserve the current setup, add one small discrimination or calibration layer, and let the next branch be selected by the question: - **taxonomy:** Paper 2, - **training-corpus prevalence:** web-corpus branch, - **encoder quality:** negatives, ranking, or margin, - **representation viability:** preservation and reconstruction later. That would keep the program moving without forcing every open branch to be resolved at once. ---
**Author:** ![k7007research](https://avatars.discourse-cdn.com/v4/letter/k/6bbea6/32.png) [@k7007research](https://discuss.huggingface.co/u/k7007research)\ **Post date:** [July 21, 2026, 6:08am UTC](https://discuss.huggingface.co/t/project-uctf-an-open-research-program-on-machine-native-ai-training-representations/177240/16 "2026-07-21T06:08:29Z")
@John6666 — README has been updated with full navigation to all artifacts. All CSVs, notebooks, and PDF reports are now linked directly from the README. Your feedback on word-shuffle has been added to the limitations section. Paper 2 taxonomy results are posted separately in this thread. [GitHub - kesavanvjp0007/project-uctf · GitHub](http://github.com/kesavanvjp0007/project-uctf) — K7007 ---
**Author:** ![k7007research](https://avatars.discourse-cdn.com/v4/letter/k/6bbea6/32.png) [@k7007research](https://discuss.huggingface.co/u/k7007research)\ **Post date:** [July 21, 2026, 6:11am UTC](https://discuss.huggingface.co/t/project-uctf-an-open-research-program-on-machine-native-ai-training-representations/177240/17 "2026-07-21T06:11:24Z")
**Project UCTF — Papers 1 & 2 Complete** **Measuring Redundancy · Characterizing Knowledge · Defining What UCTF Can Compress** * * * Two papers complete. Here is the combined results report. All code, data, notebooks, and PDF reports are available on GitHub: > **[GitHub - kesavanvjp0007/project-uctf](https://github.com/kesavanvjp0007/project-uctf)** > > Contribute to kesavanvjp0007/project-uctf development by creating an account on GitHub. * * * ## Paper 1 — Measuring Semantic Redundancy in Multilingual Training Corpora **Research question:** How much semantic duplication exists across languages in real AI training datasets? **Six experiments — v0 through v4:** Version | Scale | Mean similarity | Key finding v0 | 20 pairs manual | 0.9570 | 100% precision at all thresholds v1 | 1,500 real pairs | 0.7474 | Real data noisier — Eng-Jap gap of 0.136 v2 | 4 encoders GPU | 0.8820 mE5 | mE5 best — zero negatives, std 0.046 v3a | 900 noisy | 0.9627 avg | mE5 robust — 98.7–100% above 0.85 v3b | 600 low-resource | 0.9045 avg | Sinhala, Nepali match high-resource v4 | 10,000 corpus | 0.8948 | 1,515 sent/sec on RTX 3050, 458 MB VRAM **Paper 1 findings:** 1. Text deduplication finds 0% cross-lingual redundancy. Embedding-based approaches are necessary. 2. mE5-small is the strongest encoder — mean 0.8820, std 0.0455, zero negative pairs, closes Eng-Jap gap from 0.136 to 0.006. 3. mE5 maintains 94.9–97.5% similarity under character noise, word shuffling, and code-switching. 4. Low-resource languages (Sinhala 91.8%, Nepali 90.4%) outperform high-resource languages (Chinese 87.6%, Japanese 86.7%) at corpus scale — reason under investigation. 5. Consumer hardware is sufficient — 20,000 sentences in 13.2 seconds on RTX 3050 laptop GPU. * * * ## Paper 2 — Characterizing Universal and Language-Specific Knowledge **Research question:** Which semantic information is universal and safely compressible — and which is language-dependent and must be preserved? **Dataset:** 60 sentences × 6 knowledge categories × 4 languages = 240 sentences Categories: factual · mathematical · named\_entity · emotional\_nuance · cultural\_idiom · grammar\_dependent **Results — similarity by category (mE5-small):** Category | Mean | Std | Above 0.90 | UCTF Safety grammar\_dependent | 0.9148 | 0.0213 | 76.7% | Safe emotional\_nuance | 0.9069 | 0.0244 | 60.0% | Safe factual | 0.8968 | 0.0216 | 46.7% | Safe mathematical | 0.8795 | 0.0309 | 25.0% | Medium named\_entity | 0.8749 | 0.0318 | 16.7% | Medium cultural\_idiom | 0.8573 | 0.0470 | 20.0% | Caution **Most universal sentence:** “She gave him a gift yesterday.” — sim=0.9459 (grammar\_dependent) **Most divergent sentence:** “It is raining cats and dogs.” — sim=0.8103 (cultural\_idiom) **Paper 2 findings:** 1. Hypothesis partially reversed. Grammar-dependent sentences scored highest (91.5% preserved) — not factual or mathematical. mE5 abstracts syntactic structure and captures semantic intent regardless of word order. 2. Cultural idioms are the primary compression risk. Lowest preservation (85.7%), highest variance (std 0.0470), only 58.3% of pairs above 0.85 threshold. Idiomatic expressions carry language-specific meaning that compression would destroy. 3. Emotions are more universal than expected. Emotional nuance scored 90.7% — human emotional concepts transfer across language boundaries more reliably than initially assumed. 4. Named entities show moderate divergence (87.5%) with high variance. Cultural references (Mona Lisa, Great Wall) diverge more than scientific proper nouns (Einstein, Amazon River). 5. Hypothesis partially reversed — grammar and emotion outperformed factual and mathematical categories, suggesting mE5 measures semantic intent rather than syntactic universality. * * * ## UCTF Design Implication — 3-Tier Compression Strategy Based on Papers 1 and 2, the UCTF encoder requires a tiered approach: Tier 1 — Safe compression Categories: factual, mathematical, grammar-dependent, emotional Preservation: 88–91% Strategy: Full UCTF compression — low risk of information loss Tier 2 — Flagged compression Categories: named entities Preservation: 87.5% Strategy: Compress with entity flagging — preserve proper nouns separately Tier 3 — Preserve as-is Categories: cultural idioms Preservation: 85.7% Strategy: Store as language-specific tokens with source language tag — do not compress * * * ## Information Preservation Estimates grammar dependent → 91.5% preserved emotional nuance → 90.7% preserved factual → 89.7% preserved mathematical → 88.0% preserved named entity → 87.5% preserved cultural idiom → 85.7% preserved * * * ## Limitations Paper 1: OPUS-100 is a parallel corpus — cleaner than web-crawled training data. Only 7 languages tested. Paper 2: 10 sentences per category is a small sample. Only 4 languages. mE5-small used — larger encoders may differ. Both: Preservation estimates are proxy measures based on similarity scores — actual UCTF information loss requires encoder-decoder experiments in Paper 4. * * * ## What Comes Next — Paper 3 Formal design specification for the UCTF encoder. Tiered compression architecture based on Papers 1 and 2 findings. UCTF token format specification. Encoder requirements per compression tier. SOSAR evaluation as candidate encoder for Tier 1. * * * ## Resources GitHub: [GitHub - kesavanvjp0007/project-uctf · GitHub](http://github.com/kesavanvjp0007/project-uctf) Paper 1 PDF: /paper1\_final/ Paper 2 PDF: /paper2/ All code, datasets, embeddings, and visualizations: /paper1\_v0 through /paper2/ Feedback on the cultural idiom finding and the 3-tier compression strategy especially welcome. Special thanks to @John6666 for the research architecture that made this program possible. — K7007