🚀 COPM: Converting clinical signals into outcome intelligence for DHTI.
CDS Embedding Model with Genomic Tokenizer (Dim128 Variant)
Preferred 128-dimensional CDS embedding model optimized for efficiency with biologically-compressed Genomic Tokenizer. Trained using contrastive learning with the Genomic Tokenizer on human protein-coding DNA sequences.
Note on Model Size: Due to the biologically-compressed nature of the Genomic Tokenizer, which operates at the codon level (26 tokens representing 20 amino acids plus special tokens), smaller embedding dimensions may actually outperform larger ones. The tokenizer's compression reduces the representation space complexity, enabling more efficient learning even with fewer parameters.
Model Details
- Variant: Dim128
- Architecture: Transformer Encoder with 2 layers, 4 attention heads
- Hidden Size: 256 dimensions
- Embedding Dimension: 128 dimensions (L2-normalized)
- Vocabulary: 26 tokens (Genomic Tokenizer)
- Max Sequence Length: 2048 tokens
- Training Objective: Contrastive learning (instance discrimination)
- Parameters: ~1M parameters
Model Variants
This model is part of a family of CDS embedding models:
| Variant | Parameters | Layers | Hidden Size | Embedding Dim | Use Case |
|---|---|---|---|---|---|
| dim128 | ~1M | 2 | 256 | 128 | ⭐ Preferred - efficient with tokenizer compression |
| dim256 | ~2M | 2 | 256 | 256 | Research |
| dim512 | ~4M | 4 | 512 | 512 | Research |
Genomic Tokenizer
This model uses the Genomic Tokenizer which tokenizes DNA sequences at the codon level:
- 26 unique tokens:
- Special tokens:
[CLS],[SEP],[BOS],[MASK],[PAD],[UNK] - 20 amino acid codons (Alanine, Cysteine, Aspartic_acid, etc.)
- Special tokens:
Each token represents a codon (3 nucleotides) that encodes a specific amino acid, enabling the model to learn at the protein-coding level rather than individual nucleotides.
Training Data
- Dataset: GENCODE v50 human protein-coding CDS sequences
- Genome Assembly: GRCh38
- Transcripts: ~17,000 protein-coding transcripts
- Preprocessing:
- Extracted CDS from GTF annotations
- Reconstructed sequences from GRCh38 genome
- Quality control: start/stop codon verification, reading frame validation
Use Cases
Recommended for most use cases - the codon-level tokenizer compression enables smaller embeddings to match or exceed larger models:
✅ Recommended:
- Multimodal Clinical Outcome Prediction for Precision Healthcare Applications (Contact me for details)
- Gene prioritization in rare disease diagnostics (as a research tool)
- Functional annotation of novel variants
- Disease gene discovery in research settings
- Building gene-disease knowledge graphs
Clinical Validation Status
- Research Phase: Model is intended for research and translational applications
- Not Clinically Validated: Has not undergone clinical validation studies
- NOT APPROVED FOR CLINICAL USE: Should not be used for patient care or diagnostics
Installation
pip install torch transformers safetensors genomic-tokenizer
Usage
Load the model and generate embeddings using Transformers AutoModel/AutoTokenizer:
import torch
from transformers import AutoModel, AutoTokenizer
# Load model and tokenizer
repo_id = "beapen/genomic-cds-embedding-dim128-v1"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModel.from_pretrained(repo_id, trust_remote_code=True).eval()
# Batch process CDS sequences
sequences = [
"ATGGCCGCCTACAAGCAGCAGTCTGGGGAAGACGGGCTACCTTGCCCCTGA",
"ATGGCCGCC",
"ATGTACGTA"
]
# Tokenize with automatic padding
inputs = tokenizer(sequences, return_tensors="pt", padding=True)
# Generate L2-normalized embeddings
with torch.no_grad():
embeddings = model(**inputs)
print(f"Embeddings shape: {embeddings.shape}") # torch.Size([3, 128])
print(f"Embedding norm: {embeddings.norm(dim=-1)}") # Should be ~1.0 for all
Each embedding is L2-normalized and has dimension 128.
Model Card Contact
- Author: Bell Eapen
- Website: https://nuchange.ca
- Date: 2026-11-16
- Framework: PyTorch 2.5.1
- Variant: Dim128
- Repository: https://github.com/dermatologist/gtembed
Limitations
- Training Data: Only includes protein-coding genes from GENCODE v50
- Sequence Length: Limited to 2048 tokens (~6144 nucleotides)
- Species: Human-only (GRCh38 assembly)
- Context: Does not include regulatory regions or non-coding sequences
- Variant: Dim128 model - dim128 and dim256 variants available. (dim128 may outperform larger models due to tokenizer compression)
Ethical Considerations
This model is intended for research purposes in genomics and bioinformatics. Users should:
- Verify biological interpretations with experimental data
- Follow ethical guidelines for genetic data usage
Citations
If you use this model in your research, please cite the following:
GTEmbed Model
Eapen, B. (2026). GTEmbed: Genomic Tokenizer Embedding Pipeline. Hugging Face. https://huggingface.co/beapen/genomic-cds-embedding-dim128-v1
Genomic Tokenizer
Eapen, B. (2026). Genomic Tokenizer: Codon-level Tokenization for CDS Sequences. GitHub. https://github.com/dermatologist/genomic-tokenizer
Training Data Sources
This model was trained using CDS sequences extracted from two publicly available datasets:
1. GENCODE v50 Human Gene Annotation
GENCODE provides comprehensive reference gene annotation for the human genome. For GENCODE v50 (released in 2023), please cite:
Frankish, A., Diekhans, M., Ferreira, A.M., Johnson, R., Jungreis, I., Loveland, J., Mudge, J.M., Sisu, C., Wright, J., Armstrong, J., Barnes, I., Berry, K., Bignell, A., Carbonell-Sala, S., Cerdán-Vélez, D., Cheng, Y., Davidson, C., Dyer, S.C., Eyrignoux, C., Fatima, R., Fernandez-Banet, J., Gall, A., Garcia Girón, C., Gonzalez, J.M., Greller, R., Hardy, M., Hunt, T., Ishiguro, H., James, B., Jayakumar, A., Juettemann, T., Kay, M., Kulesha, E., Mangan, R.M., Martin, F.J., Mohanan, S., Muir, P., Murphy, S., Nag, A., Overduin, B., Parker, A., Perry, M., Petty, T.L.P., Pujara, P., Raj, S., Sanchez-Castillo, D., Schmitt, B., Schuilenburg, H., Sealy, I., Shafer, R., Sharma, L., Slater, T., Smedley, D., Sobala, A., Spudich, A., Steed, E., Tang, Y., Thakuria, J., Tress, M.L., Vazquez, J., Wilder, S., Zadissa, A., Zerbino, D.R., Cunningham, F., Lander, A., Flicek, P., and Harrow, J. (2023). GENCODE: reference annotation for the human and mouse genomes in 2023. Nucleic Acids Research, 51(D1), D942-D949. https://doi.org/10.1093/nar/gkac1071
Alternative citation for latest GENCODE release (2025):
Mudge, J.M., Carbonell-Sala, S., Diekhans, M., Martinez, J.G., Hunt, T., Jungreis, I., Loveland, J.E., Arnan, C., Barnes, I., Bennett, R., Berry, K., Bignell, A., Cerdán-Vélez, D., Cochran, K., Cortés, L.T., Davidson, C., Donaldson, S., Dursun, C., Fatima, R., Hardy, M., Hebbar, P., Hollis, Z., James, B., Jiang, Y., Johnson, R., Kaur, G., Kay, M., Mangan, R.M., Maquedano, M.G., Gómez, L.M., Mathlouthi, N., Merritt, R., Ni, P., Palumbo, E., Perteghella, T., Pozo, F., Raj, S., Sisu, C., Steed, E., Sumathipala, D., Suner, M.M., Uszczynska-Ratajczak, B., Wass, E., Yang, Y., Zhang, D., Finn, R.D., Gerstein, M., Guigó, R., Hubbard, T.J.P., Kellis, M., Kundaje, A., Paten, B., Tress, M.L., Birney, E., Martin, F.J., and Frankish, A. (2025). GENCODE 2025: reference gene annotation for human and mouse. Nucleic Acids Research, 53(D1), D966-D975. https://doi.org/10.1093/nar/gkae1078
Data source:
- File:
gencode.v50.annotation.gtf.gz - Download URL: https://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_50/
- Website: https://www.gencodegenes.org/human/release_50.html
2. GRCh38.p14 Human Reference Genome Assembly
The Genome Reference Consortium (GRC) maintains the human reference genome assembly. For GRCh38.p14 (accession: GCF_000001405.40), cite as:
Genome Reference Consortium. (2022). Human genome assembly GRCh38.p14. RefSeq assembly accession: GCF_000001405.40. https://www.ncbi.nlm.nih.gov/assembly/GCF_000001405.40
For the original GRCh38 release (2013), you may also cite:
Church, D.M., Dunn, D.M., Hodgson, A., Loveland, J.E., Smith, A.N., Bevan, P., Cunningham, F., Earhart, S., Farrell, C.M., Gildert, J., Graf, S., Kay, M., Khurana, E., Li, Y., Thakuria, J., Trevanion, S.J., Veyrieras, J.-B., Vogel, J., White, S.J., Aken, B.L., Barrell, D., Brent, S., Chen, Z., Delcher, A., Frankish, A., Gietzen, D., Gilbert, J.G., Graham, U., Hammond, S., Haussler, D., Hermjakob, H., Hume, D.A., Kasprzyk, A., Kulesha, E., Lander, A., Long, J., Melsens, S., Meidl, P., Murphy, S., Nigro, J., Park, T., Rosenfeld, J.A., Schuster, S.C., Sealy, I., Souvorov, A., Souvorova, A., Steward, C., Taudien, S., Tress, M.L., Ureta-Vidal, A., Wallen, T., Weber, R., White, O., Woodward, J.R., Zadissa, A., Zdobnov, E.M., Zerbino, D.R., Blankenberg, D., Kent, W.J., Korf, I., Lander, E.S., Hubbard, T.J.P., Flicek, P., Harrow, J., Weber, R.J., Cooper, D.M., Eichler, E.E., Felsenfeld, G., Green, E.D., Grimwood, J., Guttmacher, C., Hardison, R.C., Istrail, S., Jaffe, D.B., Myers, R.M., Pennacchio, L.A., Ponting, C.P., Rubin, E.M., Ryder, O., Venter, J.C., Waterston, R.H., Wilson, R.K., and Collins, F.S. (2014). Grading the quality of genome assemblies. Nature Biotechnology, 32, 101-104. https://doi.org/10.1038/nbt.2783
Data source:
- File:
GRCh38.primary_assembly.genome.fa.gz - Assembly: GRCh38.p14 (patch release 14)
- RefSeq Accession: GCF_000001405.40
- GenBank Accession: GCA_000001405.29
- Download URL: ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/000/001/405/GCF_000001405.40_GRCh38.p14/
- Website: https://www.ncbi.nlm.nih.gov/assembly/GCF_000001405.40/
- GRC Website: https://www.ncbi.nlm.nih.gov/grc/human
Best practices for citing genome assemblies:
- Always include the assembly accession number (GCF_000001405.40) for reproducibility
- Specify the patch level (p14) when relevant
- Reference the Genome Reference Consortium as the maintainer
License
Apache 2.0
Tags
genomics, dna-sequences, embedding, transformer, contrastive-learning, protein-coding, cds, genomic-tokenizer, bioinformatics, healthcare, dim128-model
- Downloads last month
- 85
