DHTI Logo

🚀 COPM: Converting clinical signals into outcome intelligence for DHTI.

CDS Embedding Model with Genomic Tokenizer (Dim512 Variant)

dim512 CDS embedding model balanced for performance and efficiency. Trained using contrastive learning with the Genomic Tokenizer on human protein-coding DNA sequences.

Note on Model Size: Due to the biologically-compressed nature of the Genomic Tokenizer, which operates at the codon level (26 tokens representing 20 amino acids plus special tokens), smaller embedding dimensions may actually outperform larger ones. The tokenizer's compression reduces the representation space complexity, enabling more efficient learning even with fewer parameters.

Model Details

  • Variant: Dim512
  • Architecture: Transformer Encoder with 4 layers, 8 attention heads
  • Hidden Size: 512 dimensions
  • Embedding Dimension: 512 dimensions (L2-normalized)
  • Vocabulary: 26 tokens (Genomic Tokenizer)
  • Max Sequence Length: 2048 tokens
  • Training Objective: Contrastive learning (instance discrimination)
  • Parameters: ~10M parameters

Model Variants

This model is part of a family of CDS embedding models:

Variant Parameters Layers Hidden Size Embedding Dim Use Case
dim128 ~1M 2 256 128 ⭐ Preferred - efficient with tokenizer compression
dim256 ~2M 2 256 256 Research
dim512 ~4M 4 512 512 Research

Genomic Tokenizer

This model uses the Genomic Tokenizer which tokenizes DNA sequences at the codon level:

  • 26 unique tokens:
    • Special tokens: [CLS], [SEP], [BOS], [MASK], [PAD], [UNK]
    • 20 amino acid codons (Alanine, Cysteine, Aspartic_acid, etc.)

Each token represents a codon (3 nucleotides) that encodes a specific amino acid, enabling the model to learn at the protein-coding level rather than individual nucleotides.

Training Data

  • Dataset: GENCODE v50 human protein-coding CDS sequences
  • Genome Assembly: GRCh38
  • Transcripts: ~17,000 protein-coding transcripts
  • Preprocessing:
    • Extracted CDS from GTF annotations
    • Reconstructed sequences from GRCh38 genome
    • Quality control: start/stop codon verification, reading frame validation

Use Cases

Suitable for most research tasks and moderate-scale applications: ✅ Recommended:

  • Gene prioritization in rare disease diagnostics (as a research tool)
  • Functional annotation of novel variants
  • Disease gene discovery in research settings
  • Building gene-disease knowledge graphs

Clinical Validation Status

  • Research Phase: Model is intended for research and translational applications
  • Not Clinically Validated: Has not undergone clinical validation studies
  • NOT APPROVED FOR CLINICAL USE: Should not be used for patient care or diagnostics

Installation

pip install torch transformers safetensors genomic-tokenizer

Usage

Load the model and generate embeddings using Transformers AutoModel/AutoTokenizer:

import torch
from transformers import AutoModel, AutoTokenizer

# Load model and tokenizer
repo_id = "beapen/genomic-cds-embedding-dim512-v1"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModel.from_pretrained(repo_id, trust_remote_code=True).eval()

# Batch process CDS sequences
sequences = [
    "ATGGCCGCCTACAAGCAGCAGTCTGGGGAAGACGGGCTACCTTGCCCCTGA",
    "ATGGCCGCC",
    "ATGTACGTA"
]

# Tokenize with automatic padding
inputs = tokenizer(sequences, return_tensors="pt", padding=True)

# Generate L2-normalized embeddings
with torch.no_grad():
    embeddings = model(**inputs)

print(f"Embeddings shape: {embeddings.shape}")  # torch.Size([3, 512])
print(f"Embedding norm: {embeddings.norm(dim=-1)}")  # Should be ~1.0 for all

Each embedding is L2-normalized and has dimension 512.

Model Card Contact

Limitations

  1. Training Data: Only includes protein-coding genes from GENCODE v50
  2. Sequence Length: Limited to 2048 tokens (~6144 nucleotides)
  3. Species: Human-only (GRCh38 assembly)
  4. Context: Does not include regulatory regions or non-coding sequences
  5. Variant: Dim512 model - dim128 and dim256 variants available. (dim128 may outperform larger models due to tokenizer compression)

Ethical Considerations

This model is intended for research purposes in genomics and bioinformatics. Users should:

  • Verify biological interpretations with experimental data
  • Follow ethical guidelines for genetic data usage

Citations

If you use this model in your research, please cite the following:

GTEmbed Model

Eapen, B. (2026). GTEmbed: Genomic Tokenizer Embedding Pipeline. Hugging Face. https://huggingface.co/beapen/genomic-cds-embedding-dim512-v1

Genomic Tokenizer

Eapen, B. (2026). Genomic Tokenizer: Codon-level Tokenization for CDS Sequences. GitHub. https://github.com/dermatologist/genomic-tokenizer

Training Data Sources

This model was trained using CDS sequences extracted from two publicly available datasets:

1. GENCODE v50 Human Gene Annotation

GENCODE provides comprehensive reference gene annotation for the human genome. For GENCODE v50 (released in 2023), please cite:

Frankish, A., Diekhans, M., Ferreira, A.M., Johnson, R., Jungreis, I., Loveland, J., Mudge, J.M., Sisu, C., Wright, J., Armstrong, J., Barnes, I., Berry, K., Bignell, A., Carbonell-Sala, S., Cerdán-Vélez, D., Cheng, Y., Davidson, C., Dyer, S.C., Eyrignoux, C., Fatima, R., Fernandez-Banet, J., Gall, A., Garcia Girón, C., Gonzalez, J.M., Greller, R., Hardy, M., Hunt, T., Ishiguro, H., James, B., Jayakumar, A., Juettemann, T., Kay, M., Kulesha, E., Mangan, R.M., Martin, F.J., Mohanan, S., Muir, P., Murphy, S., Nag, A., Overduin, B., Parker, A., Perry, M., Petty, T.L.P., Pujara, P., Raj, S., Sanchez-Castillo, D., Schmitt, B., Schuilenburg, H., Sealy, I., Shafer, R., Sharma, L., Slater, T., Smedley, D., Sobala, A., Spudich, A., Steed, E., Tang, Y., Thakuria, J., Tress, M.L., Vazquez, J., Wilder, S., Zadissa, A., Zerbino, D.R., Cunningham, F., Lander, A., Flicek, P., and Harrow, J. (2023). GENCODE: reference annotation for the human and mouse genomes in 2023. Nucleic Acids Research, 51(D1), D942-D949. https://doi.org/10.1093/nar/gkac1071

Alternative citation for latest GENCODE release (2025):

Mudge, J.M., Carbonell-Sala, S., Diekhans, M., Martinez, J.G., Hunt, T., Jungreis, I., Loveland, J.E., Arnan, C., Barnes, I., Bennett, R., Berry, K., Bignell, A., Cerdán-Vélez, D., Cochran, K., Cortés, L.T., Davidson, C., Donaldson, S., Dursun, C., Fatima, R., Hardy, M., Hebbar, P., Hollis, Z., James, B., Jiang, Y., Johnson, R., Kaur, G., Kay, M., Mangan, R.M., Maquedano, M.G., Gómez, L.M., Mathlouthi, N., Merritt, R., Ni, P., Palumbo, E., Perteghella, T., Pozo, F., Raj, S., Sisu, C., Steed, E., Sumathipala, D., Suner, M.M., Uszczynska-Ratajczak, B., Wass, E., Yang, Y., Zhang, D., Finn, R.D., Gerstein, M., Guigó, R., Hubbard, T.J.P., Kellis, M., Kundaje, A., Paten, B., Tress, M.L., Birney, E., Martin, F.J., and Frankish, A. (2025). GENCODE 2025: reference gene annotation for human and mouse. Nucleic Acids Research, 53(D1), D966-D975. https://doi.org/10.1093/nar/gkae1078

Data source:


2. GRCh38.p14 Human Reference Genome Assembly

The Genome Reference Consortium (GRC) maintains the human reference genome assembly. For GRCh38.p14 (accession: GCF_000001405.40), cite as:

Genome Reference Consortium. (2022). Human genome assembly GRCh38.p14. RefSeq assembly accession: GCF_000001405.40. https://www.ncbi.nlm.nih.gov/assembly/GCF_000001405.40

For the original GRCh38 release (2013), you may also cite:

Church, D.M., Dunn, D.M., Hodgson, A., Loveland, J.E., Smith, A.N., Bevan, P., Cunningham, F., Earhart, S., Farrell, C.M., Gildert, J., Graf, S., Kay, M., Khurana, E., Li, Y., Thakuria, J., Trevanion, S.J., Veyrieras, J.-B., Vogel, J., White, S.J., Aken, B.L., Barrell, D., Brent, S., Chen, Z., Delcher, A., Frankish, A., Gietzen, D., Gilbert, J.G., Graham, U., Hammond, S., Haussler, D., Hermjakob, H., Hume, D.A., Kasprzyk, A., Kulesha, E., Lander, A., Long, J., Melsens, S., Meidl, P., Murphy, S., Nigro, J., Park, T., Rosenfeld, J.A., Schuster, S.C., Sealy, I., Souvorov, A., Souvorova, A., Steward, C., Taudien, S., Tress, M.L., Ureta-Vidal, A., Wallen, T., Weber, R., White, O., Woodward, J.R., Zadissa, A., Zdobnov, E.M., Zerbino, D.R., Blankenberg, D., Kent, W.J., Korf, I., Lander, E.S., Hubbard, T.J.P., Flicek, P., Harrow, J., Weber, R.J., Cooper, D.M., Eichler, E.E., Felsenfeld, G., Green, E.D., Grimwood, J., Guttmacher, C., Hardison, R.C., Istrail, S., Jaffe, D.B., Myers, R.M., Pennacchio, L.A., Ponting, C.P., Rubin, E.M., Ryder, O., Venter, J.C., Waterston, R.H., Wilson, R.K., and Collins, F.S. (2014). Grading the quality of genome assemblies. Nature Biotechnology, 32, 101-104. https://doi.org/10.1038/nbt.2783

Data source:

Best practices for citing genome assemblies:

  • Always include the assembly accession number (GCF_000001405.40) for reproducibility
  • Specify the patch level (p14) when relevant
  • Reference the Genome Reference Consortium as the maintainer

License

Apache 2.0

Tags

genomics, dna-sequences, embedding, transformer, contrastive-learning, protein-coding, cds, genomic-tokenizer, bioinformatics, healthcare, dim512-model

Downloads last month
51
Safetensors
Model size
12.9M params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support