- 🚀 COPM: Converting clinical signals into outcome intelligence for DHTI.
CDS Embedding Model with Genomic Tokenizer (Minimal Variant)
Minimal CDS embedding model for quick prototyping and testing. Trained using contrastive learning with the Genomic Tokenizer on human protein-coding DNA sequences.
Model Details
- Variant: Minimal
- Architecture: Transformer Encoder with 2 layers, 4 attention heads
- Hidden Size: 256 dimensions
- Embedding Dimension: 256 dimensions (L2-normalized)
- Vocabulary: 26 tokens (Genomic Tokenizer)
- Max Sequence Length: 2048 tokens
- Training Objective: Contrastive learning (instance discrimination)
- Parameters: ~2M parameters
Model Variants
This model is part of a family of CDS embedding models:
| Variant | Parameters | Layers | Hidden Size | Use Case |
|---|---|---|---|---|
| Minimal | ~2M | 2 | 256 | Development, testing |
| Small | ~10M | 4 | 512 | Research, moderate tasks |
| Regular | ~50M | 12 | 768 | Advanced, high accuracy |
Genomic Tokenizer
This model uses the Genomic Tokenizer which tokenizes DNA sequences at the codon level:
- 26 unique tokens:
- Special tokens:
[CLS],[SEP],[BOS],[MASK],[PAD],[UNK] - 20 amino acid codons (Alanine, Cysteine, Aspartic_acid, etc.)
- Special tokens:
Each token represents a codon (3 nucleotides) that encodes a specific amino acid, enabling the model to learn at the protein-coding level rather than individual nucleotides.
Training Data
- Dataset: GENCODE v50 human protein-coding CDS sequences
- Genome Assembly: GRCh38
- Transcripts: ~17,000 protein-coding transcripts
- Preprocessing:
- Extracted CDS from GTF annotations
- Reconstructed sequences from GRCh38 genome
- Quality control: start/stop codon verification, reading frame validation
Use Cases
Best for development, debugging, and small-scale experiments: ✅ Recommended:
- Gene prioritization in rare disease diagnostics (as a research tool)
- Functional annotation of novel variants
- Disease gene discovery in research settings
- Building gene-disease knowledge graphs
Clinical Validation Status
- Research Phase: Model is intended for research and translational applications
- Not Clinically Validated: Has not undergone clinical validation studies
- NOT APPROVED FOR CLINICAL USE: Should not be used for patient care or diagnostics
Installation
pip install torch genomic-tokenizer pydantic
Loading the Model
import torch
import json
from pathlib import Path
from huggingface_hub import hf_hub_download
from genomic_tokenizer import GenomicTokenizer
# You need the model classes (install gtembed package or copy model.py)
# from src.gtembed.model import CDSEmbeddingModel
# from src.gtembed.config import ModelConfig
# Download model files
# Replace with your variant: minimal, small, or regular
repo_id = "beapen/gt-embed-minimal-v1" # e.g., beapen/gt-embed-minimal-v1
config_path = hf_hub_download(repo_id=repo_id, filename="config.json", repo_type="model")
model_path = hf_hub_download(repo_id=repo_id, filename="model.pt", repo_type="model")
# Load config
with open(config_path) as f:
config_dict = json.load(f)
# Initialize model (requires gtembed package)
# config = ModelConfig(**config_dict)
# model = CDSEmbeddingModel.from_config(config)
# model.load_state_dict(torch.load(model_path, map_location="cpu"))
# model.eval()
# Load tokenizer
tokenizer = GenomicTokenizer(model_max_length=2048)
Generating Embeddings
import torch
# Sample CDS sequence (protein-coding DNA)
cds_sequence = "ATGGCCGCCTACAAGCAGCAGTCTGGGGAAGACGGGCTACCTTGCCCCTGA"
# Tokenize
tokens = tokenizer(cds_sequence)
input_ids = torch.tensor([tokens['input_ids']])
# Generate embedding
with torch.no_grad():
embedding = model(input_ids)
print(f"Embedding shape: {embedding.shape}") # [1, 256]
print(f"Embedding norm: {embedding.norm().item():.4f}") # Should be ~1.0 (L2 normalized)
Batch Processing
sequences = [
"ATGGCCGCCTACAAGCAGCAGTCTGGGGAAGACGGGCTACCTTGCCCCTGA",
"ATGGCCGCCTACAAGCAGCAGTCTGGGGAAGACGGGCTACCTTGCCCCTGA",
# ... more sequences
]
# Tokenize all
tokenized = tokenizer(sequences, padding=True, return_tensors="pt")
input_ids = tokenized['input_ids']
attention_mask = tokenized['attention_mask']
# Generate embeddings
with torch.no_grad():
embeddings = model(input_ids, attention_mask=attention_mask)
print(f"Batch embeddings shape: {embeddings.shape}") # [batch_size, 256]
Model Card Contact
- Author: Bell Eapen
- Website: https://nuchange.ca
- Date: 2026-11-16
- Framework: PyTorch 2.5.1
- Variant: Minimal
- Repository: https://github.com/dermatologist/gtembed
Limitations
- Training Data: Only includes protein-coding genes from GENCODE v50
- Sequence Length: Limited to 2048 tokens (~6144 nucleotides)
- Species: Human-only (GRCh38 assembly)
- Context: Does not include regulatory regions or non-coding sequences
- Variant: Minimal model - consider Small or Regular variants for production use
Ethical Considerations
This model is intended for research purposes in genomics and bioinformatics. Users should:
- Verify biological interpretations with experimental data
- Follow ethical guidelines for genetic data usage
Citations
If you use this model in your research, please cite the following:
GTEmbed Model
Eapen, B. (2026). GTEmbed: Genomic Tokenizer Embedding Pipeline. Hugging Face. https://huggingface.co/beapen/gt-embed-minimal-v1
Genomic Tokenizer
Eapen, B. (2026). Genomic Tokenizer: Codon-level Tokenization for CDS Sequences. GitHub. https://github.com/dermatologist/genomic-tokenizer
Training Data Sources
This model was trained using CDS sequences extracted from two publicly available datasets:
1. GENCODE v50 Human Gene Annotation
GENCODE provides comprehensive reference gene annotation for the human genome. For GENCODE v50 (released in 2023), please cite:
Frankish, A., Diekhans, M., Ferreira, A.M., Johnson, R., Jungreis, I., Loveland, J., Mudge, J.M., Sisu, C., Wright, J., Armstrong, J., Barnes, I., Berry, K., Bignell, A., Carbonell-Sala, S., Cerdán-Vélez, D., Cheng, Y., Davidson, C., Dyer, S.C., Eyrignoux, C., Fatima, R., Fernandez-Banet, J., Gall, A., Garcia Girón, C., Gonzalez, J.M., Greller, R., Hardy, M., Hunt, T., Ishiguro, H., James, B., Jayakumar, A., Juettemann, T., Kay, M., Kulesha, E., Mangan, R.M., Martin, F.J., Mohanan, S., Muir, P., Murphy, S., Nag, A., Overduin, B., Parker, A., Perry, M., Petty, T.L.P., Pujara, P., Raj, S., Sanchez-Castillo, D., Schmitt, B., Schuilenburg, H., Sealy, I., Shafer, R., Sharma, L., Slater, T., Smedley, D., Sobala, A., Spudich, A., Steed, E., Tang, Y., Thakuria, J., Tress, M.L., Vazquez, J., Wilder, S., Zadissa, A., Zerbino, D.R., Cunningham, F., Lander, A., Flicek, P., and Harrow, J. (2023). GENCODE: reference annotation for the human and mouse genomes in 2023. Nucleic Acids Research, 51(D1), D942-D949. https://doi.org/10.1093/nar/gkac1071
Alternative citation for latest GENCODE release (2025):
Mudge, J.M., Carbonell-Sala, S., Diekhans, M., Martinez, J.G., Hunt, T., Jungreis, I., Loveland, J.E., Arnan, C., Barnes, I., Bennett, R., Berry, K., Bignell, A., Cerdán-Vélez, D., Cochran, K., Cortés, L.T., Davidson, C., Donaldson, S., Dursun, C., Fatima, R., Hardy, M., Hebbar, P., Hollis, Z., James, B., Jiang, Y., Johnson, R., Kaur, G., Kay, M., Mangan, R.M., Maquedano, M.G., Gómez, L.M., Mathlouthi, N., Merritt, R., Ni, P., Palumbo, E., Perteghella, T., Pozo, F., Raj, S., Sisu, C., Steed, E., Sumathipala, D., Suner, M.M., Uszczynska-Ratajczak, B., Wass, E., Yang, Y., Zhang, D., Finn, R.D., Gerstein, M., Guigó, R., Hubbard, T.J.P., Kellis, M., Kundaje, A., Paten, B., Tress, M.L., Birney, E., Martin, F.J., and Frankish, A. (2025). GENCODE 2025: reference gene annotation for human and mouse. Nucleic Acids Research, 53(D1), D966-D975. https://doi.org/10.1093/nar/gkae1078
Data source:
- File:
gencode.v50.annotation.gtf.gz - Download URL: https://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_50/
- Website: https://www.gencodegenes.org/human/release_50.html
2. GRCh38.p14 Human Reference Genome Assembly
The Genome Reference Consortium (GRC) maintains the human reference genome assembly. For GRCh38.p14 (accession: GCF_000001405.40), cite as:
Genome Reference Consortium. (2022). Human genome assembly GRCh38.p14. RefSeq assembly accession: GCF_000001405.40. https://www.ncbi.nlm.nih.gov/assembly/GCF_000001405.40
For the original GRCh38 release (2013), you may also cite:
Church, D.M., Dunn, D.M., Hodgson, A., Loveland, J.E., Smith, A.N., Bevan, P., Cunningham, F., Earhart, S., Farrell, C.M., Gildert, J., Graf, S., Kay, M., Khurana, E., Li, Y., Thakuria, J., Trevanion, S.J., Veyrieras, J.-B., Vogel, J., White, S.J., Aken, B.L., Barrell, D., Brent, S., Chen, Z., Delcher, A., Frankish, A., Gietzen, D., Gilbert, J.G., Graham, U., Hammond, S., Haussler, D., Hermjakob, H., Hume, D.A., Kasprzyk, A., Kulesha, E., Lander, A., Long, J., Melsens, S., Meidl, P., Murphy, S., Nigro, J., Park, T., Rosenfeld, J.A., Schuster, S.C., Sealy, I., Souvorov, A., Souvorova, A., Steward, C., Taudien, S., Tress, M.L., Ureta-Vidal, A., Wallen, T., Weber, R., White, O., Woodward, J.R., Zadissa, A., Zdobnov, E.M., Zerbino, D.R., Blankenberg, D., Kent, W.J., Korf, I., Lander, E.S., Hubbard, T.J.P., Flicek, P., Harrow, J., Weber, R.J., Cooper, D.M., Eichler, E.E., Felsenfeld, G., Green, E.D., Grimwood, J., Guttmacher, C., Hardison, R.C., Istrail, S., Jaffe, D.B., Myers, R.M., Pennacchio, L.A., Ponting, C.P., Rubin, E.M., Ryder, O., Venter, J.C., Waterston, R.H., Wilson, R.K., and Collins, F.S. (2014). Grading the quality of genome assemblies. Nature Biotechnology, 32, 101-104. https://doi.org/10.1038/nbt.2783
Data source:
- File:
GRCh38.primary_assembly.genome.fa.gz - Assembly: GRCh38.p14 (patch release 14)
- RefSeq Accession: GCF_000001405.40
- GenBank Accession: GCA_000001405.29
- Download URL: ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCF/000/001/405/GCF_000001405.40_GRCh38.p14/
- Website: https://www.ncbi.nlm.nih.gov/assembly/GCF_000001405.40/
- GRC Website: https://www.ncbi.nlm.nih.gov/grc/human
Best practices for citing genome assemblies:
- Always include the assembly accession number (GCF_000001405.40) for reproducibility
- Specify the patch level (p14) when relevant
- Reference the Genome Reference Consortium as the maintainer
License
Apache 2.0
Tags
genomics, dna-sequences, embedding, transformer, contrastive-learning, protein-coding, cds, genomic-tokenizer, bioinformatics, healthcare, minimal-model
- Downloads last month
- 40
