Instructions to use inference-optimization/DeepSeek-V4-Flash-DSpark-2.1B-A1.9B-small with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use inference-optimization/DeepSeek-V4-Flash-DSpark-2.1B-A1.9B-small with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="inference-optimization/DeepSeek-V4-Flash-DSpark-2.1B-A1.9B-small")# pip install -U transformers accelerate # Load model directly from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained("inference-optimization/DeepSeek-V4-Flash-DSpark-2.1B-A1.9B-small") model = AutoModelForCausalLM.from_pretrained("inference-optimization/DeepSeek-V4-Flash-DSpark-2.1B-A1.9B-small", device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use inference-optimization/DeepSeek-V4-Flash-DSpark-2.1B-A1.9B-small with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "inference-optimization/DeepSeek-V4-Flash-DSpark-2.1B-A1.9B-small" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "inference-optimization/DeepSeek-V4-Flash-DSpark-2.1B-A1.9B-small", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/inference-optimization/DeepSeek-V4-Flash-DSpark-2.1B-A1.9B-small
- SGLang
How to use inference-optimization/DeepSeek-V4-Flash-DSpark-2.1B-A1.9B-small with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "inference-optimization/DeepSeek-V4-Flash-DSpark-2.1B-A1.9B-small" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "inference-optimization/DeepSeek-V4-Flash-DSpark-2.1B-A1.9B-small", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "inference-optimization/DeepSeek-V4-Flash-DSpark-2.1B-A1.9B-small" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "inference-optimization/DeepSeek-V4-Flash-DSpark-2.1B-A1.9B-small", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use inference-optimization/DeepSeek-V4-Flash-DSpark-2.1B-A1.9B-small with Docker Model Runner:
docker model run hf.co/inference-optimization/DeepSeek-V4-Flash-DSpark-2.1B-A1.9B-small
DeepSeek-V4-Flash-DSpark-2.1B-A1.9B-small
This is a small version of deepseek-ai/DeepSeek-V4-Flash-DSpark created for testing and development. Unlike the previous "tiny" models, all hidden dimensions and hyperparameters are kept identical to the base model; only the layer count, layer types per position, and expert count are reduced.
Model Details
- Base Model: deepseek-ai/DeepSeek-V4-Flash-DSpark
- Architecture: deepseek_v4 (DeepseekV4ForCausalLM)
- Total Parameters: 2.085B
- Activated Parameters: ~1.934B (6 of 8 routed experts per token + 1 shared expert, topk unchanged from base)
Configuration Changes
The following parameters were reduced from the original model:
| Parameter | Original | Small |
|---|---|---|
| num_hidden_layers | 43 | 3 |
| layer_types | 43 entries (sliding_attention / compressed_sparse_attention / heavily_compressed_attention) | [sliding_attention, compressed_sparse_attention, heavily_compressed_attention] |
| mlp_layer_types | 43 entries ([hash_moe x3, moe x40]) | [hash_moe, moe, moe] |
| dspark_target_layer_ids | [40, 41, 42] (last 3 of 43 layers) | [0, 1, 2] (last 3 of 3 layers) |
| n_routed_experts | 256 | 8 |
All other hyperparameters are identical to the base: hidden_size 4096, vocab_size 129280, moe_intermediate_size 2048, num_experts_per_tok 6 (topk unchanged), n_shared_experts 1, q_lora_rank 1024, o_lora_rank 1024, o_groups 8, head_dim 512, index_n_heads 64, index_head_dim 128, index_topk 512, compress_rates {csa: 4, hca: 128}, hc_mult 4, sliding_window 128, rope parameters, scoring_func sqrtsoftplus, routed_scaling_factor 1.5, expert_dtype: fp4, DSpark parameters (dspark_block_size 5, dspark_markov_rank 256, dspark_noise_token_id 128799), and num_nextn_predict_layers: 1.
All three distinct attention types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention) and both MLP types (hash_moe, moe) are preserved, using (attention, mlp) combinations that occur in the base (layer 0 of the base is sliding_attention+hash_moe; compressed_sparse_attention+moe and heavily_compressed_attention+moe occur from layer 3 on), preserving the base's ordering style (sliding + hash_moe layers first). As in the base, sliding-attention layers carry no attention compressor, CSA layers carry a compressor + Lightning Indexer, and HCA layers carry a compressor without indexer.
Note: the parameter count (~2.1B, of which the two untied 129280x4096 embedding matrices are ~1.06B) is expected with fixed base dims; the model is still far smaller than the base and uses the minimal layer set (one layer per distinct attention type) and a minimal expert count that still exercises routing (8 >= topk 6).
Checkpoint Structure
Single-file bf16 checkpoint (model.safetensors, 155 tensors) using the original base naming convention (no model. prefix): layers.N.attn.wq_a.weight, layers.N.ffn.experts.E.w1/w2/w3.weight, layers.N.ffn.gate.tid2eid (hash-moe layers), embed.weight, norm.weight, head.weight, hc_head_fn/base/scale, layers.N.hc_attn_base/fn/scale, etc. Tensor-name structure matches the base model.safetensors.index.json modulo layer count: all 41 tensor-name patterns match exactly (MTP tensors excluded, FP8 .scale tensors not present in this unquantized bf16 checkpoint). Per-layer key sets are identical to the corresponding base layer types (sliding+hash_moe = base layer 0; csa+moe = base layer 4 pattern; hca+moe = base layer 3 pattern).
Usage
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
"inference-optimization/DeepSeek-V4-Flash-DSpark-2.1B-A1.9B-small",
device_map="auto", dtype=torch.bfloat16,
)
# transformers keeps deepseek_v4 norm/hc modules in fp32 when loading bf16
# (`_keep_in_fp32_modules_strict`), which breaks the bf16 forward pass;
# cast everything to bf16 (norms still compute statistics in fp32 internally):
model = model.to(torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/DeepSeek-V4-Flash-DSpark-2.1B-A1.9B-small")
input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))
(Alternatively, load with dtype=torch.float32.)
Creation Process
This model was created using the llm-compressor create-tiny-model skill.
- Inspected the DeepSeek-V4-Flash-DSpark config: 43 layers with 3 distinct attention types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention) and 2 distinct MLP types (hash_moe, moe), 256 routed experts with topk 6, fp8
quantization_config,expert_dtype: fp4. - Built the small model keeping all dims identical: 3 layers [(sliding_attention, hash_moe), (compressed_sparse_attention, moe), (heavily_compressed_attention, moe)], 8 routed experts,
quantization_configstripped (unquantized bf16),expert_dtype: fp4kept in config. - Fine-tuned on the copypasta toy dataset (bf16, batch size 2, lr 1e-4) until perplexity < 3.0 (final train loss ~0.017).
- Converted the checkpoint to the original base tensor naming and validated the tensor-name structure against the base
model.safetensors.index.json(pattern sets match exactly modulo layer count). - Validated perplexity and generation (output below).
Notes
- Backbone-only checkpoint:
num_nextn_predict_layers=1is kept from the base config, but (as with all transformers-saved deepseek_v4 checkpoints) no MTP weights are materialized — matching the convention of e.g.inference-optimization/GLM-5.2-0.8B-A0.8B. - The base model is FP8-quantized;
quantization_configwas removed to produce this unquantized bf16 checkpoint (matching the convention for these entries). The per-tensor FP8.scaletensors of the base are therefore not present. expert_dtype: fp4is kept from the base config; withquantization_configremoved the experts are stored as regular bf16 parameters.- The hash-moe router's
tid2eid(token-id -> expert-id) table is a persistent buffer that zero-initializes (real values come from the base checkpoint); it was randomly initialized over the 8 experts here so hash routing exercises multiple experts instead of collapsing to expert 0. dspark_target_layer_ids([40, 41, 42] in the base = the last 3 of 43 layers) was remapped to [0, 1, 2] (the last 3 of 3 layers) to stay consistent with the reduced layer count. The DSpark parameters are not consumed by transformers 5.17 or vLLM 0.31 (verified); they are kept for config fidelity.- Validation output:
Success: 1.0000675916671753 <= 10.0 Generating sample text: According to all known laws of aviation, there is no way a bee should be able to fly.
- Downloads last month
- 89
Model tree for inference-optimization/DeepSeek-V4-Flash-DSpark-2.1B-A1.9B-small
Base model
deepseek-ai/DeepSeek-V4-Flash-DSpark