DeepSeek-V4-Flash-DSpark-2.1B-A1.9B-small

This is a small version of deepseek-ai/DeepSeek-V4-Flash-DSpark created for testing and development. Unlike the previous "tiny" models, all hidden dimensions and hyperparameters are kept identical to the base model; only the layer count, layer types per position, and expert count are reduced.

Model Details

  • Base Model: deepseek-ai/DeepSeek-V4-Flash-DSpark
  • Architecture: deepseek_v4 (DeepseekV4ForCausalLM)
  • Total Parameters: 2.085B
  • Activated Parameters: ~1.934B (6 of 8 routed experts per token + 1 shared expert, topk unchanged from base)

Configuration Changes

The following parameters were reduced from the original model:

Parameter Original Small
num_hidden_layers 43 3
layer_types 43 entries (sliding_attention / compressed_sparse_attention / heavily_compressed_attention) [sliding_attention, compressed_sparse_attention, heavily_compressed_attention]
mlp_layer_types 43 entries ([hash_moe x3, moe x40]) [hash_moe, moe, moe]
dspark_target_layer_ids [40, 41, 42] (last 3 of 43 layers) [0, 1, 2] (last 3 of 3 layers)
n_routed_experts 256 8

All other hyperparameters are identical to the base: hidden_size 4096, vocab_size 129280, moe_intermediate_size 2048, num_experts_per_tok 6 (topk unchanged), n_shared_experts 1, q_lora_rank 1024, o_lora_rank 1024, o_groups 8, head_dim 512, index_n_heads 64, index_head_dim 128, index_topk 512, compress_rates {csa: 4, hca: 128}, hc_mult 4, sliding_window 128, rope parameters, scoring_func sqrtsoftplus, routed_scaling_factor 1.5, expert_dtype: fp4, DSpark parameters (dspark_block_size 5, dspark_markov_rank 256, dspark_noise_token_id 128799), and num_nextn_predict_layers: 1.

All three distinct attention types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention) and both MLP types (hash_moe, moe) are preserved, using (attention, mlp) combinations that occur in the base (layer 0 of the base is sliding_attention+hash_moe; compressed_sparse_attention+moe and heavily_compressed_attention+moe occur from layer 3 on), preserving the base's ordering style (sliding + hash_moe layers first). As in the base, sliding-attention layers carry no attention compressor, CSA layers carry a compressor + Lightning Indexer, and HCA layers carry a compressor without indexer.

Note: the parameter count (~2.1B, of which the two untied 129280x4096 embedding matrices are ~1.06B) is expected with fixed base dims; the model is still far smaller than the base and uses the minimal layer set (one layer per distinct attention type) and a minimal expert count that still exercises routing (8 >= topk 6).

Checkpoint Structure

Single-file bf16 checkpoint (model.safetensors, 155 tensors) using the original base naming convention (no model. prefix): layers.N.attn.wq_a.weight, layers.N.ffn.experts.E.w1/w2/w3.weight, layers.N.ffn.gate.tid2eid (hash-moe layers), embed.weight, norm.weight, head.weight, hc_head_fn/base/scale, layers.N.hc_attn_base/fn/scale, etc. Tensor-name structure matches the base model.safetensors.index.json modulo layer count: all 41 tensor-name patterns match exactly (MTP tensors excluded, FP8 .scale tensors not present in this unquantized bf16 checkpoint). Per-layer key sets are identical to the corresponding base layer types (sliding+hash_moe = base layer 0; csa+moe = base layer 4 pattern; hca+moe = base layer 3 pattern).

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "inference-optimization/DeepSeek-V4-Flash-DSpark-2.1B-A1.9B-small",
    device_map="auto", dtype=torch.bfloat16,
)
# transformers keeps deepseek_v4 norm/hc modules in fp32 when loading bf16
# (`_keep_in_fp32_modules_strict`), which breaks the bf16 forward pass;
# cast everything to bf16 (norms still compute statistics in fp32 internally):
model = model.to(torch.bfloat16)
tokenizer = AutoTokenizer.from_pretrained("inference-optimization/DeepSeek-V4-Flash-DSpark-2.1B-A1.9B-small")

input_ids = tokenizer("According to all known laws", return_tensors="pt").input_ids.to(model.device)
output = model.generate(input_ids, max_new_tokens=20)
print(tokenizer.decode(output[0]))

(Alternatively, load with dtype=torch.float32.)

Creation Process

This model was created using the llm-compressor create-tiny-model skill.

  1. Inspected the DeepSeek-V4-Flash-DSpark config: 43 layers with 3 distinct attention types (sliding_attention, compressed_sparse_attention, heavily_compressed_attention) and 2 distinct MLP types (hash_moe, moe), 256 routed experts with topk 6, fp8 quantization_config, expert_dtype: fp4.
  2. Built the small model keeping all dims identical: 3 layers [(sliding_attention, hash_moe), (compressed_sparse_attention, moe), (heavily_compressed_attention, moe)], 8 routed experts, quantization_config stripped (unquantized bf16), expert_dtype: fp4 kept in config.
  3. Fine-tuned on the copypasta toy dataset (bf16, batch size 2, lr 1e-4) until perplexity < 3.0 (final train loss ~0.017).
  4. Converted the checkpoint to the original base tensor naming and validated the tensor-name structure against the base model.safetensors.index.json (pattern sets match exactly modulo layer count).
  5. Validated perplexity and generation (output below).

Notes

  • Backbone-only checkpoint: num_nextn_predict_layers=1 is kept from the base config, but (as with all transformers-saved deepseek_v4 checkpoints) no MTP weights are materialized — matching the convention of e.g. inference-optimization/GLM-5.2-0.8B-A0.8B.
  • The base model is FP8-quantized; quantization_config was removed to produce this unquantized bf16 checkpoint (matching the convention for these entries). The per-tensor FP8 .scale tensors of the base are therefore not present.
  • expert_dtype: fp4 is kept from the base config; with quantization_config removed the experts are stored as regular bf16 parameters.
  • The hash-moe router's tid2eid (token-id -> expert-id) table is a persistent buffer that zero-initializes (real values come from the base checkpoint); it was randomly initialized over the 8 experts here so hash routing exercises multiple experts instead of collapsing to expert 0.
  • dspark_target_layer_ids ([40, 41, 42] in the base = the last 3 of 43 layers) was remapped to [0, 1, 2] (the last 3 of 3 layers) to stay consistent with the reduced layer count. The DSpark parameters are not consumed by transformers 5.17 or vLLM 0.31 (verified); they are kept for config fidelity.
  • Validation output:
    Success: 1.0000675916671753 <= 10.0
    Generating sample text:
    According to all known laws of aviation, there is no way a bee should be able to fly.
    
Downloads last month
89
Safetensors
Model size
2B params
Tensor type
I64
·
BF16
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for inference-optimization/DeepSeek-V4-Flash-DSpark-2.1B-A1.9B-small

Finetuned
(4)
this model

Collection including inference-optimization/DeepSeek-V4-Flash-DSpark-2.1B-A1.9B-small