_{by Mia'a AI Lab}

Serve deepseek-ai/DeepSeek-V4.1-Flash with SGLang on three DGX Spark boxes (GB10, SM121, 121.7 GiB unified memory each) over a ConnectX-7 RoCE triangle: native MXFP4 experts, FP8 dense weights, NVMe-resident Engram tables, DSpark speculative decoding, tool calling and vision, behind one OpenAI-compatible endpoint.

On disk the checkpoint is 476 GiB. After moving the two Engram n-gram tables (189 GiB) to NVMe, the GPU-resident part is ~305 GiB of MXFP4 experts plus FP8 dense weights:

Host RAM is GPU memory on a Spark. Everything that pins host memory (NCCL buffers, row caches, tokenizer processes) competes with the model, and a head node that runs out of memory stalls all three ranks because TP is synchronous. Most of what this repo does beyond the upstream recipe is about that.

Never use OFFLOAD_MODE=ram here. The Engram tables would evict the model.

spark1  10.0.0.1  rank 0   API :8888, NFSv4 export of the checkpoint, head-node processes
spark2  10.0.0.2  rank 1   mounts spark1's checkpoint over ConnectX (10.0.22.1)
spark3  10.0.0.3  rank 2   mounts spark1's checkpoint over ConnectX (10.0.23.1)

• TP=3, EP=3 (world size 3), NCCL over RoCE (NCCL_NET=IB), Gloo bootstrap on the LAN NIC.
• Workers do not copy the checkpoint: ./start.sh share publishes spark1's tree via the vllm-fn-nfs exporter and creates the dsv41-weights docker volume on each worker.
• Each node keeps its own rank's Engram rows on local NVMe (./start.sh pack, ~63 GiB/node).
• heads=64, o_groups=8, vocab=129280 and the 128 DSpark draft experts are not divisible by 3. adapter/tp3_pad.py pads heads 64→96, groups 8→12 (32 local heads, the smallest size the SM120 sparse-MLA kernels instantiate), vocab to a TP-aligned size and draft experts 128→129, zero-filling the extra shards. Rank 2's attention shard is therefore entirely padding; see Padded shards below for why that matters.

cd ~/NewModels/DS4.1
cp -n .env.example .env        # IPs, users and knobs; already set for spark1/2/3
./start.sh doctor              # ssh, docker, IB, checkpoint shards, image arch, busy GPU
./start.sh build               # base image + this repo's overlay, on all three nodes
./start.sh share               # NFSv4 export + docker volumes on the workers
./start.sh pack                # Engram shards onto each node's NVMe (once; ~10 min)
./start.sh serve               # workers first, then the head; streams the engine log

A full boot takes 12-13 minutes (8 min to read 476 GiB, then the draft model, KV pool and CUDA-graph capture). ./start.sh status, ./start.sh logs [worker1|worker2] [N], ./start.sh smoke, ./start.sh stop.

curl http://10.0.0.1:8888/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"deepseek-v4.1-flash","messages":[{"role":"user","content":"What is 19 + 23?"}],
       "max_tokens":32, "chat_template_kwargs":{"thinking":false}}'

enable_thinking is accepted as an alias for thinking (vLLM / EXL3 clients). Either key turns thinking off; if both are set and disagree, thinking wins. Always send max_tokens (the EXL3 recipe does this on every smoke). Requests that omit it are filled and clamped at DSV41_MAX_NEW_TOKENS (default 32768) so a generation loop cannot run to the 1M context. A live token stream also cannot spin forever: an n-gram or same-line repeat aborts the request (see DSV41_LOOP_*). --watchdog-timeout 1800 does not fire while tokens keep arriving.

Set API_KEY in .env to require a bearer token (state/api-key holds it).

TP4 production line: docs/tp4.md. An opt-in serving line for four Sparks on top of this profile: the upstream SGLang dsv4.1 branch image with RoCEnante (Dockerfile.canary-roce), EP 1 with the routed MoE on b12x, prefill sequence parallel, the fast loader and gated decode adapters. Measured from a fresh clone (sparkDash 1.8.8, greedy, switched fabric): prose c1 87.7 tok/s, code c1 124.8, prose c16 342.7 aggregate, cold prefill 4,059 / 5,855 / 5,900 / 5,925 / 5,797 / 5,394 tok/s at 4k–262k, real text 4.9k–5.0k tok/s at 16k–128k, qeval 72/75, a 1,011,084-token needle passes. The profile, images, results, rollback and credits are on that page.

The improvements for now are only for TP=4 and do not affect TP=3. Thanks to @majewskizby for the awesome PR, who brought the opt-in four-Spark production line: the RoCEnante image, EP 1 with the routed MoE on b12x, prefill sequence parallel, the fast loader and the gated decode adapters, and measured it from a fresh clone. ./start.sh is unchanged.

start-tp4.sh is the same engine and image with a profile of its own: .env.tp4 (copied from .env.tp4.example on first run), state-tp4/ and logs-tp4/, so one checkout can drive a 3-node and a 4-node fleet. Everything TP4-specific lives in .env.tp4.example: 4 workers (WORKER_IPS, WORKER_HOSTS, NFS_SERVER_IPS, one address for all workers behind a switch or one per worker), TP_SIZE=4 (EP_SIZE=2 on the base and canary images, 1 on the production image), and a runtime sized for the ~40 GB per rank that TP4 leaves free (~77 GiB of weights per rank instead of ~101). The profile ships with these defaults:

The KV pin. At TP4 the boot log's budget line — DSV4 memory calculation: bytes_per_full_token=1670.75 ... full_token= — reported room for ~16M tokens (~26.7 GB per rank) at MEM_FRACTION_STATIC=0.90. This profile runs 0.80, which takes ~13 GB off that budget, and pins 8,000,000 tokens (13.4 GB of KV per rank) under it. These settings are tested: the profile boots and serves as shipped. With MAX_RUNNING_REQUESTS=8 the pool holds 8 requests averaging 1,000,000 tokens each (8 × 1Mi = 8,388,608 would be the pin for all 8 at the full context). The 0.80 fraction is what keeps ~24 GiB per rank outside the static pool for the prefill transient, so if the pool ever has to shrink, lower the pin rather than raise the fraction. Long prefills: the indexer's per-chunk transient peaks at about 14 B × chunk × prefix (docs/chunked-prefill-memory.md): ~15 GB for a 1024-token chunk at 1M context and ~60 GB for 4096, which is why a 4096 chunk needs the chunked indexer (DSV41_INDEXER_CHUNKED=1, on in this profile) that scores it in bounded row chunks. A 1M-context needle-in-a-haystack test passed on TP4 with these settings, so the 1M envelope is verified end to end rather than extrapolated; scripts/verify/memguard.py remains the tool for ramping new prompt shapes, and dropping the chunk to 512 is the lever if it ever fires. The budget is not fixed: it is computed from MemAvailable after weights, which moves with the page cache between identical boots, and a pin the budget cannot cover either kills the boot or is silently clamped down — both have been seen here. Read full_token= from the boot log after any change to weights, TP size or NCCL buffers. KV pages are touched only when used, so the pool costs nothing until it fills.

Admission at concurrency 8. SGLang classifies DSpark as DFlash-family (is_dflash_family = is_dflash or is_dspark), and for that family with --max-running-requests >= 8 it auto-enables an admission delayer that holds new prefills until min(4, max(2, (max_running_requests + 5) // 6)) slots are free — 2 at 8 requests, 3 at 16. Under steady arrivals the running batch then never sustains the configured maximum: about 7 of 8, about 13–14 of 16. The TP4 profile passes --min-free-slots-delay 1 (an explicit value wins; ≤1 disables) so 8 means 8. The TP3 profile is unaffected — the formula is off below 8 running requests. Credit for finding this: reported publicly against the 3-Spark recipe, with +43% aggregate at 8 streams; the mechanism is verified here against srt/managers/min_free_slots_delayer.py, the throughput number is not yet reproduced on our fleet.

cp -n .env.tp4.example .env.tp4      # IPs, ssh user, NFS addresses
./start-tp4.sh doctor && ./start-tp4.sh build && ./start-tp4.sh share
./start-tp4.sh pack                  # engram-l*-r<rank>of4.bin shards, once per node
./start-tp4.sh serve                 # ./start-tp4.sh stop|status|logs worker3

What changes against the 3-node profile. The TP4 column is the production line (docs/tp4.md):

Measured on 4× DGX Spark with sparkDash (greedy, 256 completion tokens). Decode is faster than the 3-node fleet at every concurrency — smaller attention GEMMs per rank and no padded shards more than pay for the extra NCCL hop. An independent switchless-ring reproduction (public OpenAI path, idle cluster, greedy code, estimated decode window 129–641) is in docs/tp4-switchless-ring-results.md: C1 window 67.4 tok/s estimated, C8 aggregate 224.8 tok/s. The production profile uses DSpark k=5. The memory headroom is what makes the 1M context possible. Prose aggregate is still climbing at 16 streams (342.7 tok/s) while each stream falls from 87.7 to 22.2 tok/s.

Decode (greedy, 256 tok; aggregate tok/s, per stream in brackets):

Prefill, cold, tok/s (sparkDash 1.8.8, synthetic filler):

The filler is one repeated token, so those 16k–128k rates sit on a hot Engram row. On real text (unique prompts, no prefix-cache hits) the same line runs 4.9k–5.0k tok/s at 16k–128k. The two-pass table is in docs/tp4.md.

Fabric: a Spark has two ConnectX-7 ports, so three nodes form a full triangle but four cannot. A 4-node fleet needs either a RoCE switch, or the opt-in switchless-ring path in docs/switchless-ring.md. The latter requires four DACs, one point-to-point subnet per cable, the patched sparkring NCCL on every rank, and NCCL_SWITCHLESS_RING_ONLY=1; setting only environment variables against stock NCCL is not enough. The guide includes physical cable diagrams, IP examples, patched-library verification, local-weight operation and the exact logs that prove Tree/PAT setup was skipped. For a switched fleet, route NCCL/NFS through the switch and use one NFS_SERVER_IPS address. In either layout, set IB_HCA, NCCL_SOCKET_IFNAME/GLOO_SOCKET_IFNAME, NFS_CLIENTS and the GID index in .env.tp4; ./start-tp4.sh doctor validates the selected path before the cold boot. NCCL_DEBUG=INFO for one boot shows the channels and protocol chosen.

Engram: ./start-tp4.sh pack writes each rank's rows as engram-l<layer>-r<rank>of4.bin (~48 GiB per node at TP4) onto local NVMe; the names carry the TP size, so ENGRAM_DIR can be shared with a 3-node profile on the same machines.

The control scripts take any number of workers (WORKER_IPS/WORKER_HOSTS lists; the legacy WORKER1_IP/WORKER2_IP pairs still work), so 5+ nodes only need a matching TP_SIZE and NNODES. Nothing in the image is TP-specific; the padded-shard repair simply finds nothing to repair at TP4.

start.sh                 doctor / build / share / pack / serve / stop / status / logs / smoke
start-tp4.sh             same commands for 4 Sparks (profile: .env.tp4, state-tp4/, logs-tp4/)
boot.py                  in-container entrypoint: download+verify (optional), launch, smoke
Dockerfile               lmsysorg/sglang:dev-dsv41 pinned by digest (sha256:3dbc3130…, arm64) + adapter/ + runtime/ overlay
Dockerfile.canary        TP4: same base, SGLang dsv4.1 branch tree (fetched at build)
Dockerfile.canary-roce   TP4 production image: + RoCEnante overlay, b12x (SG17 + roce_ring), b12x main as b12x_next
runtime/sources.manifest pinned commits + sha256 of everything the TP4 images fetch (scripts/fetch_runtime.sh)
adapter/
  sitecustomize.py       import hooks that install the pieces below at process start
  encoding_compat.py     enable_thinking alias, publisher effort table, max_tokens cap
  loop_abort.py          n-gram / identical-token / same-line stop on a live decode loop
  tp3_pad.py             TP=3 padding of heads / groups / vocab / draft experts
  engram_backend.py      EngramEmbedding replacement: exact rows from NVMe via a host callback
  row_store.cpp          the row store: O_DIRECT reads, 96-thread miss servicing, packed shards
  mxfp8_b12x.py          routes MXFP8 dense linears to FlashInfer's b12x kernel; repairs the
                         block scales of padded shards
  prefill_empty_cache.py empties the allocator cache after each long prefill chunk (memory)
  wo_a_w8.py             fp8 twin of wo_a (W8 / MID / DROP)            } from knapcio's TP4 fork,
  verify_cap.py          confidence cap on the DSpark verify window     } see Attribution:
  block_verify.py        block verification for sampled rows            } which are verbatim,
  draft_head_fp8.py      fp8 copy of the draft LM head                  } which were changed
  engram_prefetch.py     Engram row lookups on a side stream            }
  autotune_keep.py       keep FlashInfer autotune caches across boots   }
  draft_tau.py, folded_result_fence.py   present, off                   }
  draft_head_fp8_tp4.py, fast_load.py, moe_b12x_next.py, prefill_sp.py, l2_prefetch.py, replicated_split.py,
  draft_main_proj.py, roce_gather.py, router_live.py, hc_fused.py, shared_pad_k.py,
  indexer_chunked*.py, spark_prefill_dense.py   TP4 production line, all gated (docs/tp4-adapters.md)
runtime/flash_mla_sm120.py  SM12x sparse-MLA dispatch (64-token page split for FlashInfer)
scripts/pack_engram.py   repack one rank's Engram rows (weight+scale adjacent) to local disk
scripts/profile/         torch-profiler helper and trace analysers (see Profiling)
files/                   NFS export helpers
benchmarks/, tests/      upstream benchmark, row-store, thinking-alias, encoder-parity, max-tokens and loop-abort tests

.env is sourced by bash: quote any value with spaces.

Measured with torch-profiler traces of all three ranks (DSpark, batch 1, 42 steps):

Where the 118 → 82 ms came from, in order of size:

1. FP8 dense GEMMs. SGLang sends any block size other than 128×128 to a Triton kernel unless --fp8-gemm-backend flashinfer_* is given; this model uses 32×32 ue8m0 blocks. The CUTLASS SM120 MXFP8 kernel FlashInfer picks by default (128×32 tile) was no faster at M=6. The kernel that fits is FlashInfer's b12x warp-level MMA kernel (16|32-row tiles), which SGLang's enum does not expose; adapter/mxfp8_b12x.py routes to it.
2. Padded shards. Rank 2's wq_b/wo_b are all zeros, and the never-written block-scale memory held denormals, so the MXFP8 re-encoding rejected them and 86 layers fell back to Triton. The adapter sets invalid scales on all-zero weight blocks to 1.0 (0 × 1 = 0).
3. Worker Engram shards. Unpacked workers read the checkpoint over NFS twice per row; rank 0 waited ~11 ms per step at the following all-reduce. ./start.sh pack on every node.
4. NCCL buffers (memory, which becomes speed on this box; see below).

• KV is tiny. Only the four kv_source layers store KV, so full-attention KV costs 1,670.75 bytes per token per rank (MLA latent, FP8, replicated across TP): 131k tokens = 0.22 GB, 1M tokens = 1.67 GB, plus a fixed 0.34 GB sliding-window pool.
• The KV budget is what is left after weights: MemAvailable_after_weights − 0.05 × MemAvailable_before_weights − 0.1 GB, and SGLang's "avail mem" on GB10 is literally the OS MemAvailable, so page cache at load time moves it by ±0.5 GB between identical boots. The pin (MAX_TOTAL_TOKENS) is silently clamped to the budget; read the DSV4 memory calculation: ... available_bytes= line of the boot log.
• NCCL allocated 512 connection buffers × 9.19 MiB (Simple + LL128 + LL) in pinned host RAM per node, 4.7 GiB that showed up as unreclaimable Shmem. Decode all-reduces are 61 KB and use the LL protocol; prefill uses Simple. The NCCL_* knobs above cut it to 139 MB and turned the head's 0.2-0.8 GB of free memory into ~6 GB, which is why the boot-time "KV lottery" and the reclaim stalls went away. NCCL logs 2 communicators × 8 channels now.
• Hierarchical cache, CPU offload and swa_full_tokens_ratio do not apply: host RAM is GPU RAM, KV is small, and the SWA pool is cap-sized.
• Long prompts and the allocator. Each 2048-token prefill chunk allocates an indexer-logits buffer proportional to the prefix so far. With PyTorch's native caching allocator a freed smaller block cannot serve the next larger request, so reserved memory grows with the square of the prompt length (8 GiB reserved with nothing live after 64 chunks in a synthetic test): a 64k-128k-token prefill exhausted the head. expandable_segments:True would coalesce that freed space (0.6 GiB in the same test) but cannot be used here: with it every prefill of more than 64 query tokens produces NaN logits (garbage for 65-~4000-token prompts and for any prefill batch of 3+ requests; REPORT.md §17). So the native allocator stays and CONTEXT_LENGTH defaults to 256k but only ~32k of it is usable on the head. On top of the fragmentation, a long prompt costs an upfront transient on the head that scales with its length (the Engram rows of the whole prompt are fetched before the first chunk runs, plus 1.67 KB/token of KV and the CHUNKED_PREFILL_SIZE × context/4 × 4 B indexer-logits buffer per chunk); the measured ramp is in REPORT.md §17.

• Fused MoE finalize is off. FlashInfer's fused finalize sums the six expert outputs with atomic bf16 adds inside the second MoE GEMM. The autotuner selected it for the 32-token bucket only (17-32-token prefills, decode at concurrency 3-4), which made greedy outputs differ run to run by up to ~1 nat on first-token logprobs and rounds to bf16 at every add. With SGLANG_FLASHINFER_MOE_FUSED_FINALIZE=0 repeated requests are byte-identical at every length tested; cost ~0.3 ms per step. Batches at concurrency >1 can still differ between runs because batch composition changes which GEMM tiles run; that is normal SGLang behaviour without --enable-deterministic-inference.
• Reasoning budget. DeepSeek-V4.1 takes a numeric reasoning effort (1-100) as a prompt prefix in thinking mode. SGLang's built-in V4.1 encoder maps the named tiers low/high/xhigh/max to 25/50/75/100 and defaults to high, so every thinking request ran at budget 50 and low meant 25; the publisher's encoding/encoding.py maps low/high/max to 50/75/100 with default 75 (evaluations used 100). adapter/encoding_compat.py remaps the tiers to the publisher's table (xhigh kept as an alias for 75) and SGLANG_DSV41_REASONING_EFFORT=75 pins the default. Per request: the OpenAI reasoning_effort field (a tier, or a 0-0.99 float = budget/100) or "chat_template_kwargs":{"reasoning_effort":N}; medium is not a V4.1 tier and falls back to the default with a warning. The budget changes reasoning length and DSpark acceptance, so state it with any thinking-mode number; the decode tables in this README are thinking-off. Same correction as local-inference-lab/rtx6kpro R37.
• Thinking toggle. SGLang reads chat_template_kwargs.thinking. vLLM-style enable_thinking is copied onto thinking before encode/parse, so either name works. python3 tests/test_thinking_alias.py covers the alias; python3 tests/test_chat_encoding.py diffs tool and thinking-off renders against the checkpoint's encoding/encoding.py (and against SGLang's encoding_dsv41 inside the image).
• Completion cap. Chat requests that omit max_tokens fill the remaining context on this OpenAI path. The 2× EXL3 recipe has no server-side cap either (vLLM does the same fill) but every one of its smokes and README curls sends max_tokens. This recipe now does both: examples send max_tokens, and adapter/encoding_compat.py fills/clamps at DSV41_MAX_NEW_TOKENS (default 32768). Set it to 0 to replay an uncapped harness.
• Loop abort. VIS-04 was a semantic loop, not a hung GPU: 714k tokens over ~2.6 h while --watchdog-timeout 1800 stayed quiet. adapter/loop_abort.py finishes the request after four consecutive copies of a ≥32-token n-gram, 64 identical tokens, or eight identical decoded lines, with OpenAI finish_reason=stop (matched=repetition). The 2× EXL3 recipe has no decode-side detector. python3 tests/test_loop_abort.py covers the detector; set DSV41_LOOP_ABORT=0 to turn it off.
• runtime/flash_mla_sm120.py sends prefills of more than 64 query tokens to FlashInfer's sparse-prefill specialisation and everything else through the decode kernel with a 256→64-token page split; both paths were verified deterministic.
• The autotune cache lives in ~/.cache/sglang/flashinfer/autotune/ on each node (root-owned, bind-mounted into the containers). Clear it on all nodes after changing a kernel flag.

./scripts/profile/profile_decode.sh 45 /tmp/prof        # arms POST /start_profile, waits
# ...send one request from a worker while it runs...
scp trace.json.gz zurih@10.0.0.3:/tmp/ && ssh zurih@10.0.0.3 'cd /tmp && python3 analyze_steps.py trace.json.gz'

analyze_steps.py prints the per-step GPU budget by category and, with a step index, the collapsed kernel sequence of that step; analyze_trace.py prints the kernel table, GPU idle gaps and NCCL latency percentiles; engram_wait.py isolates the all-reduce that follows each Engram gather (how long rank 0 waits for the workers' NVMe reads). Analyse on a worker: no second CUDA context can be created on a node while its rank is up (cudaMemGetInfo itself fails), and the head has little RAM to spare.

• Run load generators and analysis from spark2/spark3, never on the head: it hosts the HTTP server, tokenizer, detokenizer and the NFS export, and is the rank the others wait for.
• Measure decode as usage.completion_tokens / elapsed on non-streaming requests or from the gen throughput field of Decode batch log lines; counting SSE chunks under-reports ~3.5× because each chunk carries a whole accepted speculative block.
• The TP4 decode and prefill tables above were produced with sparkDash, which drives the OpenAI endpoint at a fixed stream count and reports aggregate and per-stream rates alongside TTFT.
• ./start.sh build rsyncs the repo to the workers with --delete; engram/ and the state, logs and .env are excluded. Do not edit start.sh while a ./start.sh command is running (bash reads scripts incrementally).
• The spare containers on spark3 add jitter to rank 2 (it was 5-8% slower on identical GEMMs). Stop what you can while serving.
• NCCL_DEBUG=INFO with NCCL_DEBUG_SUBSYS=INIT,ENV for one boot shows the communicator, channel and protocol choices.

• After the 2026-09-24 changes the bs=1 step is ~59 ms (docs/overnight-results.md): MoE ~21 ms, b12x dense FP8 ~17 ms, NCCL ~5.6 ms, wo_a twin 3.5 ms, the draft's bf16 wo_a 2.4 ms.
• The MoE cost scales with the 6-token verify window. A profiled SPS table (python -m sglang.benchmark.dspark_sps_profiler all --base-url ..., pushed to every rank by start.sh) turns on compact ragged verify, which mainly pays at concurrency ≥2.
• Fewer, fused small kernels (~2000 per step) and the 13-16 ms of cross-node latency are the remaining floor at TP=3.

• The recipe skeleton, boot.py, the Engram row-store idea, benchmarks/, tests/ and the overall Docker layout originate from 0xSero/deepseek-v4.1-flash-4x-rtx-pro-6000 (MIT, notice retained in LICENSE.upstream-MIT). Everything Spark-specific (TP=3 padding, NFS sharing, packed shards, the b12x route, the padded-scale repair, the NCCL and finalize settings) was added here.
• Decode adapters from knapcio/DeepSeek-V4.1-Flash-4x-DGX-Spark-TP4 (AGPL-3.0, the same licence; that repository builds on this one). Written by knapcio for a 4-Spark TP4 fleet, taken at commit 4d8f4c0 on 2026-09-24. Thank you to knapcio for the kernels, the measurements behind them, and the public write-ups that made them easy to evaluate.
  • Used verbatim: verify_cap.py, block_verify.py, draft_head_fp8.py, engram_prefetch.py, draft_tau.py, folded_result_fence.py and their tests. draft_tau and folded_result_fence ship off here (not demonstrated on TP3).
  • Adapted here: wo_a_w8.py: generalised from TP4's 2 local o_groups to TP3's 4 padded groups, an einsum bridge for the older dev-dsv41 engine (which lacks the kernel module the original patches), a fix that keeps the DSpark draft's wo_a out of DROP (on this engine the draft reads the weight directly and DROP turned every draft into garbage), and the optional DSV41_WO_A_W8_DRAFT route. autotune_keep.py: a stale-cache fix (with no matching sidecar on any rank the stock gate still loaded every rank's old file) and exclusion of the per-boot SGLANG_RUN_ID from the launch fingerprint (with it, no cache was ever reused); test extended.
  • Configuration ideas from knapcio's measurements: DSpark k=5 with the conf:0.1 cap. Each was re-measured on TP3 one switch per boot before it became a default.
  • Not adopted for the TP3 profile: RoCEnante transport, the display-reservation (DRM) row cache, shared-expert padding, fast load, the chunked-indexer prefill variants and the canary image. They are part of the opt-in TP4 production line below.
• TP4 production line (docs/tp4.md): contributed by knapcio from knapcio/DeepSeek-V4.1-Flash-4x-DGX-Spark-TP4 (v2.1; per-change history and measurements there), with work from b12x / local-inference-lab (RoCEnante, fused MoE), rhys101 (SG17 RoCEnante overlay, SG18 prefill TP split), LuZ, sumsliu, FujitsuPolycom/sparkring and rsync (RoCEnante on a switchless ring), Saolence, kpham-sgl (sglang#39187) and the SGLang dsv4.1 branch; full credits in docs/tp4.md. Third-party sources are fetched at build from their pinned commits (runtime/sources.manifest).
• Originally this repository's (unchanged by the above): TP3 padding (tp3_pad.py), the b12x MXFP8 route and padded-scale repair (mxfp8_b12x.py), the NVMe Engram row store and packed shards (engram_backend.py, row_store.cpp, scripts/pack_engram.py, which the prefetch adapter builds on), the prefill empty-cache hook, loop abort, the encoder/effort compatibility layer, the NCCL buffer and fused-finalize settings, the NFS sharing and multi-node launcher, the TP3 measurements, and the TP3/EP1 finding (EP 1 was this repository's own A/B, not a knapcio setting). The 2026-09-24 campaign that selected and validated the combination is in docs/overnight-results.md.
• runtime/flash_mla_sm120.py is Apache-2.0 from SGLang (LICENSE.sglang).
• Serving engine: SGLang dev build with FlashInfer kernels.
• Model weights stay on Hugging Face / local NVMe and are not redistributed. Pinned checkpoint: deepseek-ai/DeepSeek-V4.1-Flash @ fb2764a5cf321eaa5070ca8f9e892818f477c16d.

This repository is licensed under the GNU Affero General Public License v3.0 or later (LICENSE). It builds on 0xSero's MIT-licensed recipe (notice kept in LICENSE.upstream-MIT) and includes one Apache-2.0 file from SGLang (LICENSE.sglang); see NOTICE for the full breakdown. Model weights are not part of this repository.