Study Says DeepSeek-V4-Flash Uses Only About Two of Four Residual Streams, Deep-Layer Mixing Degrades to Identity
A study covered by a machine-intelligence outlet and published on arXiv in September 2026 reports that forward analysis and inference-time interventions on the public DeepSeek-V4-Flash-0731 weights found that mHC's four residual streams concentrate their read-write weights on about two streams, that the residual mixing matrices in layers 22 to 42 are close to the identity matrix, and that replacing deep-layer mixing with identity leaves the six-task average score almost unchanged.
What changed
According to the paper and the machine-intelligence report, DeepSeek-V4-Flash uses Manifold-Constrained Hyper-Connections (mHC) to maintain four parallel residual streams, with read-write mappings and residual mixing matrices dynamically generated from input tokens, and the mixing matrices constrained to be doubly stochastic via Sinkhorn-Knopp iteration. The researchers used the public DeepSeek-V4-Flash-0731 weights, sampled 512 sequences of length 1024 from C4, and recorded the read-write weights and mixing matrices for each token at every sublayer. The reported dominant-stream consistency averages 0.871 for the read mapping and 0.905 for the write mapping; the average effective number of streams is 1.998 for the read mapping and 1.775 for the write mapping, meaning the architecture provides four streams but typical computation uses only about two. The average cosine similarity among the four streams is about 0.404, with representational differentiation completed in the first few layers. Using the mean absolute deviation between the mixing matrix and the identity matrix to measure cross-stream mixing strength, 32 of the 40 sublayers in layers 23 to 42 have a deviation below 0.01. Inference-time interventions show: removing the lowest-weight read-write path per token raises PPL by at most 2.7% and lowers the average score on six zero-shot tasks by at most 0.38 percentage points; replacing the mixing matrices in layers 22 to 42 with the identity matrix raises PPL by 1.9% and changes the average score by +0.04 percentage points; replacing across the whole network raises PPL by 42.3% and lowers the average score by 3.28 percentage points; fixing shallow layers to the token-averaged matrix over the C4 calibration set raises PPL by 0.2% and lowers the average score by 0.25 percentage points. The paper also mentions that Tencent Hy4-preview's iHC and Qwen3.8-Flash-Next's Gated Residual use simpler residual pathways. All of the above are reported conclusions of the paper and have not yet been independently reproduced.
What it means for developers
The value of this kind of analysis lies not in leaderboards but in providing a basis for decisions about deploying multi-stream residual architectures and about in-house development routes. If the conclusions hold, the practical sparsity of reads and writes means that a substantial portion of mHC's inference overhead is spent on paths with very small weights, which bears directly on memory usage under batching and on the room for fusing attention and FFN operators; at the same time, the near-identity mixing in deep layers suggests that cross-stream communication in that range can be simplified, while cross-stream mixing in shallow layers remains necessary, so simplification can only be applied non-uniformly along the depth dimension. For teams fine-tuning or distilling with these weights, the finding suggests that the four streams' representations are not redundant (cosine similarity around 0.4) but the read-write paths are highly concentrated, so the benefits and risks of pruning the weakest paths need to be assessed layer by layer. It should be noted that the intervention metrics in the paper come from C4 perplexity and six zero-shot tasks, an academic evaluation protocol that differs from production workloads oriented toward dialogue, agents, and long-context retrieval; the reported scores cannot be directly converted into business quality. In addition, the study covers only a single public weight version, and the transferability of its conclusions to other versions in the same family has not been verified. The paper cites two domestic multi-stream models' simplified designs in the same paragraph, which can serve as architectural comparisons but do not mean those designs have been verified by the same experiments.
Integration and evaluation
The following is conditional analysis based on the report, not a conclusion of the paper. First, if a team is evaluating whether to perform operator-level optimization on mHC-like structures, it is advisable to first reproduce the statistics on 'dominant-stream consistency and effective number of streams': the basis for this choice is that the reported effective number of streams for the read-write mappings is about 1.8 to 2.0, and a practical judgment is to compare the metric for the model to be tested with the measured values on one's own traffic data; if the measured value is clearly higher than the reported value, the version's routing is more dispersed, and the premise for operator pruning based on the sparse assumption does not hold. A measurable validation criterion is to record the PPL change before and after pruning on one's own fixed-length sequences, and to set the threshold of concern at whether the reported order of 2.7% is exceeded. Second, if considering an alternative implementation that applies identity mixing to deep layers in order to avoid per-token generation of the mixing matrices, the precondition is that a deep-layer range with deviation below 0.01 actually exists in the model; the basis for this choice is that the reported average score changes by +0.04 percentage points after deep-layer replacement, whereas whole-network replacement lowers it by 3.28 percentage points, so only layer-wise treatment is possible. A measurable validation criterion is to run six tasks or one's own task set by depth, confirming that the score difference from deep-layer replacement falls within evaluation noise while shallow layers retain dynamic mixing. Third, if the plan is only a low-cost experiment, one can start with the reported intervention of 'fixing shallow layers to the calibration-set average matrix', replacing per-token dynamism with a static matrix, because the report says PPL rises by only 0.2%; a measurable criterion is to compare first-token latency and throughput between the static and dynamic versions, to judge whether the dynamism saved is worth the cost of 0.25 percentage points in average score. All thresholds come from the paper's academic evaluation setting and need to be recalibrated before being transferred to one's own workloads.
Limits and open questions
The specific version studied is DeepSeek-V4-Flash-0731, and as of the KimiSeek catalog check on October 24, 2026, this version was not found, so it cannot be confirmed whether this exact version is available on the site; the weight and API availability mentioned in this report refer to vendor-side information, and readers should rely on vendor channels. All numbers, ratios, and scores described in this article come from the paper's reported account, have not been independently reproduced, and the evaluation is limited to C4 perplexity and six zero-shot tasks: ARC-Easy, ARC-Challenge, PIQA, HellaSwag, MMLU, and GSM8K. This is an academic protocol and does not represent performance in Chinese, long-context, or agent scenarios. The weights used in the intervention experiments are a specific public snapshot, and the conclusions cannot be presumed to apply to other versions in the same family or to other vendors' multi-stream residual structures; the paper's references to Hy4-preview and Qwen3.8-Flash-Next are only architectural analogies, not experimental conclusions about those models. In addition, architecture-level analysis does not directly indicate server-side API pricing, rate limits, or context configuration; for related costs, please consult vendor or platform pricing pages.