FWKV ASCII Art Banner

Recurrent language models · 2026

FWKV FWKV ⨉ Bench Labs Bench Labs

FWKV

Simplicity is all you need.

Avg. 62.3 tok/s on a 2-vCPU instance Avg. 168.4 tok/s on a RTX PRO 6000 BLACKWELL ~56M parameters Apache 2.0

What it is

A fixed-decay recurrent core.
Nothing more complicated than that.

FWKV (Feed-forward Weighted Key Value) is a small recurrent language model architecture proposed in May 2026 by FlameF0X. It began as a side project inspired by RWKV, then turned into something faster than any dense or MoE Transformer of similar size on ordinary CPUs.

The recurrence is a single accumulated state with a learned, constant per-channel decay — computed by a standard parallel scan. No pairwise time-mixing, no data-dependent gates, no custom CUDA kernels. The ROSA variant adds an exact, training-free suffix-automaton copy signal so the model can reuse long-range patterns without attention.

It is still early and unstable. Use it if you care about simplicity and CPU speed; skip it if you need a production chat model tomorrow.

Why it exists

Speed from constraints, not from kernels.

  • Decay Fixed per-channel W. The decay is learned once and constant at inference. That single choice lets the whole history collapse into a vectorized Hillis-Steele scan.
  • ROSA Exact causal copy. A suffix automaton answers “what followed this longest prior match?” and injects the answer as an embedding. No attention, no parameters for the predictor itself.
  • Size Factorized embeddings. Vocab projects through a 128-d bottleneck instead of a full d_model matrix. Fewer parameters, cheaper logits, still tied input/output weights.
  • Honest It is not ready for everything. At 50–56M parameters the model confabulates freely. The point of the release is the architecture and the measured CPU throughput, not a claim of intelligence.

Models

Named simply so you can find them.

Pattern: FWKV-[size or type]. Everything lives under the FWKV organization on Hugging Face.

Language models

Vision

Other modalities

  • FWKV-Bert NAP
  • Text-to-speech NAP

Paper

FWKV: Simplicity Is All You Need for Efficient Language Modeling

Technical report · July 2026 · FlameF0X

Read the paper →