Open Source Long Context LLMs Went 1M. Demand Proof.

Qwen 4 now ships a native 1M-token context window under an Apache 2.0 license. And the open source long context LLM field around it filled in fast: DeepSeek V4 Pro at 1M-class, Gemma 4.5 at 1M, Mistral Voyage Pro at 512K. That’s your shortlist if you’re picking this week. And there’s a 256K model you can pull locally with one Ollama command.

The catch you need before committing: window size is the easiest spec to print and the hardest to trust.

And the benchmarks that actually stress a 1M window mostly live in research papers, not model cards.

The pace is the real story.

Eight flagship open-weight releases landed between May 9 and June 6, 2026: the full Qwen 4 release, Qwen 4 Coder, Qwen 4 4B, Llama 5 70B, Mistral Voyage Pro 70B, Gemma 4.5 12B, Phi-5 Medium. And Grok 4 Open, per llmcheck.net’s June 2026 roundup. Meta had already released Llama 4 Scout and Maverick in April 2026. The roundup’s own summary is the part I’d tattoo on the wall: “The 70B dense tier has become a real battleground, the 14B tier is competitive again. And for the first time every major Western foundation lab has at least one model with downloadable weights.”

What Actually Shipped, and What You’re Allowed to Do With It

Here’s the field as the sources report it, sorted by context window:

The license column is not decoration.

Qwen 4 under Apache 2.0 and Kimi K2.5 under Modified MIT, released January 27, 2026 per a review at nxcode.io, mean you can ship client deliverables on these weights without a procurement email chain.

That’s a genuine change from even a year ago, when “open” often meant “open until the usage clause bites.”

The 16GB Number That Changes the Math

The spec everyone quotes is context length. The number I care about is 16GB. An arXiv paper on sparse long-context inference demonstrates inference on context windows up to 1M tokens using approximately 16GB of GPU RAM, with the code released as an open GitHub project called `topk-decoding`. The mechanism: attending to less than 2% of input tokens still achieved over 95% of model performance on benchmarks including RULER, AlpacaEval.

And the Open LLM Leaderboard.

A related experiment from the same line of work took GradientAI’s Llama-3-8B, trained to a 1M-token context. And solved RULER’s needle-in-a-haystack task at 1M tokens on a single GPU, finding that k = 1 attention was sufficient in that test.

Read that again: one token of attention, one GPU, a million tokens of context.

Here’s why a small operator should care. A 16GB card is a gaming rig, not a cluster. If sparse attention holds up on your workload, whole-repo code review and multi-document analysis stop being a per-token API line item and become a box under your desk.

My rule for pricing client work: no context-heavy job goes to an API until the local version has been benchmarked against it on real data.

Window Size Is a Spec. RULER Is Evidence.

Every roundup lists the window.

Almost none tell you what degrades inside it. The evidence that does exist comes from the research side, and it’s more encouraging than the marketing.

A 2025 arXiv paper extended LLaMA-3.1-8B-Instruct to a 1 million token context using step-by-step RoPE scaling, performing well on RULER and InfiniteBench and surpassing gradientai/Llama-3-8B-Instruct-Gradient-1048k on long-context benchmarks.

The SelfLong-8B-1M work showed lightweight post-training with synthetic data can push a Llama-3 family model to 1M tokens while keeping near-perfect needle-in-haystack performance and outperforming other open-source long-context LLMs on RULER. Tooling: public too: Princeton’s ProLong hosts open models and released a 210 GB ArXiv (SlimPajama) subset of its training data. And LongAlign describes itself as “the first full recipe for LLM alignment on long context,” with models in its linked repos even though its Releases tab reads “No releases published.”

The frontier is already past 1M.

The Awesome-LLM-Long-Context-Modeling list tracks PI-Mem pushing long-context reasoning to 3.6M tokens and Long-VITA scaling multi-modal models to 1M tokens without giving up short-context accuracy.

So when a vendor model card gives you a window size and nothing else, that’s not a benchmark, it’s a sticker. Demand RULER numbers or run needle-in-a-haystack yourself before you believe any window, including the ones in my table above.

What a Small Shop Should Actually Do

Start with the model you can try tonight. laguna-xs-2.1, released by Poolside on July 2, 2026, is a 33B total / 3B active MoE built for agentic coding and long-horizon local work, with 256K context, HumanEval 70.9%.

And a one-line install: `ollama run laguna-xs-2.1`. If your job is one codebase or one log dump, 256K is probably enough. And it’s the difference between a free evening experiment and a spec debate.

If you need smaller and multilingual, AMD and Silo AI announced Poro 2 8B Long in August 2026 with a “significantly expanded context window,” available on HuggingFace. And before you stuff a million tokens into anything, look at Hugging Face’s mDenseOn and mLateOn: two open 307M-parameter retrieval models trained on a 2.8B-pair corpus for long-context and code retrieval. Pre-filtering with a 307M model and reasoning over the survivors is often cheaper and more accurate than brute-forcing the full window. That trade is the actual engineering decision, not which model has the biggest number.

The Bottom Line

Open source long context crossed the 1M line in 2026, the licenses are permissive enough for client work.

And the sparse-attention research says the hardware bar is lower than the spec sheets imply.

Treat every context number as a claim, not a fact, and test with your own documents. Pick one workflow where you’re currently pasting files into a chat window, run the 256K local model against it this week. And compare output quality against what you’re paying for now. If you want help deciding which workflows deserve local long-context and which should stay on an API, that’s what Mediascout does. Tell me what you’re feeding your model and I’ll tell you where the money is.