the lost malloc()
[R] There is no free lunch in RLVR: under strategy specialization, every pass@1 gain provably costs some prompt, and the pass@k crossover has a tight bound [R]
Paper: https://arxiv.org/abs/2610.39295
Code + verification package (every number in the paper is checked by a script): https://github.com/yliusci/no-free-lunch-rlvr
Browser diagnostic: https://huggingface.co/spaces/yliusci/passk-price
I kept running into the same pattern in RLVR papers: the RL model wins at pass@1 and the base model catches up or wins at large k (Yue et al. 2025 and many follow-ups). I wanted to know whether more training fixes this or whether it is structural. This is a theory paper, no new training runs.
Setup: model RLVR as an exponential tilt of the base policy over solution strategies shared across prompts (exact for mirror descent, and GRPO stays within 3e-4 TV of it at matched entropy in simulation).
What comes out:
No free lunch. Under exclusive specialization, every update that raises pass@1 strictly lowers success on some prompt. The exact condition is a Stiemke duality statement (a strictly positive price on prompts that equalizes strategy fitness). Being the best strategy on each prompt is provably NOT enough: there is a 3x3 counterexample with a free lunch.
The pass@k price. Once the losses reach the hardest prompts, a crossover is forced. The number of crossings is at most the sign variation of the shift in per-prompt success (Karlin total positivity). The bound is tight for 1 to 6 crossings, and the order-preserving case gives exactly one, which recovers the known single-crossover result.
A frontier, not a bug. With a per-prompt floor, the constrained update traces the optimal pass@1 vs pass@k trade. Forbidding any per-prompt loss removes the crossover and the whole pass@1 gain with it.
The same tilt gives the entropy-performance law with an identifiable exponent.
Practical part: base and RL per-prompt success rates (already in any eval log) are enough to read off the crossing count, the bound, and the price (worst pass@k loss). On published GRPO numbers for Omni-MATH (transcribed from 2607.20543, so indicative only) it finds one crossing between k=64 and k=256.
Where I expect pushback: the shared-strategy model is a population-limit abstraction, and the forced crossover needs a coverage assumption (the paper gives the regime where it fails). I would really like to see per-prompt logs where people observe 2+ crossings, since that is the signature of RL reordering prompt difficulty.
[link] [comments]
Benchmarking: Qwen 27B, 35B, QFN180B; Nemotron Bolt 74B, Flint40B & Clover 35B (Q4) on M5 Max 2026 on Kapitsa with KFlash [P]
![Benchmarking: Qwen 27B, 35B, QFN180B; Nemotron Bolt 74B, Flint40B & Clover 35B (Q4) on M5 Max 2026 on Kapitsa with KFlash [P]](https://external-preview.redd.it/lD506kUiZbfR2XmsJJ8yNSBVY6lCisMuJn2qf-Dfx7s.png?width=140&height=70&auto=webp&s=d3781be07761f18fd70c3cf16dc245eb19de2a79 "Benchmarking: Qwen 27B, 35B, QFN180B; Nemotron Bolt 74B, Flint40B & Clover 35B (Q4) on M5 Max 2026 on Kapitsa with KFlash [P]") | hey folks, I'm Ye, one of the contributors to Kapitsa, an open-source inference engine optimized specifically for Apple Silicon to run Qwen, Nemotron, Flint and Clover models locally. Our custom-written Metal kernels let us tune for each chip and each model. https://github.com/nace-ai/kapitsa/ The latest update, 0.2.26, is out. Since our last M5 Max numbers, Flash-Next 180B reads prompts 46% faster and the 27B writes 8% faster, and DiffuSpace's Clover 35B joins the catalog. Kapitsa also runs Qwen Flash-Next 180B with SSD offloading: on a Mac that can't hold it, the experts that don't fit in RAM stream from the SSD, so it runs on a 48 GB MacBook Pro at 61 tok/s. Here are the end-to-end numbers on a 2026 MacBook Pro 16 with an M5 Max and 128 GB: six targets (dense, MoE, hybrid Mamba-2 MoE and a block-diffusion MoE), from 1K context up to 32K, against the next-fastest engine for each. Try it (any Apple Silicon Mac; models in this post: ResultsDecode is a 512-token coding answer, prefill a 4K prompt, median of 3 runs:
From 1K context all the way up to 32K. At 32K the 35B still writes 175 tok/s, and Flash-Next barely drops (90 to 78): Against the other enginesAgainst the next-fastest engine for each model on the same Mac: 1.23x Splash on the 35B, 2.2x oMLX on Bolt, 1.18x rec on Flint and 1.12x mlx-serve on the 27B. Flash-Next is a tie with mlx-serve, Splash still reads prompts faster on the two Qwens, and nothing else runs Clover on a Mac. Every cell with versions and flags: benchmarks. Raw runs: results. Why it's fast: KFlashOther engines verify the drafter's guesses as one chain and stop at the first miss. KFlash verifies a best-first tree of the drafter's candidates in the same 8-row pass and keeps the longest branch the target agrees with. Three details make it fit Apple silicon:
Send us your benchmarksRun the same test on your Mac and add it to the community results. It takes 3-15 minutes: Plug in and close heavy apps while it runs. At the end it asks once: Enter sends the result as a pull request from your GitHub fork, and One more askDo you know prior work where tree-verified sampling matches plain sampling token for token under the same seed? Please point me to it so we credit it. We cite SpecInfer, Medusa, EAGLE-2 and Sequoia. [link] [comments] |
Meet TokenEconomizer: Microsoft’s Plugin That Cuts Coding Agent Costs by Up to 30.9%
A team of researchers from Microsoft Research and UC Berkeley have released TokenEconomizer, an open-source cost controller for coding agents. TokenEconomizer is a plugin that learns from past sessions in a repository to cut waste in later ones. It works with Claude Code, Codex, GitHub Copilot CLI and Pi. It does not touch model weights or agent code. On 277 SWE-Bench tasks, it lowered mean cost by 18.4% to 30.9% across 5 agent-model settings.
TL;DR
- Package: A standard-library Python package and agent plugins, with 3 switchable mechanisms.
- Runs on: Any machine running Claude Code, Codex, Copilot CLI or Pi with Python 3.10+. No GPU and no API key of its own.
- Performance: 18.4% to 30.9% lower mean SWE-Bench cost, and cheaper in all 30 repository-setting cells.
- Best: 30.9% lower cost on Claude Code with Opus 4.8 (33.2% when combined with RTK).
- Worst: Only 6.4% to 9.3% savings on ReposVul audits; Pass@1 drops 1.0 point on Opus 4.8.
Why do coding agents waste tokens?
Every agent turn re-bills the system prompt, every loaded tool schema and the full history. The research team profiled 60 SWE-Bench tasks across 6 repositories using Claude Code with Opus 4.8. They found 3 recurring sources of waste:
- Repeated discovery: once 5 tasks precede the current one, 30% of tool calls repeat an earlier action.
- Unused tools: 23 tool schemas load every turn. 16 are never called, and 3 cover 98% of calls.
- Failure loops: 10 failed runs were 17% of tasks but 32% of total spend.
They also tested 3 fixed open-source strategies: RTK, Serena and Aider’s RepoMap. None was cheaper everywhere. All 3 cut cost on Django and raised it on SymPy.
How does TokenEconomizer work?
TokenEconomizer is an external controller that treats the agent as a black box. It keeps a state per repository, built from completed traces. Before each task, it sets 3 policies through the prompt, the tool settings and the event stream.
1. Micro-skills
An LLM proposer turns trace digests into 1-line facts, such as the working test command. Each fact carries regex detectors that check later traces without a model call. A fact enters the prompt after 3 confirmations on tasks with non-overlapping edited files. Its need rate must also stay at or above 0.4. One contradiction counts as 5 confirmations, so wrong facts retire quickly.
2. Tool pruning
Shell, read, write and edit tools are always kept. Other tools are masked after 3 consecutive unused tasks, or when an LLM judge marks them irrelevant. The full catalog returns every 5 tasks for a re-audit. On Django with Claude Code, the first request on task 2 fell from 43,351 to 16,295 prompt tokens.
3. Failure-loop intervention
The controller learns a turn budget: 1.25 times the 90th percentile of successful run lengths. Cheap rule checks flag repeated failures or dense errors. An LLM judge then decides if the agent is stuck, at 0.7 confidence or higher. A stuck agent gets 1 redirect, then a stop if the loop recurs.
What do the SWE-Bench results show?
| Setting | Cost vs default | Pass@1 default | Pass@1 TokenEconomizer |
|---|---|---|---|
| Claude Code + Sonnet 5 | -23.6% | 74.7 | 76.0 |
| Claude Code + Opus 4.8 | -30.9% | 74.0 | 73.0 |
| Codex + GPT-5.6 Terra | -24.7% | 73.8 | 74.7 |
| Codex + GPT-5.6 Sol | -26.1% | 80.1 | 81.7 |
| Copilot CLI + Sonnet 5 | -18.4% | 75.0 | 77.6 |
Source: arXiv 2610.22732, Table 1. Costs include all controller calls. 3 seeds per setting.
Pass@1 improved in 4 of 5 settings. Lower reasoning effort saved cost but cut Pass@1 by 1.8 to 10.3 points. RTK alone raised cost in 12 of 30 repository-setting cells, and RepoMap in 15.
Savings build over time. Most repositories cost more at first, but all were cheaper by the sixth task. Alone on Claude Code, tool pruning saved 13.2% to 17.7%, micro-skills 11.8% to 14.3%, and the loop guard 5.4% to 7.1%.
Results also held outside SWE-Bench. Costs fell 16.0% to 21.2% on Web-Bench and 16.5% to 19.6% on CredData. ReposVul saw only 6.4% to 9.3%, because shorter runs leave less to remove.
How does it compare with other token-saving tools?
| Feature | TokenEconomizer | RTK | Serena | Aider RepoMap |
|---|---|---|---|---|
| Approach | Learns per-repository policies from past sessions | Compresses shell command output | MCP server with symbol-level code tools | Ranked map of repository symbols in the prompt |
| Learns from earlier tasks | Yes | No | No | No |
| License | Apache 2.0 | Apache 2.0 CLI; BUSL-1.1 engine | GPL-3.0-or-later (MIT for SolidLSP) | Apache 2.0 |
| Agents | Claude Code, Codex, Copilot CLI, Pi | 18 AI coding tools | Any MCP client | Aider |
| Mean SWE-Bench cost change (paper, 5 settings) | -18.4% to -30.9% | -8.9% to +4.5% | Not in main runs (raised pooled cost in profiling) | -6.1% to +6.5% |
| Cells with higher cost (of 30) | 0 | 12 | Not disclosed | 15 |
Cost rows: TokenEconomizer paper. Licenses and claims: each project’s repository or docs.
How do you install TokenEconomizer?
The plugin needs one of the 4 agents and Python 3.10 or newer. On Claude Code, clone the repo, add it as a marketplace, then run claude plugin install tokeneconomizer@tokeneconomizer. Learned data stays local in ~/.tokeneconomizer. Tool p
Meet Factory Droid 1.0: A Model-Independent Coding Agent Harness Built on Script Mode
Factory has released Droid 1.0, the largest update yet to the Factory Droid agent harness. The main change is programmatic tool calling, now on by default. The model writes a program that runs its tools. One model request can now drive hundreds of tool calls.
TL;DR
- Size: Base prompt is 13,055 input tokens, down 37% from 20,887. Only 2 tool definitions ship up front, down from 11.
- Runs on: macOS and Windows desktop app, a terminal UI, and the Node SDK
@factory/droid-sdk. Works with any supported model, including your own API keys. - Performance: Highest pass rate of every harness Factory tested, on the same model, at lower cost.
- Best: 65.2% on Terminal-Bench 4 with Claude Opus 5.5, at $4.73 per passed task (self-reported).
- Worst: With GPT-6.1 Sol, Droid takes 10% longer per passed task than Codex.
What is Droid 1.0?
Droid 1.0 is a model-independent coding agent harness from Factory. Factory defines a harness as everything around the model. That means the prompts, the tools, how tool results return, and the loop. Users pick a model or let Factory Router choose a cost-efficient mix.
How does programmatic tool calling work?
In a standard loop, every tool output lands in the model’s context. With programmatic tool calling (PTC), the model writes a program instead. Droid implements this as a Script tool. The model writes JavaScript that runs in a sandbox. The sandbox has no file, network, or system access of its own.
Raw files, search results, and logs stay inside the program. Only printed output returns to the model. Independent calls run concurrently.
The idea builds on CodeAct (Wang et al., 2024). Factory reports these results from its own sessions:
- Unit test fix: 15 tool calls took 5 model calls instead of 10. The task finished in 41.6 s instead of 56.8 s.
- Linear cleanup: 151 issues moved and verified in 312 tool calls. The Script used 2 requests and about 61,000 tokens. Factory models direct calls at 36 requests and about 3.1M tokens.
- CI sweep: 5,384 jobs across 547 runs, in 2 requests. Under 0.1% of tool results reached the model’s context.
On Claude Opus 5.5, Script mode cuts Droid’s cost by 15% versus direct tool calling. It is on by default for 10 models, including GPT-6.1 Sol and Kimi K3.
DeepSeek Open-Sources dsh-superagent, a Router-Worker Coding Agent
The DeepSeek Harness team just released dsh-superagent, an MIT-licensed research prototype. DeepSeek dsh-superagent is a proactive coding agent for one GitHub repository. It is packaged as a bundle for DeepSeek Harness. It listens to GitHub and, optionally, Feishu. It turns allowed requests into parallel work items, each in its own agent session. The release shipped alongside a design post, Exploring Super-agent Design: Towards a More Autonomous Agent System.
TL;DaR
- What is it?: A TypeScript bundle, v0.1.0, released as 1 commit under MIT. It uses whatever model the Harness profile is set to.
- Runs on: Your own machine, as a dedicated DeepSeek Harness profile. Needs Node.js 22.19+ or 24+, git 2.32+, Python 3.8+, tmux and GitHub CLI 2.48+.
- Performance: In DeepSeek’s simulation, it finished tasks far faster under a noisy message stream than a single-session agent.
- Best: p95 task completion of 15.7 minutes vs 29.5 minutes for the single-session baseline (about 1.9x faster).
- Worst: Tested only on synthetic messages and simulated tools. Real coding quality was not measured.
What is DeepSeek dsh-superagent?
It is a GitHub agent that waits to be asked. When an allowed person addresses it, it opens a work item, called a Ticket. Each Ticket runs in its own session and directory. Results return to the original conversation, often as a pull request. A local web panel lists only what needs a person.
The README page states the core rule plainly: take as little human attention as possible, and give as much value as possible. The philosophy doc applies it to behavior. The agent confirms once, stays quiet while working, and writes again only with a result, a question or a failure.
Why did DeepSeek build a Super-agent?
DeepSeek team argues that mainstream agents treat 1 session as the basic unit. According to the report, as reported by ifeng, a person can actively follow only 3 to 5 sessions at once. Human attention, not model capability, becomes the bottleneck for multitasking.
How does the Router-Worker architecture work?
The research team describes 2 layers. A long-running Router handles interaction and dispatch. Isolated Worker sessions do the actual work.
In the prototype, that flow is concrete, per the domain model:
- Source → Trigger: GitHub or Feishu produces events.
- Eval: decides whether a batch of Triggers asks for new work.
- Ticket: a unit of authorized work, with an immutable Mandate stating why it may proceed.
- Subscription: later comments on a pull request reach the right Worker without a fresh mention.
The main session runs the dsh-superagent-router preset with a smaller tool set. Each Worker runs dsh-superagent-worker. Shell commands move to the background after 5 seconds, so Workers keep receiving messages, per the execution doc.
Proactive work is gated. The authorization doc defines 3 accept modes: off, observe (default) and auto. In observe, the agent only records what it would have accepted.
The research team also lists ideas beyond the prototype: an external Watchdog, storage and compute separation, and an asynchronous ReAct loop.
How does it perform?
DeepSeek tested with simulated group chat and GitHub CI streams. Each run had about 300 messages, of which about 20 were real work requests. Each scheme ran 5 times, per ifeng’s account of the report.
- Super-agent: p50 of 12.4 minutes, p95 of 15.7 minutes.
- Single session plus subagents, neutral prompt: p95 of 29.5 minutes. No p50 was given.
- Why: under the neutral prompt, the baseline called a subagent 0 times in 5 runs.
- Router load: as arrivals rose from about 1 to 40 per minute, batch size grew from 1.0 to 2.8. Per-request latency rose from 2.5 to 3.9 seconds.
An ablation removed direct event pass-through. Router decisions rose about 30%, and cache-read tokens rose about 37%, from 23.23M to 31.77M. All figures are self-reported and come from secondary reports of the report.
How does dsh-superagent compare with other GitHub coding agents?
| Feature | DeepSeek dsh-superagent | GitHub Copilot cloud agent | Claude Code Action | OpenAI Codex |
|---|---|---|---|---|
| Trigger | Mention by allowed person; written rules | Issue assignment, chat, API, Slack and more | @claude mention, issue assignment, workflow prompt | @Codex in Slack; cloud tasks |
| Unasked (proactive) work | Yes, rule-gated; observe by default | Yes, via automations | Yes, via workflow events | Not disclosed |
| Parallel isolated sessions | Yes, 1 per Ticket | Not disclosed | Not disclosed | Not disclosed |
| Where it runs | Your machine (Harness profile) | GitHub, in the background | Your GitHub Actions runner | Codex cloud |
| Chat source | Feishu (optional) | Slack, Teams | Not disclosed | Slack |
| License | MIT | Not disclosed | MIT | CLI agent open source |
| Availabil |
NVIDIA Releases Auto Ontology: An Open Semantic Layer That Grounds AI Agent SQL
NVIDIA has released Auto Ontology, an open library under the Apache 2.0 license. NVIDIA Auto Ontology is a semantic layer framework that builds a context layer from enterprise metadata. Agents use that layer to write SQL against certified business definitions. The repository states the problem plainly: a fast revenue answer is worthless if the number is wrong.
TL;DR
- Library: Default LLM is
nvidia/nemotron-3-nano-30b-a3b, with Nemotron embedding and reranking models. - Runs on: Docker Compose for a local stack, or Kubernetes via Helm. Inference uses hosted NVIDIA NIM endpoints or self-hosted NIMs.
- Performance: Strong engineering, thin public evidence.
- Best: Ontology creation time cut by 90% in NVIDIA testing (self-reported).
- Worst: 0 published text-to-SQL accuracy scores. The eval repo ships a harness, not results.
- Bottom line: An open, MCP-ready context layer that any agent harness can query. But you must run the benchmarks yourself to trust it.
What is NVIDIA Auto Ontology?
It is a context layer that sits between an agent and the data warehouse. It tells the agent where data lives and what fields mean. It also says whether to use a field, how to use it, and where it came from.
Here are the five jobs:
- Discover: generate ontologies from metadata and connect existing semantic models.
- Ground: give agents trusted meaning for metrics, entities and relationships.
- Validate: check generated queries against certified definitions and lineage.
- Expose: serve context through MCP, CLI and API.
- Learn: fold certified corrections back into the ontology.
NVIDIA’s developer page frames the output as a starting ontology. Business experts still test, refine and certify it.
How does Auto Ontology work?
The stack has 4 core services. A Next.js 16 frontend handles the web UI and authentication on port 3000. A FastAPI backend with LangChain runs chat, NL-to-SQL and catalog APIs on port 3001. An ingestion worker writes embeddings on port 3002. Postgres with pgvector stores the catalog, the ontology and the vectors.
Source databases can be Databricks, Postgres, Snowflake or DuckDB. Models are set through environment “triplets” of endpoint, key and model. Separate roles exist for reasoning, non-reasoning, embedding and reranking. Any OpenAI-compatible endpoint works.
&&What does the MCP server expose?
The MCP server docs describe 9 read-only tools. ask_question returns the answer, the SQL and the rows as separate fields. Results cap at 100 rows. check_readiness confirms a compiled semantic layer and a live database connection exist.
There is no tool for browsing raw schemas. NVIDIA chose that on purpose, so agents reach data through the semantic layer. The underlying OpenAPI spec publishes 87 operations. Sign-in runs through OAuth, and every call carries the signed-in user’s permissions.
How is it benchmarked?
NVIDIA ships a separate harness, auto-ontology-eval. It runs 2 arms in NeMo Gym: a schema-only LLM control and the ontology-grounded agent. Scoring is deterministic execution match, with no LLM judge.
Bundled datasets include BIRD Dev (1,534 questions, 11 databases) and Mini-Dev (500 questions). FDABench-Lite adds 169 questions, and BEAVER defaults to a 100-question sample.
The harness repo adds 2 limitations. Optional few-shot exemplars and value anchors go to the ontology arm only. A delta measured with them covers those inputs, not grounding alone. Its tested configuration also routes both arms through a Claude model on NVIDIA’s inference gateway.
How does it compare with other semantic layers?
| Feature | NVIDIA Auto Ontology | Wren AI | Cube Core | dbt MetricFlow |
|---|---|---|---|---|
| License | Apache 2.0 | Apache 2.0 core (open core) | Apache 2.0 backend, MIT client | Apache 2.0 (v0.209.0+) |
| How the layer is built | Generated from metadata, then certified | Agent-assisted context files in Git | Defined in code | Defined in code in a dbt project |
| Agent interface | MCP, CLI, REST API | MCP, CLI, SDKs | SQL, REST, GraphQL APIs | Python library and CLI |
| Data sources | Databricks, Postgres, Snowflake, DuckDB | 22+ | All SQL data sources | Via dbt adapters |
| Default models | Nemotron via NVIDIA NIM | Not disclosed | Not applicable | Not applicable |
Deployment and availability
Local setup is docker compose up -d --build after filling in .env. Kubernetes users get a Helm chart and a DEPLOYMENT.md guide. NVIDIA team also publishes the backend as an NGC container. An NVIDIA API key from build.nvidia.com covers the default models.
Key Takeaways
- Auto Ontology generates a starting semantic layer from enterprise metadata.
- Agents query it through 9 read-only MCP tools, plus CLI and API.
- NVIDIA reports 90% faster ontology creation in its own testing.
- No text-to-SQL accuracy scores are published yet.
- Defaults to Nemotron on NIM, but accepts any OpenAI-compatible endpoint.
The Letter Company Releases Letterpress-1: A Coding Agent Built for Growth Engineering
The Letter Company has released the beta of Letterpress-1, a coding agent, harness and IDE for growth engineering. The Letterpress-1 coding agent reads a website and its market, then decides what should exist. It builds the change in a sandbox and opens a pull request with a live preview.
TL;DR
- Size: Letter team says it combines frontier models with fine-tuned small models; names and sizes are not disclosed.
- Runs on: Letter’s hosted platform against your git repository. No local GPU needed. Setup takes under 5 minutes, per Letter.
- Performance: No public benchmark. Letter shows 1 self-reported, side-by-side page build against Codex.
- Best: 760 tools exposed through 1 registry over MCP, REST and CLI.
- Worst: No measurable eval; entry plan is $100/month with overage at 5x the list rate.
What is Letterpress-1?
Letterpress-1 is a growth engineering agent from The Letter Company. It turns site and market signals into pull requests. The letter company team argues that websites, unlike apps, react to outside forces. These include agent behavior, Google rankings, human visits and competitor activity. General coding agents were built to scale codebases, not to act on those signals.
The agent is aimed at teams that host their marketing site in git. It is proactive: it suggests the next thing to build. Examples include de-anonymization, landing pages, warmed emails, blog automation and agent telemetry.
How does Letterpress-1 work?
Letterpress-1 runs in 3 modes:
- Build: create a page or feature from a prompt, with brand copy and visuals kept consistent.
- Plan: pick a goal and get a build plan of growth actions.
- Fix: find site issues and dispatch pull requests to fix them.
Every change follows the same loop. The agent reads the site, plans and builds in a sandbox. It reviews its own change, then opens a pull request. Each change returns a preview link of your own site. Nothing lands until you choose a preview, a pull request, or a merge after checks pass.
Letter team lists 6 surfaces. These are the web platform, a Slack app, the letterpress CLI and a website portal. A standalone app and your own agent round out the list. The in-site IDE works on a live page. Press ⌘L, edit text where you see it, and a pull request follows. Showcase examples also cover A/B tests from live edits.
For developers, agents.md documents an MCP server at app.letterstory.com/api/mcp with OAuth. The MCP manifest, OpenAPI spec and API reference form 1 registry of 760 tools. Claude Code, Codex or any MCP-capable agent can call the same functions the app uses.
What are Proofs, Simulations, Benches and Phantoms?
The launch post highlights 4 primitives aimed at winning over AI agents as buyers:
- Proofs: signed evidence of customer success, built from telemetry and billing data, for agent visitors.
- Simulations: mirror versions of your API or MCP with simulated data, so agents can try before buying.
- Benches: self-hosted, reproducible benchmark apps with open-source code and independent governance.
- Phantoms: independent, autonomous blogs used to sandbox growth strategies.
The site adds more: Kernels, Letterbases, Visitors and Mailers. Then it watches competitor sitemaps and flag new pages. Letter team positions these as replacements for tools like G2, Semrush, Profound, RB2B and Instantly.
A model guide for the GPT-6 family
Can You Run Your Own AI Locally on a Gaming GPU?
Run powerful AI locally with MSI RTX 50-Series GPUs
Explore the range and find the right GPU for your needs
AI might be everywhere and in everything right now, but for most people, they interact with it in only one place: The cloud. That’s still the best place to access frontier models with flagship capabilities. But where the latest ChatGPT and Claude models might be the best solutions for agentic AI or vibe coding entire applications, they also represent major privacy and security concerns. Token limits and ever-changing subscription fees can make them unreliable solutions to regular problems, too.
Running AI locally on your own hardware represents a significant improvement in AI privacy, as it keeps all your data on your own devices; Out of reach of the major AI developers who may, depending on plan and settings, use even proprietary and sensitive data for model training without the right safeguards. But where older AI models needed bespoke hardware to run and thousands of dollars of investment in dedicated AI systems, the latest open source models run fantastically well on graphics cards primarily designed for gaming.
How do MSI RTX 50-Series GPUs Make Local AI Possible?
The latest RTX graphics cards from MSI are fantastic at running modern games with support for hardware accelerated path tracing, and the latest DLSS upscaling and frame generation technologies. But all that graphical horsepower also makes them capable of running powerful local language models that can enable impressive AI intelligence that can be incredibly useful for everyday work and creative endeavours.
With 12GB or more of VRAM from the mid-range of the RTX-50 series, these cards have the onboard memory they need to store mid-size large language models right out of the box. Higher-end cards with 16GB of VRAM add headroom for running more capable models, while the top-tier RTX 5090 with 32GB of GDDR7 opens up new models and greater fine-tuning capabilities.
Local AI runs fast and responsively thanks to 5th generation tensor cores, providing powerful hardware acceleration for even demanding AI workloads. PyTorch and TensorFlow frameworks are optimized for Nvidia CUDA cores, too, making it much easier to get up and running with local AI on your own machine.
All these cards are supported by MSI’s smart power and thermal management systems that help them operate coolly and quietly, ensuring you stay focused on the task at hand; Whether that’s fragging enemies or prompting your very own local intelligence chatbot.
Which MSI RTX 50 Series GPU is Best For Local AI?
MSI RTX 50-series graphics cards are all excellent for gaming, but for powerful large language models, the RTX 5070 offers a fantastic entry-point with its 12GB of VRAM. Priced from $810, this card can run double duty for 1440p and light 4K gaming with upscaling and frame generation, and unlock local AI capabilities as well.
For running more demanding models and higher gaming frame rates, the MSI GeForce RTX 5070 Ti 16G Shadow 3X OC starts at $1,170 and comes with 16GB of VRAM. Its faster sibling, the MSI GeForce RTX 5080 (starting at $1,650) 16G Ventus 3X OC White, offers a truly premium gaming experience for native 4K and ultra settings with path tracing – perfect for scoring big on major benchmarks. Those extra CUDA cores help make it a more responsive card for running local AI models, too.
For those looking to really explore local AI development or who want to run some of the most capable local models without compromise, MSI’s GeForce RTX 5090 32GB Suprim SOC model starts at $5,200, delivering the ultimate in gaming and local AI capabilities.
For more information on MSI’s exciting range of RTX 50-Series graphics cards, you can find them all on the MSI store or at Amazon. Need more information to pick the right card for you and your needs? MSI’s detailed RTX 50-Series buying guide can help.
| Card | VRAM | Gaming Performance | Local AI Capabilities |
|---|---|---|---|
| RTX 5070 | 12GB GDDR7 | 1440p gaming at high detail settings | Mid-size quantized LLMs, image generation |
| RTX 5070 Ti | 16GB GDDR7 | 1440p and 4K gaming at ultra settings, with ray tracing | Larger models, more headroom |
| RTX 5080 | 16GB GDDR7 | Native 4K, path traced lighting, ultra settings | Faster responses at the same VRAM |
| RTX 5090 | 32GB GDDR7 | Maximum settings, full path tracing, high frame rate | Support for the largest local models, model fine-tuning |
FAQ
Is the MSI GeForce RTX 5070 12GB enough to run local AI?
Yes. The MSI RTX 5070 is the entry point for local AI, covering everyday chat, summarizing, coding help and image generation with mid-size models, while still handling 1440p gaming.
Which MSI RTX 50-series graphics card is best for running AI locally?
For most people, an MSI RTX 5070 Ti 16G (such as the GAMING TRIO OC or VENTUS 3X) balances price and VRAM. Developers who want to run the largest models without compromise should choose the MSI RTX 5090 32G SUPRIM SOC.
What’s the difference between the MSI RTX 5070 Ti and MSI RTX 5080 for local AI?
Both MSI cards have 16GB of GDDR7 memory, so they support the same siz
Claude won’t reverse-engineer games for you anymore
“Hey Claude, build Grand Theft Auto. Make no mistakes.”
That’s the kind of prompt some Claude users have been using to get the chatbot to reverse-engineer their favorite games, but you’re now more likely to get a “no” instead of a finished product.
New instructions discovered in Claude Code direct Claude models not to “build a recreation of a specific existing game,” even if the user suggests a new name or claims the game is original.
The new Claude Code system prompt, first spotted by a user on X, was later confirmed by a member of Claude Code’s development team.
“Do build original games, games in a genre, and games from the user’s own designs and assets,” the Claude Code prompt reads. “Don’t build a recreation of a specific existing game, even if it is renamed or called original.”
“Is this the end of game rewrites, modding and everything in between,” wondered the X user who highlighted the new Claude instructions.
Addy Osmani, an ex-Google Chrome engineer who now works at Anthropic, replied that the revised Claude Code prompt is intended to “steer it away from recreating someone else’s specific game because using Claude to infringe IP is against our usage policy.”
Osmani noted that “original games, genre games, and games built from your own designs and assets are meant to keep working,” while adding “if Claude turns down something that is yours, send it my way.”
Getting Claude, ChatGPT, and other AI models to reverse-engineer games, gadgets, and other products has become a buzzy trend over the past few weeks. One of the most popular AI hacks is aimed at trimming the bloatware on smart TVs, although Claude has reportedly started refusing TV-hacking prompts, too.
Of course, reverse-engineering a game with Claude involves more than just asking it to “build me Grand Theft Auto.” A variety of GitHub projects have recently sprung up with detailed instructions and skills to help Claude, ChatGPT, and other AI models recreate existing video games.
There are also galleries displaying examples of games successfully reengineered by Claude, including everything from Diablo II and Grand Theft Auto 2 to The Legend of Zelda: Link to the Past and Harvest Moon.
The process generally involves having the AI “decompile” the ROM or binary that a PC or game console uses to run the game, thus working backward to get the human-readable source code. There’s nothing new about decompiling the machine code of old video games, but AI has made the process way easier and faster.
But whether you decompile a binary by hand or with AI help, reverse-engineering a copyrighted video game can land you in legal trouble.
Back in 2021, Grand Theft Auto developer Take Two filed suit against the developers of decompiled mods for Grand Theft Auto 3 and Grand Theft Auto: Vice City. The lawsuit was eventually settled after the mods were pulled from release.
Coinbase hit with 50% price target cut as post-earnings rout deepens
Slowing trading activity and fallout from the Binance hack are clouding Coinbase’s revenue prospects, prompting Morgan Stanley to lower its rating on the stock.
AI poses greater threat to Bitcoin than quantum computing, Swan founder says
The Swan Bitcoin founder warned that hackers are using large language models to exploit security flaws with just a few prompts.
How to Build Durable Chat Memory for RAG Using ScyllaDB and LangChain
How to replace LangChain’s in-memory chat history with ScyllaDB — so your RAG chatbot retains context across restarts and scales across replicas
This post demonstrates how to integrate ScyllaDB Vector Search into your LangChain project for RAG use cases, as well as how to use ScyllaDB as a durable conversation memory within LangChain.
Background
Large language models are trained on a fixed snapshot of the world. RAG patches that gap by retrieving relevant documents at query time and injecting them into the prompt. The pipeline has two phases:
- Index: load documents, split them into chunks, embed each chunk, store embeddings in a vector store.
- Retrieve & Generate: enrich and embed the user’s question, find the nearest vectors (ANN search), pass the matching chunks as context to the LLM.
But RAG alone is not enough. You still need to keep track of all inputs provided by the user.
Persistent Chat Memory
LLMs are stateless. Every call starts with a blank slate unless you replay the conversation history. LangChain’s BaseChatMessageHistory abstraction lets you plug in any backend as the storage layer for that history. The default in-memory implementation vanishes on process exit. A database-backed implementation survives restarts, scales across replicas, and lets you inspect or audit conversations later.
That’s where ScyllaDB comes in.
ScyllaDB + LangChain
ScyllaDB is a NoSQL database optimized for high-throughput, low-latency workloads. Combined with LangChain, you can build reliable and always-on AI applications:
- High availability: data is automatically replicated across nodes, so there is no single point of failure. The cluster continues serving reads and writes even if a node goes down.
- Predictable P99 latency: ANN queries return results fast enough that retrieval doesn’t dominate your chain’s total latency.
- Horizontal scalability: add nodes to the cluster to increase throughput without schema changes or downtime.
- Built-in vector search: a native
vector<float, N>type and HNSW index are created automatically in ScyllaDB.
In this example, embeddings are generated locally with sentence-transformers (all-MiniLM-L6-v2, 384 dimensions). For production use, you can swap in OpenAI embeddings, Cohere, or any other provider supported by LangChain.
In each conversation turn, the chatbot is reading data from ScyllaDB and then writing back into it using LangChain.
Setup Example
With the release of ScyllaDB 2026.2, you can now integrate LangChain with ScyllaDB seamlessly by reusing the existing Cassandra connector.
Install Dependencies
pip install langchain langchain-community \
sentence-transformers langchain-groq \
langchain-text-splitters cassio \
scylla-driver python-dotenv
Environment Variables
Copy demo/.env.example to demo/.env and fill in your credentials:
# ScyllaDB Cloud
SCYLLADB_CONTACT_POINTS=node-0.your-cluster.cloud.scylladb.com
SCYLLADB_DATACENTER=AWS_US_EAST_1
SCYLLADB_USERNAME=scylla
SCYLLADB_PASSWORD=your-password-here
SCYLLADB_KEYSPACE=demo
# Groq (LLM)
GROQ_API_KEY=gsk_...
Retrieve the contact points, datacenter name, username, and password from the Connect tab of your cluster in ScyllaDB Cloud.
Connect to ScyllaDB Cloud
from cassandra.cluster import Cluster
from cassandra.auth import PlainTextAuthProvider
from cassandra.policies import DCAwareRoundRobinPolicy
import cassio
cluster = Cluster(
contact_points=["node-0.your-cluster.cloud.scylladb.com"],
auth_provider=PlainTextAuthProvider(
"scylla", "your-password"
),
load_balancing_policy=DCAwareRoundRobinPolicy(
local_dc="AWS_US_EAST_1"
),
)
session = cluster.connect()
cassio.init(session=session, keyspace="demo")
The cassio.init() call registers the session and keyspace globally so all downstream integrations (the vector store and the chat history) pick it up without further configuration.
Although cassio was originally built as a Cassandra integration, it is session-agnostic. It works with whatever driver session you hand it, so passing a session created by scylla-driver works just as well.
Integrating ScyllaDB with LangChain Vector Store
As an example, consider a chatbot that ingests articles, answers questions using retrieved context, and preserves conversation history in the database.
from langchain_community.document_loaders import WebBaseLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_community.vectorstores import Cassandra
loader = WebBaseLoader([
"https://docs.scylladb.com/stable/get-started/scylladb-basics.html",
"https://docs.scylladb.com/stable/get-started/data-modeling/query-design.html",
])
docs = loader.load()
chunks = RecursiveCharacterTextSplitter(
chunk_size=500,
chunk_overlap=50,
).split_documents(docs)
embeddings = HuggingFaceEmbeddings(
model_name="all-MiniLM-L6-v2"
)
vectorstore = Cassandra(
embedding=embeddings,
table_name="rag_docs"
)
vectorstore.add_documents(chunks)
retriever = vectorstore.as_retriever(
search_kwargs={"k": 4}
)
WebBaseLoader fetches and parses each URL into a Document.
RecursiveCharacterTextSplitter then breaks each document into 500-token chunks with a 50-token overlap so sentences are not severed at boundaries.
Cassandra(table_name="rag_docs") creates the table on first use, including a vector<float, 384> column (matching all-MiniLM-L6-v2’s output dimensions) and an HNSW index. Subsequent runs reuse the existing table; you only pay the embedding cost once unless you call add_documents again.
Persistent Chat Memory Implementation
from langchain_community.chat_message_histories import CassandraChatMessageHistory
def get_chat_history(session_id: str) -> CassandraChatMessageHistory:
return CassandraChatMessageHistory(
session_id=session_id,
table_name="chat_history",
)
CassandraChatMessageHistory stores each message as a row keyed on session_id. Because rows are written to ScyllaDB, the history survives process crashes, server restarts, and horizontal scaling. Any replica that opens the same session_id sees the same history.
Change session_id to start a fresh conversation. Keep it the same to resume where you left off — that is the entire persistence mechanism.
Putting It Together
from langchain_groq imp
Making Local LLMs Usable on 8GB GPUs
A local model can manage a brief question without any issues, but it may have difficulty when dealing with a complete document. The model's internal settings stay the same, but the effort needed and the memory required to process the request increase a lot.
In Part 1, I explained how to run a quantized model on a smaller GPU.
The next step is to choose a task, measure how long it takes, and find a setup that works within the GPU's limits.
This guide uses Qwen2.5-7B-Instruct along with llama.cpp.
It includes memory calculations, a fixed starting configuration, and a Python-based benchmark that you can run on your own computer. The numerical memory estimates are calculated values, and this article does not claim to have measured improvements in GPU performance.
To understand the actual delay you experience, there are two main waiting periods in a conversation. The first occurs when the model processes the initial prompt, often called the prefill phase. The second happens when the model generates the answer one word at a time, known as the decode phase.
If a document takes several seconds to process, fast decoding won't eliminate that initial delay. However, if the first words appear quickly but the rest of the response is slow, then the decoding phase is the area that needs optimization.
Keep these measurements separate:
Measurement | What it tells you |
Time to first text | Delay before the client receives the first nonempty text chunk |
Decode tokens per second | The server's reported generation rate |
Total request time | How long that request takes to finish |
Output token count | Whether two runs produced comparable amounts of text |
A target such as "start a short answer within two seconds" can guide your own experiments. It is a product requirement you choose, not a promised result for every 8GB card.
Budget for the KV cache as well as the weights
The model file is only part of the memory requirement. During generation, the runtime also stores attention keys and values for tokens already processed. This KV cache avoids recomputing those states, but it needs memory of its own. Working buffers and other GPU applications need space too. [1]
For a conventional full-attention transformer, an approximate KV-cache payload is:
KV bytes = 2 x layers x stored tokens x KV heads x head dimension
x bytes per cache element
The initial 2 accounts for keys and values. Use the number of KV heads, which can differ from the number of query heads.
Qwen2.5-7B-Instruct has 28 layers, 4 KV heads, and a head dimension of 128, obtained from its hidden size of 3,584 divided by 28 query heads. With FP16 cache elements, the calculation becomes 2 x 28 x tokens x 4 x 128 x 2. [2]
Stored tokens | Calculated FP16 KV payload |
4,096 | 224 MiB |
8,192 | 448 MiB |
16,384 | 896 MiB |
32,768 | 1,792 MiB or 1.75 GiB |
These figures describe one sequence and exclude allocation padding, runtime buffers, and model weights. They are not measurements of total VRAM usage. Other architectures, cache layouts, and attention types need different calculations.
This is why a large advertised context window needs its own memory budget. On this model, moving from 4K to 32K tokens adds about 1.53 GiB of calculated FP16 cache payload. Whether that increase fits depends on everything else already occupying the GPU.
Weight quantization and cache quantization are separate choices. A Q4_K_M model can still use an FP16 KV cache. Reducing cache precision may free up space, but it needs a quality check and a latency measurement. Hugging Face's cache documentation also cautions that cache quantization can hurt latency in some short-context situations. [1]
Establish one repeatable baseline
The shell commands below target Linux or WSL2 with an NVIDIA GPU. Install a working NVIDIA driver, a compatible CUDA Toolkit, Git, CMake, and a C++ compiler first. The benchmark itself requires Python 3.10 or newer and no third-party Python packages.
Use the current CMake build route in the llama.cpp build documentation: [3]
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j 4
git rev-parse HEAD > llama-commit.txt
./build/bin/llama-server --version > llama-version.txt
./build/bin/llama-server --help > llama-help.txt
nvidia-smi > gpu-environment.txt
Keep that checkout unchanged throughout the experiment. Save the model filenames and their SHA-256 hashes too, including every shard if the download is split. A model tag alone is not enough to reproduce an experiment after its files change.
I use the official Qwen GGUF repository because its model configuration is available to inspect. It is a concrete baseline, not a claim that this is the best available model. [4]
Start the server from the llama.cpp directory:
./build/bin/llama-server \
-hf Qwen/Qwen2.5-7B-Instruct-GGUF:Q4_K_M \
--n-gpu-layers all \
--ctx-size 4096 --parallel 1 \
--batch-size 512 --ubatch-size 128 \
--flash-attn off \
--cache-type-k f16 --cache-type-v f16 \
--fit off --no-context-shift --perf \
--host 127.0.0.1 --port 8080 \
2> baseline-server.log
The first launch downloads the selected model. Exclude that download and startup from request timings. Once the server is ready, inspect its log for the CUDA device, layer placement, cache allocation, and effective context settings. If an installed version rejects an option, consult its saved help output before continuing. The current server options are documented upstream. [5]
This configuration requests full GPU offload; it does not guarantee that every 8GB system has enough free memory. If startup fails, first close another GPU workload or test a smaller context. If partial offload is necessary, record the actual GPU layer count and use it consistently across the comparison.
The baseline has one request slot. Keep other inference clients stopped while measuring. With automatic fitting disabled, a failed configuration is visible instead of becoming a different experiment through an adjusted setting.
Run a benchmark that preserves the evidence</
Turning Product Info into Video Ads: Our Experience with Vizard Agent
Nowadays, having enough product material is rarely the main problem for e-commerce operators. Amazon listings may already contain product images, descriptions, key features, pricing information, and other commercial assets. The harder part is turning those materials into enough video creatives for advertising campaigns without spending hours on scripting, editing, and producing variations.
Therefore, we tested Vizard Agent from this perspective, focusing on a common scenario in which an e-commerce operator has plenty of product information but needs to produce a large number of promotional videos. The test looked particularly at its Amazon Product Link to Promo Video workflow, as well as its UGC-style and advertising function.
What We Tested
The starting point was quiet simple. Instead of preparing a complete video production package, the workflow begins with an Amazon product listing URL. Vizard's Ads mode allows sellers to paste an Amazon listing URL and have Vizard Agent extract product visuals and product details as the basis for video creation. The workflow can also start from uploaded product images, descriptions, and key features when a URL is not the preferred input.
This approach fits the broader design of Vizard Agent. The Agent can work with links, existing footage, files, or combinations of user materials and generated assets. It interprets the request, plans the video, selects or creates assets, makes creative decisions, assembles the result, and reviews the finished video. For an e-commerce operator, that changes the starting point considerably. The product listing itself can become the creative brief rather than simply a reference that someone has to manually read before beginning production.
From Amazon Listing to Promo Video
The workflow is useful because it connects product information with the actual video production process. After providing the listing, the user can specify the desired video type by giving the prompts, such as a product showcase, lifestyle-oriented video, or promotional advertisement. Vizard Agent then adapts the visual treatment to the requested format. Product images and descriptions provide the foundation, while subtitles, voiceover, visual elements, and other creative components can be incorporated into the finished video. This is where the workflow becomes more practical than simply converting an image into a slideshow. The purpose is to turn product information into a structured promotional story. Instead of deciding manually which product feature should appear first, writing the script, selecting supporting visuals, and arranging the scenes one by one, the operator can give the Agent the product and the intended advertising goal.
Vizard's documented product-ad workflow also covers product pages to video ads, product images to UGC-style ads, multiple advertising angles for one product, variations from a winning advertisement, and seasonal campaign variations.
For high-volume e-commerce campaigns, the ability to move from one product to several creative directions is arguably more important than producing one polished video.
UGC Adds Another Layer
The UGC mode was one of the more practical parts of the test. We started with the same product materials rather than preparing a separate batch of creator footage, and used the product images as the basis for a UGC-style ad. The result was noticeably different from a conventional product showcase, with the product positioned within a more creator-oriented advertising format.
What worked particularly well here was the amount of preparation Vizard Agent handled on its own. We did not have to build a separate UGC script, work out the structure manually, or assemble the product visuals scene by scene. Vizard Agent allows product images to be turned into UGC-style ads and treats the creative process as a broader video-production task rather than a sequence of individual editing operations.From a testing perspective, the more interesting point was that the same product assets could be taken in different directions without restarting the production process. We could use the product information for a conventional promotional video, then move toward a UGC-style treatment when a different advertising angle was needed. For an e-commerce operator producing creatives at scale, that flexibility was more valuable than simply having another editing option.
Why This Matters for High-Volume Advertising
The biggest advantage we observed was not a single editing feature. It was the reduction in the number of production decisions required from the operator. A conventional workflow might require a marketer to read the listing, identify selling points, write a hook, develop a script, gather product visuals, edit the footage, add captions and voiceover, and then repeat much of the process for another advertising concept.
Vizard Agent approaches these activities as one broader production task. Its documented workflow covers understanding the request, planning the video, selecting or creating assets, making creative and editing decisions, assembling the video, and reviewing the result. That becomes particularly valuable when one product needs ten creative variations rather than one finished advertisement. The platform's documented support for multiple ad angles, variations from winning ads, and seasonal versions directly addresses this type of production volume.
Who Should Consider It?
Vizard Agent is most relevant to Amazon sellers, DTC brands, e-commerce marketers, agencies, and performance marketing teams that already have substantial product information but need to increase their video output. It is also useful when the same product needs to appear in different creative formats across short-form channels. Vizard's Amazon product video workflow is not limited to Amazon itself. Generated videos can also be used across social and advertising channels such as Instagram, TikTok, and YouTube Shorts.
What Still Requires Human Review
We should declare that AI’s work does not remove the need for human oversight. Product claims, pricing, brand requirements, campaign objectives, and platform policies still need to be checked before publication. The quality of the source listing also matters because the Agent is working from the information and assets available to it. That limitation is important for Amazon sellers in particular. Automating production does not automatically make an advertisement accurate or commercially appropriate. Human review remains necessary, especially when a video makes specific product claims.
Our Assessment
For the e-commerce scenario we tested, Vizard Agent is more useful to think of as a video production agent than as another AI editing tool. The Amazon Product Link to Promo Video workflow provides a particularly straightforward entry point. An existing product listing can become the starting material for a promotional video, while UGC-style creation and multiple advertising angles allow the same product information to support different creative approaches. For e-commerce teams that already have plenty of product material but struggle with the volume of video they need to produce, that workflow reduction is the main value. The interesting shift is not simply that AI can edit a video faster. It is that the product listing, creative brief, asset selection, scripting, editing, and variation process can increasingly be treated as o