https://www.princetekki.com/blog Technical writing on large language models, with sources cited. en https://www.princetekki.com/blog/mixture-of-experts https://www.princetekki.com/blog/mixture-of-experts Sat, 26 Sep 2026 12:00:00 GMT In 2020 Google trained a 600 billion parameter MoE model in four days, then left its trillion parameter sibling out of the paper because it kept hitting numerical trouble. Read in order, the MoE papers are a chain of problems and fixes, ending with a finding about what experts actually learn. https://www.princetekki.com/blog/cost-token-observability https://www.princetekki.com/blog/cost-token-observability Thu, 24 Sep 2026 12:00:00 GMT In 2024 a Princeton team re-ran published coding agents and found that a simple retry loop was as accurate as the best of them and far cheaper than most. Here is the cost of one task built up term by term from papers, then the cascades, routers and caches that measurably cut it. https://www.princetekki.com/blog/what-to-log-and-alert https://www.princetekki.com/blog/what-to-log-and-alert Mon, 21 Sep 2026 12:00:00 GMT Between March and June 2023, GPT-4 went from 84% to 51% on a prime number task and from 52% to 10% on code that runs as written, while every request still came back as normal text. Read as an incident report, the drift studies and ML monitoring papers say which signals would have caught it. https://www.princetekki.com/blog/tracing-llm-requests https://www.princetekki.com/blog/tracing-llm-requests Sat, 19 Sep 2026 12:00:00 GMT Google built Dapper because one search query touched thousands of machines and nobody could say which one was slow. An agent request has the same shape on a smaller scale. This post dissects one span tree, field by field, and then asks whether Dapper's sampling rules still hold when every span is a model call. https://www.princetekki.com/blog/evaluation-vs-observability https://www.princetekki.com/blog/evaluation-vs-observability Wed, 16 Sep 2026 12:00:00 GMT In a 2024 Berkeley study, developers kept rewriting their own grading rules as they read more model outputs. That finding, plus research on LLM judges, benchmark leakage, and real user traffic, makes a case that offline evaluation can only be a draft of what production observability later corrects. https://www.princetekki.com/blog/autoscaling-cold-starts https://www.princetekki.com/blog/autoscaling-cold-starts Mon, 14 Sep 2026 12:00:00 GMT In the ServerlessLLM paper, KServe took 128 seconds to return the first token from a cold OPT-6.7B replica, and 114 of those seconds went to downloading the weights. This post times each phase of a GPU cold start with numbers from the papers, then shows what each paper does to shrink it and when those fixes stop helping. https://www.princetekki.com/blog/serving-frameworks-compared https://www.princetekki.com/blog/serving-frameworks-compared Fri, 11 Sep 2026 12:00:00 GMT One A100 served a 13B model at 1.6 requests per second within its latency targets. Split across three GPUs by phase, the same model reached 3.3 per GPU. The frameworks differ mostly in which of three scheduling ideas they implement, and each idea helps a different workload. https://www.princetekki.com/blog/vllm-continuous-batching https://www.princetekki.com/blog/vllm-continuous-batching Wed, 09 Sep 2026 12:00:00 GMT When the vLLM team profiled existing LLM servers, as little as 20.4% of the memory reserved for the KV cache held actual token states. Walk four requests through two schedulers by hand, then follow the memory, and the 2 to 4 times throughput gain stops looking like magic. https://www.princetekki.com/blog/why-llm-deployment-is-different https://www.princetekki.com/blog/why-llm-deployment-is-different Sun, 06 Sep 2026 12:00:00 GMT On Azure production traces, a 1,500-token prompt on BLOOM-176B took as long as generating six output tokens. That one measurement, and a few others from Splitwise and Google's PaLM inference paper, knock over most of what ordinary web-service capacity planning assumes. https://www.princetekki.com/blog/rag-evaluation https://www.princetekki.com/blog/rag-evaluation Fri, 04 Sep 2026 12:00:00 GMT A Deakin University team built three RAG systems and catalogued seven ways they failed. Those failures sort into four branches, from "never retrieved" to "used but misstated", and each branch has its own metric and its own record of agreeing, or not, with human judges. https://www.princetekki.com/blog/reranking https://www.princetekki.com/blog/reranking Tue, 01 Sep 2026 12:00:00 GMT In January 2019 a BERT reranker took the top of the MS MARCO passage leaderboard with a 27% relative jump in MRR@10, and, as a later paper timed it, about 33 seconds of GPU time per query. The reranking papers that followed are a fight over that bill. https://www.princetekki.com/blog/cag-vs-rag https://www.princetekki.com/blog/cag-vs-rag Sun, 30 Aug 2026 12:00:00 GMT Preloading 85k tokens of HotPotQA into a precomputed KV cache cut answer time from 92 seconds to 2.3 on Llama 3.1 8B. On that same test set, plain BM25 retrieval scored higher. A head-to-head readout of what three papers measured, and where each approach won. https://www.princetekki.com/blog/hyde https://www.princetekki.com/blog/hyde Thu, 27 Aug 2026 12:00:00 GMT In December 2022 a CMU and Waterloo team trained nothing and still nearly matched a retriever fine-tuned on MS MARCO. Their trick was to embed a made-up answer instead of the question. Here is one query followed through every step, then the results, the Query2doc variant, and where the papers say it breaks. https://www.princetekki.com/blog/agentic-rag https://www.princetekki.com/blog/agentic-rag Tue, 25 Aug 2026 12:00:00 GMT Handed the top retrieved passages, ChatGPT got worse on a health fact-checking benchmark, falling from 70.1% to 54.7%. The fix the research tried was to let something other than the pipeline decide when to search. Here are four answers to that question, each with its trigger rule, its results, and what it costs in retrieval calls. https://www.princetekki.com/blog/vector-databases-explained https://www.princetekki.com/blog/vector-databases-explained Sat, 22 Aug 2026 12:00:00 GMT In 2017 FAISS searched a billion image vectors at 0.0133 ms per query while storing each one in 20 bytes. Getting there took three ideas, each one a fix for what the previous one still cost: inverted lists, product quantization, and layered graphs. https://www.princetekki.com/blog/agent-improvement-flywheel https://www.princetekki.com/blog/agent-improvement-flywheel Thu, 20 Aug 2026 12:00:00 GMT In 2023 an agent built on gpt-3.5-turbo went from 40% to 59% on ALFWorld household tasks with no weight updates, only by studying its own past runs. This post follows that loop one stage at a time (collect, label, distill, redeploy) using the papers that measured each stage. https://www.princetekki.com/blog/agent-deployment https://www.princetekki.com/blog/agent-deployment Mon, 17 Aug 2026 12:00:00 GMT In 2024 the tau-bench authors ran gpt-4o on the same customer service tasks again and again. It solved about 61% on average, but fewer than 25% of tasks on all eight tries. Here is the math behind that drop, what ToolEmu measured about risky actions, and what the guardrails in these papers actually bought. https://www.princetekki.com/blog/a2a-protocol https://www.princetekki.com/blog/a2a-protocol Sat, 15 Aug 2026 12:00:00 GMT A 2026 analysis found 11 attacks on the Agent2Agent protocol that work without breaking a single rule of the spec. Follow one task from Agent Card to artifact and you can see where each one lives. https://www.princetekki.com/blog/multi-agent-orchestration https://www.princetekki.com/blog/multi-agent-orchestration Wed, 12 Aug 2026 12:00:00 GMT Three copies of gpt-3.5-turbo that read and revise each other's answers solved 81.8% of an arithmetic test the same model solved 67.0% of alone. Pipelines, managers, group chats and debates each come from a paper that measured them. Here is the wiring, the numbers, and the bill. https://www.princetekki.com/blog/single-vs-multi-agent https://www.princetekki.com/blog/single-vs-multi-agent Mon, 10 Aug 2026 12:00:00 GMT A Berkeley team annotated 1,642 traces from seven multi-agent frameworks and found failure rates between 41% and 86.7%. Here is what the controlled studies say about when splitting a task across agents helps, when it hurts, and whether the gains survive an equal token budget. https://www.princetekki.com/blog/levels-of-agentic-autonomy https://www.princetekki.com/blog/levels-of-agentic-autonomy Fri, 07 Aug 2026 12:00:00 GMT Three papers rank AI agents by how much the human still decides. Walked rung by rung, they show what the user actually does at each level, which risk arrives with it, and why one of them argues the top rung should never be built. https://www.princetekki.com/blog/build-an-mcp-server https://www.princetekki.com/blog/build-an-mcp-server Wed, 05 Aug 2026 12:00:00 GMT A walk through one stdio session of the Model Context Protocol, one JSON-RPC message at a time, with the measured problems researchers found in real open-source servers placed at the exact message where each one enters. https://www.princetekki.com/blog/mcp-tool-overload https://www.princetekki.com/blog/mcp-tool-overload Sun, 02 Aug 2026 12:00:00 GMT When RAG-MCP buried one correct MCP server among thousands of distractors, success held above 90% for small pools and collapsed past about 100. Here is the curve, the reasons the papers give for it, and what retrieval, two-stage routing, and fine-tuning on API docs actually recovered. https://www.princetekki.com/blog/mcp-primitives https://www.princetekki.com/blog/mcp-primitives Fri, 31 Jul 2026 12:00:00 GMT MCP gives each server primitive an owner: the model fires tools, the application picks resources, the user picks prompts. Here is what the specification says, what attack studies measured against each primitive, and why no paper can yet say how often real servers use the last two. https://www.princetekki.com/blog/mcp-architecture https://www.princetekki.com/blog/mcp-architecture Tue, 28 Jul 2026 12:00:00 GMT A security benchmark pointed a web page at a local MCP server and reached it every time, on Claude Desktop, OpenAI and Cursor alike. A tour of MCP's three roles, what the spec makes each one responsible for, and what the research found breaking at each boundary. https://www.princetekki.com/blog/mcp-vs-function-calling https://www.princetekki.com/blog/mcp-vs-function-calling Sun, 26 Jul 2026 12:00:00 GMT In the Berkeley Function Calling Leaderboard paper, o1 scored 91.5 on parallel calls when the tools were described in its prompt and 0.0 when they went through its native tools field. Same model, same functions. That gap only makes sense once you see function calling, MCP and a plain API as separate layers. https://www.princetekki.com/blog/what-mcp-solves https://www.princetekki.com/blog/what-mcp-solves Thu, 23 Jul 2026 12:00:00 GMT The argument for MCP, that M apps times N tools should become M plus N, was first made for code editors. Here is what research found when the Language Server Protocol tried it, and what ecosystem measurements show about MCP so far. https://www.princetekki.com/blog/manual-rag-vs-agentic-context https://www.princetekki.com/blog/manual-rag-vs-agentic-context Tue, 21 Jul 2026 12:00:00 GMT Search-R1 kept the retriever, the corpus and the three passages per call fixed, and changed only who writes the queries. Exact match on seven QA sets went from 0.304 to 0.431. The papers that measured the bill, and the cases where the fixed pipeline still won, tell the rest. https://www.princetekki.com/blog/context-rot https://www.princetekki.com/blog/context-rot Sat, 18 Jul 2026 12:00:00 GMT With 20 documents in its prompt, GPT-3.5-Turbo answered 75.8% of questions when the right one came first and 53.8% when it came tenth, below its score with no documents at all. Four papers explain why more context can make answers worse: position, sheer length, and questions that share no words with their answers. https://www.princetekki.com/blog/context-budgeting https://www.princetekki.com/blog/context-budgeting Thu, 16 Jul 2026 12:00:00 GMT LLMLingua cut a 2,366-token math prompt to 117 tokens and GPT-3.5-Turbo lost 1.5 points of accuracy. Here is a token budget worksheet, and what three papers measured for each way of making things fit: dropping, summarizing, and deleting tokens. https://www.princetekki.com/blog/six-types-of-agent-context https://www.princetekki.com/blog/six-types-of-agent-context Mon, 13 Jul 2026 12:00:00 GMT Generative Agents lost believability each time a memory component was switched off. CoALA gives the vocabulary for why: working, episodic, semantic and procedural memory, plus the tool results that flow back in. Here is each one as the papers define it, and how MemGPT and Generative Agents build and measure it. https://www.princetekki.com/blog/context-vs-prompt-engineering https://www.princetekki.com/blog/context-vs-prompt-engineering Sat, 11 Jul 2026 12:00:00 GMT Changing only the separators and spacing of a prompt swung LLaMA-2-7B from 3.6% to 80.4% accuracy on one task. Wording clearly matters. The evidence on what it can and cannot fix, set against what changing the information in the window does, points to where the effort belongs. https://www.princetekki.com/blog/information-extraction-prompts https://www.princetekki.com/blog/information-extraction-prompts Sun, 05 Jul 2026 12:00:00 GMT On 20-page FDA device reviews, a single-prompt LLM extractor skipped about 27.5% of the fields a human would fill and added values the documents never state. The papers that measured this show four distinct ways extraction goes wrong, and none of them breaks the JSON. https://www.princetekki.com/blog/capabilities-that-matter https://www.princetekki.com/blog/capabilities-that-matter Sat, 04 Jul 2026 12:00:00 GMT An audit of MMLU found errors in 57% of sampled Virology questions and an estimated 6.49% error rate overall. Add HELM's other metrics and Chatbot Arena's preference votes, and a single leaderboard score says little about which model fits your application. https://www.princetekki.com/blog/evaluation-pipeline https://www.princetekki.com/blog/evaluation-pipeline Fri, 03 Jul 2026 12:00:00 GMT In 2019, three commercial sentiment APIs failed nearly every sentence like "I thought the plane would be awful, but it wasn't." Build an evaluation pipeline in order: capability tests from CheckList, per-case scores, standard errors and paired comparisons from Miller, and a case count large enough to see the difference you care about. https://www.princetekki.com/blog/build-vs-buy-model https://www.princetekki.com/blog/build-vs-buy-model Thu, 02 Jul 2026 12:00:00 GMT Bloomberg spent a 1.3 million GPU hour budget training a 50 billion parameter finance model. Months later, GPT-4, prompted and never retrained, beat it on four of the five public benchmarks it had reported. A line by line ledger of what each path costs a team, from the papers' own figures. https://www.princetekki.com/blog/jailbreaking https://www.princetekki.com/blog/jailbreaking Wed, 01 Jul 2026 12:00:00 GMT In 2023 Berkeley researchers found, for every one of 32 red-team prompts, an attack that got GPT-4 and Claude v1.3 to answer it. Their two failure modes still explain automated suffix attacks, many-shot prompts, and why every defense measured since moves the numbers without closing the gap. https://www.princetekki.com/blog/ai-as-a-judge https://www.princetekki.com/blog/ai-as-a-judge Tue, 30 Jun 2026 12:00:00 GMT In 2023, GPT-4 judged the same 80 pairs of answers twice with only their order swapped and contradicted itself on 37 of them. A catalog of the biases papers have actually measured in LLM judges, how big each one was, which fixes were tested, and when the grades came close to human ones. https://www.princetekki.com/blog/functional-correctness https://www.princetekki.com/blog/functional-correctness Mon, 29 Jun 2026 12:00:00 GMT EvalPlus gave each HumanEval problem about 80 times more tests, and GPT-4's greedy pass rate fell from 88.4% to 76.2%. What the original tests missed, how pass@k is actually computed, and what changes when the task is a whole repository. https://www.princetekki.com/blog/detection-without-a-detector https://www.princetekki.com/blog/detection-without-a-detector Mon, 29 Jun 2026 12:00:00 GMT OWL-ViT found LVIS rare categories at 31.2 AP after every box for them was removed from its training data. How image-text training turns class names into vectors, how OWL-ViT and Grounding DINO turn those vectors into boxes, and where the papers say it still breaks. https://www.princetekki.com/blog/domain-specific-models https://www.princetekki.com/blog/domain-specific-models Sun, 28 Jun 2026 12:00:00 GMT Flan-PaLM passed a medical licensing exam benchmark, yet clinicians judged only 61.9% of its answers to match scientific consensus. Med-PaLM, Med-PaLM 2 and Medprompt show what specialization changes, and which yardstick each result was measured with. https://www.princetekki.com/blog/multilingual-quality https://www.princetekki.com/blog/multilingual-quality Sat, 27 Jun 2026 12:00:00 GMT The tokenizer behind ChatGPT and GPT-4 needs about 15 times as many tokens for a sentence in Shan as for the same sentence in English. What that premium is made of, what it does to cost, context and latency, how accuracy falls with a language's share of the training data, and which fixes the papers measured. https://www.princetekki.com/blog/probabilistic-nature https://www.princetekki.com/blog/probabilistic-nature Fri, 26 Jun 2026 12:00:00 GMT Researchers sent ChatGPT the same coding prompt five times and, on 75.76% of CodeContests problems, got five programs with no test output in common. Where that variation comes from: temperature, top-k, nucleus sampling, and the batching arithmetic that keeps temperature 0 from being deterministic. https://www.princetekki.com/blog/next-word-sampling https://www.princetekki.com/blog/next-word-sampling Thu, 25 Jun 2026 12:00:00 GMT Widening the beam dropped a translation model's BLEU from 36.42 to 14.66, and exact search picked the empty string for most sentences. Why the most probable output is not the best one, and how locally typical sampling, contrastive search and min-p each answer that, with their measured results and the dispute over min-p. https://www.princetekki.com/blog/kv-cache https://www.princetekki.com/blog/kv-cache Wed, 24 Jun 2026 12:00:00 GMT DeepSeek-V2 reported a KV cache 93.3% smaller than its predecessor's. Why generation stores keys and values at all, how to size that store, and what multi-query, grouped-query and latent attention each gave up to make it smaller. https://www.princetekki.com/blog/retrieval-algorithms https://www.princetekki.com/blog/retrieval-algorithms Wed, 17 Jun 2026 12:00:00 GMT Dense Passage Retrieval beat BM25 by 19 points of top-20 accuracy on Natural Questions. On BEIR's 18 unseen datasets, the same model averaged 47.7% worse. What each formula actually computes, term by term, why each wins where it does, and what the papers report when you fuse them. https://www.princetekki.com/blog/chunking-for-rag https://www.princetekki.com/blog/chunking-for-rag Wed, 10 Jun 2026 12:00:00 GMT Re-indexing Wikipedia as one-fact propositions lifted an unsupervised retriever's Recall@5 on EntityQuestions from 36.3% to 51.7% with no retraining. What the research found about chunk size, semantic boundaries, late chunking and RAPTOR's summary trees, including where each one did not pay. https://www.princetekki.com/blog/context-length https://www.princetekki.com/blog/context-length Wed, 03 Jun 2026 12:00:00 GMT In 2023 a Meta team stretched LLaMA from 2,048 to 32,768 tokens with about 1,000 fine-tuning steps. The number had held because of how models encode position. This is the story of RoPE, ALiBi, position interpolation and YaRN, told through the perplexity each one measured. https://www.princetekki.com/blog/chain-of-thought https://www.princetekki.com/blog/chain-of-thought Wed, 27 May 2026 12:00:00 GMT Eight worked examples took PaLM 540B from 17.9% to 56.9% on grade-school math. The papers that followed added four conditions: it needs scale, one sentence can trigger it, voting over many chains helps, and the written reasoning may not be why the model answered. https://www.princetekki.com/blog/reading-benchmarks https://www.princetekki.com/blog/reading-benchmarks Wed, 20 May 2026 12:00:00 GMT In early 2025 researchers counted 27 private Meta variants on Chatbot Arena in the run-up to Llama 4, and showed that picking the best of several tries lifts a score. The maintainers pushed back. The argument is a good checklist for reading any benchmark claim.