Why JSON is Inflating Your LLM Bill and How TOON Slashes Costs by 60%” In the field of Large Language Model (LLM) orchestration, the methodology of “prompting” has undergone a rapid professionalization. What began as conversational experimentation has matured into a rigorous discipline of structured data engineering. The first phase of LLM interaction was dominated by Natural Language Prompting . This approach relied on descriptive instructions Traditional (Natural Language) Prompt Traditional prompting relies on free-form sentences. While intuitive, it treats the LLM like a human coworker who might misunderstand “implied” instructions. Ex: “Extract the names and prices of the products from the text below. List them clearly and tell me if they are expensive (over $100).” Issues Encountered: Inconsistent Format - One run might give a bulleted list; the next might give a paragraph. LLM often adds conversational “filler” (Ex “Here is the list you asked for…”). If you are building an app, it is nearly impossible for code to reliably “read” a sentence to find a price. The model might miss a product if it’s mentioned casually. JSON ( J ava S cript O bject N otation) transforms the prompt into a data contract . It forces LLM to shift from “creative writer” mode to “ data processor” mode. { “task”: “extract_product_data”, “input_text”: “The SmartMug costs $120, while the basic Lid is only $5.”, “output_schema”: { “products”: [ {“name”: “string”, “price”: “number”, “is_luxury”: “boolean”} ] }, “constraints”: [“is_luxury is true if price > 100”] } Result ? LLM will return a clean, machine-readable JSON object that your app can immediately use to update a database or UI. JSON Prompt A robust JSON prompt typically follows a hierarchical structure that mirrors software architecture. Here is “Golden Template” for a JSON prompt: task - High-level objective Ex: summarize, classify, extract context - Background info or the “ persona “ the AI should adopt. input - Raw data or text to be processed. rules - Strict constraints Ex max 50 words, “use professional tone”. output_format - Specific JSON structure you expect back. Some of JSON templates for common tasks Document Summarization & Knowledge Pipelines In enterprise settings, a “summary” is often less useful if it’s just a paragraph. JSON allows you to create multi-faceted summaries that can populate a dashboard instantly. { “document_id”: “REF-9920”, “analysis_requirements”: { “summary_length”: “3 sentences”, “extract_entities”: [“People”, “Organizations”, “Dates”], “action_items”: “list” }, “output_format”: { “executive_summary”: “string”, “metadata”: {“author”: “string”, “priority”: “high|medium|low”}, “key_takeaways”: [“string”] } } 2. Multi-Agent Coordination When Agent A (Researcher) talks to Agent B (Writer), natural language causes “instruction drift” JSON acts as a State Machine , ensuring Agent B only receives exactly what it needs to function. Agent A Output: {“status”: “complete”, “raw_data”: “…”, “next_step”: “analyze”} Agent B Input: Takes the raw_data field only, reducing noise and “hallucination” potential. 3. Log and Error Triage Sending raw logs to an LLM often hits token limits. JSON prompting allows you to categorize errors so your monitoring tools (like Datadog or PagerDuty) can trigger specific scripts. { “log_input”: “[ERROR] 2026-01-04 19:40:15 - ConnectionTimeout at /api/v1/auth”, “analysis”: { “severity_score”: 0.9, “component”: “Authentication Gateway”, “remediation_steps”: [“Check VPC peering”, “Restart Auth Service”] } } 4. Semantic Data Enrichment It’s used to turn a messy sentence into a searchable database entry. Input: “I bought these boots in Seattle last week; they’re great but a bit heavy.” JSON Output { “sentiment”: “positive”, “product_attributes”: { “type”: “footwear”, “pros”: [“quality”], “cons”: [“weight”] }, “location_context”: “Seattle, WA”, “tags”: [“outdoor”, “winter”, “heavy-duty”] } We have moved beyond simply asking an LLM for “JSON”. We now use Schema-Driven Development (SDD) . In this paradigm, the prompt isn’t a suggestion; it is Type Contract . By providing a formal schema (usually JSON Schema or TypeScript Interfaces ), you shift LLM’s behavior from “predicting the next word” to “validating against a structure”. Core Components of Schema-Driven Prompt includes 3 critical layers: Definition Layer (“What”) You define the keys, data types (string, integer, boolean), and the required fields. Constraint Layer (“Rules”) You define valid ranges. Ex: enum: [“urgent”, “low”] or minimum: 0. This prevents the LLM from “making up” categories. Description Layer (“Why”) Inside schema, you use “description” field to give the LLM specific instructions on how to interpret that specific piece of data. Ex: Imagine you are building Medical Prescription Parser . “loose” JSON prompt might fail on edge cases. Schema-Driven prompt looks like this: System Prompt: Extract prescription data from provided text. You MUST strictly adhere to the following JSON Schema: { “type”: “object”, “properties”: { “medication_name”: { “type”: “string” }, “dosage”: { “type”: “object”, “properties”: { “value”: { “type”: “number” }, “unit”: { “type”: “string”, “enum”: [“mg”, “ml”, “g”] } }, “required”: [“value”, “unit”] }, “frequency”: { “description”: “How often patient takes the med, Ex’twice daily’”, “type”: “string” }, “is_controlled_substance”: { “type”: “boolean” } }, “required”: [“medication_name”, “dosage”] } While JSON Prompting is industry standard for enterprise workflows, it is not a “silver bullet” . As we push LLMs toward higher efficiency and deeper reasoning, 3 specific bottlenecks have emerged: 1. Context Switching Penalty When an LLM encounters JSON syntax, it doesn’t just “see” data; it shifts its internal attention mechanism. Problem - LLMs were trained on vast amounts of code. When you wrap a prompt in JSON, model often switches into “Code Interpreter” or “Data Processor” persona. Consequence - This is great for extraction, but lethal for creativity . If you ask for a “warm, empathetic customer response” inside a JSON field, the model often produces “flat,” robotic, or overly technical text because it is stuck in “technical documentation” mode. Ex: Natural Prompt: “Write a comforting email to a user who lost their data.” → Results in a heartfelt, human response. JSON Prompt: {“task”: “write_email”, “tone”: “empathetic”} → Often results in: “We regret to inform you of your data loss. We are working to resolve this.” 2. Token Inefficiency ( “Syntax Tax” ) In 2026, where we process millions of tokens daily, JSON is essentially “expensive air.” Problem - Every double quote (“), curly brace ({), and colon (:) is a token. In a large list of 100 items, you are paying for the same keys (Ex: “id”, “name”) 100 times. Impact - Syntax can occupy 20–40% of your total context window. This leaves less room for actual data or reasoning. 3. Wrong Distribution (Distribution Shift) Forcing an LLM into a strict JSON structure can actually lower its IQ for certain complex tasks. Problem - Reasoning techniques like Chain-of-Thought (CoT) require the model to “talk to itself” and think out loud. JSON is too rigid for this. If you force the model to output only JSON, you prevent it from doing the very “thinking” that makes it smart. “Hallucination” Trap - When a model is forced to fill a JSON field (like confidence_score: float), but it doesn’t actually know the answer, the pressure to maintain the valid JSON structure often forces it to hallucinate a value rather than admitting uncertainty. Token Economy & “Syntax Tax” To understand why JSON is expensive, we must examine Tokenization process. LLMs do not process words; they process Tokens — chunks of characters ranging from a single space to a sub-word Ex: “happi” and “ness”. In JSON structure, every brace {, quote “, and colon : is a token that consumes the model’s Context Window and increases API billing. API providers (OpenAI, Anthropic, Google) charge per 1,000 or 1,000,000 tokens. Memory (Context Window) - Models have “limit” (Ex 128k tokens). If your JSON is bloated, you hit that limit faster, meaning AI “forgets” earlier parts of your conversation. More tokens = longer processing time (Latency). Let’s look at standard nested JSON log. If you send 10 records like the one below, you aren’t just sending data; you are paying a heavy tax for the formatting. { “id”: “LOG-88291”, “timestamp”: “2026-01-04T20:15:00Z”, “metadata”: { “service”: “auth-gateway”, “version”: “v2.1.0”, “region”: “us-east-1” }, “level”: “ERROR”, “message”: “Connection timeout during handshake” } Token Breakdown Punctuation ~15–20 tokens Every {, }, “, :, and , is often its own token. Repeated Keys ~30 tokens “id”, “timestamp”, “metadata” are sent 10 times for 10 records. Whitespaces ~5-10 tokens If “Pretty Printed” (with indents), spaces and newlines add up. For 10 records, Total Tokens for 10 JSON records: ~800–1,000 tokens. Wasted Tokens (Syntax/Redundancy): ~600 tokens (60% of your bill). LLM is smart; it knows the second record has a “timestamp” after seeing it once. Yet, JSON forces you to tell LLM “timestamp”: over and over. Problem isn’t linear — it’s an Efficiency Trap . As your data grows, “Signal-to-Noise Ratio” drops. 1 Record: 80% Data, 20% Noise. (Acceptable) 100 Records: 30% Data, 70% Noise. (Expensive) 1,000 Records: You are now paying mostly for quotes and brackets. This is why TOON was developed. TOON (Token-Oriented Object Notation) TOON is first data format designed from the ground up for Token Economy. While JSON was built for human-machine readability over web protocols, TOON is optimized for mathematical “attention” mechanisms of LLMs. Think of TOON as a lossless translation layer : you keep your JSON code in your backend, but you translate it to TOON right before sending it to LLM to slash costs and improve focus. Source : TOON TOON Syntax: Hybrid of YAML and CSV TOON borrows the best parts of two worlds to create a “scannable” structure for AI: Indentation (from YAML): Replaces heavy curly braces {} and brackets [] with whitespace, which LLMs process more efficiently. Headers (from CSV): Declares key names once at the top of an array so they don’t repeat for every record. Ex: Converting JSON to TOON If you have a list of users, the difference in “noise” is immediate. JSON: [ {“id”: 1, “name”: “Alice”, “role”: “admin”}, {“id”: 2, “name”: “Bob”, “role”: “user”} ] TOON: users[2]{id,name,role}: 1,Alice,admin 2,Bob,user This works better as: Zero Repetition: Keys id, name, and role are sent once. Explicit Counting: [2] tells LLM exactly how many rows to expect, which acts as a “guardrail” against model accidentally skipping data or cutting off mid-sentence. Structural Cues: : and , are used as minimal delimiters that LLMs already understand from their training on code and data tables. “Translation Layer” Workflow In production pipeline, your code would look like this: Fetch data from your database as a standard JSON object. Encode that JSON into TOON using a library (like toon-format). Inject the TOON string into your LLM prompt. Receive the answer (usually back in JSON for easy parsing in your app). Let’s compare a standard nested JSON log of 10 records against the same data in TOON to see where those tokens actually go. In a typical production environment, a single log record often looks like this: { “id”: 2001, “timestamp”: “2026-01-04T08:14:23Z”, “level”: “error”, “service”: “auth-api”, “message”: “Authentication failed”, “metadata”: { “ip”: “172.16.4.21”, “code”: “401” } } Token Breakdown — Total for 10 Records: ~450–500 tokens. TOON uses a Header-First approach. It declares keys once and then lists data rows. logs[10]{id,timestamp,level,service,message,metadata{ip,code}}: 2001,2026-01-04T08:14:23Z,error,auth-api,Authentication failed,172.16.4.21,401 2002,2026-01-04T08:15:12Z,warn,billing-worker,Retrying… ,172.16.4.88,503 … (8 more records) Token Breakdown — Total for 10 Records: ~180–210 tokens. Comparison Table: JSON vs. TOON In JSON, “Syntax-to-Data Ratio” stays constant. If your keys take up 60% of the record, they will always take up 60%, whether you have 10 records or 10,000. In TOON, “Syntax-to-Data Ratio” improves as you add more data. Because the header is only written once, as you add more records, “cost per record” drops significantly. This makes TOON the superior choice for: RAG Pipelines - Fitting more search results into a single prompt. Log Triage - Processing thousands of lines of system errors at a fraction of the cost. Agentic Workflows - Allowing AI agents to pass large state objects between each other without hitting context limits. Python implementation that demonstrates the conversion from JSON to TOON and calculates the token savings using the tiktoken library (which mirrors OpenAI’s tokenization). Installation To run this, you will need the toon-format package and tiktoken. pip install toon-format tiktoken This script takes a sample of 10 nested log records, converts them, and prints a side-by-side comparison of the token counts. import json import tiktoken from toon import encode # Prepare your structured JSON data (Ex 10 log records) data = { “logs”: [ { “id”: i + 2000, “ts”: “2026-01-04T20:15:00Z”, “level”: “ERROR”, “service”: “auth-api”, “meta”: {“ip”: f“192.168.1.{i}“, “code”: 500} } for i in range(10) ] } # Setup Tokenizer (cl100k_base is used by GPT-4) encoding = tiktoken.get_encoding(“cl100k_base”) def count_tokens(text): return len(encoding.encode(text)) # Perform Conversions json_str = json.dumps(data, separators=(‘,’, ‘:’)) # Minified JSON toon_str = encode(data) # TOON Format # Compare Results json_tokens = count_tokens(json_str) toon_tokens = count_tokens(toon_str) savings = 100 * (1 - (toon_tokens / json_tokens)) print(“— TOON Conversion Summary —”) print(f“JSON (Minified) Tokens: {json_tokens}“) print(f“TOON Tokens: {toon_tokens}”) print(f“Token Savings: {savings:.2f}%“) print(”\n— TOON Output Snippet —“) print(toon_str[:250] + “…”) # Preview the first 250 characters Notice that encode(data) automatically identifies that your logs are a “uniform array.” It pulls keys (id, ts, level, etc.) into a single header line so they aren’t repeated 10 times. It replaces JSON’s heavy punctuation ({ } [ ]) with simple indentation and commas. For repetitive logs or RAG results, you will typically see savings between 40% and 60%, allowing you to fit nearly double the data into the same prompt context. Source : Playground Key features that make TOON a powerhouse for LLM interactions: 📊 Token-Efficient & Accurate TOON isn’t just about saving money; it’s about increasing Signal-to-Noise Ratio . In mixed-structure benchmarks across major models like GPT-5 and Gemini 3, TOON achieved 74% accuracy compared to JSON’s 70% . By removing syntactic clutter, the model’s “attention” is focused entirely on the data values rather than parsing brackets and quotes. On average, it uses ~40% fewer tokens , though this can climb to 60% for repetitive tabular data (like e-commerce catalogs or logs). 🛤️ LLM-Friendly Guardrails Unlike CSV, which is just “dump” of data, TOON includes explicit structural markers that act as guidelines for the model: [N] Lengths - Telling model users[50] prevents it from “truncating” the list or hallucinating an extra 10 items. { fields} Headers: Declaring schema once at top of an array provides a strict roadmap for the model to follow during retrieval or extraction tasks. 📐 Minimal Syntax & Tabular Arrays TOON achieves its compactness through two primary design shifts: Indentation Hierarchy - Similar to YAML, it uses two-space indents to show relationships, which models find highly intuitive. Schema Amortization - In standard JSON, keys like “price” are repeated for every item. TOON “amortizes” that cost by declaring it once, allowing the data to stream line-by-line like a high-speed table. 🌐 Multi-Language Ecosystem TOON project is fully open-source with production-ready libraries available for almost every modern stack: TypeScript/JS - Reference implementation (fastest for Node.js edge functions). Python - Standard for RAG and data science pipelines. Go/Rust - Used for high-throughput log processing and backend infrastructure. IDE Support - Plugins for VS Code and JetBrains provide syntax highlighting to make TOON as readable for you as it is for the AI. Knowing when to switch gears is critical for balancing API costs , model intelligence , and system latency . When to Use TOON: “Sweet Spots” TOON is most effective when your data has high Tabular Eligibility — meaning it contains repetitive structures that JSON would otherwise tax with redundant keys. A. Large Uniform Arrays (RAG & Logs) If you are sending 50+ items that share the same keys (Ex: search results for a RAG pipeline or system logs), TOON is the undisputed winner. Why ? It declares headers once. In a list of 100 products, JSON repeats “price”: 100 times; TOON says it once. Impact -> 40–60% token reduction and ~4.2% higher accuracy in extraction tasks because the model isn’t “distracted” by quotes and braces. B. Context-Constrained Workflows When you are hitting “ceiling” of your model’s context window (Ex trying to fit a massive legal document and 200 metadata tags into single prompt), TOON is your “compression” tool. It allows you to pack nearly double the information into the same space without losing data integrity. C. Real-Time “Stream” Processing For high-frequency agents that communicate back and forth, TOON’s minimal syntax reduces the “Payload Weight,” leading to faster processing in bandwidth-sensitive environments. When Not to Use TOON: “Red Flags” Despite its efficiency, there are 3 scenarios where JSON (or even CSV) is actually superior. A. Deeply Nested or Irregular Data If your data looks like a complex “tree” with different keys at every level (Ex: deeply nested application config file), TOON’s indentation overhead can actually exceed the cost of minified JSON. Rule of Thumb: If “Tabular Eligibility” is B. Pure Flat Tables (Use CSV) If your data is a simple 2D table with no nesting at all, CSV is still the most token-efficient format in existence. TOON adds about 5–10% overhead compared to CSV to provide “guardrails” like explicit array lengths ([N]). Choose TOON over CSV only if: You need those guardrails to prevent LLM from “skipping” rows or if you have some light nesting within the columns. C. Local or Quantized Model Latency Recent benchmarks show that while TOON uses fewer tokens, some local inference engines (like Ollama or vLLM) are highly optimized for JSON parsing at the C++ level. Sometimes a model can process 1,000 tokens of JSON faster than 600 tokens of TOON because of hardware-level acceleration for common formats. Measure your Time to First Token (TTFT) . If JSON is faster despite the cost, latency-critical apps should stay on JSON. Decision Matrix: Choosing Your Format “Hybrid” Strategy The most sophisticated pipelines now use “JSON-In, TOON-Between” architecture: Storage - Keep your data in JSON database. Prompting — Convert to TOON right before the API call to save money. Output - Instruct the LLM to respond in JSON (since your backend code already knows how to parse it perfectly). To determine if the switch to TOON is “worth it” for your specific pipeline, we use a metric called Tabular Eligibility (TE) . Tabular Eligibility is the percentage of your data that consists of uniform arrays of objects (where every item shares the exact same keys). The higher the TE, the more you save. Python script compares your data across 7 formats and calculates the cost efficiency of each using tiktoken (standard for OpenAI/Gemini models). import json import yaml import tiktoken import csv import io from toon_format import encode as toon_encode # Your Dataset data = { “records”: [ {“id”: i, “status”: “active”, “ref”: f“REF-{i*100}“, “val”: i * 0.5} for i in range(50) ] } def get_token_count(text): # o200k_base - high-context models enc = tiktoken.get_encoding(“o200k_base”) return len(enc.encode(text)) # Format Conversions results = {} # JSON variants results[“JSON (Pretty)”] = json.dumps(data, indent=2) results[“JSON (Minified)”] = json.dumps(data, separators=(‘,’, ‘:’)) # YAML results[“YAML”] = yaml.dump(data, default_flow_style=False) # TOON results[“TOON”] = toon_encode(data) # CSV & TSV (Requires flattening for nested data) output = io.StringIO() writer = csv.DictWriter(output, fieldnames=data[“records”][0].keys()) writer.writeheader() writer.writerows(data[“records”]) results[“CSV”] = output.getvalue() # Benchmark Execution print(f“{‘Format’: print(“-” * 45) base_tokens = get_token_count(results[“JSON (Minified)”]) for fmt, content in results.items(): tokens = get_token_count(content) percentage = (tokens / base_tokens) * 100 print(f“{fmt: 8.1f}%“) # Calculate Tabular Eligibility num_arrays = sum(1 for v in data.values() if isinstance(v, list)) uniform_arrays = sum(1 for v in data.values() if isinstance(v, list) and all(isinstance(i, dict) for i in v)) te_score = (uniform_arrays / num_arrays) * 100 if num_arrays > 0 else 0 print(f”\nTabular Eligibility Score: {te_score}%“) Interpreting Your Results Here is how to read your output: “Worth It” Threshold If TE > 80% Switch to TOON immediately. You will likely save ~50% on API costs and potentially increase extraction accuracy by 3–5%. If TE Stay with JSON (Minified) . The overhead of converting to TOON for irregular data isn’t worth the complexity. If Cost is No Object Stick to YAML . It is the most human-readable format for debugging prompts in real-time. Conclusion If you’re ready to transition your production workflows, here is the recommended 5-step roadmap: Audit for Tabular Eligibility: Identify datasets with high repetition (logs, product catalogs, RAG chunks). If the structure is >80% uniform, it’s a prime TOON candidate. Define Your Type Contract: Use TypeScript interfaces or Pydantic models in your backend. This ensures your internal logic remains strict. Deploy a Translation Layer: Use the encode() function at the very boundary where your data leaves your server and enters the LLM API. Prompt for JSON Output: While TOON is the best input format, most models are still optimized to output valid JSON. Use TOON to “tell” the model the facts, and ask for JSON to “hear” its answer. Monitor the Savings: Track your “Tokens per Request” metric. Expect a 40–60% reduction in input costs almost immediately. By adopting the “JSON for Code, TOON for Prompts” strategy, you aren’t just saving money; you’re expanding “brainpower” (context window) of every request you send. Thank you for journeying with me through the architecture of the modern token economy! Transitioning from natural language to structured frameworks like JSON and TOON isn’t just about saving bits — it’s about building a more precise, cost-effective, and scalable future for artificial intelligence. If you found this deep dive valuable, please clap (you can go up to 50! ) and share it across. Have you tried other formats like XML or YAML in production? Are you seeing similar “syntax tax” in your pipelines? Drop a comment below — I’d love to hear about your results or any challenges you’re facing. TOON Prompting: Moving Past Natural Language and JSON to Token-Optimized Data was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.