✂️ Tokenization - tomasz · Scour

Compute Optimal Tokenization (2 minute read) 🔢Kolmogorov Complexity

arxiviq.substack.com

Learning Faster with Better Tokens: Parameter-Efficient Vocabulary Adaptation for Specialized Text Summarization 📝TextRank

AI Paper Review: Language Models are Few-Shot Learners (GPT-3) 🤖Transformer Architecture

freecodecamp.org·10h

Mistral's Open TTS, Anthropic's Activation Translator, and Matt Pocock's Skills Repo: Tokenizer #28 🎭Anthropic Claude

newsletter.artofsaience.com·1d

Compute Optimal Tokenization: Scaling Laws for Data Compression in LLMs 🔢Kolmogorov Complexity

co-tok.github.io·4h·Hacker News

Tokenizer Tampering 🌱Stemming

hiddenlayer.com·23h

NLP · Machine Learning 💬Natural Language Processing

·6d

I Built a Tokenizer From Scratch. 🏭Code Generation

·3d

Text Analysis for Hybrid Search: Tokenization, Stopwords & Accent Folding 🌱Stemming

weaviate.io·5d

Empowering Language Model Applications: Understanding and Evaluating Vector Databases in Production 💬Natural Language Processing

mlops.community·5d

Reinforcing Recursive Language Models (18 minute read) 🧠LLM Reasoning

alphaxiv.org·6d·Hacker News

SomaliWeb v1: A Quality-Filtered Somali Web Corpus with a Matched Tokenizer and a Public Language-Identification Benchmark 🌱Stemming

Structure tokens sharpen the feature vocabulary of protein language models 🌱Stemming

biorxiv.org·3d

A More Word-like Image Tokenization for MLLMs 🔗RAG

Guided Generation for LLM Outputs 🏭Code Generation

mlops.community·5d

WinTok: A Win-Win Hybrid Tokenizer via Decomposing Visual Understanding and Generation with Transferable Tokens 🔗RAG

OmniGene-4: A Unified Bio-Language MoE Model with Router-Level Interpretability 🤖Transformer Architecture

biorxiv.org·3d

Vision Foundation Models as Generalist Tokenizers for Image Generation 🔗RAG

Transformer Scalability Crisis: The First Comprehensive Empirical Analysis of Performance Walls in Modern Language Models 🤖Transformer Architecture

RoPE Distinguishes Neither Positions Nor Tokens in Long Contexts, Provably 🔢Kolmogorov Complexity

Log in to enable infinite scrolling