🚀 Tokenizer Performance - abnv · Scour

How Long Is a Piece of String? A Brief Empirical Analysis of Tokenizers

arxiv.org·3d

⚡Tokenizer Benchmarks

A Comprehensive Benchmark of Language Models on Unicode and Romanized Sinhala

arxiv.org·5h

⚡Tokenizer Optimization

ANN v3: 200ms p99 query latency over 100 billion vectors

turbopuffer.com·1d·

Discuss: Hacker News

⚡Cache-Aware Algorithms

Harper: Free, Open Source Grammar Checker

writewithharper.com·8h·

Discuss: r/webdev

🔄Incremental Lexing

You Probably Don’t Need a Vector Database for Your RAG — Yet

towardsdatascience.com·1d

Automatic compilation of a Pre-Qin philosophy lexicon via large language models

nature.com·1d

🔤Language Tokenizers

Exploring Text Compression

denvaar.dev·1d

📦Compression Algorithms

Making a Language

thunderseethe.dev·11h

🔬programming language theory

How Does ChatGPT Work? A Guide for the Rest of Us

producttalk.org

·20h

🎯Finite Automata

t2x - a CLI tool for AI-first text operations

shruggingface.com·1d

🔄Incremental Lexing

Iterative multi-word anagram solver

boulter.com·1d

🤖Suffix Automata

The RAG System That Retrieved Perfect Chunks (But Answered Wrong Anyway)

dev.to·34m·

Discuss: DEV

🔧Error Recovery

A Guide to TF-IDF

pub.towardsai.net

·6h

⚡Tokenizer Benchmarks

Explainer: Tree-sitter vs. LSP

lambdaland.org·1d

🌲Tree-sitter

MIT’s new ‘recursive’ framework lets LLMs process 10 million tokens without context rot

venturebeat.com·1d·

Discuss: Hacker News, r/technews

🪜Recursive Descent

PHPStan Now 25~40% Faster For Static Analysis

phoronix.com·23h

🏃Escape Analysis

istmarc/tenseur: C++23 Tensor, neural networks and mathematical library

github.com·15h·

Discuss: r/cpp

🦀MIR Optimization

I [[musttail]] You About a Tokenizer

neilhenning.dev·2d

How poor chunking increases AI costs and weakens accuracy

blog.logrocket.com·20h

⚡Tokenizer Benchmarks

Trying Out Claude Code with Ollama

slicervm.com·2d·

Discuss: Hacker News

Loading more...