🔍 Tokenizers

Lexical Analysis, Token Recognition, State Machines, Parsing Pipeline

Feeds to Scour
SubscribedAll
Scoured 77944 posts in 3.18 s
Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Models
arxiv.org·15h
🔤Language Tokenizers
Preview
Report Post
t2x - a CLI tool for AI-first text operations
shruggingface.com·17h
🔄Incremental Lexing
Preview
Report Post
Reducing Tokenization Premiums for Low-Resource Languages
arxiv.org·15h
⚡Tokenizer Benchmarks
Preview
Report Post
LLMs - Custom Tokenizers
dev.to·2d·
Discuss: DEV
🔤Language Tokenizers
Preview
Report Post
Iterative multi-word anagram solver
boulter.com·22h
🤖Suffix Automata
Preview
Report Post
I [[musttail]] You About a Tokenizer
neilhenning.dev·2d
📚Factor
Preview
Report Post
Patterns All the Way Down: A Generalization for Graph-Like Things
medium.com·4h·
Discuss: Hacker News
🪢Rope Data Structures
Preview
Report Post
MIT's Recursive Language Models Improve Performance on Long-Context Tasks
infoq.com·1d
🪜Recursive Descent
Preview
Report Post
Building a Regulatory Risk Copilot with Databricks Agent Bricks (Part 1: Information Extraction)
databricks.com·49m
🎮Language Ergonomics
Preview
Report Post
🔬 Research-Grade Context Engineering This material represents high-performance prompt research results designed based on Advanced Logical Architectures. 📜 Ma...
freederia.com·1d
💬Interactive REPLs
Preview
Report Post
Wispr Flow-inspired voice input for any web app
speechos.ai·4h·
Discuss: Hacker News
🔄Incremental Tokenizers
Preview
Report Post
MIT’s new ‘recursive’ framework lets LLMs process 10 million tokens without context rot
venturebeat.com·1d·
Discuss: r/technews
🪜Recursive Descent
Preview
Report Post
istmarc/tenseur: C++23 Tensor, neural networks and mathematical library
github.com·2h·
Discuss: r/cpp
🦀MIR Optimization
Preview
Report Post
Want TradFi to embrace tokenization? Crypto's distribution strategy must mature
coindesk.com·5h
🔤Language Tokenizers
Preview
Report Post
Building Chatbots That Don't Annoy Users: A Developer's Guide
dev.to·1h·
Discuss: DEV
🎮Language Ergonomics
Preview
Report Post
How Does ChatGPT Work? A Guide for the Rest of Us
producttalk.org
·6h
🚀Tokenizer Performance
Preview
Report Post
Wikipedia:Lists of common misspellings/For machines
en.wikipedia.org·1d
🪢Rope Algorithms
Preview
Report Post
FlashLabs Chroma 1.0: A Real-Time End-to-End Spoken Dialogue Model
huggingface.co·4h·
Discuss: Hacker News
✨Gleam
Preview
Report Post
How poor chunking increases AI costs and weakens accuracy
blog.logrocket.com·7h
⚡Tokenizer Benchmarks
Preview
Report Post
Your Brain on ChatGPT: Accumulation of Cognitive Debt when Using an AI Assistant for Essay Writing Task
media.mit.edu·1h
⚡Tokenizer Optimization
Preview
Report Post

Keyboard Shortcuts

Navigation
Next / previous item
j/k
Open post
oorEnter
Preview post
v
Post Actions
Love post
a
Like post
l
Dislike post
d
Undo reaction
u
Recommendations
Add interest / feed
Enter
Not interested
x
Go to
Home
gh
Interests
gi
Feeds
gf
Likes
gl
History
gy
Changelog
gc
Settings
gs
Browse
gb
Search
/
General
Show this help
?
Submit feedback
!
Close modal / unfocus
Esc

Press ? anytime to show this help