🔍 Tokenizers

Lexical Analysis, Token Recognition, State Machines, Parsing Pipeline

Feeds to Scour
SubscribedAll
Scoured 256137 posts in 2.15 s
The Transformer Architecture: A Deep Dive into How LLMs Actually Work
dev.to·8h·
Discuss: DEV
📊LR Parsing
Preview
Report Post
Natural language processing for word sense disambiguation and information extraction
arxiv.org·19h·
Discuss: r/compsci
🧠Semantic Parsing
Preview
Report Post
TRUNAJOD: A text complexity library for text analysis built on spaCy — TRUNAJOD 0.1.1 documentation
trunajod20.readthedocs.io·13h
🔤Language Tokenizers
Preview
Report Post
aniket-work/intelligent-support-system: AI-Powered Multi-Agent System for Automated Customer Support Ticket Resolution
github.com·35m·
Discuss: DEV
🎮Language Ergonomics
Preview
Report Post
TOON for LLMs: A Comparative Performance Analysis against JSON
gist.github.com·12h·
Discuss: DEV
📋JSON Parsing
Preview
Report Post
Small wordle project I made!
reddit.com·19h·
Discuss: r/rust
🌱Tiny Languages
Preview
Report Post
No, Small Models Are Not the "Budget Option" (English)
mostlylucid.net·2h
🚂Error Propagation
Preview
Report Post
Advanced Calendar Parsing towards an 3rd Party App.
icloud.com·1d·
Discuss: r/shortcuts
⚡Incremental Parsing
Preview
Report Post
A Validated Semantics for LLVM IR
sf.snu.ac.kr·16h
🔧LLVM IR
Preview
Report Post
LangChain & LangGraph: Beginner’s Guide to AI Workflows
javacodegeeks.com·1d
🌊Dataflow Languages
Preview
Report Post
[Discussion] The "Noise" Bottleneck in Local 8B RAG – A comparison of cleaning strategies (Regex vs. Unstructured vs. Entropy)
reddit.com·9h·
Discuss: r/LocalLLaMA
🧪Parser Testing
Preview
Report Post
Building an internal agent: Context window compaction
lethain.com·1d
🌊Streaming Lexers
Preview
Report Post
Final (?) version of my LLM proofreading macro
leancrew.com·11h
💬Interactive REPLs
Preview
Report Post
Eertree – An Interactive Guide
ufukhaciogullari.com·12h·
Discuss: Hacker News, r/programming
🌿Trie Structures
Preview
Report Post
SMART SLM: Structured Memory and Reasoning Transformer, A Small Language Model for Accurate Document Assistance
arxiv.org·2d
🌱Minimal ML
Preview
Report Post
Hj Hornbeck
freethoughtblogs.com·19h
🔍ML Language
Preview
Report Post
SDSL : a new/old shader programming language
stride3d.net·3d·
Discuss: r/programming
📊Pratt Parsers
Preview
Report Post
Docling + Go + Bob: The Modern Document Stack
dev.to·1d·
Discuss: DEV
🔄Incremental Lexing
Preview
Report Post
Prompts are becoming Code, but we still treat them like Strings
dev.to·20h·
Discuss: DEV
💬Interactive REPLs
Preview
Report Post
Deno 2 vs Node.js vs Bun in 2026: The Complete JavaScript Runtime Comparison
pockit.tools·17h·
Discuss: DEV
🚀V8 Internals
Preview
Report Post