🔍 Tokenizers

Lexical Analysis, Token Recognition, State Machines, Parsing Pipeline

Feeds to Scour
SubscribedAll
Scoured 259254 posts in 7.85 s
The Transformer Architecture: A Deep Dive into How LLMs Actually Work
dev.to·2h·
Discuss: DEV
📊LR Parsing
Preview
Report Post
Natural language processing for word sense disambiguation and information extraction
arxiv.org·14h·
Discuss: r/compsci
🧠Semantic Parsing
Preview
Report Post
TRUNAJOD: A text complexity library for text analysis built on spaCy — TRUNAJOD 0.1.1 documentation
trunajod20.readthedocs.io·7h
🔤Language Tokenizers
Preview
Report Post
TOON for LLMs: A Comparative Performance Analysis against JSON
gist.github.com·7h·
Discuss: DEV
📋JSON Parsing
Preview
Report Post
A Validated Semantics for LLVM IR
sf.snu.ac.kr·11h
🔧LLVM IR
Preview
Report Post
Advanced Calendar Parsing towards an 3rd Party App.
icloud.com·1d·
Discuss: r/shortcuts
⚡Incremental Parsing
Preview
Report Post
Hj Hornbeck
freethoughtblogs.com·14h
🔍ML Language
Preview
Report Post
LangChain & LangGraph: Beginner’s Guide to AI Workflows
javacodegeeks.com·1d
🌊Dataflow Languages
Preview
Report Post
[Discussion] The "Noise" Bottleneck in Local 8B RAG – A comparison of cleaning strategies (Regex vs. Unstructured vs. Entropy)
reddit.com·4h·
Discuss: r/LocalLLaMA
🧪Parser Testing
Preview
Report Post
Building an internal agent: Context window compaction
lethain.com·1d
🌊Streaming Lexers
Preview
Report Post
Final (?) version of my LLM proofreading macro
leancrew.com·5h
💬Interactive REPLs
Preview
Report Post
Eertree – An Interactive Guide
ufukhaciogullari.com·7h·
Discuss: Hacker News, r/programming
🌿Trie Structures
Preview
Report Post
SMART SLM: Structured Memory and Reasoning Transformer, A Small Language Model for Accurate Document Assistance
arxiv.org·2d
🌱Minimal ML
Preview
Report Post
SDSL : a new/old shader programming language
stride3d.net·3d·
Discuss: r/programming
📊Pratt Parsers
Preview
Report Post
kalolad1/llm-sorter: A Python library that uses LLMs as comparators for semantic sorting. Sort any list by meaning, tone, complexity, urgency, or any criteria expressible in natural language.
github.com·6h·
Discuss: Hacker News
⚡Incremental Parsing
Preview
Report Post
Docling + Go + Bob: The Modern Document Stack
dev.to·1d·
Discuss: DEV
🔄Incremental Lexing
Preview
Report Post
Prompts are becoming Code, but we still treat them like Strings
dev.to·15h·
Discuss: DEV
💬Interactive REPLs
Preview
Report Post
Deno 2 vs Node.js vs Bun in 2026: The Complete JavaScript Runtime Comparison
pockit.tools·12h·
Discuss: DEV
🚀V8 Internals
Preview
Report Post
Predictive text input for English + Pinyin/Chinese
reddit.com·14h·
Discuss: r/meshcore
🚀Tokenizer Performance
Preview
Report Post
<p>**Abstract:** This paper introduces a novel framework, Automated Protocol Refinement for Enhanced Algorithm Validation (APREAV), to significantly improve the...
freederia.com·12h
🤖Abstract Machines
Preview
Report Post