Tokenizers

🔍 Tokenizers

Lexical Analysis, Token Recognition, State Machines, Parsing Pipeline

Feeds to Scour
SubscribedAll
Scoured 30 posts in 38.0 ms

Finding Optimal Tokenizers

 🌱Mini Languages  Content type: Blog
blog.aqnichol.com··Hacker News·Cited by 1 article

musab05/osirisdb: Production-oriented SQL database engine written from scratch in Rust.

 📚PL/0 Compilers  Content type: Code
github.com··DEV

DNA Compression with Genomic Language Models: Tokenization, Benchmarking, and an Information-Content Map

 📦Compression Algorithms  Content type: Academic
biorxiv.org·

Back to Basics: Build Your Own LLM from Scratch

 🌱Mini Languages
thejeshgn.com·

[NEW MODEL] SupraLabs just released Supra1.5-50M Base (Experimental)!

 🌱Mini Languages
huggingface.co··r/LocalLLaMA

A system programmer’s guide to LLM inference

 🌱Mini Languages  Content type: Blog
blog.xiangpeng.systems··Hacker News

Designing CherryScript: Optimizing Data-Driven Workflows via Custom Python-Based Interpreters​​​​‌‍​‍​‍‌‍‌​‍‌‍‍‌‌‍‌‌‍‍‌‌‍‍​‍​‍​‍‍​‍​‍‌​‌‍​‌‌‍‍‌‍‍‌‌‌​‌‍‌​‍‍‌‍‍‌‌‍​‍​‍​‍​​‍​‍‌‍‍​‌...

 📚PL/0 Compilers  Content type: Blog
stackoverflow.blog·

Sub_pl: My first functionnal programming language using zig

 🛠programming language development
ziggit.dev·

Security updates for Tuesday [LWN.net]

 ⚡Tokenizer Benchmarks
lwn.net·

Vibe Diaries: Training Nanochat

 🌱Minimal Languages
vibediary.dev··Hacker News

frankkk96/FlashQwen: From-scratch C++/CUDA inference engine for Qwen3-8B, with zero external libraries

 🧮Linear Algebra  Content type: Code
github.com·

GPUsnek is Python on nVidia’s CUDA

 🗣interpreters  Content type: Blog
blog.adafruit.com·

How Far Apart Does a Model Think Its Tokens Are?

 🌱Mini Languages
lesswrong.com·

CudaText Portable 1.234.4.0 (developer's text editor) Released

 🔧Error Recovery
portableapps.com·

smile/deep at master · haifengl/smile

 🌉Language Bridges  Content type: Code
github.com··Hacker News

Architecting the Control Plane for Intelligence: System Design of an Enterprise AI Gateway

 📏Linear Memory  Content type: Blog
medium.com·

Agentic RL: Token-In, Token-Out Done Right

 🌱Mini Languages
qgallouedec-tito.hf.space··Hacker News

Run an Apache Airflow DAG with Docker Compose and PostgreSQL

 🌱Mini Languages
pyimagesearch.com·

Introducing Lightstep UQL to PromQL Translator

 🔤Language Tokenizers  Content type: Blog
engineering.zalando.com·

How to write a compiler ?

 ⚙compilers  Content type: Code
github.com
··DEV

Keyboard Shortcuts

Navigation

Next / previous item
j/k
Open post
oorEnter
Preview post
v

Post Actions

Love post
a
Like post
l
Dislike post
d
Undo reaction
u
Save / unsave
s

Recommendations

Add interest / feed
Enter
Not interested
x

Go to

Home
gh
Interests
gi
Feeds
gf
Likes
gl
History
gy
Changelog
gc
Settings
gs
Browse
gb
Search
/

General

Show this help
?
Submit feedback
!
Close modal / unfocus
Esc

Press ? anytime to show this help