🔤 Language Tokenizers - abnv · Scour

Stop Taking Tokenizers for Granted: They Are Core Design Decisions in Large Language Models

arxiv.org·1d

Reducing Tokenization Premiums for Low-Resource Languages

arxiv.org·1d

⚡Tokenizer Benchmarks

Harper: Free, Open Source Grammar Checker

writewithharper.com·4h·

Discuss: r/webdev

🔄Incremental Lexing

LLMs - Custom Tokenizers

dev.to·2d·

Discuss: DEV

⚡Tokenizer Benchmarks

Automatic compilation of a Pre-Qin philosophy lexicon via large language models

nature.com·22h

🌳Tree Diffing

t2x - a CLI tool for AI-first text operations

shruggingface.com·1d

🔄Incremental Lexing

Making a Language

thunderseethe.dev·8h

🔬programming language theory

A Guide to TF-IDF

pub.towardsai.net

·2h

⚡Tokenizer Benchmarks

I [[musttail]] You About a Tokenizer

neilhenning.dev·2d

subtitlesdog/Subtitles.Translate.Agent: A disruptive AI subtitle engine using Multi-Agent collaboration to deliver near-human level translation with unified style and terminology. 基于多智能体协同的颠覆性 AI 字幕引擎，实现风格统一、术语一致的准人工级翻译交付。

github.com·3h·

Discuss: Hacker News

🔄Incremental Lexing

Alconost Launches Free MQM Annotation Tool for MQM-Based Quality Analysis

multilingual.com·4h

📈Performance Tools

Explainer: Tree-sitter vs. LSP

lambdaland.org·1d

🌲Tree-sitter

DevUtils, a Mac app for text conversion and processing

blakewatson.com·1d

🔄Incremental Lexing

**Abstract:** This paper presents a novel system, “LexiGuard,” for automated review and anomaly detection within large-scale collaborative LaTeX ...

freederia.com·2d

🔄Incremental Lexing

Coding Alpha: How We Use Python to Scrape Fintech Sentiment in Brazil

velthorneassetmanagement.com·19h·

Discuss: DEV

📋JSON Parsing

Iterative multi-word anagram solver

boulter.com·1d

🤖Suffix Automata

LINGUIST List 37.275 Calls: Workshop: Variation and Change in Endangered Languages (Germany)

linguistlist.org·2h

🌉Language Bridges

Rainbow Query Language

rbql.org·1d

📊Query Optimizers

Dates and Times in DH An annotated application profile of ISO 8601:2019 for use with TEI and other DH systems

wwp.neu.edu·6h

🔢Binary Formats

Patterns All the Way Down: A Generalization for Graph-Like Things

medium.com·14h·

Discuss: Hacker News

🪢Rope Data Structures

Loading more...