✂️ Tokenization - tomasz · Scour

Semantic Reranking at Inference Time for Hard Examples in Rhetorical Role Labeling 📝TextRank

GiLT: Augmenting Transformer Language Models with Dependency Graphs 🤖Transformer Architecture

Artificial Aphasias in Lesioned Language Models 🤖Transformer Architecture

Learning Variable-Length Tokenization for Generative Recommendation 🔗RAG

TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignment 🌱Stemming

Effective Context in Transformers: An Analysis of Fragmentation and Tokenization 🤖Transformer Architecture

Tokenizer Fertility and Zero-Shot Performance of Foundation Models on Ukrainian Legal Text: A Comparative Study 🌱Stemming

Pretraining Language Models with Subword Regularization: An Empirical Study of BPE Dropout in Low-Resource NLP 🤖Transformer Architecture

Language Modeling with Hyperspherical Flows 🔢Kolmogorov Complexity

Seed Bank, Co-op, Stoop Swap: Metaphors for Governing Language Model Data for Creative Writing 🧩Cognitive Architecture

Spectral Vision Transformer for Efficient Tokenization with Limited Data 📊TF-IDF

Probabilistic Calibration Is a Trainable Capability in Language Models 🔢Kolmogorov Complexity

Correct Answers from Sound Reasoning: Verifiable Process Supervision for Language Models 🧠LLM Reasoning

Images in Sentences: Scaling Interleaved Instructions for Unified Visual Generation 🤖Transformer Architecture

Dywave: Event-Aligned Dynamic Tokenization for Heterogeneous IoT Sensing Signal 🔢Kolmogorov Complexity

Exploring Token-Space Manipulation in Latent Audio Tokenizers 🌱Stemming

Multi-Stream LLMs: Unblocking Language Models with Parallel Streams of Thoughts, Inputs and Outputs 🧠LLM Reasoning

InsightTok: Improving Text and Face Fidelity in Discrete Tokenization for Autoregressive Image Generation 🔗RAG

Discrimination Is Generation: Unifying Ranking and Retrieval from a Tokenizer Perspective 🔗RAG

Sampling from Flow Language Models via Marginal-Conditioned Bridges 🔢Kolmogorov Complexity

Log in to enable infinite scrolling