A lightweight LLM Token Counter to manage context limits
Hey everyone,
Like many of you, I use Obsidian alongside LLMs (GPT, Claude, etc.) to help me with my notes. One frustration I always had was not knowing exactly how big my context was before pasting it into an LLM. And in big projects, this can cost money.
So, I built this plugin.
It adds a subtle indicator to your status bar showing the real-time token count of your current note.
It calculates BPE tokens (what GPT uses) and this can be CPU-intensive on massive files. To solve this, I implemented a strategy.
- Doesn't calculate on every keystroke (which causes lag).
- Waits 500ms after you stop typing to update the count.
- You can type in a 10k-word file, and the UI remains smooth.
- Uses js-tiktoken (OpenAI's official tokenizer implementation).
- Supports encoders for GPT-4/GPT-3.5 (cl100k_base) and legacy models.
- Minimalist: Sits quietly in the status bar (e.g., 1.2k tokens).
Repo & Code:
It's open-source, and I'd love your feedback before I submit it to the official community list.
link -> [github.com/leourl/obsidian-llm-token-counter]
Let me know what you think and if there are other features you'd like!
PS: I'm not a programmer, I'm a graphic designer that know some javascript cause I helped some websites projects in the past. I built this with the help of AI cause of my lack of modern state of JS/TS/backend knowledge. If you are a professional programmer and wants to send a PR or remake the thing the "right" way (good practices and stuff), you are welcome.
submitted by