Have you ever wondered how your editors and IDEs are able to support so many programming languages? Perhaps you’ve been thinking about designing your own language and wanted to know how you can give it editor support?
This talk is for you - I’ve spent over a year building a small language and integrating it with code editors, and I’d like to share some of the challenges I’ve faced, as well as lessons I’ve learned in that time.
I’ll also show how easy it is to build a new Language Server project in Scala 3 thanks to the Langoustine library.
More Decks by Jakub Kozłowski
Other Decks in Programming
Featured
Transcript
[Storytime - meet Jane. 👩💻](https:/…
Have you ever wondered how your editors and IDEs are able to support so many programming languages? Perhaps you’ve been thinking about designing your own language and wanted to know how you can give it editor support?
This talk is for you - I’ve spent over a year building a small language and integrating it with code editors, and I’d like to share some of the challenges I’ve faced, as well as lessons I’ve learned in that time.
I’ll also show how easy it is to build a new Language Server project in Scala 3 thanks to the Langoustine library.
More Decks by Jakub Kozłowski
Other Decks in Programming
Featured
Transcript
Storytime - meet Jane. 👩💻
Jane is tasked with building a language... • A small
DSL meant to be used in an existing application • The application is JVM-based (Scala) • the compiler also needs to work on the JVM • 2-3 weeks later, the compiler is done and embedded in the application • Jane’s life is good! • until... 1.
😳 Jane starts getting users And they want editor support
😭 More users And this time they want VS Code
Jane decides to shell out 🐚 Client-server architecture • The
editor extension (language frontend) will start a server in a separate process • A compilation server (language backend) will run on the JVM ☕ • The editor extension will send RPCs to the server ☎ • When the editor is closed, the extension will kill the server process 💀 1.
How it’ll all work • The editor extension (language frontend)
will: • Listen to user events 👂 (hover, clicks, keyboard shortcuts...) • Ask the server for information 🙋 (completions at position...) • Provide parameters 🧱 (cursor position, fi le contents...) • Apply actions and present results 🏋 (complete statement, go to de fi nition...) Client responsibilities 1. 1.
Jane needs an API for the communication • Jane wants
to call it "Language Server Protocol", but a quick Google search tells her it’s taken... • She decides to call it "Language Backend Protocol" ✅ • It’ll use HTTP over local TCP ✅ trait LanguageBackend { def definition( cursor: Position, file: Path ): Location def rename( position: Position, file: Path, newName: String ): List[TextEdit] //... } 1. 1.
🤔 Jane gets a DM Meet Trish
Server capabilities A means for a server to describe what
it can do • A new method in the protocol: initialize • Called by the client after server launches • The server responds with a list of supported methods • Unsupported features aren’t advertised by the client ✅ • Servers and clients can add features at their own pace ✅ • Only features that make sense have to be implemented ✅ 1.
Jane and Trish get to work... • Jane’s server declares
its capabilities • Trish’s new methods are added to the protocol • Trish implements a language backend • The extensions are updated to hide unsupported features • The extensions can be con fi gured to launch di ff erent backends • Jane’s life is good • until... 1.
😱 User complaint Some actions are visibly slow...
Users can’t see any feedback In the middle of an
action • Some actions are inherently slow ⏳ • Sometimes you just need to tell the user to wait a little more, or ask them for con fi rmation • Only the client can initiate calls ↪ • The protocol doesn’t give the server the ability to respond with intermediate results ↩ 1. 1. 1.
Updates to the initialize request • Not every client can
handle features like noti fi cations • Clients will advertise their capabilities in the initialize request • Servers will adjust behavior based on client capabilities trait LanguageBackend { def initialize( clientCapabilities: ClientCapabilities ): ServerCapabilities //... } 1.
Updates to the initialize request • Not every client can
handle features like noti fi cations • Clients will advertise their capabilities in the initialize request • Servers will adjust behavior based on client capabilities trait LanguageBackend { def initialize( clientCapabilities: ClientCapabilities ): ServerCapabilities //... } 1.
Jane gets to work • The editor extensions are updated
to advertise their capabilities • The server and extensions add support for the server -> client requests • Language backends can now provide estimations, noti fi cations, logs ✅ • Jane’s life is good • until... 1.
💀 User complaint The editors are often breaking...
Users can’t do anything without saving the file Jane immediately
knows the cause of the issue • The protocol identi fi es fi les by their disk path • Editors don’t save fi les on each keystroke (disk I/O is slow) • The fi les aren’t always up to date with the editor state ❌ • The protocol needs to account for unsaved fi les 💾 trait LanguageBackend { def definition( cursor: Position, file: Path ): Location def rename( position: Position, file: Path, newName: String ): List[TextEdit] //... } 1. 1.
Syncing text New methods in the protocol • New methods:
onChanged/onSaved/onClosed • When the fi le changes, editor extension sends updates • These can be patches (if the server is capable) or entire fi les • The server will keep these in memory • When the fi le is closed/saved, the extension informs the server • The server can delete these from memory 1. 1. 1.
🥰 Giving back Why keep this to ourselves?
Jane will return (Not really, this is the end)
🔄 LSP - a summary
No LSP: M * N integrations Source: https://code.visualstudio.com/api/language-extensions/language-server-extension-guide
Yes LSP: M + N integrations Source: https://code.visualstudio.com/api/language-extensions/language-server-extension-guide
Example: go to definition https://github.com/kubukoz/badlang
Go to definition in LSP
Go to definition in LSP
Go to definition in LSP
Go to definition in LSP
Go to definition in LSP
🤺 Challenges I’ve faced
🔎 Parsing
Parsing Making text structured
Parsing, more honest Ranges included
Parsing test harness Derive tests from directory structure
Parsing test harness Derive tests from directory structure
More ideas for graceful parsing "we want parsing to always
succeed at producing some kind of structured result. The result can contain error nodes inside it, but the error nodes don’t have to replace the entire result" "(...) every string matches a rule, by adding rules for erroneous inputs" https://duriansoftware.com/joe/constructing-human-grade-parsers 1.
