You might have heard about Prism, the new Ruby parser. Perhaps you’ve heard it’s faster, more reliable, and more powerful than what we had before. Or maybe you never took a compilers class and aren’t sure about what this actually means.

I’m here to tell you all about it, and how it’s changing our lives as Ruby developers. Today, I want to take you from square one to writing your first transpiler.

Interpreters 101

Before we begin our journey, let’s start with the basics of how an interpreter works, so we’re all on the same page. Interpreting a programming language usually involves three main steps:

1. Tokenizing input (a.k.a. lexing): Breaking the input text into a list of meaningful tokens. That’s like converting your code into something like this:

2. Parsing: Analyzing the tokens to understand the program structure (what to do and in which order) and building a data representation that holds that information (known as an Abstract Syntax Tree). For example:

3. Evaluating: Executing the parsed input and producing an output. This is where your code actually runs.

For a deeper dive into this topic, I recommend the Crafting Interpreters book or my RailsConf talk on the subject.

Now let’s dive into what Prism can do and how it helps with parsing.

Why Is Prism Useful for Ruby Parsing?

Ruby historically used a parser called parse.y, built with Yacc. The catch? It was made specifically for CRuby, forcing other Ruby implementations (like JRuby and TruffleRuby) to create their own parsers from scratch.

That’s why tools like RuboCop, code editors, and even other Ruby implementations often lagged behind or had incompatibilities with newer Ruby syntax. Developers building Ruby analysis tools had to write their own parsers too, spawning projects like whitequark/parser and ruby_parser.

Prism solves this by becoming the de facto parser for all Ruby tools and implementations. And it’s working: it is now used in CRuby, JRuby, TruffleRuby, Rails, RuboCop, and more.

Okay, enough talk. Prism can lex and parse Ruby, which allows us to build fun things. How about we build a ✨ transpiler ✨ with it? Wait! Don’t go away. This will be simple. I promise.

Your First Transpiler

The full code for the examples in this post is available in this repository.

First, we’ll build a tool that converts our Ruby code into Emoruby. If you have never seen Emoruby, this is what it looks like:

Equivalent to this in Ruby:

Ready? Ok, here we go. We’ll need a Gemfile to install emoruby and prism:

After bundle installing, let’s create the entry point for our transpiler — the Rubyemo.ruby_to_emoji method:

For now, it only tokenizes the input with Prism. The lex method returns a result object that contains either the tokens or errors. If the source code contains invalid Ruby, we’ll just raise an exception.

Then we get the value attribute, which contains a list of tokens and some other stuff. We only care about the tokens, so we grab them with map(&:first).

Emojify the Ruby Tokens

Now onto the fun part. How do we emojify our Ruby tokens? Emoruby has a very simple design, so we can basically replace tokens one by one with an emoji alternative:

To do that mapping, we’ll use Emoruby’s translation file and EmojiData to translate token values to emojis by name.

If a token has no mapping, we leave it as-is so the code still runs. Pattern matching is pretty handy here to match a particular type and grab its value at the same time.

And... that’s it! Let’s see our little transpiler in action:

Fix Indentation and Spacing

You might not have noticed, but our code doesn’t handle indentation or spacing. See how the space between puts and the opening quote got lost? We can do better than this, so let’s handle that. Luckily, the Prism::Token instances have information about their location, including the start/end line and column.

Let’s change emojify to this:

And implement indentation_for:

Try emojifying this now:

Done — for real.

Onto Parsing with Prism for Ruby

Rubyemo helped us to learn Prism lexing, but we didn’t need any parsing. Let’s try another example. Let’s say you learned about Ruby 3.2’s Data class and you want to rewrite your old structs with it. Why spend 10 minutes doing it manually when you can write a script in one hour that does it?

Note: If you think this sounds like a RuboCop cop, you’re 100% correct! You could turn this into a custom cop if you wanted.

While Prism has a method to parse code, for this, we’ll use its Visitor class instead. The visitor design pattern allows us to add new operations to objects (in this case, the Prism AST nodes) without changing their classes. In other words, for each node type it finds, the Visitor class will call a method, and we decide what to do in that situation.

We need to act when we find Struct.new, so that’s a method call, which in Prism is identified by the CallNode type. Let’s filter those:

Note that we need to call super on the visit methods, which makes Prism keep walking inner nodes (like the body of a class definition).

Now we need to collect the struct arguments and build our fix object with the replacement code. We’ll also skip named structs, as those don’t map 1:1 to Data classes:

Make Fixes with a Method

Now let’s write a method to apply the fixes. We’ll process them in ascending order of start position, so offsets remain correct as we build the new source.

Note: We have to use byteslice because Prism offsets are in bytes, while replacing using something like String#[]= would fail on multibyte characters.

This is enough for us to test our code. Let’s see it in action:

It works!

Mutation: The Source of All Evil

There’s one big problem with our current approach: Data objects are immutable, while structs aren’t, so we can’t always convert them. We have to also skip structs that mutate internal state.

We’ll use an inner visitor to check if a struct body contains mutations:

This catches many cases, but there are many more ways to mutate a value in Ruby (i.e., by writing to an instance variable): writing to ivars directly, using attribute writers, or memoization, to name a few. It would be a chore to define methods for each one manually.

Luckily, Prism consistently names the nodes for these mutation methods, so we’ll just flag any nodes that perform a write:

So we get all "write" constants and define methods, but ignore writing to constants, local variables, and global variables (as those don’t change a struct’s internal state).

Let’s wire up this scanner now:

That’s it! Now our rewriter is ready to dance. Try it out with some of your structs.

Wrapping Up

Prism has already reshaped the Ruby landscape by making our tools faster, more portable, and more consistent. But its real impact will come from what you build with it.

Think bigger than just parsing: a Ruby-to-JS transpiler, a test runner that knows exactly which test to run from a file and line number, or even something that turns your code into pixel art. The parser is no longer the bottleneck — your imagination is.

Go make something amazing!

Matheus Richard

Guest author Matheus is a Brazilian software developer and a member of the Rails Issues team. He is passionate about open source and enjoys creating games and designing interpreters to explore new ideas and push his creativity.

All articles by Matheus Richard