I’m working on a development environment for Python based on Emacs’ SLIME mode for Common Lisp. In this talk I’ll demonstrate some of its features, like an object inspector, interactive backtrace buffer, thread and async task viewer, and function tracer. I’ll also discuss its implementation and limitations, along with future directions for the project.

This project aims to bring a Lisp and Smalltalk inspired style of development to Python. You get a faster feedback loop by developing inside a running python process without needing to restart your program and lose state on changes, allowing you to immediately inspect the results of code you write. We can also provide more advanced tooling based on runtime introspection, as we have more information available at runtime than is available to traditional tools based on static analysis of source code, mainly we have the actual values of variables rather than just their types.

Python is eating the world. Emacs is eating my computing environment. I’m attempting to get them working together.

Transcript

Hello everyone, I’m Scott and I’ll be talking about Swanky Python, which is a development environment for Python based on Emacs’ Slime package. So what is that and why might you find it interesting? SLIME is the Superior Lisp Interaction Mode for Emacs. It’s an Emacs package for developing Common Lisp, and it’s a bit different from the way we develop most languages in that you’re always connected to a running instance of your application, and you kind of build up your application, piece by piece, modifying one expression at a time without ever having to restart your application. So why might you want to develop this way? One advantage is that you can get a faster feedback loop. For some kinds of software, it doesn’t make a big difference. Like, if you’re developing a web backend where all state is stored externally in a database, then you can have a file watcher that just restarts the whole Python process whenever you make any edit, and you’re not really losing anything, because all the state is stored outside the Python process in a database. So it works great. But for other kinds of software, like let’s say you’re developing an Emacs package or a video game, then it can be a real pain to restart the application and recreate the state it was in before just to test the effect of each edit you want to make. Another advantage is the runtime introspection you have available. So since you’re always connected to a running instance of your application, you can inspect the values of variables, you can trace functions, and all sorts of other information to help you understand your application better. And lastly, it’s just a lot of fun to develop this way, or at least I find it fun developing with SLIME, so I wrote a SLIME backend for Python so I could have more fun when I’m coding in Python. As for the name swanky-python, within SLIME, swank is the name of the Common Lisp backend that runs within your Common Lisp application and connects to Emacs. So I’m not too creative. swanky-python is just a swank implementation in Python.

[00:02:08.000] Demo

So let’s see it in action. So we started up with M-x slime. And what that does is it starts a Python process, starts swanky-python within it, and connects to it from Emacs. And you can configure how exactly it runs Python. Or you can start swanky python manually within a Python application running on a remote server and forward the port locally and connect to it in Emacs, from Emacs remotely. Within the README, there’s more documentation on other ways to start it. But just M-x slime is the basic way that works most of the time. So within the REPL, the first thing you’ll notice is that REPL outputs are clickable buttons, what SLIME calls presentations. So you can do things like inspect them. And for each presentation, in the Python backend, it holds on to the reference to the object. So for an int, it’s not too interesting, but let’s do a more complex object like a file. Then we can inspect the file. We can describe it, which will bring up documentation on that class. We can use it in further expressions like if we copy it, it will use the actual Python object in this expression. We can assign it to a variable. SLIME uses presentations everywhere that a Python object would be displayed. So instead of just their string representation, when you have a backtrace on an exception, or you... within the inspector or anywhere else really, anywhere that the string representation of an object would be displayed, it displays a presentation that you can go on to inspect, reuse, or send to the REPL and so on. One useful utility function is pp for print presentation. We haven’t imported it yet. So when we get a name error exception and SLIME sees that that name is available for import somewhere, it’ll give us the option of importing it. Since it’s available for import from multiple modules, it’ll prompt us for which one we want to import it from. We want to import it from swanky-python, not from the standard library. Then it will print a presentation of that object. Within the REPL, this is not really useful because all REPL outputs are already presentations. But I use this now whenever I would use print debugging, just whenever I would use insert print statements in my program to see what’s going on, I have it print a presentation because that way I can go back and inspect it later, copy it to the REPL and further manipulate it and so on.

[00:05:16.600] Inspector

Next up, let’s look at the inspector more. If we go back and inspect the file object, you can write custom inspector views for different kinds of objects. So far, I just have a couple. One for sequences, one for mappings, and one for every other kind of object. Like if we inspect a mapping, there’s a shortcut inspect last result, which is what I normally use to open the inspector. Then we see the values, and each value in the inspector is a presentation that we can go on to inspect, and so on. Let’s go back to inspecting the file object. Again, we can inspect each of the values, we can copy them back to the REPL and so on. It just displays all the attributes for the class and their values. We can configure what attributes we want to show. There’s a transient menu where we can toggle if we want to show private attributes, dunder attributes, doc strings, so on, or everything, which is a bit much to show by default. So we’ll reset it to the default. In the future, I want to add graphical inspector views for different kinds of objects, and also support showing plots in both the inspector and the REPL, but that’s future work I haven’t started on yet.

[00:06:47.720] Evaluating Python

Let’s look at the different options for evaluating Python. So we can evaluate a whole file. We can evaluate just a class. We can evaluate just the method we’re working on. We can evaluate a Python statement, and it will show the result in an overlay next to the cursor. We can select some code and just evaluate the highlighted region. We can sync the REPL to the active file. So now everything we evaluate in the REPL will be in the context of the eval_demo module. We can also set the module that the REPL is in. We can go back to main. But let’s go back to the eval_demo module for now.

[00:07:43.680] Updating

One useful thing is when you update a class or a function, it updates old instances of that class or function. So right now, f.bar is foobar. But if we edit that class, it will actually edit the code for the old instance of that class. And that’s provided by code I copied from IPython’s autoreload extension. It helps when you’re trying to develop in Python without having to restart the Python process whenever you make a change. Auto reload in Python is a big topic that I don’t really have time to go into here, but right now it is more limited than what is done in Common Lisp. Like for example, if you have a data class in Python and you add a new field to the data class, it won’t automatically update old instances of the data class with a new field. So there’s more that needs to be done with that, but I am perhaps naively optimistic that Python’s runtime is quite dynamic and flexible, and that I can fully implement autoreload in Python, but there’s still work to be done, and it’s a big topic to go into. Next up, let’s look at the backtrace buffer. But as it is right now, autoreload is actually useful. I mostly develop in Python without having to restart the process and without running into issues from old state that hasn’t been updated properly.

[00:09:22.900] Backtraces

So if we go on to look at the backtrace buffer, whenever we get an exception in Python... Let’s go back to it. Whenever we get an exception, it will... let’s change the code so that it actually gets an exception... we will get an interactive backtrace buffer where we can browse the source code for the different stack frames and the local variables within the stack frames, which are all presentations that we can inspect and so on. We can also open a REPL in the context of any stack frame. Or we can, when we go to the source for a given stack frame, we can select some Python code and evaluate it within the context of that stack frame. One major limitation compared to SLIME for Common Lisp is that in Common Lisp, you have the option to restart or resume execution from a given stack frame after an exception happens, where in Python, what we have right now is pretty much equivalent to the postmortem debugger. You can view the state that the call stack was in at the time of the exception, but you can’t actually resume execution, which you often might want to do, because when you’re coding in a dynamic language, you’re going to get runtime errors. So if you’re writing a script that does like some sort of long-running computation or processes a ton of files and gets an exception parsing one file halfway through, normally you’d have to fix the script, and then rerun it and have it process all the same files all over again, and lose a bunch of time for every bug you run into and fix you have to make. So right now we’ve got a kind of mediocre workaround which is you can add the restart decorator to a function and then... where in the case of a script processing a bunch of files, you would add the restart decorator to the function that processes a single file. You’d add it to the function that represents kind of the smallest unit of work that might fail with an exception, Then, when you get an exception, you can actually edit the function. Like, if we edit it so it doesn’t throw an error, and then we can resume execution, then it will return from foo using the the new version of baz, without having to run the script from the beginning again. So in the example of a script that processes a bunch of files, that would let you, as you run into files that cause an exception, fix your code to deal with it and resume execution without having to restart the script from the beginning. But this is obviously a pretty terrible hack, having to add the restart decorator to the function. I would like it to be able to restart from any function. without needing the decorator, as you can in Common Lisp, but I think that will require patching CPython and I really have no idea how to do that. So if you do know anything about CPython internals and are interested in helping, please reach out.

[00:13:03.721] pydumpling

Another feature we have with the backtrace buffer is there’s this library called PyDumpling which can serialize a traceback and store it to a file. So you can use PyDumpling with your applications running in production to serialize a traceback whenever they have an exception and save it to a file. Then you can transfer the file locally and load it into your local Emacs with slime-py-load-pydumpling. This will load the same backtrace buffer, and you see all the same local variables at the time of the exception. You can inspect them and get a REPL in the context of the stack frame. Well, this will only work for variables that can be serialized with pickle. Or actually, the library uses dill, which can serialize a bit more than pickle can. But yeah so this can help you inspect and debug errors for applications running in production remotely that you don’t want to have SLIME connected to 24-7.

[00:14:20.060] Documentation browser

Next up, let’s look at the documentation browser. We can bring up documentation for any module, and all this information is generated from runtime introspection, from the doc strings for the module and the classes and so on. So you won’t see documentation for libraries that you don’t have actually loaded into your running Python process. Then you can go browse to classes. It’ll show all the attributes, their methods, and so on. By each method to the right, it will show the base class where the method was originally inherited from. You can also bring up a screen with all the Python packages that are installed, and browse that with imenu, and bring up information on any package and so on.

[00:15:20.360] Thread view

Next up, let’s take a look at the thread view. So let’s run this and then bring up the thread view and this will show information on all running threads. You can configure it to refresh after a given interval, like every second, but I don’t have that set up right now, so I have to manually refresh it. Probably the most useful thing is that you can bring up a backtrace for any thread which won’t pause the thread or anything, but will just give you the call stack at the time you requested the backtrace. You can again view the stack frames, local variables, open a REPL in the context of the thread, and so on. There’s also a viewer for async tasks, but I’m not going to demo that right now, because for that to work, you have to start swanky-python after the async event loop has started, from within the same thread. If you go to the project readme, there’s a demo of how to use the async task viewer with a fastapi project.

[00:16:27.440] Tracing functions

Next up, let’s look at tracing functions. So here we got some random error, because this is still very much a work in progress. But it looks like it executed correctly this time. So now let’s mark the fibonacci function for tracing and execute it. We can see, every time the function is called, all its arguments and return values. Again, there are presentations that we can inspect and so on. But let’s inspect a more complex object, like a file object. If we trace the count_lines function and run that code, then we can inspect the file it was passed, or the file object. One pitfall is that in Python, objects are mutable. So in the trace buffer, the string representation that’s printed is the string representation at the time it was passed to the function. But when we go to inspect it, we’re inspecting the object as it is right now, which can be different than it was at the time the function saw it. So for this file object, for example, it’s closed now, when it was open at the time the function used it.

[00:17:47.800] AI integrations

Next up, let’s look at AI integrations. So if you’re used to SLIME with Common Lisp, Emacs actually has a built-in AI that can help with the transition. So it’s just a joke, I actually really like Python. And for more serious AI integrations, I have some ideas for the future but I haven’t implemented anything yet. I think right now, people are mostly passing source code to LLMs but since we’re embedded in the Python process at runtime, we have a lot of more information available, like maybe we can trace all calls to functions, and when we have a bug, we can feed the trace to the LLM, and the LLM can point out maybe when this function was called with these arguments, its return value doesn’t make sense, so maybe that’s the root cause of your bug. If you have any ideas of potential LLM or AI integrations, let me know. I’m happy to discuss.

[00:19:06.000] LSP-type features

Next up, let’s look at standard LSP-type features. So we’ve got completions. It’s fuzzy completions right now, so it’s showing everything with a PR in the name. We can bring up documentation for each one. When we start calling a method in the minibuffer at the bottom it’ll show the signature. There’s some refactoring available. We can extract a function or variable, or rename something, like, let’s rename fib to fib2, and it will rename all the uses of it. All these features are based on Jedi, which is the Python library used by IPython. But as it is right now, if you want the most complete Python development experience in Emacs, I’d probably recommend using LSP for everything LSP can do, and then just using swanky-python for the object inspector and backtrace buffer, and the interactive features it has that an LSP can’t provide.

[00:20:18.032] Wrapping up

And that’s it really. Shortly we’ll have questions and answers as part of EmacsConf, and later on, if you have any questions, ideas, or issues feel free to reach out over email or create an issue on the repository. I should probably warn you, if you want to try out the project: so far I’m probably the only user of it and I’ve only tested it on my own Emacs setup, so it’s quite likely you’ll run into issues trying to get it installed and working. But if you do run into problems, please reach out, let me know. I’m happy to help and try and fix them. So that’s it. Thanks for listening.