By Artyom Bologov

So there are half a dozen of Common Lisp package/system/project managers:

• Quicklisp (with Ultralisp distribution)
• CLPM
• OCICL
• Qlot
• Vend
• Qi

These mostly fall into two categories:

1. Package/project managers fetching things from the Internet right into the running Lisp image and not touching the project itself.
2. Project-local package managers that include all the dependencies into the project tree.

Project-local solutions bloat the checkout size and slow down repository cloning. Online ones require network connection. I’m not satisfied with either of the trade-offs, and I think I have a solution to that.

Okay, I’ll need a more involved problem statement than that. The features I’m looking for in my dependency management journey are:

• Fully local build options
• CLI/UNIX-friendliness, especially for running tests
• Reproducibility, of a weaker kind—Lisp deps that don’t change underneath me
• Not needing bloated Swiss knife binaries just to manage dependencies
• Option to use both local deps and Quicklisp / Ultralisp / CLPM / whatever
• Custom REPLs with dependencies pre-loaded
• Not reinventing the wheel

Submodules #

Git submodules are this kind of solution.

• They allow the library developer to pin dependencies precisely
• The library user only needs to fetch the dependencies they require, when they need them
• There’s no need to install any heavyweight tools like Roswell
• Repository size doesn’t grow unboundedly for those that don’t want it
• It’s CI-friendly and well-integrated with the UNIX tooling

So my setup is:

• Make a project/library and write the code interactively using Quicklisp for dependencies.
• Once the library is more or less stable dependency-wise, run a 100 LoC dependency-resolving script using Quicklisp metadata.
• Run all the commands it suggests (it doesn’t run anything itself, by design!)
• Add some leftover deps that Quicklisp doesn’t have or that are special in some other way. (I tend to use Nyxt’s .gitmodules to get dependency locations (seems like working on Nyxt left an imprint, huh?))
• And then set CL_SOURCE_REGISTRY to the path with dependencies (usually repository root) when I need to run tests or a custom REPL.

Stripped-down project Makefile (yes, I use Make for Lisp projects) then looks like:

LISP ?= sbcl
LISP_FLAGS ?= ...

export CL_SOURCE_REGISTRY ?= $(PWD)//

executable:
$(LISP) $(LISP_FLAGS) --eval '(require "asdf")' --eval '(asdf:make "system")' --eval '(uiop:quit)'

.PHONY: tests
tests:
$(LISP) $(LISP_FLAGS) --eval '(require "asdf")' --eval '(asdf:test-system "system")' --eval '(uiop:quit)'

.PHONY: repl
repl:
$(LISP) $(LISP_FLAGS) --eval '(require "asdf")' --eval '(asdf:load-system "system")'

Makefile for submodule vendoring approach Things to note:

exported/shared CL_SOURCE_REGISTRY with double slash That’s a vital hack mentioned by Alexander Artemenko on Reddit. This double slash means a given directory is an ASDF “tree” to be searched for systems recursively. Here’s the syntax explanation in ASDF manual. –eval ‘(require "asdf")’ –eval ‘(asdf:load-system "system")’ All the bootstrapping is basically two expressions, all reliant on ASDF. Given that CL_SOURCE_REGISTRY is set up properly, all the dependencies are there. Build, test, and launch REPLs all you want. No Quicklisp needed. repl REPL is the same call to $LISP, but without the --eval '(quit)' It’s both rlwrap compatible and has all the necessary libraries accessible. Swank/Slynk integration Easy to support by adding two more ASDF subsystems and running them from the Makefile.

This setup gives me:

• Ability to clone the project without dependencies for use with Quicklisp—git clone project-url
• Ability to clone the project with all the dependencies—git clone --recursive project-url
• Ability to get only a subset of dependencies I need (say, for libraries not on Quicklisp yet)—git submodule update --init -- deps/something or git submodule update --init -- deps/ to update them all
• Global reuse of dependencies with custom ASDF registries (as in .config/common-lisp/source-registry.conf.d/asdf.conf)
• Option to run tests from CLI and on CI with all the dependencies accessible to them
• Precisely pinned dependencies possibly going fresher than Quicklisp
• Custom REPLs in the dev environment
• Total Quicklisp / CLPM / OCICL independence, with the only hard project dependency being ASDF (and Quicklisp project repositories at vendoring time, but that’s minor and easy to have locally)
• An option to use Quicklisp etc. nonetheless, just do make CL_SOURCE_REGISTRY=path/to/systems// and your dependencies are replacing the submodule ones

What not to like about it?

The Problem With ~/common-lisp #

Thanks to Konrad Hinsen for making me think about that! One scenario where this submodule-driven workflow might break is when you clone the repo into ~/common-lisp. And then there’s another version of one of the deps in there, provoking a version clash.

Cloning into ~/common-lisp is a bad practice that should be eradicated in favor of properly compartmentalized registries. But alas. Anyway, if you clone into ~/common-lisp or ~/.local/share/common-lisp/source/, then do a non-recursive clone instead. If you cloned the library there, you likely have all the dependencies available already. In the very same ~/common-lisp, no doubt. So no need for submodules.

But better not use ~/common-lisp at all, global state bad.

---

To see how the whole setup around submodules looks, you can check out my Quickproject template and its makefile and .asd file.

Leave feedback! (via email)