I like to draw sketchnotes when I want to untangle a thought or build up a thought over several posts.1 Following up on Playing around with org-db-v3 and consult: vector search of my blog post Org files, with previews, I want to be able to search my sketches using approximate text matches. I also want to have an approximate search interface that includes both sketches and blog posts.

Here’s what I’ve gotten working so far:

Screencast of my-sketch-similar and my-consult-similar

• 0:00: Text preview: All right, so here’s a preview of how I can flip through the images that are similar to my current text. Let’s say, for example, I’ve got my-sketch-similar. I’m passing in my whole blog post. This one is a text preview. It just very quickly goes through the OCR text.
• 0:27: Image preview: I can use the actual images since Emacs has image support. It’s a bit slower and sometimes it doesn’t work. So far it’s working, which is nice, but you can tell there’s a bit of a delay.
• 0:51: External viewer like Geeqie: The third option is to use an external program like Geeqie. Geeqie? Anyway, that one seems a lot faster.
• 1:12: my-consult-similar: Then I can use that with this new multiple source consult command that I’ve just defined. So, for example, if I say my-consult-similar, I can then flip through the related blog posts as well as the related sketches in one go. I was calling it with a C-u universal prefix argument, but I can also call it and type in, for example, parenting and anxiety. Then I can see my recent blog posts and sketches that are related to that topic. I think I should be able to just… Yes, if I press enter, it will insert the link. So that’s what I hacked up today.

The data

As part of my sketchnote process,2 I convert my sketches to text files.3 I usually use Google Cloud Vision to automatically convert the images to text. By keeping .txt files beside image files, I can easily search for images and include them in blog posts.4 I usually edit the file afterwards to clean up the layound and fix misrecognized words, but even the raw text output can make files more searchable.

I indexed the sketches’ text files with:

(defun my-org-db-v3-index-recent-sketches (after)
(interactive (list
(when current-prefix-arg
(org-read-date nil nil nil "After: " nil "-2w"))))
(setq after (or after (org-read-date nil nil "-2w")))
(mapcar #'org-db-v3-index-file-async
(seq-remove
(lambda (o) (string> after (file-name-base o)))
(directory-files "~/sync/sketches" t "\\.txt$"))))

Completion code

Writing the Consult completion code for the sketches was pretty straightforward because I could base it on my blog posts.

(defun my-org-db-v3-sketch--collection (input)
"Perform the RAG search and format the results for Consult.
Returns a list of cons cells (DISPLAY-STRING . PLIST)."
(mapcar
(lambda (o)
(cons (file-name-base (alist-get 'source_path o)) o))
(seq-uniq
(my-org-db-v3-to-emacs-rag-search input 100 "%sync/sketches%")
(lambda (a b) (string= (alist-get 'source_path a)
(alist-get 'source_path b))))))

(defun my-sketch-similar (&optional query hide-initial)
"Vector-search blog posts using `emacs-rag-search' and present results via Consult.
If called with \\[universal-argument\], use the current post's text.
If a region is selected, use that as the default QUERY.
HIDE-INITIAL means hide the initial query, which is handy if the query is very long."
(interactive (my-11ty-interactive-context current-prefix-arg))
(consult--read
(if hide-initial
(my-org-db-v3-sketch--collection query)
(consult--dynamic-collection
#'my-org-db-v3-sketch--collection
:min-input 3 :debounce 1))
:lookup #'consult--lookup-cdr
:prompt "Search sketches (approx): "
:category 'sketch
:sort nil
:require-match t
:state (my-image--state)
:initial (unless hide-initial query)))

(defun my-sketch-similar-insert (link)
"Vector-search sketches and insert a link.
If called with \\[universal-argument\], use the current post's text.
If a region is selected, use that as the default QUERY.
HIDE-INITIAL means hide the initial query, which is handy if the query is very long."
(interactive (list
(if embark--command
(read-string "Sketch: ")
(apply #'my-sketch-similar
(my-11ty-interactive-context current-prefix-arg)))))
(my-insert-sketch-and-text link))

(defun my-sketch-similar-link (link)
"Vector-search sketches and insert a link.
If called with \\[universal-argument\], use the current post's text.
If a region is selected, use that as the default QUERY.
HIDE-INITIAL means hide the initial query, which is handy if the query is very long."
(interactive (list
(if embark--command
(read-string "Sketch: ")
(apply #'my-sketch-similar
(my-11ty-interactive-context current-prefix-arg)))))
(when (and (listp link) (alist-get 'source_path link))
(setq link (my-image-filename (file-name-base link))))
(insert (org-link-make-string (concat "sketchLink:" link) (file-name-base link))))

(From Handle sketches too in my config)

Previewing images

Displaying images in Emacs can be a little bit slow, so I wanted to have different options for preview. The fastest way might be to preview just the text to see whether this is a relevant image.

(setq my-sketch-preview 'text)

Another way to preview to load the actual image, if I have a bit more patience.

(setq my-sketch-preview t)

Figure 2: Screenshot of image preview

Sometimes Consult says “No partial preview of a binary file”, though, so I can probably look into how to get around that.

Using an external program is another option. Here I have some code to use Geeqie to display the images.

(setq my-sketch-preview 'geeqie)

Figure 3: Using Geeqie to flip through images

Using Geeqie feels faster and more reliable than using Emacs to preview images.

The preview is implemented by the following function in the Completing sketches part of my config.

(declare-function 'my-geeqie-view "Sacha.el")

(defvar my-sketch-preview 'text
"*Preview sketches.
'text means show the associated text.
'geeqie means open image in Geeqie.
t means open image in Emacs.")

(defun my-image--state ()
"Manage preview window and cleanup."
;; These functions are closures captured when the state is initialized by consult--read
(let ((preview (consult--buffer-preview))
(open (consult--temporary-files)))
;; The returned lambda is the actual preview function called by Consult
(lambda (action cand)
(unless cand
(funcall open))
(when my-sketch-preview
(let ((filename (cond
((and (eq my-sketch-preview 'text)
(listp cand)
(alist-get 'source_path cand))
(alist-get 'source_path cand))
((and (listp cand)
(alist-get 'source_path cand))
(my-image-filename (file-name-base (alist-get 'source_path cand))))
(t cand))))
(when filename
(pcase my-sketch-preview
('geeqie (my-geeqie-view (list filename)))
(_ (funcall preview action
(and cand
(eq action 'preview)
(funcall open filename)))))))))))

The following function calls geeqie. It’s in the Manage photos with geeqie part of my config.

(defun my-geeqie-view (filenames)
(interactive "f")
(start-process-shell-command
"geeqie" nil
(concat
"geeqie --remote "
(mapconcat
(lambda (f)
(concat "file:" (shell-quote-argument f)))
(cond
((listp filenames) filenames)
((file-directory-p filenames)
(list (car (seq-filter #'file-regular-p (directory-files filenames t)))))
(t (list filenames)))
" "))))

Multiple sources

Now I can make a Consult source that combines both blog posts and sketches using semantic search. I wanted to have the same behaviour as my other functions. If I call it interactively, I want to type in text. If I call it with a region, I want to search for that region. If I call it with the universal prefix argument C-u, I want to use the current post text as a starting point. Since this behaviour shows up in several functions, I finally got around to writing a function that encapsulates it.

Then I can use that for the interactive arguments of my new my-consult-similar function.

(defvar my-consult-source-similar-blog-posts
(list :name "Blog posts"
:narrow ?b
:category 'my-blog
:state #'my-blog-post--state
:async (consult--dynamic-collection
(lambda (input)
(seq-take
(my-org-db-v3-blog-post--collection input)
5)))
:action #'my-embark-blog-insert-link))

(defvar my-consult-source-similar-sketches
(list :name "Sketches"
:narrow ?s
:category 'sketch
:async (consult--dynamic-collection
(lambda (input)
(seq-take (my-org-db-v3-sketch--collection input) 5)))
:state #'my-image--state
:action #'my-insert-sketch-and-text))

(defun my-consult-similar (query hide-initial)
(interactive (my-11ty-interactive-context current-prefix-arg))
(if hide-initial
(let ((new-sources
(list
(append
(copy-sequence my-consult-source-similar-blog-posts)
(list :items (seq-take (my-org-db-v3-blog-post--collection query) 5)))
(append
(copy-sequence my-consult-source-similar-sketches)
(list :items (seq-take (my-org-db-v3-sketch--collection query) 5))))))
(dolist (source new-sources)
(cl-remf source :async))
(consult--multi new-sources))
(consult--multi '(my-consult-source-similar-blog-posts
my-consult-source-similar-sketches)
:initial query)))

(defun my-org-db-v3-index-recent-public (after)
(interactive (list
(when current-prefix-arg
(org-read-date nil nil nil "After: " nil "-2w"))))
(setq after (or after (org-read-date nil nil "-2w")))
(mapc #'org-db-v3-index-file-async
(my-blog-org-files-except-reviews after))
(my-org-db-v3-index-recent-sketches after))

This is what it looks like given this whole post:

Figure 4: Screenshot of semantic search for both blog posts and sketches

Thoughts and next steps

The vector search results from my sketches don’t feel as relevant as the blog posts, possibly because there’s a lot less text in my sketches. Handwriting is tiring, and I can only fit so much on a page. Still, now that I’m sorting results by similarity score, maybe we’ll see what we get and how we can tweak things..

It might be nifty to use embark-become to switch between exact title match, full-text search, and vector search.

plz supports asynchronous requests and org-db-v3.el has examples of doing this, so maybe I can replicate some of Remembrance Agent’s functionality by having an idle timer asynchronously update a dedicated buffer with resources that are similar to the current paragraph, or maybe the last X words near point.

I wonder if it makes sense to mix results from different sources in the same list instead of splitting it up into different categories.