Taking Notes on Nicole van der Hoeven

Non-Lazy DMs use Obsidian for D&D

Thu, 30 Sep 2021 16:29:54 +0200

Last month, I wrote about how I use Obsidian for D&D as a player in a 4e campaign (I play a half-elf hybrid wizard/cleric). This is about my second weekly game, in which I am the Dungeon Master for a 5e campaign. I’m a relatively new DM (I’ve got 39 games under my belt at the time this writing, and Obsidian is at the center of my workflow for improving my DMing skills, getting inspiration, preparing for a game, running the game, and making sure my players are having fun.

In many ways, my setup is the Non-Lazy DM’s approach to preparing for D&D games: high-effort, high-touch, and high-return. But I only do it because I enjoy it!

Plugins I use

This is going to be a long article, so if you’re just after the plugins I use, here they are up front!

I go into detail for some, but not all of these, in the article below.

Additionally, feel free to skip around to different sections by clicking on the hamburger menu on the upper right-hand corner and selecting the session that interests you.

Outside the game

Although I use Obsidian while playing D&D, I use it most for D&D between games: fleshing out the world, getting inspiration for new adventures, or preparing for specific sessions.

World-building

Many DMs run published campaigns, but I’m not one of them. I don’t think there’s anything wrong with running published campaigns; I just personally enjoy world-building and being able to tailor the environment to what I know my specific players will like. However, I still do read campaign books, for D&D as well as for other systems, for inspiration. I am also a voracious fantasy reader, and consider everything fair game for “stealing” for my games.

One thing I’ve begun experimenting with is processing my fiction reads the way I do my nonfiction reads. What better place to look for inspiration about creating a fantasy world than other fantasy worlds? How could I use a Bene Gesserit-like faction in my campaign?

I recently reread the first Dune book in preparation for the movie coming out later this year, but while I was reading it, I remembered how much I love the world it’s set in. I read books on Kindle, either on my actual physical Kindle ereader or on the Kindle app on my mobile, and I took highlights while I was reading Dune. You can see my full highlights here, synced via Readwise. I highlight turns of phrases I like, character descriptions that sounded cool, or passages that are particularly revealing of the type of world the story of Dune is set in. Below is an example of a quote I highlighted:

Schools were started to train human talents.” “Bene Gesserit schools?” She nodded. “We have two chief survivors of those ancient schools: the Bene Gesserit and the Spacing Guild. The Guild, so we think, emphasizes almost pure mathematics. Bene Gesserit performs another function.” “Politics,” he said. (Location 248)

It really resonated with me as a good characterization of two important factions in the Dune universe.

Then, just as I do with nonfiction books, I create a separate note about the book itself, attempting to summarize the parts of it that might be interesting or useful to me in the future. Below is a screenshot of my page on the first book of Dune. I identify “Key truths” about the world, because I also like to use that terminology when creating secrets or themes for a fantasy world or city I’ve created.

Here are my full notes on Dune.

Then, I also think about tropes or patterns of character, plot, or setting present in the book. I’ve started collecting these in a a page called Forms in my vault. I think it’s useful to spot things that occur again and again in fantasy fiction, whether they’re overused or not. If they’re tropes that I’d rather not repeat, having a “formula” of the trope helps me either avoid it or consciously subvert it in my games. Sometimes, I just take a form wholesale and incorporate it as is.

Here’s an example of a page I created that was inspired by the Bene Gesserit– a short note about “Nuns who think and fight”.

I haven’t been doing it for very long, but it’s already come in useful when thinking of major events and players in my world.

Adventure creation

Adventures are smaller-scale and higher-resolution than campaigns or worlds, so I often need more specific notes than forms.

I read and highlight adventure blog posts and articles using Readwise’s recently announced Reader app, and then they, along with tweets, books, and PDFs, get sent to Readwise and then Obsidian using the Official Readwise plugin for Obsidian. I’ve written about this process in depth, but here’s what that looks like for Mike Shea’s Return of the Lazy Dungeon Master - affiliate link.

An advantage of using Reader (which is currently in a private beta, so I’m not allowed to share screenshots of it) is that I can even highlight text locked behind a paywall. For example, I have a D&D Beyond subscription and I’ve bought all the books available on their store, but most reading apps don’t know that, so it’s been hard to find one that will let me highlight text from one of the WotC sourcebooks. Not so with Reader. Here’s what my highlights from the first chapter of Volo’s Guide to Monsters looks like in my Obsidian vault after it’s synced from Reader:

Readwise is great because I can highlight something on my Kindle and worry about processing it later in Obsidian. I process incoming Obsidian notes daily from 9-10pm– that doesn’t mean I get through them all, just that I have a scheduled time to go through highlights and notes every day.

For D&D-specific notes, that also involves summarizing what I’ve learned into the notes I already have on how to be a dungeon master:

The referencing feature on Obsidian lets me confidently paraphrase without stealing. All I need to do is put a [^lazydm] where I want the superscript to be within the text, and then at the end of the note, I put something like:

[^lazydm]: Shea, M. (2018). _Return of the lazy dungeon master_. [[Return of the Lazy Dungeon Master|My highlights]].

That gets parsed as a footnote by Obsidian and lets me properly credit authors.

When I find good tools that help me flesh out adventures, I try to update my page on DM tools and inspiration, so they’re all in one place.

While creating adventures, I like to use Obsidian Admonition to call out specific parts of the text for me as I’m running the adventure, like this:

Random tables

I definitely have overarching plans about the wider plot in the world at large, but I also like to be surprised. Throwing random elements into games is one of the ways that I can keep myself on my toes and remind myself that I’m not the only one making the story.

There are many random tables in both official and unofficial D&D sourcebooks, but I only use those as starting points. For me, creating random tables that are tailored to my characters and to the things they care about is where it’s at!

I’ve experimented with creating my own random generators for various things, as well as using existing ones like Donjon, but it’s an Obsidian plugin, Jeremy Valentine’s Dice Roller, that I have settled on.

The name “Dice Roller” makes you think it’s only for rolling dice, but it’s much more than that. Its REAL value, in my opinion, is in how it lets me create tables for ANYTHING, and then randomly pull one row out of it. Here’s how it works.

First, you create a table. I had a session where my players discovered a place I called Shipwreck Beach, and had a lot of fun going through several of the ships that were there. Here’s a table of “encounters” I put together for that session.

### Ship encounters
| Roll | Encounter |
| ---- | --------------------------------------------------------------------- |
| 1 | Carnivorous Ship, ToB II p. 56 |
| 2 | Map Mimic: https://www.dndbeyond.com/monsters/1624929-map-mimic |
| 3 | Random curse: p.249 of GM's Book of Random Encounters - d100 |
| 4 | Clockwork weaving spider (1, ToB p. 66) |
| 5 | 1d6 Cobbleswarm (2, ToB p. 68) |
| 6 | a feral little girl wearing a periapt of proof against poison |
| 7 | A large gold coin bearing the crest of the Kryn Dynasty and the queen |
| 8 | Book about the Arms of the Betrayer Gods |
^ship

Note the ^ship identifier at the bottom there– it can be called whatever you want, but you’ll use that identifier when rolling on that table. I also included the Roll column, in case I decided on the spot to let players roll instead of doing it myself.

Next, in another note (or the same one, if you want), include this line:

Shipwreck encounter: `dice: [[Shipwreck Beach#^ship]]`

When you flip to Preview mode, you’ll see this in the place of the line above:

That’s it!! This has made a HUGE impact on my games, because it’s so easy to create or modify tables. I use random encounter tables sparingly during the game, but often use it before a game, to introduce something fresh that I hadn’t thought of.

Session prep

As you might have guessed, I mostly use Sly Flourish’s Lazy DM steps for encounter creation, saved as an Obsidian template:

---
type: session
world: Exandria
campaign: "The Slayers Five"
date: {{date}}
game_date: 836-03-
location: Rumblecusp
characters: [Boris, Lira, Ora, Jamedi, Flint]
tags: [TVZ]
---
## [[Session Summary]]:
^summary
---
## Housekeeping
## Recap
![[^summary]]
## Strong start
>
## Scenes
- [ ]
- [ ]
- [ ]
- [ ]
## Secrets and Clues
- [ ]
- [ ]
- [ ]
- [ ]
- [ ]
- [ ]
- [ ]
- [ ]
- [ ]
- [ ]
## [[Loot]]
- [ ]
---
## Log

Before each session, I’ll go through each of these sections.

I add metadata to every session for use with the Dataview plugin later, in case I want to do a search for all sessions that involved a particular NPC or that took place in a certain location, for example.

I fill out the Session Summary section after the game, but I put it at the top so I can immediately see what happened in the session in case I need to refresh my memory on it in the future. I also identify this section with a ^summary so that I can embed it in the next session, as a recap - more on that below.

In the Housekeeping section, I write down things I want to mention before we dive into the narrative of the session. Usually, these topics are scheduling- or rules-related. For example, if my players have just levelled up and I see that one of them has chosen a feat, I might make a note of it here to make sure that they know what it means. My players are pretty new, so I think it’s better to err on the side of more communication rather than less.

In the Recap section, I call the sumary from the previous session and embed it into the current session using something like ![[38_8360310-8360311 Seduction at the Moonlight Chateau#^summary]]. When in preview mode, Obsidian shows just that section labelled with ^summary from the previous game. I like to start the game by either reading out the summary myself, or asking one of the players to recount what they remember. That way we’re all on the same page when we begin.

In Strong start, I write a paragraph or two of prepared text that I use to kick off the story. This is usually the only text I’ll read out word for word during the game.

Scenes are a list of events that I THINK will happen. My players frequently surprise me, which I welcome, so I also list these with checkboxes so that I can keep track of where the story deviated from the plan.

Sly Flourish does recommend a separate section for Fantastic Locations, but I tend to prepare these as separate pages. For example, here are my notes on a possible location, The Charred Circus. I don’t include them within the session note itself, but I do make sure I have a few that characters could go to. Then, I link to all of them from the note I have for the city they’re in, like this one for Port Damali.

Secrets and Clues is where I put a list of 10 things that players MIGHT find out somehow. This is where I look when a player rolls a natural 20 on a Perception check. The idea is to describe what the secrets are, but not get too caught up in trying to plan when or how the players will find out those secrets. I try to include things that might be interesting for the players, as well as a few potential adventure hooks. Here’s a list that I made for a previous game. I have never used all 10 up in one game, but I like having more than I need.

In Loot, I put in any items I know the party will likely get to. If they’re in a boss fight, I usually save in this section all the things I’ve decided beforehand they’ll get. If I don’t have something specific like that in mind, I sometimes randomize a bunch of non-magical trinkets. The Loot page contains items I’ve picked out for specific characters.

Post-session processing

If your players are anything like mine, you’ll often find yourself with a bunch of unused material and a lot of stuff you still had to improvise on the spot, despite adequate preparation. That’s okay! I actually enjoy when my players do something unexpected. I encourage and reward creative play… but it does mean that I have to do a bit of a debrief after each session.

I try not to prepare things that may really be “wasted”. Things like locations, NPCs, or general plot hooks can usually be recycled. After games, I spend some time tying off loose ends, noting down decisions the party made and the reasoning behind those decisions, and tweaking connections between NPCs or places based on how the party reacted.

Fleshing out bits of the world, with each city or NPC getting a page
Tagging cities and NPCs with Location or NPC, and then within the city, doing a query for NPC and city name to get everyone living in that city.

I use metadata for every page in my campaign so that I can use the excellent Dataview plugin later to find whatever I want. You already saw my session template, but here’s an NPC template that I use:

---
type: NPC
faction: []
location: []
world: Exandria
campaign: "The Slayers Five"
date: [{{date}}]
description: ""
race: []
gender: []
class: []
---

I insert this template for every single NPC in my world, whether my players get to it or not. Because I use it so often, I’ve set up a hotkey for inserting templates in the Obsidian settings:

That way, all I have to do is hit CMD + ; and select the right template from the list, and the text above is inserted at the beginning of the note as frontmatter, with all the fields I need to fill out. The date is also automatically generated.

Using the Dataview plugin, I can then embed queries on location pages so that I can see NPCs in that place at a glance. For example, here’s the snippet I put under the People section in my Port Damali page:

\```dataview
table shop as Shop, faction as Faction, description as Description from "Exandria"
WHERE contains(type, "NPC") and contains(location,"Port Damali")
SORT file.name ASC
\```

(If you try this, remove the backticks \ in the snippet above– those are just to show you the syntax for a dataview query.)

That query, when rendered in Preview mode in Obsidian, looks like this:

Having the metadata for every NPC, monster, item, and place in my world is essential for creating queries like this. I’ve also found it useful to maintain a model of the metadata I use:

- `type`
- `NPC`
- `PC`
- `shop`
- `place`
- `item`
- `session` for D&D play sessions
- `quest`
- `race`
- `faction`
- `family`
- `monster`
- `condition`
- `deity`
- `other` for anything else
- `characters` (for sessions): characters involved in this session
- `deceased`
- `true`
- `false`
- `family`
- `faction`
- `race`
- `gender`
- `class`
- `description`
- `date`
- `game_date`
- `world`
- `Exandria`
- `Forgotten Realms`
- `Exandya`
- `campaign`
- `Order of the Wandering Mind`
- `The Slayers Five`
- `Die vergessene Schwester`
- `Temporary White Circle`
- `shop` (for proprietors)
- `shop_type` (for shops)
- `general` : general goods store
- `alchemy`: potions
- `magical items`: sells magical items
- `services`: magical spells, messaging, brothel
- `tavern`: food and drink
- `inn`: lodging
- `herbalism`
- `smithy`
- `location`: city of residence
- `origin`: city of origin

During the game

I keep a “World” page for each of my D&D campaigns, including a list of sessions:

During the game, I use the session page that I mentioned above, and start filling in the Log section. I tend to take way fewer notes as a DM than as a player, just because I’m busy doing other things, but I do always write down things that the players do that I might need to remember later.

One thing that comes up frequently in my games is the need to improvise encounters, NPCs, or locations. I’ve had great success in creating a Random Rolls page that collates many of these random tables in one note. If you click on that link, you unfortunately won’t be able to see the results rendered, because the Dice Roller plugin isn’t available for Obsidian Publish, but maybe you’ll get an idea of things I’m rolling on. Most of the tables referred to are from this Random Tables note, but every now and then I’ll include tables in location notes instead.

I always have this Random Rolls page up as part of my DM workspace, in case I need to quickly improvise something. Here’s what the NPC section looks like:

I can use any or all of those things that I want, or use them as inspiration for something else I make up on the spot. Either way, it means I’ll never be completely at a loss for the name of an NPC. ;)

Initiative Tracker

I use D&D Beyond for character sheets, but I use Jeremy Valentine’s initiative tracker to handle combat because there are a few critical features that D&D Beyond’s Encounter Tracker doesn’t yet support:

Status conditions
(Easy to tweak) homebrew monsters
Hiding monsters’ names from the players

The Initiative Tracker plugin supports all of these. When activated, it appears as a side panel in Obsidian:

I can easily see everyone’s HP and AC at a glance, and also apply special status conditions when necessary. My players don’t have to know whose turn it is– they just have to know when their turn is. I warn them verbally when they’re “on deck”, and they haven’t had an issue with that.

D&D Beyond does allow you to create homebrew monsters, but the only real benefit to doing so is to be able to use it in their Combat Tracker, so I’ve been using another plugin by Jeremy Valentine, the author of this one, to create monster statblocks

Statblocks

The 5e statblocks plugin lets me easily create homebrew creatures just by using this format:

\```statblock
name: string
size: string
type: string
subtype: string
alignment: string
ac: number
hp: number
hit_dice: string
speed: string
stats: [number, number, number, number, number, number]
fage_stats: [number, number, number, number, number, number, number, number, number]
saves:
- <ability-score>: number
skillsaves:
- <skill-name>: number
damage_vulnerabilities: string
damage_resistances: string
damage_immunities: string
condition_immunities: string
senses: string
languages: string
cr: number
spells:
- <description>
- <spell level>: <spell-list>
traits:
- [<trait-name>, <trait-description>]
- ...
actions:
- [<trait-name>, <trait-description>]
- ...
legendary_actions:
- [<legendary_actions-name>, <legendary_actions-description>]
- ...
reactions:
- [<reaction-name>, <reaction-description>]
- ...
\```

(Again, remove the backticks \ at the starting and ending lines of the snippet above if you copy and paste this into your vault.)

I usually start with a “real” monster from official or unofficial sourcebooks, then tweak it from there. For example, here’s a sample one that I tweaked from Tome of Beasts:

The best part about this plugin is that it integrates with the Initiative Tracker. When you create a statblock and save it, that monster then becomes available for use within the Initiative Tracker, as you can see from the screenshot above.

Publishing

I subscribe to Obsidian Publish, partly to give back to the fine Obsidian developers who’ve made a tool I use daily and partly because I’ve found that it’s reduced friction in publishing. I use it primarily for non-D&D-related things, but D&D is part of my life, so it’s there as well. You can find my notes on the D&D campaign I DM here.

Some of the plugins that I use unfortunately don’t render correctly when Published, but the developers are working on that.

I publish my campaign notes mainly because I love seeing other DM’s thought processes and systems, so I thought I’d return the favour by sharing mine as well. In case you were wondering, my players don’t know I publish these notes– or, at least, I hope they don’t!

Player-facing campaign vault

I used to drop handouts into a WhatsApp group with my players, but it became increasingly unwieldy to look for something. We switched to Slack, but that had the same problem of needing to scroll back through lots of photos before finding the right one. Photos aren’t very searchable in those apps.

A few months ago, I switched to Obsidian for handouts. I created a separate vault for the campaign and shared it with my players via Dropbox. I did have to talk them through the setup, but it’s worked out very well for us as a free alternative.

In the vault, I put things like:

Pictures and other handouts
Session videos and summaries
Links for everything we use as a group
House rules

Session page

I create a page for every session we have, which has a section for the video recording of that session (unlisted and just for us, sorry), a player-friendly summary, and a log. The log is where I dump photos I give them during the session. Here’s what that looks like:

I don’t put any other notes in the log; that’s the players’ jobs (if they want to).

I like to have a “Story Thus Far” page that embeds the video and summary sections of all the sessions, so any of us can read through it to remember what’s happened. It’s fun to see how far we’ve come as a group.

About our game

The About our game page has a list of links to all the tools we use, our schedule, and a few things about the campaign and the world.

Rules

I keep a house rules page that shows all the extra rules I’ve put into the game. I mostly play with all the WoTC-produced sourcebooks, but I’ve added some extra rules about things that my players are interested in, such as strongholds (for which I use Matt Colville’s Strongholds & Followers book) and smith’s tools.

I also have some other pages on rules that come up often, such as Ship statblocks (since the campaign is set on a coast) and suggested downtime activities.

Conclusion

I call my approach to game preparation the Non-Lazy DM’s approach as a tongue-in-cheek nod to Mike Shea’s content, but there’s truth to the moniker, too. My setup is extremely detail-oriented, time-consuming, and fun… at least for me! And that’s the key: I wouldn’t be doing any of this if I didn’t find it truly enjoyable to world-build to this level of detail.

If you’re a non-lazy DM too, thanks for sticking around! I hope this has given you some ideas for how you could use Obsidian to run your games and stimulate your imagination, too.

How to import Shortform highlights to Obsidian via Readwise

Mon, 16 Aug 2021 21:40:16 +0200

I’ve always been dubious of those book summary apps. You know the ones– they tend to have obnoxious ads that promise to help you “read 5 books a day” or learn the world’s secrets in 15 minutes. Having tried a few of them, I’ve been disappointed with the quality of their summaries (or lack thereof). They attempt to distill the message of a 300+ page book into a handful of bullet points, which I’ve found neither accurate nor useful. Recently, though, I found myself happily subscribing to one, and here’s why.

What’s Shortform and why is it awesome?

Shortform really took me aback because it isn’t just a book summary. I should mention here that I’m not affiliated with Shortform and gain nothing by recommending them. I’m just a happy subscriber.

When I read their guide for the book Why We Sleep, I was pleasantly surprised to find that Shortform not only summarized it succinctly, but they also added intelligent counterarguments to the author’s points. In some cases, when the author made a false claim, there were Shortform notes that linked to studies that showed why the claim was false. In other cases, Shortform mentioned related concepts proposed in other books that strengthened the author’s viewpoint. I found myself looking up their guides of books I’d already read, just because I was curious to see if I had missed anything. And I had!

In short, Shortform is the only book summary reader I’ve found that goes beyond summarizing and actually adds value to the reading experience, even if you’ve read the book.

They also have a one-page summary of the book AND a summary per chapter, so information is presented at different levels of detail. In my time with it, my most frequent workflow with it has involved reading the one-page summary of a book, buying it if it was interestering enough, reading the book, and then going back to Shortform to go read the chapter summaries. And both the one-page and detailed chapter summaries, including Shortform notes, can be highlighted!

Naturally I wanted to take it further than that. I already sync my Kindle highlights to Obsidian automatically and incrementally process them in Obsidian, so I wrote a script to have my Shortform highlights going to Readwise as well.

My Shortform highlights for the book Mindset, in my Obsidian vault.

The screenshot above shows the final result: highlights I made in the Shortform web or mobile app that have been synced to a page in my Obsidian vault. Here’s how I did that.

Note: Since this article was published, Shortform and Readwise have created an integration, so the script I mention here is no longer necessary. Click here to see what I’d recommend doing today.

What you need

A Shortform account. (affiliate link)
A Readwise account. (This is an affiliate link. If you sign up with this link, you’ll get a two-month free trial instead of one, and I get a small percentage if you sign up for a plan.)
A tool that Readwise can export to, such as Obsidian, Roam Research, Notion, or Evernote. You can also use their CSV and Markdown exports.
A local installation of Python 3, which you can download here.

Obsidian is my Personal Knowledge Management tool of choice, and I use the Readwise Official community plugin to get Readwise to send highlights to Obsidian. However, using Obsidian is not necessary as my script connects only Shortform to Readwise.

The setup

Clone my Shortform-to-Readwise repository on GitHub or download it to your computer.

Enter your tokens

The file variables.py requires two tokens: one from your Shortform account and another from your Readwise account:

# Your authorization token for Shortform
authToken = ''

# Your Readwise API token
# Retrieve this token from https://readwise.io/access_token
readwiseToken = ''

Getting your Shortform token

This is a little hacky, because Shortform doesn’t expose an API token in their web app. So you’ll need to log in to your Shortform account and open up DevTools in your browser. In Chrome, go to Options > More Tools > Developer Tools (or, on Mac, hit CMD+OPTION+I). Click on the Network panel.

You should now see Shortform on one side, and DevTools on another in the same browser window.

Then, go to your Highlights page. Doing so will populate the Network panel with a bunch of requests that look like this:

DevTools open in Chrome, showing requests from Shortform.

Click on the request with the name ?sort=date, highlighted above. That’ll open another side panel that will show Headers. Scroll down to the section marked “Request Headers”, and look for where it says Authorization: Basic followed by a bunch of characters.

Authorization headers for Shortform, in DevTools.

I’ve removed my personal authorization token from the screenshot above, but it’s a long series of characters that comes after Authorization: Basic . Go ahead and right click on it and click Copy Value. Paste it into variables.py as the value for authToken, and then _remove Basic (including the space after it) so that you’re saving only everything that comes afterwards.

Getting your Readwise token

Thankfully, this one’s a little easier. Go to this page on Readwise after logging in. Click “Get Token”, click “Copy to clipboard”, and then paste it into variables.py.

Save variables.py.

Note: Don’t share either of these tokens with anyone else, as they will then be able to access your Shortform or Readwise accounts.

Run the script

Open up your terminal, and cd into the /shortform-to-readwise repository. Then, run:

python3 getHighlights.py

It should finish in a few seconds (depending on how many Shortform highlights you have), and then you can verify that it worked by going to your Readwise library.

Shortform exports showing up in your Readwise library.

You should see books that have the Shortform- prefix. If you do, then you’re done! If you’ve got your Readwise account set up to push your highlights elsewhere (such as Obsidian), then you’re all set!

In case you’re wondering why I added the Shortform- prefix, it’s because I often read the Shortform guide, and then read the book itself. Rather than end up in a sticky situation where the Shortform guide highlights might override the actual book highlights, I titled it differently so that I get two pages for the two sets of highlights.

What about running the script regularly?

The whole point of this, at least for me, was to set it up so that I’ve got highlights going to Obsidian automatically, without needing my intervention. What I did was to use crontab on macOS to run the script every 30 minutes, but you can set up your own schedule.

Give cron full system access

Cron needs to have the right permissions to run scripts on your behalf, so you may have to go to System Preferences > Security & Privacy > Privacy > Full Disk Access on your laptop and add /usr/bin/cron to the “allowed” list.

Find the path for your python3 installation

In your terminal, run:

which python3

This will give you a path. For me, it was /Library/Frameworks/Python.framework/Versions/3.9/bin/python3. Copy this.

Open crontab

Then, from your terminal, run the command env EDITOR=nano crontab -e to open up a list of cronjobs. If you don’t have any other jobs, this will be blank.

Set a schedule

Add a line to crontab for the schedule you want to create. This line has to be in the format

* * * * * command

where the five asterisks correspond to:

minute
hour
day of the month
month
day of the week

In my case, I wanted it to run every 30 minutes, so here’s what my schedule looks like:

*/30 * * * * /Library/Frameworks/Python.framework/Versions/3.9/bin/python3 /Users/nic/python/shortform-to-readwise/gethighlights.py

Make sure you change /Users/nic/python/ to the path where you put the repository and your python3 path instead of /Library/Frameworks/Python.framework/Versions/3.9/bin/python3. Now exit out of it and save it (CTRL+X, and yes). The Python script should automatically run every 30 minutes.

That’s it!

I consume a lot of content daily, but without systems in place to capture insights from that content and process it, it all just gets forgotten eventually. That’s why I love services like Shortform and Readwise– they let me get what I’ve read/watched/heard out of my head and into Obsidian, where I can review them more critically and truly learn the important parts.

Both are paid services (which I paid for out of my own money), but I really think they’re worth it.

Note: This post contains affiliate links. Check out My Ethics Statement to see my standards for promotions.

The easiest way to sync Kindle highlights to Obsidian: Readwise to Obsidian Redux

Tue, 10 Aug 2021 19:50:40 +0200

Almost exactly six months ago, I wrote a blog post detailing my workflow for processing content. Specifically, I wrote about how I used a python script to take my highlights from Instapaper, Kindle, Twitter, and Airr and automatically put them into Obsidian.

That method worked, and you can read the original article in the link above. However, today, Readwise announced that they now have an official Obsidian community plugin, and I disabled my trusty old python script for good.

You see, although my old script worked, there were quirks– not the least of which was that making changes to it meant programming in Python. I’m not a developer, so while I made do, it was never a painless experience. I also ran into issues with time zones and invalid characters being used in titles. So I’m thrilled to have something significantly better from Readwise itself! Here’s a video from the Readwise team, or read on for instructions on how to get started.

From Readwise on YouTube

Install the Obsidian plugin.

In your Obsidian vault, go to Settings > Community plugins. You’ll need to turn Safe mode off for community plugins if you haven’t already.

Then, click Browse and search for Readwise Official.

Install it, and then enable it.

Connect your Readwise account to the Obsidian plugin

Head back to the Settings menu and scroll down to the “Plugin Options” section, and then click Readwise Official.

Click connect as highlighted in orange above, and a browser window will open, showing the Obsidian Export options. You’ll need to sign into your Readwise account if you haven’t already.

At this point, your Readwise account will have been connected to the Readwise Official plugin, but you won’t have triggered any syncs yet. If you don’t want to change any of the configuration settings, you can then skip ahead to triggering the sync on Obsidian, but let’s run through the options anyway.

Modifying configuration settings

On the Obsidian Exports page, you’ll be presented with a few options.

Group Files in Category Folders

By default, this will be enabled. When enabled, Readwise will put your highlights in different folders depending on its source. For example, Kindle highlights will appear in the folder Books. Disabling this option means all new notes will be created in the directory specified in the Obsidian plugin settings, which we’ll cover later. I left this enabled in case I read a book that has the same name as a blog post about the book, for example.

Use Custom Formatting

Custom formatting is disabled by default, and lets you choose how the final note looks. Here are the things you can change in this section

Page title
Page metadata
Highlights header
Highlight
YAML front matter
Sync notification

I did make some changes in this section just to tweak how I want information to be displayed.

Page metadata

I don’t see the need for Page metadata since this is what I use YAML front matter for, so I removed the matadata section and the document tags (I never use those either). So my Page metadata section looks like this:

{% if image_url -%}
![rw-book-cover]({{image_url}})
{% endif -%}
{% if url -%}
URL: {{url}}
{% endif -%}

Highlight

By default, highlights are entered in bullet-point form. That’s fine when I’m taking notes, but when I’m highlighting someone else’s words, I like to be very careful to use blockquotes. I frequently do block searches in Obsidian, and I do NOT want to accidentally pass off a quote as something I said.

To this end, I added a > before each highlight and two newlines (carriage returns) at the end to delineate each highlight. That way, I can quote just one highlight at a time.

> {{ highlight_text }}{% if highlight_location and highlight_location_url %} ([{{highlight_location}}]({{highlight_location_url}})){% elif highlight_location %} ({{highlight_location}}){% endif %}{% if highlight_note %}
- Note: {{ highlight_note }}{% endif %}

As you can see above, I also removed highlight tags, which I never use.

YAML front matter

Metadata has to be quite specific, so I added parameters I already use for other notes in this section. The TVZ is a tag I use that means “To Verzettelen”, meaning pages that I need to process further.

Title: {{full_title}}
Author: {{author}}
Tags: TVZ, readwise, {{category}}
date: {{date}}

Sync notification

As someone who has worked on a syncing plugin for Readwise to Obsidian, I know that things go wrong sometimes. Enabling this outputs a synchronization log to a note, which has come in handy MANY times in the past. Has that book been synced, or did something go wrong? So I copied what the Readwise team suggested:

- [[{{date|date('Y-m-d')}}]] {{time}} — Synced {{num_highlights}} highlight{{num_highlights|pluralize}} from {{num_books}} document{{num_books|pluralize}}.
{% for book in books %} - {{ book.num_highlights_added}} highlights from {{ book.title }}
{%endfor %}

Select Items to be Exported

In this section, you can choose which books/podcasts/other content in Readwise you want to export. It is disabled by default. If you haven’t synced Readwise to Obsidian before, and you just want to pull in everything, leave it disabled.

In my case, I have already synchronized everything using the plugin I mentioned, and didn’t want to create another few hundred duplicate notes in my vault. So I enabled it and selected a few more recent books.

A weird thing about this Export settings page is that there is no saving– it just automatically saves it. Just a note, because I was scrolling up and down that page looking for an indication that it had been saved before I realized it is done automatically.

Initiate Sync from Obsidian

Head back to Obsidian and to the Readwise Official plugin options page. You should see a “Initiate Sync” button where the “Connect” one was before.

That’s a good sign - that means your Readwise account has been connected, and your configuration settings from the previous step will be fetched.

Before you click the button, note that the Customize base folder option is set to Readwise. This is the directory that all your Readwise notes will be created. So if you enabled Group Files in Category Folders earlier, your Kindle highlights will be saved in Readwise/Books/booktitle.md. You can change the base folder if you prefer, but I just kept the default.

Time to click the button!

You’ll see a few messages popping up on the upper right corner of your screen, and you’ll also see a Synced! message appear next to the button you just clicked.

Verifying results

In your Obsidian vault, search for one of the items you synced. Here’s what my note for Pat Flynn’s Superfans looks like:

Great! It looks exactly how I want it to look.

Set up automatic sync

If you leave it there, the plugin will only sync when you tell it to, so you’d need to go into the plugin options each time. That’s not particularly useful for me, because I want it to just happen.

If you feel similarly, head back into Plugin Options one more time and select your desired frequency in the Configure resync frequency dropdown menu. I chose 1 hour, the minimum. I also enabled the automatic sync when Obsidian opens.

Conclusion

I love this new plugin! It was very much needed, because existing options for accomplishing the same thing were flawed in some way, mainly because they were community-driven– meaning the people who created these workarounds (myself included) were “just” users who had other jobs. I’m grateful that the Readwise team has acknowledged the popularity of Obsidian and taken on the maintenance burden. I’ve experimented with the plugin quite a bit and haven’t found any major issues so far. It is a seamless way to get the best of the content I’ve consumed from various sources into one single source of truth: Obsidian. I highly recommend it to anyone looking to streamline content processing and creation processes.

Note: This post contains an affiliate link for Readwise. Clicking on that link will give you a two-month trial of Readwise, double the usual amount, and they give me a small percentage if you sign up for a plan. Check out My Ethics Statement to see my standards for promotions.

Downloading files uploaded to Roam Research for use with Obsidian

Wed, 02 Jun 2021 00:27:59 +0200

If you’ve recently transitioned from Roam Research to Obsidian, you might run into a few issues, even after you’ve run the “Markdown format importer” core plugin. One of the most annoying ones that I ran into was that files uploaded to Roam (images, pdfs, etc.) are automatically uploaded to Google Firebase.

Roam assets are linked to in the Markdown files like this:

![](https://firebasestorage.googleapis.com/v0/b/firescript-471a1.appspot.com/o/imgs%2Fapp%2FMyNotes%2FxRFXx7l-ZK.png?alt=media&token=1e03b680-5472-33dc-9b06-a127301af37f)

Now, since Roam relies on security by obscurity on this front, the links to your assets will still be accessible in your Obsidian vault, and they’ll still render correctly. However, I still think it’s worth downloading the files for three reasons:

If you don’t intend to use Roam anymore and cancel your subscription, they will likely delete those assets at some point, leaving you with a lot of broken links in Obsidian.
Part of Obsidian’s appeal is that you own your assets. Not so if they’re on someone else’s server.
You can’t access Firebase files without an internet connection.

So, instead, you want to download the contents of that link and replace it in your notes with something like this:

![](assets/newfile.png)

That way, you’ll have full control over all your assets, and you won’t be dependent on those assets being stored on Firebase by Roam.

To fix this, I wrote a Python script that will:

Go through every file and every folder within a specified root vault directory
Look for links to Google Firebase
Download all assets from Firebase to a local folder (/assets by default), using the timestamp and index number as a filename
Replace the Firebase links in your vault with links to the newly downloaded files, accounting for PDFs, whose links are formatted differently in Roam as {{ pdf: ... }}

Before you run this script, I highly recommend backing up your vault or trying this out on a dummy vault.

After downloading and saving the script, run python3 downloadfirebase.py on your terminal.

This worked really well for me, and I can breathe a little more easily now that my vault is self-contained.

Readwise to Obsidian

Sat, 06 Feb 2021 09:55:36 +0100

August 10th, 2021: Readwise just released their own Obsidian plugin, Readwise Official, which is available now! I now recommend this method over the one outlined in this article.

I’m about 9 months into heavily using Roam Research as my primary note-taking, task management, capture-everything system, and for the most part, that’s gone very well. Obsidian is another excellent personal knowledge management (PKM) system, though, and I’ve been curious about it for almost as long as I’ve been using Roam. I’ve had it installed for a while, but I never really seriously attempted to make the switch. Until now. I’ve decided to use Obsidian as my primary PKM and zettelkasten for the next month, to see if it’s something I can switch to.

Why I’m considering switching from Roam to Obsidian is another post, probably one I’ll write at the end of this experiment, when my thoughts and opinions are more solidified. For now, I want to address one of the things I struggled with when switching over: Readwise integration.

Readwise is an app that’s all about importing notes and highlights you’ve made from all over the internet and exporting them all into a single format. Readwise supports far more import sources than I use, but here are the ones I’ve got hooked up to it:

Instapaper (for highlights and notes on online articles)
Amazon Kindle (for highlights on ebooks)
Twitter (for interesting tweets)
(Readwise) Web highlighter (for straight highlights off any webpage)
Airr (for quotes and notes from audio podcasts)

Readwise collates my activity in all of these input sources and automatically sends them to my Roam Research account. Here’s what that looks like when it gets pulled into my Daily Note:

Readwise also pulls metadata into Roam, depending on the source. Here’s what the page for the book “Ready Player Two” looks like in Roam:

Readwise integration is an essential feature for me in any PKM, because I can take notes wherever I’m consuming my content and just have it automagically appear in my notes, tagged and ready for processing. So it’s absolutely something I need to have in my Obsidian workflow.

That’s where things get tricky, because Readwise doesn’t have an Obsidian integration.

Source: Readwise Exports

The closest thing is the Markdown export, but that’s not automatic. You can manually trigger an export, and then copy those exports over into your Obsidian vault, and thne make sure they’re in the right format with the right tags. I knew I was never going to do that, so I found a way to automate it.

readwise2directory

I discovered that Readwise has an API, and briefly thought about building my own integration. Happily, I found this Python project on GitHub by nicrivard, who had already done that! Why rebuild the wheel? His solution already did what I needed:

It connects to Readwise via your Readwise access token
Checks for new highlights or updated ones
Exports the new/updated highlights to a local folder of your choosing (presumably your Obsidian vault)

It even pulls in highlight tags from Readwise, if you’ve created any (which I haven’t).

I forked the project here to add just a few minor touches:

Changed date format of imported notes
Added a section to output the import date to a comment block (an Obsidian Insider build feature) rather than YAML frontmatter, so that the date would actually be linked and backlinked.
Customized tags and filename for my use
Sort imports into folders within my Obsidian vault depending on source (Article/, Tweet/, and so on).

This works out pretty well. Here’s an example of an import from the podcast app Airr into Obsidian, in Edit mode.

It’s placed within my Podcast folder inside my value, is tagged appropriately, and contains links to the source. Switching to Preview mode shows an imported cover photo and even a playable recorder for the part of the podcast that I snipped on Airr:

So that works out well, except for one thing: firing off the script that does all this is still manual.

Scheduling the script

Luckily, it’s pretty easy to schedule a Python script to run at certain intervals. I’m on macOS, so I used a built-in utility called cron.

Give cron full system access

Cron needs to have the right permissions to run scripts on my behalf, so I had to go to System Preferences > Security & Privacy > Privacy > Full Disk Access on my laptop and add /usr/bin/cron to the “allowed” list.

Open crontab

Then, from my terminal, I ran the command env EDITOR=nano crontab -e to open up the list of cronjobs I had currently.

Set a schedule

I added a line to crontab for the schedule I wanted to create. This line has to be in the format

* * * * * command

where the five asterisks correspond to:

minute
hour
day of the month
month
day of the week

In my case, I wanted it to run every hour, so here’s what my schedule looks like:


0 * * * * cd /Users/nic/git/readwise2directory/ && /Library/Frameworks/Python.framework/Versions/3.9/bin/python3 /Users/nic/git/readwise2directory/readwise-GET.py >> /Users/nic/git/readwise2directory/readwiseGET.log

Python issues

I had some trouble getting cron to work until I realized it was because I had multiple versions of Python installed. I ran a which python3 to find the path to the version I wanted to use and added that path to the command. I also had to cd to the project directory, for some reason.

While I was troubleshooting, I found it really useful to output to a log so I could see what had gone wrong.

Once I added the schedule as in the example above, though, it worked a treat!

Conclusion

This solution did require some setup at the beginning, but doesn’t need any maintenance once it’s going. I’m pretty happy with it as it means I can keep taking notes on various places from anywhere, and it’ll all be waiting for me in my Obsidian vault whenever I’m ready to sit down and process them into my system. Thanks to nicrivard for the repo that made it possible!

Taking Notes on a Video: my YiNote to Roam Research Workflow

Tue, 15 Dec 2020 22:00:34 +0100

Listen to the audio version of this blog post here:

I was working on an article yesterday comparing the performance of k6 with JMeter, and I had a chat to Mihail, one of the devs of k6. He mentioned that a large part of why k6 is so performant compared to other tools (particularly Java-based ones) is that it’s written in Go. He talked a bit about Goroutines, and when I told him I was a Go noob, he recommended this very excellent video by Kavya Joshi, which explains a bit more about the Go Scheduler and how it manages to achieve performance for Go apps that exceeds the typical 1 Virtual User: 1 kernel thread paradigm that many load testing tools (like JMeter) use.

Anyway, I got around to watching the video today, and because it was a video I knew I’d learn from, I took notes - and ended up improving part of the workflow (more on that later). Other people have commented on my notes before, so I thought I’d share how I process videos into my personal wiki, currently held in Roam Research.

Install YiNote Chrome extension

I use a free app called YiNote to take video notes in particular. I use their Chrome extension.

Take Notes

I was pretty dubious about using an app just to take notes for video specifically, but there were a few things that sold me:

When you click on the text field to take notes in the extension, YiNote automatically pauses the video so that you can type without missing what’s being said.
YiNote also automatically saves the timestamps associated with notes you create, so that you can click on it afterwards to jump to the specific section that you wrote a note on.
YiNote has an export to Markdown option - great compatibility with Roam or Obsidian
You can also export your notes to a PDF, which is great for sharing. It saves screenshots of the presentation and puts your notes next to the relevant screenshot.

Here’s what it looks like while taking notes:

Export YiNote notes

After watching the video, I click the Settings wheel from the Chrome extension panel.

I click the Export as Markdown button.

That lets me save a .md file to my local drive. I always save YiNote files in the same folder as the script below.

Run reformatter script

This is the part that I tweaked today. YiNote’s exported Markdown file is formatted in a way that doesn’t really play well with Roam. Here’s what it looks like, fresh from YiNote:

<!-- Generated by <a href="https://www.yinote.co/#installation">YiNote</a> -->

<!-- # [GopherCon 2018: Kavya Joshi - The Scheduler Saga](https://www.youtube.com/watch?v=YHRO5WQGh0k) -->

<!-- ## [1:19](https://www.youtube.com/watch?v=YHRO5WQGh0k&t=79) -->

The scheduler is what makes sure the Goroutines run concurrently.

<!-- ## [1:29](https://www.youtube.com/watch?v=YHRO5WQGh0k&t=89) -->

All the information is there, but importing that to Roam, with the comments, isn’t really useful. I want to see those timestamps as clickable links, and I want my notes to be nested underneath the corresponding timestamp so I know what belongs where.

So I dusted off my Python skills tonight and wrote up this little script to do just that. Warning: I am not a developer; I just occasionally code. So I’m sure that this isn’t the most concise way to do it (contact me and let me know how it could have been better if you have any ideas). But it’s functional. And, in the spirit of learning in public, here it is:

The script results in an output file that is better formatted for Roam.

Import resulting file to Roam

On Roam, I hit Options > Import Files, select the output file, hit OK, and that creates a new page in a way better format than the one directly from YiNote. After adding the date, author, and “Recommended By” fields, this is what it looks like in Roam:

That’s it!

I don’t do this for every video I watch, but I find this process useful for more technical videos that I really want to learn from. It’s really important to me to get all my notes from disparate sources into Roam, so that they then get processed like any other note into my Zettelkasten - but that’s a process for another day.

How I use Roam Research for D&D: Session Notes

Mon, 14 Dec 2020 20:13:13 +0100

In case you thought this site was all about performance and load testing, surprise! It’s not.

If you’ve talked to me personally for longer than ten minutes in the last year, there’s a pretty good chance I’ve mentioned Roam Research, or the note-taking and learning methodology called Zettelkasten. Sorrynotsorry. I’ve been obsessed since I discovered it.

To say Roam Research is a note-taking device is like saying Bitcoin is currency: yes, it IS that, but also so much more. To me, it’s my second brain. It’s a record of what I know and what I want to know, and sometimes even things I don’t know yet. The real reason that Roam and the Zettelkasten idea resonated with me so much is that I can use it a lot for my work. But I’ve discovered another use case that Roam is perfect for: Dungeons & Dragons.

This became longer than I thought it would be, so I’ll just talk about how I handle Session Notes in particular.

Starting a new session

Everything starts with sitting down to play D&D. I write a log of events and information that comes out while I’m playing. Here’s what a new session log looks like:

I preface pages in this campaign with D&D/ to make it a separate namespace. A namespace is like a folder, and I use it mainly because I have two D&D campaigns, both of which would have the page Campaign, Characters, Locations, etc, even though the two campaigns are on different worlds. To keep them separate, and to keep from getting mixed up (Was Gloomtoes an NPC in this campaign or that one?) I use different namespaces for each campaign.

I add the real-world date in the page name, and then an in-game date reference. The in-game date reference is useful for keeping track of time since I might go a couple weeks (okay, maybe one week; let’s not get too crazy) without playing this campaign, but no time in the in-game world has elapsed. Then, I end the page name with a descriptive title, which I always end up changing after the session. Sometimes I’ll title a page Exploring Castle Never, but then we end up getting carried away in another part of the city instead. That’s okay - when I change a page name, Roam then updates all references to that page that I’ve made from other pages.

I don’t type all this out manually - I use a TextExpander template so that to start a new session, all I have to do is create a page in Roam, type ;s.dnd and that is automatically replaced with prompts for a few fields.

Date: This is the real-world date of when I played this particular session.
Tags: When I first create session notes, this is prepopulated with #TVZ, which stands for To Verzettelen. It’s just my personal way of marking a note as one that needs to be processed.
Location: Here I note down the in-game places that my character visited this session. Usually I’ll just put city names, but I sometimes also split locations up into districts within a city or rooms within a dungeon.
Characters: I’ve found it helpful to keep track of which characters (players and NPCs) I’ve come across in this session.
Session Summary: This is a section I leave blank when the session is initially created.

Filling in the session log

The rest of the page, after Session Summary, is where I’ll spend most of the time writing. I just write whatever comes to mind and whatever I have time to write, in bullet form. I don’t worry about formatting or grammar or linking to other pages. I just type all my thoughts out in plain text.

HP

I like to play healers, which means I care a lot about how everyone in the party is doing so that I can figure out who best needs healing. {{[[table]]}} in Roam lets me create a custom table, and I have a ;hp snippet in TextExpander that recreates this table, with all the names of the people currently in my party, so that I just worry about filling in the values. Sometimes I still get it wrong, or miss a bit of damage that the DM calls out, but it’s enough to give me an idea of who needs heals.

I also write down initiative orders, the defenses/vulnerabilities/immunities of monsters we face, descriptions of NPCs we meet, suspicions or questions I might have… basically anything and everything that comes to mind. Session time is pretty much a brain dump, with no formatting.

Equipment

If I acquire a new item during a session, I’ll tag that line with #[[D&D/Carrying]]. This lets me stay on the Session Notes page while still maintaining an up-to-date record of what I’m carrying.

A tag in Roam is the same as a page, so when I click on that link, it takes me to the D&D/Carrying page, which conveniently also shows me all the references to that page from other pages.

Roam shows me not just the page, but the exact block where I made the reference to D&D/Carrying, so it’s easy to see at a glance what I have and where I got it.

Next Actions and Waiting For

I use David Allen’s GTD methodology in D&D too. When I want to remember to do something later, I tag it with #[[D&D/Agenda]] and #[[Next Actions]] or #[[Waiting For]]. This presents me with the same opportunities as D&D/Carrying, where I can see references to those pages later as well.

Processing the session

Sometime after the session, I’ll go and fill in the metadata from my template, including the session summary, which is just a paragraph or two describing what happened in the session at a high level. This is mainly to jog my memory in the next session, when I’ve already forgotten what we did in the previous one.

Here’s what it looks like, all filled in:

Adding links and fleshing out pages

I then go back through the notes themselves and add turn any people, places, or interesting objects that I mention into links. In Roam, that’s a matter of highlighting them and hitting [[. Then, I click on the page I’ve just created and try to add information that I’ve learned. This isn’t strictly necessary, since all mentions of that page will show up as a linked reference, but it’s an opportunity to consolidate my information on that subject and see how it fits into what else I’ve written there.

Here’s a page for an NPC:

Having these links means that when an NPC says “I’m from Leilon”, I can type Leilon into Roam:

… and say, “Oh, Leilon. I heard you had a dragon problem there recently. Did your family get out okay?” Boom! Instant advantage on a Diplomacy check.

Sometimes we’re in a dungeon with many rooms, and this is the point where I draw a quick map on my iPad Pro and Apple Pencil using Notability, and then I upload it right from my iPad to Roam.

This map is in my page for Castle Never:

The little blue “8” on the upper left of the map is versions. Information rarely comes all at once in D&D; it usually comes in trickles. As I learned more about the castle, I revised the map and added a new version to it, so that I can still see previous versions and what I thought the map was like when we first explored it.

At this stage, I also add any tags I forgot to add during the session.

Reviewing quests and questions

After I go through the session notes, I go to the page called D&D/Quests and Questions, where I keep a running list of all the open quest hooks and any unanswered questions I might have.

Not everything on this page is meant to be resolved, but I find it comforting to track everything anyway, in case I happen to find myself in a city with nothing in particular to do. I mark these with a checkbox ({{[[TODO]]}}) in Roam) and tick the box when one is resolved. Roam lets me filter out the resolved quests so I only see things that are outstanding.

This is where I get the most value out of taking notes. It’s at this point where I’ll add our progress towards quests, make connections between two seemingly disconnected quests, and situate what I’ve just learned in the session within the broader context of the world and the campaign. I also review and update items on the Next Actions and Waiting For.

This is the part that takes the longest, but it’s also what most prepares me for when I get to the table again next week.

Tying it all together

I use a D&D/Campaign page as my landing page for this campaign. It includes D&D/Quests and Questions as well as a list of all sessions to date, so that I always know what happened and when.

Why do I take notes for a game?

You might be wondering why I go to so much trouble. Why do I take complicated notes about something I do “just” for fun?

Taking notes helps me offload this all safely from my brain.
I take my fun as seriously as I take my work.