Python on Nicole van der Hoeven

Downloading files uploaded to Roam Research for use with Obsidian

Wed, 02 Jun 2021 00:27:59 +0200

If you’ve recently transitioned from Roam Research to Obsidian, you might run into a few issues, even after you’ve run the “Markdown format importer” core plugin. One of the most annoying ones that I ran into was that files uploaded to Roam (images, pdfs, etc.) are automatically uploaded to Google Firebase.

Roam assets are linked to in the Markdown files like this:

![](https://firebasestorage.googleapis.com/v0/b/firescript-471a1.appspot.com/o/imgs%2Fapp%2FMyNotes%2FxRFXx7l-ZK.png?alt=media&token=1e03b680-5472-33dc-9b06-a127301af37f)

Now, since Roam relies on security by obscurity on this front, the links to your assets will still be accessible in your Obsidian vault, and they’ll still render correctly. However, I still think it’s worth downloading the files for three reasons:

If you don’t intend to use Roam anymore and cancel your subscription, they will likely delete those assets at some point, leaving you with a lot of broken links in Obsidian.
Part of Obsidian’s appeal is that you own your assets. Not so if they’re on someone else’s server.
You can’t access Firebase files without an internet connection.

So, instead, you want to download the contents of that link and replace it in your notes with something like this:

![](assets/newfile.png)

That way, you’ll have full control over all your assets, and you won’t be dependent on those assets being stored on Firebase by Roam.

To fix this, I wrote a Python script that will:

Go through every file and every folder within a specified root vault directory
Look for links to Google Firebase
Download all assets from Firebase to a local folder (/assets by default), using the timestamp and index number as a filename
Replace the Firebase links in your vault with links to the newly downloaded files, accounting for PDFs, whose links are formatted differently in Roam as {{ pdf: ... }}

Before you run this script, I highly recommend backing up your vault or trying this out on a dummy vault.

After downloading and saving the script, run python3 downloadfirebase.py on your terminal.

This worked really well for me, and I can breathe a little more easily now that my vault is self-contained.

Readwise to Obsidian

Sat, 06 Feb 2021 09:55:36 +0100

August 10th, 2021: Readwise just released their own Obsidian plugin, Readwise Official, which is available now! I now recommend this method over the one outlined in this article.

I’m about 9 months into heavily using Roam Research as my primary note-taking, task management, capture-everything system, and for the most part, that’s gone very well. Obsidian is another excellent personal knowledge management (PKM) system, though, and I’ve been curious about it for almost as long as I’ve been using Roam. I’ve had it installed for a while, but I never really seriously attempted to make the switch. Until now. I’ve decided to use Obsidian as my primary PKM and zettelkasten for the next month, to see if it’s something I can switch to.

Why I’m considering switching from Roam to Obsidian is another post, probably one I’ll write at the end of this experiment, when my thoughts and opinions are more solidified. For now, I want to address one of the things I struggled with when switching over: Readwise integration.

Readwise is an app that’s all about importing notes and highlights you’ve made from all over the internet and exporting them all into a single format. Readwise supports far more import sources than I use, but here are the ones I’ve got hooked up to it:

Instapaper (for highlights and notes on online articles)
Amazon Kindle (for highlights on ebooks)
Twitter (for interesting tweets)
(Readwise) Web highlighter (for straight highlights off any webpage)
Airr (for quotes and notes from audio podcasts)

Readwise collates my activity in all of these input sources and automatically sends them to my Roam Research account. Here’s what that looks like when it gets pulled into my Daily Note:

Readwise also pulls metadata into Roam, depending on the source. Here’s what the page for the book “Ready Player Two” looks like in Roam:

Readwise integration is an essential feature for me in any PKM, because I can take notes wherever I’m consuming my content and just have it automagically appear in my notes, tagged and ready for processing. So it’s absolutely something I need to have in my Obsidian workflow.

That’s where things get tricky, because Readwise doesn’t have an Obsidian integration.

Source: Readwise Exports

The closest thing is the Markdown export, but that’s not automatic. You can manually trigger an export, and then copy those exports over into your Obsidian vault, and thne make sure they’re in the right format with the right tags. I knew I was never going to do that, so I found a way to automate it.

readwise2directory

I discovered that Readwise has an API, and briefly thought about building my own integration. Happily, I found this Python project on GitHub by nicrivard, who had already done that! Why rebuild the wheel? His solution already did what I needed:

It connects to Readwise via your Readwise access token
Checks for new highlights or updated ones
Exports the new/updated highlights to a local folder of your choosing (presumably your Obsidian vault)

It even pulls in highlight tags from Readwise, if you’ve created any (which I haven’t).

I forked the project here to add just a few minor touches:

Changed date format of imported notes
Added a section to output the import date to a comment block (an Obsidian Insider build feature) rather than YAML frontmatter, so that the date would actually be linked and backlinked.
Customized tags and filename for my use
Sort imports into folders within my Obsidian vault depending on source (Article/, Tweet/, and so on).

This works out pretty well. Here’s an example of an import from the podcast app Airr into Obsidian, in Edit mode.

It’s placed within my Podcast folder inside my value, is tagged appropriately, and contains links to the source. Switching to Preview mode shows an imported cover photo and even a playable recorder for the part of the podcast that I snipped on Airr:

So that works out well, except for one thing: firing off the script that does all this is still manual.

Scheduling the script

Luckily, it’s pretty easy to schedule a Python script to run at certain intervals. I’m on macOS, so I used a built-in utility called cron.

Give cron full system access

Cron needs to have the right permissions to run scripts on my behalf, so I had to go to System Preferences > Security & Privacy > Privacy > Full Disk Access on my laptop and add /usr/bin/cron to the “allowed” list.

Open crontab

Then, from my terminal, I ran the command env EDITOR=nano crontab -e to open up the list of cronjobs I had currently.

Set a schedule

I added a line to crontab for the schedule I wanted to create. This line has to be in the format

* * * * * command

where the five asterisks correspond to:

minute
hour
day of the month
month
day of the week

In my case, I wanted it to run every hour, so here’s what my schedule looks like:


0 * * * * cd /Users/nic/git/readwise2directory/ && /Library/Frameworks/Python.framework/Versions/3.9/bin/python3 /Users/nic/git/readwise2directory/readwise-GET.py >> /Users/nic/git/readwise2directory/readwiseGET.log

Python issues

I had some trouble getting cron to work until I realized it was because I had multiple versions of Python installed. I ran a which python3 to find the path to the version I wanted to use and added that path to the command. I also had to cd to the project directory, for some reason.

While I was troubleshooting, I found it really useful to output to a log so I could see what had gone wrong.

Once I added the schedule as in the example above, though, it worked a treat!

Conclusion

This solution did require some setup at the beginning, but doesn’t need any maintenance once it’s going. I’m pretty happy with it as it means I can keep taking notes on various places from anywhere, and it’ll all be waiting for me in my Obsidian vault whenever I’m ready to sit down and process them into my system. Thanks to nicrivard for the repo that made it possible!

Note: This post contains an affiliate link for Readwise. Clicking on that link will give you a two-month trial of Readwise, double the usual amount, and they give me a small percentage if you sign up for a plan. Check out My Ethics Statement to see my standards for promotions.

Python for Everybody Specialization

Tue, 26 Jan 2021 23:01:18 +0100

I’m starting off the new year by learning something new. I am a big fan of non-traditional education, and Coursera is a platform I’ve used a few times before. You may not get courses that qualify as university credit, but I’ve found it to be a useful resource for learning. I might be biased (my university degree is in Economics, which I’ve never used), but in some ways, I actually prefer a MOOC (that’s a Massively Open Online Course) to a typical formal university course for a few reasons:

MOOCs are entirely online, which means it’s more accessible - I don’t need to physically be in the US to take a course at Harvard
MOOCs are usually self-paced - perfect for someone working full-time like me
MOOCs are free or cheap - no huge student loans here; MOOCs are democratizing education and making it available to more people
Good online education tends to be more practical. There is none of the intellectual posturing that I’ve seen in physical classrooms; there are no random classes you need to take to get a fancy piece of paper. I love being able to choose a topic and learn ONLY that– and get right down to business in the first lesson.
Formal education is often centered around getting students jobs, not helping workers do their jobs well. I already have a job, and I want to learn real, usable skills, not “marketable” ones.

My main complaint with MOOCs was that they tend to be short (in comparison to a formal course), but Coursera has addressed that somewhat with their specializations. A specialization is a series of courses, but it’s more than just disparate courses that are strung together; it’s a thoughtfully designed curriculum that builds up on knowledge from previous courses. Last year, I began learning Python.

Why Python? Others might have different reasons, but here were mine:

Python is one of the best languages to learn for data analysis (the other being R). I’ve always been interested in data visualization, and as a load tester, I’m frequently exposed to large quantities of data that I somehow need to make sense of.
Python is pretty good at retrieving data from websites, and I wanted to explore how good it is to use in load testing. I’m always eager to learn different ways to test, and I’ve wanted to try Locust for a while.
Python is increasing in popularity, and I wanted to see why.

So I decided to take the Python for Everybody Specialization from the University of Michigan (via Coursera) to get an introduction to this cool language. The specialization consists of five modules, each of which I had to take separately:

Programming for Everybody (Getting Started with Python)
Python Data Structure
Using Python to Access Web Data
Using Databases with Python
Capstone: Retrieving, Processing, and Visualizing Data with Python

The specialization was really fun, and the teacher, Charles Severance, is fantastic. I’d taken courses from him before, so I knew what to expect. He didn’t disappoint. He has a way of explaining daunting technical concepts in a way that’s easy to digest for beginners.

You can check out my certificate here. I’m pretty happy with my progress. I’m by no means a Python expert now, but I’m definitely eager to keep learning how to do more things with this new knowledge.