List of Excellent Tiny Desk Concerts

Published: 2020-08-12 1 minute read

Kishi Bashi
Max Richter
St. Paul and the Broken Bones
Hobo Johnson and the Lovemakers
Moon Hooch
Florence and the Machine
Sudan Archives
Ages and Ages
Alicia Keys
Ólafur Arnalds
Mother Falcon

Lessons learned while adopting Snakemake

Published: 2020-01-12 4 minute read

Recently, I decided to adopt the Snakemake workflow system, as I had started to hit some pain points in my project that it reportedly solves. While I believe that this was ultimately a great decision in the long term, it did take me a week to fully move things over when I had expected it to just take a day. Much of this work was adapting my code to the framework and was really only necessary because I was not adhering to the Unix philosophy in the first place, which is entirely my fault, but Snakemake fortunately incentivizes this. That said, it was very frustrating at points, which I think is a combination of the inherent complexity of the problem it's solving, some documentation issues, and unhelpful error messages. I decided to describe these problems and how I solved them.

Input referenced a variable that wasn't referenced by the output

This was the most conceptually surprising part of the framework for me. I had assumed that if you had a chain of rules, that one rule's input could reference the output of the previous rule using a pattern. However, this is only true if any variables that the input section has are also used in the output! Even when directly using rules.some_rule.output as the input, Snakemake couldn't figure out what I meant, since it had a variable {run} that I wasn't using in the output filename. The solution was to refactor how I was building up the file paths, so that the unique thing that identifies each dataset is the only string represented by the variable.

A pseudo-rule that lists every file you want to make is required

One of the more confusing aspects of Snakemake to me (which effectively explains the previous issue) is that while it can use globbing to infer all the inputs to a rule, it can only do that because the outputs are explicitly defined my subsequent rules, with an all-encompassing pseudo-rule (usually called rule all) at the very end. My mental framework going in was working in the opposite direction and assumed it was just inferring what work was necessary, but this isn't the case. I feel like this should be a solvable problem but I have a gut feeling it runs into some sort of halting problem issues.

An input is not necessary

Virtually every example rule in the documentation has an input, but this isn't strictly necessary. At the beginning of my pipeline I take sequencing data and merge the paired end reads, and while the script that's run does take actual inputs, there was no reason to reference them.

Referencing a directory that contains outputs as an input will cause Snakemake to rerun the pipeline unnecessarily

I had mistakenly referenced an output directory as an input to a rule when it was completely unnecessary (it was actually the input mentioned in the previous section that wasn't necessary at all). Because files were being written to that directory, the last-modified timestamp of the directory was being updated, so it always appeared to Snakemake that the earlier rule had a fresher input than the rest of the pipeline. Unfortunately, this was also my very first rule, so the entire pipeline was running from scratch even after it had completed, which took several hours. Although this is not as likely to happen to other people, it's definitely what motivated this post.

Script sections referenced variables that are surprisingly out of scope

I had rules with a script: section, and I tried to reference a variable called {run} that was used in the input and output sections. The solutions is simply to use {}, but the error message doesn't make this clear. This is in the documentation so I was fundamentally the problem there.

Not escaping Python formatting properly

When using .format() to build up strings in my Snakefile, I didn't realize that you had to escape the Snakemake variables with double curly braces. This has nothing to do with Snakemake, I was just not too familiar with how the built-in format() function worked. However, the error message didn't make it clear that this was a Python syntax issue and not a Snakemake issue.


I feel that there's probably a better way to present the logic of Snakemake to new users such that they won't encounter these errors, and in some cases the error messaging might be able to be improved. Overall, however, this is so much better than GNU Make for a host of reasons, and is orders of magnitude better than not having any build system at all. Once I get enough experience with it I might write a tutorial or propose changes to the official documentation.

DS9 Viewing Guide

Published: 2017-08-26 13 minute read

I loved Star Trek: Deep Space Nine, but it really takes a long time for it to come into its own, and there are a lot of terrible episodes, especially in the first few seasons. However, since DS9 has a long-term plot and since there are a lot of great episodes in every season, I wrote this guide so friends could know which episodes they could skip and which ones they really do need to see.

Bold and underlined episodes are critical to the long-term plot or character development
Italic episodes are good and worth watching but do not affect the long-term plot
Red episodes are terrible
All other (unlisted) episodes are neither good nor bad, watch them if you're just really into the show.

World building episodes from Star Trek: The Next Generation

These episodes set the stage for DS9.

S4E12 - The WoundedIntroduces the Cardassians, one of the main antagonists in DS9. Also features O’Brien and Worf, who become main characters in DS9. The actor that plays the Cardassian captain in this episode plays a different Cardassian in DS9, but character-wise they’re very similar.
S5E03 - Ensign Ro: Introduces the Bajorans, who basically had a multi-decade occupation/holocaust done to them by the Cardassians. Ensign Ro was supposed to appear on DS9 but the actress declined; the character Kira is essentially the same person.
S6E10/11 - Chain of Command: Allegory to 1984. One of the highest rated episodes of TNG.
S7E24 - Preemptive Strike: More good background, terrorism stuff.

Season 1

01/02 - Emissary: First episode so mandatory. Some context: the reason Sisko doesn’t like Picard is because in an episode of TNG, Picard was kidnapped by cyborgs (the Borg), turned into a cyborg, and he became their leader and went on a killing spree (which killed Sisko’s wife) before ultimately being saved and reverted to a human.
03 - Past Prologue: Introduces Garak, my favorite character.
08 - Dax: Character building for Jadzia, and they explain the whole Trill thing in detail. Not a great episode but the multi-life thing comes up a lot.
10 - Move Along Home: So awful, do not watch this even for ironic reasons
11 - The Nagus: The first Ferengi-heavy episode. Typically, these serve as comic relief when the show gets really dark, and they’re generally good episodes. Still I’m going to mark them as optional. But this one is a good one to watch just because so many of the main characters are Ferengi, and a lot of them defy their culture, so it’s good to know where they come from.
13 - Battle Lines: Nice episode about war and peace, important development with the Kai.
14 - The Storyteller: Someone on the Internet wrote an essay about why this is the worst episode of any Star Trek series. It was long. Needless to say, skip this one.
15 - Progress: Eminent domain, former rebel realizes she is now the establishment. Kira character-building.
19 - Duet: Great episode about war criminals.
20 - In the Hands of the Prophets: Allegory for teaching evolution in schools, imposing cultural values onto others, terrorism, power struggles. Two important characters are introduced.

Season 2

01-03 - The Homecoming/The Circle/The Siege: An insurgent group engages in shenanigans.
05 - Cardassians: Good Garak episode
06 - Melora
07 - Rules of Acquisition: The Ferengi go to the Gamma quadrant to do stuff. Some important foreshadowing but no game-changers.
13 - Armageddon Game: One-off about biological weapons and the power/danger of information.
14 - Whispers: A nice one-off episode, pretty science-fictiony.
16 - Shadowplay: Another nice one-off episode, but includes some more foreshadowing.
18 - Profit and Loss: Cardassian rebels, Garak and Quark.
19 - Blood Oath: There’s a really good line in this episode that I think of often
20/21 - The Maquis: Rebels, war, secret plans, etc.
(highly recommended) 22 - The Wire: Mostly about addiction, but also goes deep into Garak’s spy-related past.
24 - The Collaborator: Power struggle to become the next space pope, new information on the culprit behind a massacre that occurred during the occupation of Bajor
25 - The Tribunal: O’Brien is accused by the Cardassians of a crime he didn’t commit. Nothing is as it seems.
26 - The Jem’Hadar: One of the defining episodes of the show. The war (effectively) begins.

Season 3

01/2 - The Search: Major plot point, great action
03 - The House of Quark: Quark’s ex-wife rolls into town, shenanigans ensue
05 - Second Skin: Kira might be a secret Cardassian
06 - The Abandoned: They find a baby Jem’Hadar
(good) 07 - Civil Defense: They accidentally trigger a program intended to thwart slave uprisings from back when the station was a mining facility run by the Cardassians
09 - Defiant: Major plot point.
11/12 - Past Tense: I’m not actually that into these episodes, but here is a quote from Wikipedia about it: “as this episode was finishing production an article appeared in the Los Angeles Times describing a proposal by the mayor to create fenced-in "havens" for the city's homeless, to make downtown Los Angeles more desirable for business. The cast and crew were shocked that this was essentially the same scenario that Past Tense warned might happen in three decades, but was now being seriously proposed in the present.”
13 - Life Support: Major plot point, nature of conscious experience
14 - Heart of Stone: I don’t like the main plot of this episode but it’s relatively important, though the B-story is great.
15 - Destiny: Literal interpretations of religious texts can be problematic
16 - Prophet Motive: Decent Ferengi episode, a total one-off
17 - Visionary: Just a good episode about time travel

A few more episodes from Star Trek: The Next Generation

To better know the Klingons and Romulans, who will be playing a much larger role in the show soon, I highly recommend watching a few episodes from The Next Generation:

S2E08 - A Matter of Honor: (the most quotable episode of TNG)
S3E10 - The Defector
S4E26/S5E1 - Redemption: To set this up, Worf’s father was falsely accused of helping the Romulans commit a massacre decades ago, which the Duras family actually orchestrated. If this were to become public knowledge however, it would lead to a civil war, so after learning the truth, Worf agreed to publicly accept that his father was to blame in order to save the Empire. He later stabbed the head of the Duras family to death, but he was the only person who could have restored Worf’s honor. So Worf is stuck in this weird limbo where all Klingons hate him and there’s nothing anyone can do.

Season 3 (continued)

20 - Improbable Cause: Major plot point, great ep
21 - The Die is Cast: Major plot point, great ep
23 - Family Business: Women’s rights
24 - Shakaar: Space pope power struggles and rebels
26 - The Adversary: War episode

Season 4

01/02 - The Way of the Warrior: New characters, major plot points
04 - Hippocratic Oath: Philosophy of giving medical care to the enemy during a war
05 - Indiscretion: They search for a missing ship
07 - Starship Down: Good ep
(highly recommended) 08 - Little Green Men: They go back in time, fun one-off episode
10 - Our Man Bashir: Holodeck shenanigans
11/12 - Homefront/Paradise Lost: The fragility of democracy during a time of war and suspicion
13 - Crossfire: Seriously I hate this entire subplot but it matters to the show
14 - Return to Grace: A little slow but important developments occur
15 - Sons of Mogh: Morality of assisted suicide
16 - Bar Association: Fun Ferengi episode about labor rights and unions
17 - Accession: Faith, caste systems, etc. I don’t care for this kinda stuff
(recommended) 18 - Rules of Engagement : Worf commits a war crime but you aren’t supposed to do those
(recommended) 19 - Hard Time: Very science-fictiony story about a novel way of punishing prisoners
22 - For the Cause: Sisko’s girlfriend might be fooling around on him
23 - To the Death: Renegade Jem’Hadar
(good) 24 - The Quickening : Bashir helps with a plague, allegory for the third world in some respects
(good) 25 - Body Parts: Ferengi episode
26 - Broken Link: Shit gets real

Season 5

01 - Apocalypse Rising
02 - The Ship
03 - Looking for par’Mach in All the Wrong Places
(recommended) 04 - Nor the Battle to the Strong: MASH in space
(recommended) 05 - The Assignment: O’Brien can’t catch a break
(recommended) 06 - Trials and Tribble-ations: Only recommended if you have seen the original Tribbles episode from original Star Trek
(recommended) 09 - The Ascent: The Odd Couple in desolate wilderness
10 - The Rapture: Sisko has prophetic visions
11 - The Darkness and the Light: Someone is murdering Kira’s friends
12 - The Begotten: Major development
(recommended) 13 - For the Uniform: A little ham-fisted with the allegory, but Sisko goes full beast mode
14 - In Purgatory’s Shadow: One of my favorite episodes of the entire series, very important, do not read the Netflix synopsis, don’t look at the preview screen cap
15 - By Inferno’s Light: Major plot point, do not read the Netflix synopsis, don’t look at the preview screen cap
16 - Doctor Bashir, I Presume?: Not that important but a secret about Bashir is revealed that comes up repeatedly from here on out
(recommended) 18 - Business as Usual: Quark gets involved with an arms dealer
19 - Ties of Blood and Water: More war crimes deliberations, callback to S3E05 “Second Skin”
(good) 20 - Ferengi Love Songs: Good Ferengi ep, as all Ferengi eps are
(good) 21 - Soldiers of the Empire: Good Klingon episode
(good) 22 - Children of Time: Science fictiony episode
(good) 23 - Blaze of Glory: Stolen nukes
(good) 24 - Empok Nor: Spooky episode
25 - In the Cards: Important strategic things with a nice comedic relief subplot
26 - Call to Arms: Shit gets real, but for real this time

Season 6

01 - A Time to Stand
02 - Rocks and Shoals: This is the most metal of all episodes. Two of my hypothetical cats are named after characters in this episode, an honor not frequently bestowed. Try to guess who!
03 - Sons and Daughters
04 - Behind the Lines
05 - Favor the Bold
06 - Sacrifice of Angels
07 - You Are Cordially Invited…: If you watch this episode and get married and still don’t have a Klingon-themed wedding then you are history’s greatest monster
09 - Statistical Probabilities: Flowers for Algernon in space
10 - The Magnificent Ferengi: Yes, a non-optional Ferengi episode! Very war-related
11 - Waltz
(recommended) 12 - Who Mourns for Morn?
13 - Far Beyond the Stars: One of the highest-rated episodes by DS9 fans. I don’t think it’s the greatest but I do like it, and it’s pretty unusual. And you get to see the actors without prosthetics. There’s also a brief reference to it in season 7 that will be super confusing if you haven’t seen it
(recommended) 14 - One Little Ship: Just a fun one-off episode that makes a really good point about a fatal flaw in a well-known children’s movie
(recommended) 15 - Honor Among Thieves
(recommended) 16 - Change of Heart
(recommended) 17 - Wrongs Darker than Death or Night
18 - Inquisition: Major downstream plot points based on this episode
19 - In the Pale Moonlight: Major plot point, Garak goes full Garak. One of my favorite episodes. Perhaps my favorite of all.
20 - His Way: I just don’t like anything about this subplot, but it does have large implications for several characters. And they introduce a holodeck program that will be featured repeatedly. Protip: you can skip the music in every episode, nothing important ever happens during a song
21 - The Reckoning
(recommended) 22 - Valiant: Hubris, ambition, elitism
(recommended) 23 - Profit and Lace: Women’s rights
(not that great) 24 - Time’s Orphan: Another terrible thing happens to O’Brien
(okay-ish) 25 - The Sound of Her Voice: Mild science fiction
26 - Tears of the Prophets: Important war episode

Season 7

01 - Image in the Sand
02 - Shadows and Symbols
03 - Afterimage
04 - Take Me Out to the Holosuite: So poorly written and executed, everyone involved with this episode should be bricked up inside a wall
05 - Chrysalis
06 - Treachery, Faith and the Great River
07 - Once More Unto the Breach
08 - The Siege of AR-558
09 - Covenant
10 - It’s Only a Paper Moon
11 - Prodigal Daughter: A few bad things happen to O’Brien, and he’s not even the focus of the story. Held by the writers to be the worst episode of the season, presumably because no one wants to mention episode 704 out loud for fear of what happens to those who do. I thought it was fine.
14 - Chimera: Odo meets a douchebag Changeling
(good) 15 - Badda-Bing, Badda-Bang: Oceans Eleven in space
16-26: The last 10 episodes are effectively one continuous plotline, and they’re all great.

Fission yeast don't age

Published: 2017-01-31 1 minute read

My colleagues and I just published our work on fission yeast aging in eLIFE. Although we had hoped to develop a more human-like single-celled model organism to study eukaryotic aging, we came to the surprising conclusion that fission yeast simply don't age. That is, the chance of a fission yeast dying is constant throughout its lifespan. Still, it's a fascinating result.


Published: 2015-05-26 1 minute read

I recently published nd2reader, a Python library that reads .nd2 image files produced by NIS Elements. Most of the credit goes to the authors of SLOTH who figured out how nd2s are actually structured.

nd2reader improves on things in a few ways. It can associate metadata with individual images, so you can know the channel, timestamp, field of view and z-level of any particular image, which I believe is novel among all the nd2 reading libraries. It also provides a simple, easy-to-use interface and (most importantly to me) does not require Java alongside Python, as Bio-Formats does.

Since it's now in PyPI, you can install it with:

pip install nd2reader

Update: I handed over control of this project and no longer maintain it.