ferd.ca – Home Alone: A Post Incident Review
It’s so many things in one: a movie review, an intro to “post incident reviews,” a lesson into which kinds of precautions work and which don’t, and a reminder that things can still go wrong even when well prepped.
Ad Fontes Media rates media outlets according to their bias and reliability. 1) factual reporting has very slight left-leaning bias (see also “Facts Have a Well-Known Liberal Bias”) and consequently 2) it doesn’t seem to be possible to be reliable and biased at the same time.
In Hard to discover tips and apps for making macOS pleasant Tristan Hume says:
Chrome and Firefox have much better sounding audio resampling for watching videos on 1.5x or 2x speed. This is the only reason I don’t use Safari.
I just tried it and oh my is this true!
For the past 4 years, just before the holidays, I’ve sent a newsletter to my colleagues about all the things I’ve read, watched, listened to, cooked, or ate in the last year. This is the 2020 edition.
Where I’m from, there are so many blueberries that people from that region are called bleuets. (The tourist info phone number is 1-877-BLEUETS (253-8387). You should go and visit.) When you go out to pick them (the blueberries, not the people) and you hit a patch with tons of them, we say “I found une talle!” That’s how it felt getting into Christopher Alexander’s ideas.
You might know of Alexander without realizing it. He’s considered as the father of the idea of design patterns, which were heavily inspired by his book architecture book A Pattern Language. Patterns are part of a larger process to solve design problems, from problem definitions, to solutions, to the evaluation of how good the solution is. It’s that larger process that I found interesting. He named the parts of the process context, form, and fit. In brief, the context is the problem to be solved together with its constraints. It’s totally independent of the solution. The form is the solution. It has a shape that fits the context more of less well. Once I had these words in my mind, I started seeing them everywhere.
Clay Christiensen’s Jobs-to-be-Done framework (from Competing Against Luck) is a way of defining product opportunities (jobs) as a context that can be fulfilled in many different ways. To get an idea of what that means, watch him telling the story of the job of a milkshake and how milkshakes compete with bananas, donuts, and bagels, and not with ice cream.
In Demand-Side Sales, Bob Moesta (who did the milkshake research) talks about selling from the buyer’s perspective, from their struggles, from their context. In contrast to selling from the supply-side, which talks about features and form words. I highly recommend this book.
On the programming side, unit testing has morphed into form testing, whereas the original intent was context testing. Behavior-driven development tried to get that spirit back. What I heard Ian Cooper say in his TDD, Where Did It All Go Wrong talk was “Write context tests.”
Basecamp’s Shape Up is basically Alexander’s ideas applied to product development and management. It makes a clear distinction between shaping (defining the problem and the context) and doing the work (creating a form). Which in turns allows a new dimension of diagnosis when a project fails: was the project badly shaped or is it that the execution failed and form was a bad fit?
Ryan Singer, head of strategy at Basecamp, did a great intro to Alexander’s work this summer.
Losing the War by Lee Sandlin is the best writing I read this year (and I read it on January 1, 2020). It’s an incredible essay on war (the Second World One), and memories. I found the “flow” to be exemplary with plenty of historical context and commentary. Well worth the time.
programming == cookingI used to tell students that I was “teaching them how to fish.” But after reading Robin Sloan’s Home Cooked App, I’ve started telling people I’m teaching them how to cook. It’s a much richer analogy.
If you don’t know how to cook, you eat pre-packaged meals or you eat out (or you live with your mom forever). But there are tons of reasons to want to learn how to cook and tons of things to do once you’ve learned how. You can cook for health reasons, to please friends, to create the perfect version of something, as a creative output, because you’re picky, to make food you want to eat, because it’s fun. You can specialize and become a baker, or open a taco truck, or start a family restaurant, or run a fine-dining restaurant. You can teach others how to cook. In the end, you can still go out and eat if you want, you have a choice. There are so many reasons to learn to cook.
If you don’t know how to program (or how to reason about programming) you use apps you bought and hire consultants. But there are tons of reasons to want to learn how to program and tons of things to do once you’ve learned how. You can create little scripts to rename files, write notebooks to make custom analyzes, write an app to enable friends and colleagues to explore data like you can, write your own text editor (!) because vim is holding you back, because it’s fun. You can specialize and build deep learning models, or design data management systems, or build custom web apps, or start a scientific software development consultancy. You can teach others how to program. In the end, you can still hire consultants if you want, you have a choice. There are so many reasons to learn to program.
The Glamorous Toolkit is an impressive “moldable development environment”. I’d describe it as a strange hybrid between a notebook editor and an IDE. Any objects can have a rich and interactive representations, which is not new (to us), but the cool part is that it’s easy to dynamically add new rich representations to the development environment. Watch this talk by Tudor Gîrba to get a taste of what’s possible.
Simon Willison’s (co-creator of Django) Datasette is an impressive project to “explore and publish data”. It relies heavily on sqlite. I look forward to playing with it some more over the holidays. He’s done super cool things with it, including finding the best photo of a pelican according to Apple Photos, and building a regex search engine across a collection of Github repos using ripgrep.
I started many books this year, but didn’t finish that many:
Thinking in Systems by Donella Meadows is a short introduction to “systems thinking,” their components (stocks, flows, feedback loops), common system configurations, common pitfalls/opportunities of those configurations, and leverage points to intervene in those systems. I had so many “Ah ha!” moments. Many about things I “knew” but hadn’t realized the consequences of. Like how “systems with similar feedback structures produce similar dynamic behaviors, even if the outward appearance of these systems is completely dissimilar.” For example, a population system (controlled by births and deaths) has a similar configuration as an economic capital system (controlled by investment and depreciation). Reading this book, it was so easy to think “Here’s a silver bullet!” but then she totally called me out on it:
People who are raised in the industrial world and who get enthused about systems thinking are likely to make a terrible mistake. They are likely to assume that here, in systems analysis, in interconnection and complication, in the power of the computer, here at last, is the key to prediction and control. This mistake is likely because the mind-set of the industrial world assumes that there is a key to prediction and control.
Elements of Typographic Style by Robert Bringhurst. Yes it’s a typography book, but it may also be the funniest thing I read all year. Bonus: the book itself is a beautiful object.
Lab Girl by Hope Jahren is a wonderful memoir of a “life in science.” She’s a good storyteller, telling a good story, about how the path to success in academia can be wild.
But I read a lot of good articles:
The Ouarzazate Solar Power Station melts salt by concentrating the power of 7400 parabolic mirrors towards a central tower. The molten salt is then used to superheat water and power steam-powered generators.
It’s possible to tell one’s position in the sea based on wave patterns. The 2016 NYT piece by Kim Tingley, The Secrets of the Wave Pilots, is an impressive testament of what our brains can perceive using our full bodies as sensors. See also these beautiful Marshall Islands navigation charts, and the Eagle Eyes Radiolab story about the vest David Eagleman is building to help deaf people hear.
I started making pizza just after the pandemic started. I can’t wait for it to be over to organize pizza parties. My source of knowledge is Ken Forkish’s Elements of Pizza. Using tipo 00 flour is worth it. So is a baking steel/stone.
Kenji Alt Lopez’s Serious Eats episode on emulsions has a really compelling demonstration of why you should put a surfactant in your emulsions. Now I put a little dollop of mustard in all my vinaigrettes.
My friend Matt shared some bardcore videos by Hildegard von Blingin’ earlier this year. Here’s Somebody That I Used To Know (Bardcore/Medieval Style Cover with Vocals). Bardcore is medieval renditions of pop songs. Obviously. So… the name Hildegard von Blingin’, is a reference to Hildegard von Bingen, an 11th century badass abbess, composer, and scientist (among others). In the early 2000, my mom gave me a CD by Garmarna, a Swedish band that does… electro-folk renditions of her 11th-century music. Hildegard von Blingin' = 1/Garmarna! Here’s one song and here’s the whole Garmarna album.
According to Spotify and Last.fm, I’ve listened to 1,700+ different artists this year, 600 of which were new to me. I won’t give you the whole list, don’t worry. Although I do have a playlist of every album I liked in 2020 and one of only the best songs I listened to in 2020.
Here are my favorite albums released in 2020, in no particular order.
Here’s more great music I discovered this year but that wasn’t made this year.
I somehow ended up listening to a lot of Japanese indie music (for a lack of better term).
I just finished reading Thinking in Systems: A Primer by Donella H. Meadows. I believe sometimes one was to be “ready” to read a book. I must have been ready for this one because I had more “Holy sh!t” moments than any other book I’ve read in years.📚
Austin, TX, 8:15. Morning walk around the neighborhood. There are many good Halloween decorations, but this one is particularly entertaining. #adayinthelife
Given what happened to the other saguaro, it makes me fear that this one will fall too.
I took this photo of a saguaro a month ago. It had been partially burnt during the fires that happened north of Phoenix earlier this summer. This weekend I realized it had fallen. According to Wikipedia, saguaros grow their first arm around 75–100 years of age. 👋🌵😔
Yesterday I read a series of posts on custom URL scheme handlers on the Zettelkasten forums. The handler registers itself to open links like zettel://202006061337, where the number is the unique ID of a zettel (here it’s a timestamp). I’m not sure everyone realized the magnitude of what that means.
Combining a custom URL with a unique ID means notes and links can become entirely independent from your apps. Only the handler needs to know about the apps you’re using.
That alone is very nice, but then I thought: URLs can have query parameters… That means I can have URLs like zettel://202006061337&edit that open in my text editor of choice: TextMate, BBEdit, WriteRoom, FoldingText, etc. Or zettel://202006061337&preview to open in Marked. Or I could even pick the app interactively with zettel://202006061337&pick.
Handling the ID alone is pretty easy since the common Zettelkasten-like apps respond to a URL scheme to search and open files. The Archive uses thearchive://match/ID, nvUltra uses x-nvultra://find/ID, and nvAlt uses nvalt://find/ID.
But how to open in a text editor or in Marked given only the ID? With Spotlight. I used mdfind -name ID to find the file. This could be further refined with the -onlyin FOLDER option but I didn’t need it. Then it’s a matter of calling open -a Marked FILEPATH.
There’s also a zettel://create special case that will create a new zettel with the current time stamp (YYYYMMDDHHMM). It’s always done with the default Zettelkasten app because the script doesn’t know where to write the file but the app does.
I wrote the handler in Applecript because it’s the easiest way I know to create something that macOS considers an “app” and that can therefore handle URLs.
The full script is below. To use it:
appChoices array.Contents/Info.plist with a text editor<dict> line. Replace zettel with the URL prefix you’ve chosen.<key>CFBundleURLTypes</key>
<array>
<dict>
<key>CFBundleURLName</key>
<string>Zettel Link Opener</string>
<key>CFBundleURLSchemes</key>
<array>
<string>zettel</string>
</array>
</dict>
</array>Here’s the full script:
-- Zettel Link Opener
-- Created by Alexandre Chabot-Leclerc
-- Source: https://alexchabot.net/2020/06/06/custom-url-handler-for-zettels/
-- URL Handler for zettelkasten unique IDs, e.g., zettel://202006061017
-- Handles options after the ID to open different apps:
-- zettel://202006061017&edit to open is a text editor like TextMate
-- zettel://202006061017&preview to open in a preview app like Marked
-- zettel://202006061017&pick to open a menu of apps to pick from
-----------------------------------------------------------
-- CONFIGURATION
-- URL prefix for your custom URL, e.g. zettel://ZETTEL_ID
property urlPrefix : "zettel"
-- Default URL to call to open a note with a given ID. The ID will be appended
property defaultZkAppUrl : "thearchive://match/"
--property defaultZkAppUrl : "nvalt://find/"
--property defaultZkAppUrl : "x-nvultra://find/"
-- URL to use to create a new zettle with the current timestamp YYYYMMDDHHMM
property urlForCreation : "thearchive://matchOrCreate/"
--property urlForCreation : "nvalt://make?txt="
--property urlForCreation : "x-nvultra://make?txt="
-- Apps to use for the different query parameters
property editApp : "FoldingText" -- App to used with "&edit" query parameter
property previewApp : "Marked" -- App to used with "&preview" query parameter
-- List of app to display in the menu with with &pick query option
-- The apps will appear in the order defined here
property appChoices : {defaultZkAppUrl, editApp, previewApp, "TextMate"}
property defaultApp : {defaultZkAppUrl}
-----------------------------------------------------------
on splitText(theText, theDelimiter)
set AppleScript's text item delimiters to theDelimiter
set theTextItems to every text item of theText
set AppleScript's text item delimiters to ""
return theTextItems
end splitText
on removeUrlPrefix(original)
-- Remove URL prefix so we're left with only the ID and the optional query parameter
return do shell script "echo " & quoted form of original & " | sed 's;" & urlPrefix & "://;;'"
end removeUrlPrefix
on getIdAndOption(resouceAndQuery)
-- Split the zettel ID and the optional parameter
-- For example 202006061012&edit or 202006061012&preview
set theItems to splitText(resouceAndQuery, "&")
if length of theItems is 1 then
-- Append an empty string if there's no option so this
-- function always returns an array of 2 elements
copy "" to the end of theItems
end if
return theItems
end getIdAndOption
on findFilepath(zk_id)
-- Finds the filepath using Spotlight.
-- It's easier than finding the proper filename given only the zettel ID
return do shell script "mdfind -name " & zk_id
end findFilepath
on createZettel()
set newZkId to do shell script "date +'%Y%m%d%H%M'"
do shell script "open " & urlForCreation & newZkId
end createZettel
on openInChoosenApp(zkId, zkFilepath)
-- From Simple List Handler by Patrick Welker <http://rocketink.net>
-- Promp the use for the app to use
set selectedApp to item 1 of (choose from list the appChoices with title "Available App" with prompt "Which app do you want to use?" default items defaultApp)
if selectedApp is false then
-- Exit prematurly if the user clicked Cancel
error number -128
end if
-- Open the URL directly, or open by app name
if selectedApp contains "://" then
do shell script "open " & selectedApp & zkId
else
do shell script "open -a " & selectedApp & " " & quoted form of zkFilepath
end if
end openInChoosenApp
on open location thisURL
set resouceAndQuery to removeUrlPrefix(thisURL)
set idAndOption to getIdAndOption(resouceAndQuery)
set zkId to item 1 of idAndOption
if zkId is "create" then
createZettel()
return
end if
set zkFilepath to findFilepath(zkId)
if item 2 of idAndOption is "edit" then
do shell script "open -a " & editApp & " " & quoted form of zkFilepath
-- Exit the script immediately so we don't also open in the default app
return
else if item 2 of idAndOption is "preview" then
do shell script "open -a " & previewApp & " " & quoted form of zkFilepath
-- Exit the script immediately so we don't also open in the default app(
return
else if item 2 of idAndOption is "pick" then
openInChoosenApp(zkId, zkFilepath)
return
end if
-- Fall back to the default handler if there was no option
-- or the option was invalid
do shell script "open " & defaultZkAppUrl & zkId
end open location
TLDR; Use %matplotlib if you want interactive plotting with matplotlib. If you’re only interested in the GUI’s event loop, %gui <backend> is sufficient.
I never really understood the difference between %gui and %matplotlib in IPython. One of my colleagues at Enthought once told me that at some point in his career, he more or less stopped reading documentation and instead went straight to the code. That’s what I did here. But let’s do a bit of history first.
In the “beginning”, there was pylab. It (still) is a module of matplotlib and was a flag to IPython designed to facilitate the adoption of Python as a numerical computing language by providing a MATLAB-like syntax.1 The reference was so explicit that before being renamed to pylab on Dec 9, 2004, the module was called matplotlib.matlab. IPython adopted the rename on the same day.2 With the ‑‑pylab flag or the %pylab magic function, IPython would set up matplotlib for interactive plotting and executed a number of imports from IPython, NumPy and matplotlib. Even thought it helped a few people transition to Python (including myself), it turned out to be a pretty bad idea from a usability point of view. Matthias Bussonnier wrote up a good list of the many things that are wrong with it in “No Pylab Thanks.”
For the 1.0.0 release of IPython in August 2013, all mentions of %pylab were removed from the examples (in a July 18, 2013 commit) and were replaced by calls to the %matplotlib magic function, which only enables interactive plotting but does not perform any imports. The %matplotlib function had already been introduced in a 2013 refactoring to separate the interatice plotting from the imports. The %gui magic command had already been introduced in 2009 by Brian Granger to “manage the events loops” (hint hint).
Now we know that the (my) confusion with %gui and %matplotlib started in 2013.
This analysis refers to IPython 7.8.0 and ipykernel 5.1.2.
Our entry point will be the %matplotlib magic command. Its source code is in the IPython.core.pylab.py file. The essential call is to shell.enable_matplotlib(gui), which is itself implemented in IPython.core.interactiveshell.InteractiveShell, and does five things:
IPython.core.pylabtools.find_gui_and_backend(gui). It encapsulates the logic to go
from a GUI name, like "qt5" or "tk", to a backend name, like "Qt5Agg" and "TkAgg".IPython.core.pylabtools.activate_matplotlib(backend), which:
matplotlib.interactive(True);matplotlib.pyplot.switch_backend(backend);matplotlib.pyplot.draw_if_interactive method with the same method, but wrapped by a flag_calls decorator, which adds a called flag to the method. That flag will be used by the new %run runner that’s introduced below at point #5;IPython.core.pylabtools.configure_inline_support(shell, backend). This is where some very interesting stuff happens. It first checks that InlineBackend is actually importable from ipykernel.pylab.backend_inline, otherwise it returns immediately. But if it’s importable and the backend is "inline", it:
ipykernel.pylab.backend_inline.flush_figures function, and register it as a callback for the "post_execute" event of the shell. As we’ll see later, callbacks for "post_execute" are called after executing every cell;"inline", it’ll unregister the flush_figures callback;shell.enable_gui(gui). This method is not implemented in the IPython.core.interactiveshell.InteractiveShell base class, but rather in IPython.terminal.interactiveshell.TerminalInteractiveShell. If a gui as specified, it gets the name of the active_eventloop and its corresponding inputhook function using IPython.terminal.pt_intputhooks.get_inputhook_name_and_func(gui). The active_eventloop is just a string, such as 'qt', but the inputhook is more interesting. It’s the function to call to start that GUI toolkit’s event loop. Let’s dig further into get_inputhook_name_and_func(gui). That function checks a few things, but it essentially:
inputhook function for the chosen GUI by importing it from IPython.terminal.pt_intputhooks.<gui_mod>. For example, the Qt inputhook is imported from IPython.terminal.pt_intputhooks.qt. Later on, when inputhook is executed for Qt, it will:
QCoreApplication;QEventLoop for that application;tk, wx, or osx, but they all essentially do the same thing. At this point we’re ready to go back up the stack to enable_matplotlib in %matplotlib;default_runner with the one defined in IPython.core.pylabtools.mpl_runner. The default_runner is the function that executes code when using the %run magic. The mpl_runner:
matplotlib.interactive state, and disables it;interactive state;plt.draw_if_interactive.called flag that was introduced at point #1.3 above.As for the other magic, %gui, it only executes a subset of what %matplotlib does. It only calls shell.enable_gui(gui), which is point #4 above. This means that if your application requires interaction with a GUI’s event loop, but doesn’t require matplotlib, then it’s sufficient to use %gui. For example, if you’re writing applications using TraitsUI or PyQt.
%gui and %matplotlibLet’s start with the “simplest” one, %gui. If you execute it in a fresh IPython session, it’ll only start the event loop. On macOS, the obvious effect of this is to start the Rocket icon.
At that point, if you import matplotlib and call plt.plot(), no figure will appear unless you either call plt.show() afterwards, or manually enable interactive mode with plt.interactive(True).
On the other hand, if you start your session by calling %matplotlib, it’ll start the Rocket and activate matplotlib’s interactive mode. This way, if you call plt.plot(), your figure will show up immediately and your session will not be blocked.
%runIf you call %run my_script.py after calling %matplotlib, my_script.py will be executed with the mpl_runner introduced above at point #5.
"inline" BackendIn the terminal the IPython.terminal.interactiveshell.TerminalInteractiveShell.interact() method is where all the fun stuff happens. It prompts you for code, checks if you want to exit, and then executes the cell with InteractiveShell.run_cell(code) and then trigger the "post_execute" event for which we’ve registered the ipykernel.pylab.backend_inline.flush_figures callback. As you might have noticed, the flush_figures function comes from ipykernel, and not from IPython. It tries to return all the figures produced by the cell as PNG of SVG, displays them on screen using IPython’s display function, and then closes all the figures, so matplotlib doesn’t end up littered will all the figures we’ve ever plotted.
To sum it up, use %matplotlib if you want interactive plotting with matplotlib. If you’re only interested in the GUI’s event loop, %gui <backend> is sufficient._ Although as far as I understand, there’s nothing very wrong with using %matplotlib all the time.
plot, figure, subplot, etc.
[return]