Currently reading: The Art of Gathering: How We Meet and Why It Matters by Priya Parker đ
Currently reading: The Art of Gathering: How We Meet and Why It Matters by Priya Parker đ
What an odd Big Sur bug. VS Code’s dock icon is minuscule and impossible to click on. It reminds me of when your Badland’s little flappy furry balls becomes tiny.
Tom7 creates the uppestcase and lowestcase letters by training two deep learning models: one to create uppercase letters from lowercase ones, and the other to create lowercase letters from uppercase ones. Then he pushes things (beyond) their logical conclusions, such as creating lowercase versions of lowercase letters, and uppercase versions of uppercase letters. The results are fonts you can download and “use.” In the process, he builds a lot of really neat custom UIs to visualize what the models are doing. It’s entertaining and interesting.
Via Macdrifter.
Two loaves of Ken Forkish’s Overnight Country Blonde. After a 15h bulk rise at 23ÂșC, the dough had almost quadrupled in volume. Much more than usual. The final bread is a tad flat. I think it’s because it ran out of energy, but maybe I could have made deeper cuts.
Simon Willison has a fascinating data-publishing and data-management project named datasette. A few months ago, he put together a plugin named datasette-ripgrep that uses ripgrep (you use ripgrep, right?) to search folders of files and display the results using datasette’s machinery.
I thought of creating a datasette-ripgrep instance to search all the packages from the Enthought Tool Suite. Using GitHub to search across this cohesive set of tools, and only this set of tools, doesn’t really work.
Setting datasette-ripgrep up locally turned out to be pretty easy. But publishing it to Google Cloud Compute (GCP) using GitHub Actions so I could automate the daily the content of the indexes repositories turned out to be a multi-month effort.
I started working off the demo deploy action which took me most of the way there. But I kept running into GCP authentication issues. It complained that “No credentials provided, skipping authentication”. That is, until I realized 2 months later (of on-and-off attempts) that I was putting GitHub secrets in Settings > Environment > Secrets, and not in Settings > Secrets. *slaps forehead* I’m sure actions can see secrets in the Environment section somehow, but I don’t know how. Another thing I learned is that when the GCP docs ask you to put the service account key in a GitHub secrets, you can just paste the whole JSON as-is.
The next hurdle was that the datasette publish cloudrun
command would fail with the error “You do not appear to have access to project [âŠ]“. I tried many things related to IAM, role, service accounts and the likes, but without success. The ah ha! moment came when I realized/remembered that datasette.publish.cloudrun
actually talks to GCP using the gcloud
command line tool. I identified that it calls the builds and deploy subcommands. Using that information I could make searches to figure out which permissions were required to execute those commands. The one I was missing was Cloud Build Editor (and maybe Viewer).
In the end, the Service Account has the following roles (I’m not 100% sure they’re all necessary):
After 100 failed deploys and much reading of mediocre Medium articles and of Google’s (seemingly) incomplete and incorrect READMEs, the 101th deploy succeeded! You can now search the ETS repos at the very unglamorous URL of https://datasette-ripgrep-ets-alicuzwd4a-uc.a.run.app and see the source on GitHub.
The discussion on symbiotic relationships between apps on the latest Core Intuition by @manton and @danielpunkass reminded me of Eastgate’s SummerFest/WinterFest for “artisanal” research/thinking/writing apps. It’s a great example of a collection of complementary apps.
What is the maximum one can be doing with someone else and still be hanging out?
Austin Kleon talks about the 13-month International Fixed Calendar. It reminds of the ISO week numbers that the Danes love so much. It always weirded me out when they ask something like “Are you free week 14?” But thinking back, they might have been onto something.
Despite all the damage done by the ice in Texas, it still created quite a bit of beauty (or at least novelty).
Icicle agave during the Texas winter storm.
I shared these notes with my friends and colleagues at the Technical University of Denmark (DTU) when I finished my PhD in December 2015.
Here are things that I either learned during my PhD, or that helped me get through it. I in no way have all the answers, but there are a few things that helped me that I think would be helpful to other people as well.
I split the content in six different sections, meta-things, writing, reading, speaking, programming, and mind and body. I included links to articles, books, and tools that are related to each section.
The main recommendation here is to have a system to deal with all the things you have to do and the things you want to do. I personally used the GTD method, which stands for Getting Things Done. I highly recommend getting the latest book. You can find it delivered at your desk for 100 kr. The main ideas of the system are: everything goes in an inbox (physical and/or on a computer). Periodically, process the inbox and decide if you’ll do the thing, trash it, or store it in your reference system. “Tasks” that require more than two action are considered projects, and a project consist of a list of physical, actionable actions. The other important thing about GTD is that it includes a weekly review of all the projects in your system, as well as your calendar and other active things in your life. I learned about this almost 10 years ago now1 and I would go as far as to say that it changed my life.
There are many applications designed to implement the GTD system. On the Mac, there’s OmniFocus, Things, and TaskPaper (really worth checking). Todoist is multi-platform.
I also recommend reading Getting Results the Agile Way, by J.D. Meier, which proposes a different take on selecting what you should work in. The basic idea is to pick 3 things to focus on every day, three per week, three per month, and three per year. The things you do every day should go towards you weekly goals, etc. The Asian Efficiency blog (weird name, I know), is a good noise-free resource about that kind of material (check the “specialty topics” in the sidebar).
Keep checklists for processes and things you do often but with lots of time in between, like how to log on to remote computers, adding printers to your computer, etc. It will save you a lot of time in the long run. Similarly, save info about things you never remember how to do, such as the behavior of the FFT implementation in MATLAB, check if a file exists using bash, undo a Git merge, etc.
I keep all my notes in individual text files. I have about 800. They live on Dropbox, which means I can edit them and view them anywhere. On my Mac I use nvAlt to search and write them. ResophNotes does the same thing on Windows. Evernote is a web-based service that can be used for a similar purpose. I write all my notes in Markdown (I even wrote a paper in it!), which also allows me to easily export my notes or view in nice previews. Both nvAlt and ResophNotes are “markdown-aware”.
I kept a daily or weekly text file in which I wrote almost everything I did within the day. I would write down either thing that I wanted to do, notes to clarify my thoughts, or the things I just did. For example, I would save the Git commit for a particular simulation and my comments about the results. It helped me when I wanted to look back at why I did certain things or to find the source of certain ideas.
I tried doing everything on the computer, but sometime paper really is the best tool for the job. The final product doesn’t have to be on paper but paper is often really helpful to develop ideas because of the freedom you have to place things wherever you want!
If you see something that is broken, or that could be better, just fix it. Especially if other people have noticed the same problem. I mean to fix both immaterial things (BitBucket, for example) but also physical things. Did you know you can report broken things to CAS and they’ll come repair it within a day or so? Blocked toilet, dead light, water leek? Report it at https://fejlmeld.cas.dtu.dk or use their iOS or Android apps (can’t find the link, but look for DTU Fejlrapportering).
At first, I did not have a method for writing but then I read this article by TimothĂ©e Poisot, which really helped me. The main idea is to start building an outline as soon as you’re working on the new project. As you read papers, or have ideas, you add quotes, citations, and snippets to your outline. You can start writing sentences and whole paragraphs whenever you want. The big advantage is that all the relevant things you find for a given project are all together; no need to go hunting for where you read that thing, six months ago.
On the Mac, I use Tree2, which I love. OmniOutliner is the most famous alternative, but it much more expensive. I don’t know about Windows or Linux. I know Microsoft Word has an outlining mode that could work. You could also outline in a normal Word document, or a text file, but good outliners give you many shortcuts to move things around and to insert new ideas. The new Manuscripts application for Mac looks amazing. Scrivener is an amazing writing tool for both Mac and Windows. It is particularly good for large projects.
Other good resources include:
Do not despair, you will get better at writing! I don’t really know how to do it, but the earlier you get good the easier your life will be. :-) A possible way to get better is certainly to write more. Maybe having a writing club would help. There are many good books about writing. I read How to write a lot, by Paul J. Silvia and really liked it.
Here are a few articles about writing:
You never find time, you’ll have to make some. Personally I found out that I could read an article on the bus and while walking. When I needed to read many article, I would take the bus mornings and nights for a week. That way I could read about 10 articles in week. It helped me to keep a stack of papers that I wanted to read on my desk; I could grab one whenever had a moment.
I find that it’s much easier to understand and remember papers if I read with a purpose, as a way of answering a question. Also, if that question is related to what you’re currently working on, you can add your discoveries in the outline of your paper.
I found that being able to associate a face with an author name helped me remember who was talking about what and made it easier to remember different papers and ideas.
That one is pretty obvious, but writing down notes and ideas while reading papers really helps. It helps while reading the paper but also later when coming back to it. I designed a highlight system for myself based on the idea of Walton Jones. I used specific colors to code different things, for example paper-specific results were yellow, new references in green, and paper summaries in red. This way could look back at a paper and have a general overview of what was important without reading the whole paper again.
On the Mac, Skim allows you to export notes including their color. It also works well with DEVONthink (see below).
I recommend having a good search system for your notes and papers. On the Mac I use an application called DEVONthink which can show you documents related to the one you’re currently reading. It also does non-exact searches, e.g., it can search for the meaning of your query, not just the exact words you typed. It really is magic. I am not aware of anything exactly the same on Windows or Linux, but The Brain (Windows) and Recoll might be good places to start.
I gave all my articles the same file name structure, authorYEARfirstword
, e.g. chabotleclerc2014predicting
. I picked this format because it has no space and it’s also very short and compact. It’s essentially a unique identifier. I used the identifier everywhere: in handwritten notes, computer notes, outlines, marginalia. This way, I knew exactly which paper I was talking about.
I tried to use spaced repetition (also see this link) for a while but it didn’t stick for really long. Anki is probably the best tool for the job.
Based on the idea of this crazy guy, Stian HĂ„klev, I tried setting up a wiki using either DokuWiki or TiddlyWiki. I managed to keep it updated for about a month and then abandoned it. It required a lot of maintenance and a lot of diligence to keep up with the linking between subjects.
2021 update: These days, I would strongly recommend looking into Zettelkasten as an alternative.
I tried building a gigantic outline or mind map or concept map of all the things I learned, but it became unmanageable. I still find these methods really useful when fleshing out an idea for a project or paper, but they were not really good tools for me to manage such a large amount of knowledge. If you want to try these tools, on the Mac I recommend MindNode (simple and cheap), and iThoughtsX (more powerful, more expensive). MindMeister is online and very popular. Docear mixes an outliner, a reference manager (JabRef) and an outliner. It’s power and complex. If you want to know everything about mind mapping and the best tools, check out Brett Terpstra’s blog.
For concept maps, I love Scapple, which works on Mac and Windows. Cmap is also multi-platform, but heavier. I use them when I want to find structure in a mess of ideas floating in my head.
This series of articles by David L Stern really influenced me: How to Give a Talk. He suggests 5 rules3:
Of course, some rules are meant to be broken, but it’s worth a read. I also liked the book Presentation Zen: Simple Ideas on Presentation Design and Delivery by Garr Reynolds.
When working on a new model, first decide what you’re optimizing for, is it a correlation, the mean square error, or some other metric. Make sure this metric is computed automatically with each simulation, so you have immediate feedback about how good your model is. Just eyeballing the results is a really bad idea.
Don’t. Repeat. Yourself. When you see that you have the same code in multiple places, it’s a good hint that you need to refactor it into a function, and to call that function. Also, if your function is hundred lines long it’s probably a sign that he should be chopped into smaller functions that have clear names.
If you’re interested in becoming a better programmer, I recommend The Pragmatic Programmer, by Andrew Hunt and David Thomas, Clean Code and Code Complete.
You should write tests for your functions to make sure that they are actually doing what you think they are doing. They help make sure that code is correct, they help detecting regressions (when you break things that used to work), they act as a specification, design and documentation, and they make refactoring easier.
I cannot tell you how many times I found small and huge bugs because of the tests I wrote. All decent programming languages, including Matlab include a framework to automatically run tests. The words you want to look for are “unit testing”. You should run tests every time you make changes. You can even set Git to run the tests before every commit. This way you can make sure you never commit broken code.
The advantage of working with a free language (Python!) and in the open is that you can use “continuous integration” system to have them run tests every time you push new changes to GitHub (or BitBucket). Travis CI is one of those services.
These articles or sites are worth reading for info about testing.
Code comments are useful to explain why you’re doing something, but they should not explain every single line of code. Otherwise one day you will change the comment or the code and then one of them will be lying and you won’t be able to know which one.
Instead, use explicit names for variables and functions. Write function help files. Write documentation. Write README files. And break up long and complicated lines of code into smaller ones where you can name concepts and variables.
Version control is a communication tool and a collaboration tool. It’s invaluable even if you don’t work with someone else, I can assure you that your future self will be really happy to know what your past self was thinking when he/she wrote that code/text. I heard your future self can get really upset. :-)
Track where experiment results and figures come from. You can do it by hand, by keeping your log book and tracking output folders and inputs commits, but you can also use automated solutions. I used Sumatra, which is written in Python but can be used with Matlab. Every time you run a simulation it saves to a database: the experimental parameter used, the input files, and the command line outputs. It also allows you to comment and tag results. You can look back and see where things are from, what worked and what didn’t, and rerun experiments with exactly the same parameters. A provenance-tracking tool like Sumatra paired with version control is a good step towards making reproducible research. Recipy is a new Python-only solution that looks interesting.
Some very bright people have written “Best practices” for scientific computing. It’s worth reading the two or three articles below and applying their recommendations. I also recommend looking at this shablona
project, which proposes a standard folder structure for a given project, I found it to work really well. It’s written with Python in mind but it could easily be adapted for Matlab.
Learn about literate programming, a useful concept where the code and the text (of, e.g., an article) live together. This can be done in Matlab using the publish
command. In Python, the Jupyter notebook is the way to go. Actually, you can use the Jupyter notebook with Matlab.
If you want to do more than just read things, you should try to organizing a Software Carpentry workshop. It’s pretty much free and would probably help a lot of people.
If you find out you’re doing the same thing over and over again it might be worth automating it. On the Mac, Keyboard Maestro can do magic (like activating menu items for you, clicking places, moving windows, etc.) On Windows, AutoHotkey can do similar things, and on Linux, I found AutoKey to be the best.
Use a shortcut expander application. For example, I never type “intelligibility”, I just type inty
and it expands to the full word. Same thing with my name, my bank account, my phone number, etc. On the Mac, Keyboard Maestro and TextExpander are the best. On Windows, PhraseExpress is compatible with TextExpander; you can sync your snippets via Dropbox. On Linux, Autokey can also do snippet expansions.
Seriously, take care of your body. If you start having wrist, forearm, or shoulder pain, don’t just live with it. Seek help and get proper a keyboard, mouse and chair. Learn some basic ergonomic practices about keyboard, mouse and screen placement. See the articles below for more info and recommendations.
If you are in pain, really look for help: find a massage therapist, a doctor, and stretch often. Here are a few good ressources.
It’s worth paying for things that make your life better, more comfortable, and easier, especially if you use those things all day long. Buy a nice keyboard, you probably write eight or nine hours a day. Buy nice headphones, good software, a mattress, a bike, etc.
Ask for help when you’re stuck. Ask for a meeting if you need one. Everyone is super busy, but everyone is super generous. People rarely say no, but they might say “later”. In a perfect world, everyone would be proactive, everyone would have their slot in the calendar, but sometimes the world is not perfect. Also talking to people if often waaaay faster than googling for an answer.
This idea is not from me, it is from ryans01 on Reddit (and it’s also in the PhD Starter Kit, below). His idea is to make sure that every day you do at least one small thing towards your goal. No day should be completely wasted, even if the thing you do is really small. Keeping a log or journal helps realizing all the things you’ve done in a day and keeping your mood up.
Other people have written really good and more in-depth guides than this. The PhD Starter Kit is simply amazing. Philip Guo’s Advice for first year PhD students is a must-read/must-watch (as well as many of his other articles. He also wrote: The Ph.D. Grind, a 115-page e-book, is the first known detailed account of an entire Ph.D. experience.” That guy is amazing.
They could have used additional time during the bulk fermentation to account for the cold, but itâs still a good rise.
Great Resident Advisor podcast episode. Noisy, glitchy, grimy, dark. RA.760 Hyph11E â RA Podcastđ”
Installing air filters in classrooms has surprisingly large educational benefits
[âŠ] Math scores went up by 0.20 standard deviations and English scores by 0.18 standard deviations [âŠ] this is comparable in scale to […] the potential benefits of smaller class sizes.â
ferd.ca â Home Alone: A Post Incident Review
Itâs so many things in one: a movie review, an intro to âpost incident reviews,â a lesson into which kinds of precautions work and which donât, and a reminder that things can still go wrong even when well prepped.
Ad Fontes Media rates media outlets according to their bias and reliability. 1) factual reporting has very slight left-leaning bias (see also “Facts Have a Well-Known Liberal Bias”) and consequently 2) it doesn’t seem to be possible to be reliable and biased at the same time.
In Hard to discover tips and apps for making macOS pleasant Tristan Hume says:
Chrome and Firefox have much better sounding audio resampling for watching videos on 1.5x or 2x speed. This is the only reason I donât use Safari.
I just tried it and oh my is this true!
For the past 4 years, just before the holidays, I’ve sent a newsletter to my colleagues about all the things I’ve read, watched, listened to, cooked, or ate in the last year. This is the 2020 edition.
Where Iâm from, there are so many blueberries that people from that region are called bleuets. (The tourist info phone number is 1-877-BLEUETS (253-8387). You should go and visit.) When you go out to pick them (the blueberries, not the people) and you hit a patch with tons of them, we say “I found une talle!” Thatâs how it felt getting into Christopher Alexander’s ideas.
You might know of Alexander without realizing it. He’s considered as the father of the idea of design patterns, which were heavily inspired by his book architecture book A Pattern Language. Patterns are part of a larger process to solve design problems, from problem definitions, to solutions, to the evaluation of how good the solution is. It’s that larger process that I found interesting. He named the parts of the process context, form, and fit. In brief, the context is the problem to be solved together with its constraints. It’s totally independent of the solution. The form is the solution. It has a shape that fits the context more of less well. Once I had these words in my mind, I started seeing them everywhere.
Clay Christiensen’s Jobs-to-be-Done framework (from Competing Against Luck) is a way of defining product opportunities (jobs) as a context that can be fulfilled in many different ways. To get an idea of what that means, watch him telling the story of the job of a milkshake and how milkshakes compete with bananas, donuts, and bagels, and not with ice cream.
In Demand-Side Sales, Bob Moesta (who did the milkshake research) talks about selling from the buyer’s perspective, from their struggles, from their context. In contrast to selling from the supply-side, which talks about features and form words. I highly recommend this book.
On the programming side, unit testing has morphed into form testing, whereas the original intent was context testing. Behavior-driven development tried to get that spirit back. What I heard Ian Cooper say in his TDD, Where Did It All Go Wrong talk was “Write context tests.”
Basecamp’s Shape Up is basically Alexander’s ideas applied to product development and management. It makes a clear distinction between shaping (defining the problem and the context) and doing the work (creating a form). Which in turns allows a new dimension of diagnosis when a project fails: was the project badly shaped or is it that the execution failed and form was a bad fit?
Ryan Singer, head of strategy at Basecamp, did a great intro to Alexander’s work this summer.
Losing the War by Lee Sandlin is the best writing I read this year (and I read it on January 1, 2020). It’s an incredible essay on war (the Second World One), and memories. I found the “flow” to be exemplary with plenty of historical context and commentary. Well worth the time.
programming == cooking
I used to tell students that I was “teaching them how to fish.” But after reading Robin Sloanâs Home Cooked App, Iâve started telling people I’m teaching them how to cook. Itâs a much richer analogy.
If you donât know how to cook, you eat pre-packaged meals or you eat out (or you live with your mom forever). But there are tons of reasons to want to learn how to cook and tons of things to do once you’ve learned how. You can cook for health reasons, to please friends, to create the perfect version of something, as a creative output, because you’re picky, to make food you want to eat, because it’s fun. You can specialize and become a baker, or open a taco truck, or start a family restaurant, or run a fine-dining restaurant. You can teach others how to cook. In the end, you can still go out and eat if you want, you have a choice. There are so many reasons to learn to cook.
If you don’t know how to program (or how to reason about programming) you use apps you bought and hire consultants. But there are tons of reasons to want to learn how to program and tons of things to do once you’ve learned how. You can create little scripts to rename files, write notebooks to make custom analyzes, write an app to enable friends and colleagues to explore data like you can, write your own text editor (!) because vim is holding you back, because it’s fun. You can specialize and build deep learning models, or design data management systems, or build custom web apps, or start a scientific software development consultancy. You can teach others how to program. In the end, you can still hire consultants if you want, you have a choice. There are so many reasons to learn to program.
The Glamorous Toolkit is an impressive “moldable development environment”. I’d describe it as a strange hybrid between a notebook editor and an IDE. Any objects can have a rich and interactive representations, which is not new (to us), but the cool part is that it’s easy to dynamically add new rich representations to the development environment. Watch this talk by Tudor GĂźrba to get a taste of what’s possible.
Simon Willison’s (co-creator of Django) Datasette is an impressive project to “explore and publish data”. It relies heavily on sqlite. I look forward to playing with it some more over the holidays. He’s done super cool things with it, including finding the best photo of a pelican according to Apple Photos, and building a regex search engine across a collection of Github repos using ripgrep
.
I started many books this year, but didn’t finish that many:
Thinking in Systems by Donella Meadows is a short introduction to “systems thinking,” their components (stocks, flows, feedback loops), common system configurations, common pitfalls/opportunities of those configurations, and leverage points to intervene in those systems. I had so many “Ah ha!” moments. Many about things I “knew” but hadn’t realized the consequences of. Like how “systems with similar feedback structures produce similar dynamic behaviors, even if the outward appearance of these systems is completely dissimilar.” For example, a population system (controlled by births and deaths) has a similar configuration as an economic capital system (controlled by investment and depreciation). Reading this book, it was so easy to think “Here’s a silver bullet!” but then she totally called me out on it:
People who are raised in the industrial world and who get enthused about systems thinking are likely to make a terrible mistake. They are likely to assume that here, in systems analysis, in interconnection and complication, in the power of the computer, here at last, is the key to prediction and control. This mistake is likely because the mind-set of the industrial world assumes that there is a key to prediction and control.
Elements of Typographic Style by Robert Bringhurst. Yes it’s a typography book, but it may also be the funniest thing I read all year. Bonus: the book itself is a beautiful object.
Lab Girl by Hope Jahren is a wonderful memoir of a “life in science.” She’s a good storyteller, telling a good story, about how the path to success in academia can be wild.
But I read a lot of good articles:
The Ouarzazate Solar Power Station melts salt by concentrating the power of 7400 parabolic mirrors towards a central tower. The molten salt is then used to superheat water and power steam-powered generators.
It’s possible to tell one’s position in the sea based on wave patterns. The 2016 NYT piece by Kim Tingley, The Secrets of the Wave Pilots, is an impressive testament of what our brains can perceive using our full bodies as sensors. See also these beautiful Marshall Islands navigation charts, and the Eagle Eyes Radiolab story about the vest David Eagleman is building to help deaf people hear.
I started making pizza just after the pandemic started. I can’t wait for it to be over to organize pizza parties. My source of knowledge is Ken Forkish’s Elements of Pizza. Using tipo 00 flour is worth it. So is a baking steel/stone.
Kenji Alt Lopezâs Serious Eats episode on emulsions has a really compelling demonstration of why you should put a surfactant in your emulsions. Now I put a little dollop of mustard in all my vinaigrettes.
My friend Matt shared some bardcore videos by Hildegard von Blingin’ earlier this year. Here’s Somebody That I Used To Know (Bardcore/Medieval Style Cover with Vocals). Bardcore is medieval renditions of pop songs. Obviously. So… the name Hildegard von Blinginâ, is a reference to Hildegard von Bingen, an 11th century badass abbess, composer, and scientist (among others). In the early 2000, my mom gave me a CD by Garmarna, a Swedish band that does… electro-folk renditions of her 11th-century music. Hildegard von Blingin' = 1/Garmarna
! Here’s one song and here’s the whole Garmarna album.
According to Spotify and Last.fm, Iâve listened to 1,700+ different artists this year, 600 of which were new to me. I won’t give you the whole list, don’t worry. Although I do have a playlist of every album I liked in 2020 and one of only the best songs I listened to in 2020.
Here are my favorite albums released in 2020, in no particular order.
Here’s more great music I discovered this year but that wasn’t made this year.
I somehow ended up listening to a lot of Japanese indie music (for a lack of better term).
I just finished reading Thinking in Systems: A Primer by Donella H. Meadows. I believe sometimes one was to be “ready” to read a book. I must have been ready for this one because I had more “Holy sh!t” moments than any other book I’ve read in years.đ
Austin, TX, 8:15. Morning walk around the neighborhood. There are many good Halloween decorations, but this one is particularly entertaining. #adayinthelife
Given what happened to the other saguaro, it makes me fear that this one will fall too.
I took this photo of a saguaro a month ago. It had been partially burnt during the fires that happened north of Phoenix earlier this summer. This weekend I realized it had fallen. According to Wikipedia, saguaros grow their first arm around 75â100 years of age. đđ”đ
Yesterday I read a series of posts on custom URL scheme handlers on the Zettelkasten forums. The handler registers itself to open links like zettel://202006061337
, where the number is the unique ID of a zettel (here it’s a timestamp). I’m not sure everyone realized the magnitude of what that means.
Combining a custom URL with a unique ID means notes and links can become entirely independent from your apps. Only the handler needs to know about the apps you’re using.
That alone is very nice, but then I thought: URLs can have query parameters⊠That means I can have URLs like zettel://202006061337&edit
that open in my text editor of choice: TextMate, BBEdit, WriteRoom, FoldingText, etc. Or zettel://202006061337&preview
to open in Marked. Or I could even pick the app interactively with zettel://202006061337&pick
.
Handling the ID alone is pretty easy since the common Zettelkasten-like apps respond to a URL scheme to search and open files. The Archive uses thearchive://match/ID
, nvUltra uses x-nvultra://find/ID
, and nvAlt uses nvalt://find/ID
.
But how to open in a text editor or in Marked given only the ID? With Spotlight. I used mdfind -name ID
to find the file. This could be further refined with the -onlyin FOLDER
option but I didn’t need it. Then it’s a matter of calling open -a Marked FILEPATH
.
There’s also a zettel://create
special case that will create a new zettel with the current time stamp (YYYYMMDDHHMM). It’s always done with the default Zettelkasten app because the script doesn’t know where to write the file but the app does.
I wrote the handler in Applecript because it’s the easiest way I know to create something that macOS considers an “app” and that can therefore handle URLs.
The full script is below. To use it:
appChoices
array.Contents/Info.plist
with a text editor<dict>
line. Replace zettel
with the URL prefix you’ve chosen.<key>CFBundleURLTypes</key>
<array>
<dict>
<key>CFBundleURLName</key>
<string>Zettel Link Opener</string>
<key>CFBundleURLSchemes</key>
<array>
<string>zettel</string>
</array>
</dict>
</array>
Here’s the full script:
-- Zettel Link Opener
-- Created by Alexandre Chabot-Leclerc
-- Source: https://alexchabot.net/2020/06/06/custom-url-handler-for-zettels/
-- URL Handler for zettelkasten unique IDs, e.g., zettel://202006061017
-- Handles options after the ID to open different apps:
-- zettel://202006061017&edit to open is a text editor like TextMate
-- zettel://202006061017&preview to open in a preview app like Marked
-- zettel://202006061017&pick to open a menu of apps to pick from
-----------------------------------------------------------
-- CONFIGURATION
-- URL prefix for your custom URL, e.g. zettel://ZETTEL_ID
property urlPrefix : "zettel"
-- Default URL to call to open a note with a given ID. The ID will be appended
property defaultZkAppUrl : "thearchive://match/"
--property defaultZkAppUrl : "nvalt://find/"
--property defaultZkAppUrl : "x-nvultra://find/"
-- URL to use to create a new zettle with the current timestamp YYYYMMDDHHMM
property urlForCreation : "thearchive://matchOrCreate/"
--property urlForCreation : "nvalt://make?txt="
--property urlForCreation : "x-nvultra://make?txt="
-- Apps to use for the different query parameters
property editApp : "FoldingText" -- App to used with "&edit" query parameter
property previewApp : "Marked" -- App to used with "&preview" query parameter
-- List of app to display in the menu with with &pick query option
-- The apps will appear in the order defined here
property appChoices : {defaultZkAppUrl, editApp, previewApp, "TextMate"}
property defaultApp : {defaultZkAppUrl}
-----------------------------------------------------------
on splitText(theText, theDelimiter)
set AppleScript's text item delimiters to theDelimiter
set theTextItems to every text item of theText
set AppleScript's text item delimiters to ""
return theTextItems
end splitText
on removeUrlPrefix(original)
-- Remove URL prefix so we're left with only the ID and the optional query parameter
return do shell script "echo " & quoted form of original & " | sed 's;" & urlPrefix & "://;;'"
end removeUrlPrefix
on getIdAndOption(resouceAndQuery)
-- Split the zettel ID and the optional parameter
-- For example 202006061012&edit or 202006061012&preview
set theItems to splitText(resouceAndQuery, "&")
if length of theItems is 1 then
-- Append an empty string if there's no option so this
-- function always returns an array of 2 elements
copy "" to the end of theItems
end if
return theItems
end getIdAndOption
on findFilepath(zk_id)
-- Finds the filepath using Spotlight.
-- It's easier than finding the proper filename given only the zettel ID
return do shell script "mdfind -name " & zk_id
end findFilepath
on createZettel()
set newZkId to do shell script "date +'%Y%m%d%H%M'"
do shell script "open " & urlForCreation & newZkId
end createZettel
on openInChoosenApp(zkId, zkFilepath)
-- From Simple List Handler by Patrick Welker <http://rocketink.net>
-- Promp the use for the app to use
set selectedApp to item 1 of (choose from list the appChoices with title "Available App" with prompt "Which app do you want to use?" default items defaultApp)
if selectedApp is false then
-- Exit prematurly if the user clicked Cancel
error number -128
end if
-- Open the URL directly, or open by app name
if selectedApp contains "://" then
do shell script "open " & selectedApp & zkId
else
do shell script "open -a " & selectedApp & " " & quoted form of zkFilepath
end if
end openInChoosenApp
on open location thisURL
set resouceAndQuery to removeUrlPrefix(thisURL)
set idAndOption to getIdAndOption(resouceAndQuery)
set zkId to item 1 of idAndOption
if zkId is "create" then
createZettel()
return
end if
set zkFilepath to findFilepath(zkId)
if item 2 of idAndOption is "edit" then
do shell script "open -a " & editApp & " " & quoted form of zkFilepath
-- Exit the script immediately so we don't also open in the default app
return
else if item 2 of idAndOption is "preview" then
do shell script "open -a " & previewApp & " " & quoted form of zkFilepath
-- Exit the script immediately so we don't also open in the default app(
return
else if item 2 of idAndOption is "pick" then
openInChoosenApp(zkId, zkFilepath)
return
end if
-- Fall back to the default handler if there was no option
-- or the option was invalid
do shell script "open " & defaultZkAppUrl & zkId
end open location
TLDR; Use %matplotlib
if you want interactive plotting with matplotlib. If you’re only interested in the GUI’s event loop, %gui <backend>
is sufficient.
I never really understood the difference between %gui
and %matplotlib
in IPython. One of my colleagues at Enthought once told me that at some point in his career, he more or less stopped reading documentation and instead went straight to the code. That’s what I did here. But let’s do a bit of history first.
In the “beginning”, there was pylab
. It (still) is a module of matplotlib and was a flag to IPython designed to facilitate the adoption of Python as a numerical computing language by providing a MATLAB-like syntax.1 The reference was so explicit that before being renamed to pylab
on Dec 9, 2004, the module was called matplotlib.matlab
. IPython adopted the rename on the same day.2 With the ââpylab
flag or the %pylab
magic function, IPython would set up matplotlib for interactive plotting and executed a number of imports from IPython, NumPy and matplotlib. Even thought it helped a few people transition to Python (including myself), it turned out to be a pretty bad idea from a usability point of view. Matthias Bussonnier wrote up a good list of the many things that are wrong with it in “No Pylab Thanks.”
For the 1.0.0 release of IPython in August 2013, all mentions of %pylab
were removed from the examples (in a July 18, 2013 commit) and were replaced by calls to the %matplotlib
magic function, which only enables interactive plotting but does not perform any imports. The %matplotlib
function had already been introduced in a 2013 refactoring to separate the interatice plotting from the imports. The %gui
magic command had already been introduced in 2009 by Brian Granger to “manage the events loops” (hint hint).
Now we know that the (my) confusion with %gui
and %matplotlib
started in 2013.
This analysis refers to IPython 7.8.0 and ipykernel 5.1.2.
Our entry point will be the %matplotlib
magic command. Its source code is in the IPython.core.pylab.py
file. The essential call is to shell.enable_matplotlib(gui)
, which is itself implemented in IPython.core.interactiveshell.InteractiveShell
, and does five things:
IPython.core.pylabtools.find_gui_and_backend(gui)
. It encapsulates the logic to go
from a GUI name, like "qt5"
or "tk"
, to a backend name, like "Qt5Agg"
and "TkAgg"
.IPython.core.pylabtools.activate_matplotlib(backend)
, which:
matplotlib.interactive(True)
;matplotlib.pyplot.switch_backend(backend)
;matplotlib.pyplot.draw_if_interactive
method with the same method, but wrapped by a flag_calls
decorator, which adds a called
flag to the method. That flag will be used by the new %run
runner that’s introduced below at point #5;IPython.core.pylabtools.configure_inline_support(shell, backend)
. This is where some very interesting stuff happens. It first checks that InlineBackend
is actually importable from ipykernel.pylab.backend_inline
, otherwise it returns immediately. But if it’s importable and the backend is "inline"
, it:
ipykernel.pylab.backend_inline.flush_figures
function, and register it as a callback for the "post_execute"
event of the shell. As we’ll see later, callbacks for "post_execute"
are called after executing every cell;"inline"
, it’ll unregister the flush_figures
callback;shell.enable_gui(gui)
. This method is not implemented in the IPython.core.interactiveshell.InteractiveShell
base class, but rather in IPython.terminal.interactiveshell.TerminalInteractiveShell
. If a gui
as specified, it gets the name of the active_eventloop
and its corresponding inputhook
function using IPython.terminal.pt_intputhooks.get_inputhook_name_and_func(gui)
. The active_eventloop
is just a string, such as 'qt'
, but the inputhook
is more interesting. It’s the function to call to start that GUI toolkit’s event loop. Let’s dig further into get_inputhook_name_and_func(gui)
. That function checks a few things, but it essentially:
inputhook
function for the chosen GUI by importing it from IPython.terminal.pt_intputhooks.<gui_mod>
. For example, the Qt inputhook
is imported from IPython.terminal.pt_intputhooks.qt
. Later on, when inputhook
is executed for Qt, it will:
QCoreApplication
;QEventLoop
for that application;tk
, wx
, or osx
, but they all essentially do the same thing. At this point we’re ready to go back up the stack to enable_matplotlib
in %matplotlib
;default_runner
with the one defined in IPython.core.pylabtools.mpl_runner
. The default_runner
is the function that executes code when using the %run
magic. The mpl_runner
:
matplotlib.interactive
state, and disables it;interactive
state;plt.draw_if_interactive.called
flag that was introduced at point #1.3 above.As for the other magic, %gui
, it only executes a subset of what %matplotlib
does. It only calls shell.enable_gui(gui)
, which is point #4 above. This means that if your application requires interaction with a GUI’s event loop, but doesn’t require matplotlib, then it’s sufficient to use %gui
. For example, if you’re writing applications using TraitsUI or PyQt.
%gui
and %matplotlib
Let’s start with the “simplest” one, %gui
. If you execute it in a fresh IPython session, it’ll only start the event loop. On macOS, the obvious effect of this is to start the Rocket icon.
At that point, if you import matplotlib and call plt.plot()
, no figure will appear unless you either call plt.show()
afterwards, or manually enable interactive mode with plt.interactive(True)
.
On the other hand, if you start your session by calling %matplotlib
, it’ll start the Rocket and activate matplotlib’s interactive mode. This way, if you call plt.plot()
, your figure will show up immediately and your session will not be blocked.
%run
If you call %run my_script.py
after calling %matplotlib
, my_script.py
will be executed with the mpl_runner
introduced above at point #5.
"inline"
BackendIn the terminal the IPython.terminal.interactiveshell.TerminalInteractiveShell.interact()
method is where all the fun stuff happens. It prompts you for code, checks if you want to exit, and then executes the cell with InteractiveShell.run_cell(code)
and then trigger the "post_execute"
event for which we’ve registered the ipykernel.pylab.backend_inline.flush_figures
callback. As you might have noticed, the flush_figures
function comes from ipykernel, and not from IPython. It tries to return all the figures produced by the cell as PNG of SVG, displays them on screen using IPython’s display
function, and then closes all the figures, so matplotlib doesn’t end up littered will all the figures we’ve ever plotted.
To sum it up, use %matplotlib
if you want interactive plotting with matplotlib. If you’re only interested in the GUI’s event loop, %gui <backend>
is sufficient._ Although as far as I understand, there’s nothing very wrong with using %matplotlib
all the time.
plot
, figure
, subplot
, etc.
[return]
Every year since 2017 I send my colleague a long email about ideas and media I’ve come across during that year. This is the 2019 edition. You can read the 2017 and [2018]() editions.
Hi all,
Time for installment #3 of my Holiday Newsletter! For once, I started a bit earlier collecting links and summarizing why things are interesting. Let me know if you’d be interested in receiving something like this more than once a year. It’s something I’m considering doing in 2020.
I hope you find some time to read/listen/watch something good over the break, even if it’s not from this list. :) Happy Holidays!
-Alex
The End of Bureaucracy by Gary Hamel and Michele Zanini at HBR is the most mind-blowing thing I’ve read this year. It features a Chinese company named Haier, which is the worlds largest appliance maker with $35 billon/year in revenue. The business is structured around 4,000 independent microentreprises (MEs) made of 10 to 15 employees. Each ME provides different of services to other MEs: marketing, R&D, design, manufacturing, HR, etc. Each ME is free to contract with any other ME, or go to an external provider if they don’t think they’d be well served internally. It’s a setup that flies in the face of pretty much any other large company. Definitely worth a read.
Dan Barber (chef and co-owner of Blue Hill at Stone Barns) writes Save Our Food. Free the Seed. at the NYT. It’s an inspiring, and also infuriating, article about seeds and how they’re at the root of our entire food supply chain. I didn’t realize that seed “designers” make their seeds work hand in hand with insecticides, which are often made by the same companies (e.g., Monsanto). That wouldn’t matter too much if we weren’t down to just two mega seed producers, and if the seeds weren’t patented.
I still haven’t read any of Michael Lewis’ books (somehow), but every time I read one of his articles I’m completely sucked in. Here are two good ones I read this year. In Jonathan Lebed’s Extracurricular Activities, Lewis tells the story of a 15-year old “stock manipulator.” In Portrait of an Inessential Government Worker talks about Art Allen, the only oceanographer inside the U.S. Coast Guardâs Search and Rescue division, and the person who’s basically single-handedly responsible for mankind’s ability to find things lost a sea. Lewis also started a podcast this year called Against the Grain. Recommended.
In Stab a Book, the Book Won’t Die Craig Mod (real name) speaks of different forms of content and their respective “contracts” with their consumers. Books have edges, a clear “I buy you, I read you or I don’t” contract. Online publications, like Netflix, Twitter, and Instagram don’t have such clear contracts. They’re infinity pools. I really liked his observation that with print newspapers, only the front page (often just above the fold) needs sensational headlines to get people to buy it. Internal headlines can afford to be factual. Online, all headlines are on the front page.
Cormac McCarthy, in his “free time”, does editing for faculty and post docs at the Santa Fe Institute. Van Savage and Pamela Yeh publish McCarthy’s advice on writing scientific papers at Nature. It’s equally good advice for writing documentations and reports.
The Japanese addressing system is strange and wonderful. When written in Japanese, the addresses start with the largest geographical entity and proceed to the most specific one.
Someone who’s name I can’t figure out writes Becoming a magician, with an interesting approach to “personal growth”: surround yourself with people who feel to you like they’re magicians.
Julia Evansâ What does debugging a program look like? is a great read about the multiple ways to approach debugging. Julia started writing zines full-time this year. If youâre interested in having a look, I have one on my desk. And on the topic of debugging, Greg Wilson recommends Why Programs Fail: A Guide to Systematic Debugging by Andreas Zeller.
In Open Source is not about you, Rich Hickey, creator of Clojure, writes a wonderful article about expectations one should have about open source software (I still find strange people who âblogâ using Github repos and Gists…)
As a user of something open source you are not thereby entitled to anything at all. You are not entitled to contribute. You are not entitled to features. You are not entitled to the attention of others. You are not entitled to having value attached to your complaints. You are not entitled to this explanation.
If you have expectations (of others) that arenât being met, those expectations are your own responsibility. You are responsible for your own needs. If you want things, make them.
And on a related topic, License Zero has a thoughtful look at funding models for open source.
I award “Data Structure of the Year” to Conflict-free Replicated Data Types (CRDTs). I learned about them through Ink and Switchâs article Local-first software: You own your data, in spite of the cloud, where they use them to write a real-time syncing engine that doesnât require a central server. But then, in pure BaaderâMeinhof, it seemed like everyone was talking about them, e.g., the Xi editor, and Figma.
Basecamp published Shape Up, the âmanualâ for their software development process. Itâs a 1-hour read that I highly recommend. Itâs a variation on Agile, with thoughtful additions, structures, and rationales for why the do things in certain ways.
Lowtech Magazine hosts their website on a solar-powered Raspberry Pi. Warning, their site might be down when you visit if it’s been cloudy for a while in Barcelona. Roel Roscam Abbing wrote about the technical details.
Jakob Nielsen explains Why You Only Need to Test with 5 Users.
Here are some people I discovered this year. They are so inspiring, and also so frustrating, because they make you wonder “And what have I done today?”
My “Podcast of the Year” is the The Amelia Project, a fiction podcast about Amelia, an organization that helps clients fake their death and then reappear as whatever they want. It’s original, beautifully produced, endearing, and funny. Do start with the 1st episode, and go get yourself a hot cocoa. And if you’re new to fiction podcasts, check out Limetown, and Passenger List.
Michael Lopp (author of Managing Humans, which was in last year’s email) has a podcast called The Important Thing. If you’re interested in management and leadership, I recommend the The One About Management (Pt. 1) and The One About Management (Pt. 2) for a good discussion of what it means and what it’s like to be a manager.
The Nevermore, Amazon episode of Rework (by Basecamp), talks about about the economics of bookstores, and how selling new hardcovers for $15 (like Amazon does) would put them out of business. Independent bookstores are the best. If you prefer the convenience of shopping online but want to support independent bookstores, use Biblio. Cool fact: they offset the carbon emissions of their shipping.
For the Austinites, and people visiting Austin, go eat at:
Make bread at home. It’s easy and it’s better than anything you can get in Austin (except maybe from Sour Duck, and from the late Miche Bread). Just buy Flour Water Salt Yeast by Ken Forkish, and make the Saturday Overnight White bread. Bring the extra loaf to the office, please.
It was a good year. It’s always a good year. Here’s a playlist of every song I “liked” in 2019, and one of every album. The music on those playlists wasn’t necessarily released in 2019 though. Just to name a few albums that were release in 2019:
Woman at War tells the story of a choir conductor and eco-activist who plans to disrupt the operations of an aluminum plant in Iceland, but whose plans are temporarily interrupted by the opportunity to finally adopt a child from Ukraine. It’s funny, beautiful, and strange.
The Marvelous Mrs. Maisel and Killing Eve were the two best series we watched this year. One great quality of Killing Eve is that it ends, in only two seasons.
Simple Made Easy by Rich Hickey. Simple is the not the same as easy.
In his 2019 State of Mozilla talk, Mike Hoye makes this wonderful point about caring about making it possible to care. Paraphrasing from his talk: There’s no Keurig machine in the Toronto office because there’s no amount of care that will make it produce good coffee. I like that a lot (coffee, but also this statement).