nathanherald.com feed

Talib Kweli & Hi Tek (Reflection Eternal) - The Blast

Sat, 19 Sep 2026 20:43:00 +0200

TIL about Solid and FOAF

Fri, 26 Jun 2026 10:56:07 +0200

TIL about Solid and FOAF

I've had a note sitting in my reminders for a few days: "write something about that solid thing and linked data and foaf." So I finally dug in. As I wrote about recently, I’m very interested in RDF-* and linked data and Solid and FOAF both come up in my searches every time. So…

Solid

Solid (Social Linked Data) is Tim Berners-Lee's project to fix the part of the web where your data lives inside someone else's box. Your stuff goes in a pod which is a little datastore you own and “apps” get decoupled from the data. So instead of an app holding your data hostage, it asks your pod for permission to read it. Switch apps, keep everything. Flip a switch, revoke access.

Which, yeah. Obviously we should own and control our own data. So cool idea I guess.

It's RDF underneath

The data in a pod is just RDF which is the same subject/predicate/object triples from my recent post. Solid doesn't reinvent how data is shaped, it only seems concerned about the container for the data.

RDF and linked data are about what your data means and how it connects. Solid is about where it lives and who gets to read it. A pod is just somewhere to keep your slice of the big graph, with a permission system bolted on.

“Linked data” is the glue: triples whose subjects and objects are URLs, and those URLs resolve to more triples. You follow from one fact to the next across the web.

FOAF

FOAF (Friend of a Friend) is the oldest real example of linked data about people. It’s a tiny RDF vocabulary: foaf:Person, foaf:name, foaf:knows, foaf:homepage. Dan Brickley and Libby Miller started it around 2000. Probably the first social-semantic-web thing.

The vocabulary's been basically frozen for two decades and the namespace still resolves. And it's what Solid leans on under the hood. A Solid identity is a WebID, and a WebID profile is more or less a FOAF file describing you.

So FOAF is the whole idea shrunk to its smallest form. No pods, no server, no protocol. Just a file about you, linking to files about your friends.

A real Solid use case

In Flanders, the public data company Athumi put citizen data on Solid pods. The first real use case was: you share your diploma with a recruiter on consent, for whatever scope and time window you pick. Before, only the government could pull your diploma from the official database. With a pod, you can hand it over yourself.

And it’s still going. They’ve extended the same platform to digital student certificates through the itsme app, with a student app called Dibbs as the first to use it.

Cool idea and I'm not sure I'd ever use it

Even at that project in Belgium: their own research found people were confused by WebIDs and pods and that the main problem was explainability. Which is why every shipping version hides Solid completely behind a normal-looking app like itsme.

FOAF me

I’ve published a FOAF file, which looks kinda like this:

@prefix foaf: <http://xmlns.com/foaf/0.1/> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .

<#me> a foaf:Person ;
    foaf:name "Nathan Herald" ;
    foaf:homepage <https://nathanherald.com/> ;
    foaf:knows [
        a foaf:Person ;
        foaf:name "A Friend" ;
        rdfs:seeAlso <https://afriend.example/profile.ttl>
    ] .

It lives at /profile.ttl, so https://nathanherald.com/profile.ttl#me is my WebID. That #me fragment matters: the file is a document, and I'm a person it describes.

So no pod for me. Just a little file and I’m now a little node in the web of people, pointing at my friends. Pretty cool.

"The Taste of Tea (茶の味) Trailer"

Sun, 21 Jun 2026 10:51:46 +0200

John Thomson’s China

Sat, 20 Jun 2026 09:20:33 +0200

It’s amazing to see these photographs from so long ago.

I’m tired of talking to AI

Sat, 30 May 2026 13:48:56 +0200

I’m tired of talking to AI. I want to talk to real people. But even when I talk to people, they forward my questions to AI and send me the AI’s answer.

Yep.

"Reel: Overview Walkthrough (Part 1)"

Sat, 16 May 2026 16:43:35 +0200

An awesome 4-track recorder for iOS.

Open House

Sat, 16 May 2026 15:17:57 +0200

📍 Coffee in Berlin.

Practical example of how cool RDF is

Fri, 15 May 2026 15:21:24 +0200

Imagine we’ve ingested a ton of maritime data. (I couldn’t come up with a better example today.)

Someone asks us “Can shipment 8821 go through the Suez Canal?”

Before: someone has to query different systems for regulations, jurisdictions, routes, cargo manifests, and permits. JOIN them if possible. Paste them into an LLM to synthesize, and hope the answer is complete.

What we could do: we can ingest all our data into a semantic knowledge system and let an agent query through at the semantic level.

Our agent might get a list of tools it can call:

Lots of JSON

{
  "actions": [
    {
      "name": "CanTransitAction",
      "description": "Determine whether a shipment may transit a jurisdiction under active maritime policy.",
      "input": {
        "type": "object",
        "required": ["shipment", "jurisdiction", "validTime"],
        "properties": {
          "shipment": {
            "type": "string",
            "description": "IRI of the shipment entity.",
            "examples": ["cargo:shipment-8821"]
          },
          "jurisdiction": {
            "type": "string",
            "description": "IRI of the jurisdiction entity.",
            "examples": ["jurisdiction:egypt-suez"]
          },
          "validTime": {
            "type": "string",
            "format": "date-time",
            "description": "The real-world time for which the transit decision should be evaluated."
          }
        },
        "additionalProperties": false
      },
      "output": {
        "type": "object",
        "required": ["type", "allowed", "reason", "citesClaim", "usedPolicy"],
        "properties": {
          "type": {
            "const": "CanTransitResult"
          },
          "allowed": {
            "type": "boolean"
          },
          "reason": {
            "type": "string"
          },
          "citesClaim": {
            "type": "array",
            "items": { "type": "string" }
          },
          "usedPolicy": {
            "type": "string"
          }
        },
        "additionalProperties": false
      }
    },
    {
      "name": "ResolveEntityAction",
      "description": "Resolve a human phrase like 'shipment 8821' or 'Suez Canal' into a substrate IRI.",
      "input": {
        "type": "object",
        "required": ["mention", "expectedType"],
        "properties": {
          "mention": {
            "type": "string",
            "description": "The phrase supplied by the user.",
            "examples": ["shipment 8821", "Suez Canal"]
          },
          "expectedType": {
            "type": "string",
            "description": "The expected entity type.",
            "enum": ["Shipment", "Jurisdiction", "Port", "Vessel", "Permit", "CargoType"]
          }
        },
        "additionalProperties": false
      },
      "output": {
        "type": "object",
        "required": ["status"],
        "properties": {
          "status": {
            "type": "string",
            "enum": ["resolved", "ambiguous", "not_found"]
          },
          "iri": {
            "type": "string",
            "description": "Resolved entity IRI when status is resolved."
          },
          "confidence": {
            "type": "number",
            "minimum": 0,
            "maximum": 1
          },
          "matchedBy": {
            "type": "array",
            "items": { "type": "string" },
            "examples": [["exact shipmentId"], ["label", "alias"]]
          },
          "citedClaims": {
            "type": "array",
            "items": { "type": "string" }
          },
          "candidates": {
            "type": "array",
            "description": "Candidate entities when status is ambiguous.",
            "items": {
              "type": "object",
              "required": ["iri", "label", "confidence"],
              "properties": {
                "iri": { "type": "string" },
                "label": { "type": "string" },
                "confidence": {
                  "type": "number",
                  "minimum": 0,
                  "maximum": 1
                }
              },
              "additionalProperties": false
            }
          }
        },
        "additionalProperties": false
      }
    }
  ]
}

The CanTransitAction looks correct, but the agent needs to first resolve the entities from plain text to IDs first:

action(%{
  action: "ResolveEntityAction",
  expectedType: "Shipment",
  mention: "shipment 8821"
})

action(%{
  action: "ResolveEntityAction",
  expectedType: "Jurisdiction",
  mention: "Suez Canal"
})

Then it can try to run the action:

action(%{
  "action": "CanTransitAction",
  "shipment": "cargo:shipment-8821",
  "jurisdiction": "jurisdiction:egypt-suez",
  "validTime": "2026-05-12T00:00:00Z"
})

Pretty straightforward and it should get back an answer.

But what is actually in our “database” though?

The claims we have might look like:

# @prefix declarations left out on purpose

GRAPH :CargoTransitPolicy-v12 {
  :CargoTransitPolicy-v12 a :TransitPolicy ;
    :blocksCargoType :SanctionedGoods ;
    :requiresPermit [
      :forCargoType :HazardousCargo ;
      :throughJurisdiction :EgyptSuez ;
      :permitType :SuezHazmatPermit ;
    ] .
}

GRAPH cargo:claim-100 {
  cargo:shipment-8821 a onto:Shipment ;
    onto:contains cargo:diesel-fuel ;
    onto:origin port:singapore ;
    onto:destination port:rotterdam .
}

GRAPH cargo:claim-101 {
  cargo:diesel-fuel a onto:HazardousCargo ;
    onto:hazardClass "flammable-liquid" .
}

GRAPH route:claim-200 {
  route:singapore-to-rotterdam onto:transitsThrough jurisdiction:egypt-suez .
}

GRAPH reg:claim-300 {
  jurisdiction:egypt-suez onto:allowsTransitOf onto:HazardousCargo ;
    onto:requiresPermit permit:suez-hazmat-permit .
}

GRAPH permit:claim-400 {
  cargo:shipment-8821 onto:hasPermit permit:suez-hazmat-permit-778 .
}

We would likely persist “quads” which is a “triple” + named graph. (RDF Quads has a nice graphic that kinda helps illustrate this.)

A triple is the smallest assertion in our system. A triple is always subject, predicate, and object. Nathan → hasEmail → <myobie@duck.com>.

That last quad in the above .trig document says: “shipment 8821 has permit 778.” By itself, it does not say what the permit allows. The meaning comes from combining it with the other claims: Suez requires a hazmat permit for hazardous cargo, the shipment contains hazardous cargo, and the shipment has the required permit. Being able to combine these claims lets us build up a world of facts which we can then answer questions from.

A quad is the smallest stored assertion: subject, predicate, object, and named graph. In this type of system, a "claim" is typically a named graph / claim packet containing one or more quads plus metadata such as provenance, policy, validation, and signatures. The subjects and objects are nodes. The predicates are the edges.

GRAPH names a collection of triples inside an RDF dataset. The graph name itself is an IRI, so the system can make statements about the graph elsewhere. A line ending in a ; means the next line still applies to the same subject. A line ending in . means that’s it for this subject.

Actions are also claims in the database:

GRAPH :CanTransitAction {
  :CanTransitAction a :ExecutableAction ;
  :inputShape :CanTransitActionShape ;
  :outputShape :CanTransitResultShape ;
  :runtime "elixir" ;
  :mfa "Maritime.Actions.CanTransit.run/2" ;
  :requiredCapability :CanEvaluateTransitPolicy ;
  :usesPolicy :CargoTransitPolicy-v12 .
}

GRAPH :ResolveEntityAction {
  :ResolveEntityAction a :ExecutableAction ;
    :inputShape :ResolveEntityInputShape ;
    :outputShape :ResolveEntityResultShape ;
    :runtime "elixir" ;
    :mfa "Substrate.Actions.ResolveEntity.run/2" ;
    :requiredCapability :CanResolveEntities .
}

And here is some elixir pseudo-code that the persisted MFA might point to:

`Maritime.Actions.CanTransit`

defmodule Maritime.Actions.CanTransit do
  @moduledoc """
  Evaluates whether a shipment may transit a jurisdiction.

  Input shape:  :CanTransitActionShape
  Output shape: :CanTransitResultShape
  Policy:       :CargoTransitPolicy-v12
  """

  def run(input, ctx) do
    projection =
      ctx.open_projection!(
        ontology: "maritime-compliance-v4",
        valid_time: input.validTime,
        capability_envelope: ctx.capabilities
      )

    shipment = projection.entity!(input.shipment)
    jurisdiction = projection.entity!(input.jurisdiction)

    cargo_types = projection.types_for_contained_cargo(shipment)
    permits = projection.permits_for(shipment)

    required_permit =
      projection.required_permit_for(
        cargo_types: cargo_types,
        jurisdiction: jurisdiction,
        policy: ":CargoTransitPolicy-v12"
      )

    cond do
      required_permit && required_permit not in permits ->
        %{
          type: ":CanTransitResult",
          allowed: false,
          reason: "Shipment requires #{required_permit} for transit through #{input.jurisdiction}.",
          citesClaim: [
            "cargo:claim-100",
            "cargo:claim-101",
            "route:claim-200",
            "reg:claim-300"
          ],
          usedPolicy: ":CargoTransitPolicy-v12"
        }

      true ->
        %{
          type: ":CanTransitResult",
          allowed: true,
          reason: "Shipment may transit because all required permits are present.",
          citesClaim: [
            "cargo:claim-100",
            "cargo:claim-101",
            "route:claim-200",
            "reg:claim-300",
            "permit:claim-400"
          ],
          usedPolicy: ":CargoTransitPolicy-v12"
        }
    end
  end
end

The result of the action can itself become a claim:

GRAPH :result-900 {
  :result-900 a :CanTransitResult ;
    :allowed true ;
    :reason "Shipment 8821 may transit because hazardous cargo is allowed with the required Suez hazmat permit." ;
    :citesClaim cargo:claim-100 ;
    :citesClaim cargo:claim-101 ;
    :citesClaim route:claim-200 ;
    :citesClaim reg:claim-300 ;
    :citesClaim permit:claim-400 ;
    :usedPolicy :CargoTransitPolicy-v12 .
}

So it’s claims all the way down.

By making everything in the system a claim we build up a world that is coherent and governed. Claims can describe policies and capabilities, and claims can also be governed by policies and capabilities. Which means every actor can only move through the substrate with their given capabilities: human or agent.

And to take it a bit further: an agent can “tend the garden of shapes” to help keep things tidy and coherent. If the system were built correctly, everything an agent does would be reviewable and reversible. Then all queries can operate at a well curated semantic layer and return real, trustable results.

This maritime example might seem far from your use cases, but this is applicable even for very personal data. Scanning PDFs and emails of all your medical records to build up the facts of your medical history, household knowledge base, job search / placement projects, etc. Pretty cool.

Mastodon kinda hates AI

Thu, 14 May 2026 11:51:57 +0200

Mastodon, as a network, leans pretty strongly anti-AI. Not just the people I follow, but the network overall. It’s something I see every day, but today I saw a very clear single example.

1Password posted a link to a blog post about how they use AI for backend dev on Masto which showed up in my feed and I immediately noticed the amount of negative replies: 42 replies, 41 of which were negative. Here is a representative slice, paraphrased so I’m not singling anyone out:

“No. Just no.”
Equating any AI use with deliberately introducing security flaws.
“Stochastic parrots.” “AI slop.” “Grey goo.”
“The ideal agent-to-human ratio is 0.”
A lot of “I’m canceling my subscription” / “guess I need to find a new password manager.”
“Set itself on fire in the race to shove AI into everything.”

I went to see if they posted that blog link anywhere else. They posted the same blog post link on LinkedIn and on X, but not on their Threads or Bluesky accounts. There are a lot fewer replies on both and they are mostly just engaging with the topic or asking for support.

I mainly read through Mastodon to learn about cool tech stuff + a lot of my real-life friends are on there, but Masto seems to be getting more anti-tech in general these days and I am finding less cool stuff there recently.

I find Threads is better for photography and general interest. How weird that photographers have migrated over to Threads from Instagram since Insta is all about Reels now…

The norms building up inside the post-twitter text-based social networks are fascinating to watch. I honestly never would have assumed a large network could have visible biases like this.

Is there an online space that is pro-“tech to help people”? Or pro-this-stuff-is-cool-check-it-out? Or just generally biased towards constructive convos?

"1990: Whatever Happened to the Cameras of the Future? | Tomorrow’s World | BBC Archive"

Sun, 10 May 2026 21:31:35 +0200

So fun to see.

"Ozu in Color – TCM x Every Frame a Painting | TCM Originals"

Thu, 07 May 2026 18:37:54 +0200

We are so lucky that Every Frame a Painting is back.

Principles matter more with agents

Tue, 05 May 2026 00:13:18 +0200

Whether working with humans or robots, it always helps to have your core principles documented and defined. I find most programmers think the specific architecture, data structures, or the language matters most and should be well defined for all projects… but I disagree.

What matters to me are the core principles and conventions. How do we agree we can read from state safely? How do we agree to mutate state, if at all? What sort of reactivity is important to us and how do we agree to use reactivity in our UI and elsewhere? Where should we keep shared code? How important is sharing code vs. isolation? What defines a unit test? Can unit tests use the disk, the database, the network? How do we avoid mocking everything so we know our app works without having slow tests? Can we tolerate any failing or skipped tests?

If these things aren’t defined, then it doesn’t matter if you choose go or javascript, your app will most likely be buggy and not understandable.

It’s easier to talk about an example to show what I mean.

When building an iOS app with an agent, I find setting some base principles around state to be the most important.

All visible state must be exposed through AppState which must be @Observable. AppState must contain and expose literally all visible state, no exceptions. If it is shown, it must be in AppState.

This takes care of how we read from state to present it, how that state is reactive, and helps us pre-decide a ton of things. Inside AppState we keep state as simple structs and enums. If we want to debounce or rate limit state updates, we debounce or rate limit assignments to parts of AppState, we don’t mess with how @Observable works. Etc. Also, AppState is fully unit testable. We can setup and test any version of AppState in a test, which means we can test any version of our app that the user would see. And creating Previews from AppState will be super straightforward.

State can only be mutated by methods on AppActions, which has init(appState: AppState). By convention, nothing else can mutate state held by AppState. All methods on AppActions must be unit tested.

AppActions should be super easy to unit test. It should be easy to setup any version of an AppState and then test how a method of AppActions changes it. Which leads me to…

The UI can only call methods on AppActions. So views must take init(appActions: AppActions) or use @Environment.

Since AppActions is super unit testable and the UI can only ever call methods on AppActions, then >90% of what the UI does can be unit tested. You can be confident most of the app works just from the tests.

This is super super important when working with others you don’t trust. And a coding agent is a developer you cannot trust 😅

All things like date formatters, string formatters, etc must be methods on AppHelpers. All methods on AppHelpers must be unit tested.

This should make sense.

Local first, offline first all the time.

Don’t ever allow a coding agent to start building some sort of online only feature. If you want to show something from a remote server, expose it via AppState. Then have a way to read from disk into AppState. Then have a way to fetch from the network and write to disk. And now you can hit the remote server, write to disk, assign into AppState, and the UI reacts automatically and everyone is good. Leave the old version cached on disk if at all possible. Don’t revert to an empty state while fetching. Keep some sort of isFetching in AppState if you need to. Even better: model your state machine with an enum so you can transition through different states which drive the UI. And unit test everything.

Even with strong principles defined, you have to review the agent’s work regularly and make sure it’s still adhering. Drift happens. An agent will start mocking things it shouldn’t, mutating state from outside AppActions, or sneaking in some online-only call. Catch it early and pull the work back in line.

This is enough to get started. And sure, one could say SwiftUI only, Swift 6.2 only, MainActor by default, and one can even start talking about data modeling and schemas… but those should be defined in a different file.

The above, and more, should go into a PRINCIPLES.md file that doesn’t try to be a plan or a spec, but instead can help us build a reliable, resilient app from any plan or spec.

Also, don’t just go from PRINCIPLES.md to SPEC.md, write an IDEA.md and make sure it’s clear that the idea is flexible, it may change as we iterate. Document the problem + the big idea, those likely won’t change. The plan will.

Locking into a super detailed spec too early can prevent you from discovering the best solutions to your problems. This is the same when working with humans or coding agents.

Instead define your core principles, your big idea, the problems you are trying to solve, and then maybe some light specs.

"Blackwater Holylight – Full Performance (Live on KEXP)"

Wed, 29 Apr 2026 22:28:05 +0200

🎧

"Casio SXC-1 portable sampler"

Thu, 23 Apr 2026 19:05:40 +0200

Let’s go! So cool.

Making Chrome livable

Sun, 19 Apr 2026 20:29:19 +0200

I’ve used Safari consistently since it came out. I’ve only maybe used something different like OmniWeb or Camino or Firefox or whatever, but always returning to Safari.

Enter vertical tabs in Chrome. I’ve wanted vertical tabs in my browser since OmniWeb. Chrome’s version isn’t perfect, but it is very good. Combined with split views in windows and I am a happy camper. I’ve been using Chrome Beta to get access to the vertical tabs (which should be shipped out in regular Chrome now) for weeks and I love them.

But there are 3 things I miss from Safari and I had to find a way to fix them.

Right click, Translate. Fixed with my own extension. Chrome wants to translate a whole page, but it misses iframes, dynamic content, etc. I almost always only want to translate a small part and in Safari I can select, right click, and Translate. Well, now I can do the same in Chrome using my own extension.
When a link opens a new tab in Safari, the back button is enabled and click back closes the new tab and returns to the original tab. Apparently this is something I use 100s of times a day and it has driven me crazy. So I made a back to opener chrome extension that does exactly this. This has made things sooo much better for me.
I miss 1Blocker. It’s my fav content blocker and I want to have it in Chrome as well. So I built a python script to analyze the rules inside 1Blocker.app and churn out a custom chrome extension that contains basically the same rules. After some trial and error, it works!

Fixing these three things has made Chrome really livable for me. They also have synced tab groups, just like Safari, so that works too.

Chrome still doesn’t feel like a Mac app, it feels like a Linux app running on my Mac, but it is much closer to my ideal browser now. I’ve been daily-ing Chrome for a few weeks and I anticipate keeping it as my default browser for many more.

Same beans, different cups

Sun, 19 Apr 2026 16:39:31 +0200

I was hunting for a Melodrip and couldn’t find one in Europe without paying 40 to 50 bucks for shipping. Kaffebox had one, and while I was there I saw the Nordics Best Roaster tasting set and added it to the cart too.

Ten 50g tubes, one per finalist, all roasted from the exact same green beans. I’ve been running them through my Hario Switch, V60 style.

The fruitiness is all over the map. Some are explosively fruity, others hardly fruity at all. The roaster’s share of the final taste is way bigger than I would have guessed.

First tasting set for me. The Sourced Coffee set is next, where each roaster picks their own beans. Can’t wait.

"The Film Everyone Passed On | The ROBOCOP Commentary"

Fri, 17 Apr 2026 17:41:38 +0200

Let’s go!

The Affordable Car Is Dead

Wed, 15 Apr 2026 10:59:11 +0200

The average transaction price for a new car now sits around $50,000.

Today, Detroit doesn’t need the average American to buy a car; better to sell affluent households their second S.U.V.

"Bill Withers – Ain’t No Sunshine (BBC In Concert, 1974)"

Mon, 13 Apr 2026 15:38:55 +0200

"THE BLADE (1995): The Edges of Wuxia"

Sun, 12 Apr 2026 11:40:57 +0200

Always great to see another Every Frame A Painting, it makes a ton of sense for Criterion to work with Tony, and the comparison of kung fu movies to westerns is so good!

How I use AI daily, April 2026

Sun, 12 Apr 2026 12:30:43 +0200

I keep a list in Reminders for every project I work on. When I sit down in the morning I scan through and pick what to work on. Usually I make a git worktree for the task, spin up claude code, and ask it to propose how best to achieve what I’m after. Then I decide: am I writing code or is the agent spiking it out?

All of my work sessions run inside pty, a tool I built for persistent terminal sessions. I can detach when I switch or take a break, reattach later, and the conversation is right where I left it and reminding me what I was doing. I jump between pty sessions throughout the day as things need attention – even remotely from my MacBook Air and iPhone.

Some days I have ten things going like a reactor pattern, bouncing between sessions all day. Some days I go deep on one thing and only touch the other sessions during breaks. I let the work decide the shape. It takes real mental energy to context switch this much, like playing multiple chess boards at once. But the sessions are patient. They will be there when I have brain space freed up.

I rarely run codex directly these days, I use it as a sub-agent. I have a skill configured in claude code to let claude drive codex as a sub-agent. Codex is better at unwinding very complex code or noticing nuanced things, so I let claude dispatch to it and report back the results. My main context stays at the project level and most of the heavy work happens in sub-agents underneath.

While I started, like everyone else, trying to build up docs and skills and workflows to prevent claude from making mistakes, I’ve ended up relaxing almost all of it. I’ve removed most of my CLAUDE.md files. Why do I need docs for the robot and not for me? Now I document things for myself and the robot can read those when it needs to.

Everything is context management now. The transformer architecture computes attention weights across the entire context to decide what to generate next. Every token in the context influences the probability distribution of the next token. So when you carefully set up the context, clear problem statement, relevant code, good examples, prior conversation, you are literally shifting the probability distribution toward the output you want.

You are not actually talking to claude. You are attempting to drag the weight of the context into the right starting point so it generates next tokens appropriately to solve your problem. A longer, more detailed context is heavier to drag but lands more precisely. A short vague prompt is light but lands anywhere. The skill is knowing how much context to load and what kind.

Context engineering is real and it is a skill, just like some people are better at googling than others.

I wrote about this on my AI page, but the short version: I run every project at the level of automation that fits it.

Tier 1, Hands Off. I am the CEO, the agent is the developer. I never read the code. My personal journal app, my podcast listening app, a handful of CLI tools. The code is not the artifact. The experience of using it is.

Tier 2, Hands On. Anything that ships to other people. new.space is in this tier. My cofounder and I review everything carefully. Sometimes I write the code, sometimes the agent spikes it, sometimes we throw the spike away and start over properly.

Tier 3, Hands Only. I write near 90% of the code myself because I want to learn. The agent watches, answers questions, handles tedium.

My side projects in Tier 1 simply would not exist. A podcast listening app, a personal journal, an image EXIF data manager. I described what I wanted, the agent built them, and I use them every day. I have never looked at the source code for any of them.

For new.space we have a JavaScript file that runs on the page in Safari when someone uses the iOS Share Sheet. It extracts metadata from the page, creates a readable version, handles all the edge cases. Building and testing across dozens of different sites would have been too much effort for our small two-person team, but letting claude hack on it and prove to us it was working made it a project we could actually do. A real increase in quality of life for our users.

Today I was working on a potential approval flow for dangerous tool calls in new.space (like "send an email for me"). I pushed claude to build it and use xcodebuildmcp to actually test it in the iOS Simulator and prove to me that it works. Then I can try the UI concept myself, give feedback, let it go again. Later today I will test the prototype on my real phone and decide if this is even the direction I want to go.

I do not have to waste time getting the details right just to figure out if the direction is correct. I can have a tireless agent implement it, try it, then decide. And I always have the agent document what it learned and what it would do differently if it had the chance, so if I want to start over I am not starting from zero.

I am building a web crawler and search engine in elixir by hand. I want to combine it with a full download of Wikipedia for a local offline agent. Crawling millions of sites with a Mac Mini is very possible these days, and with an external drive I think I can easily get to hundreds of millions of pages. I will seed it with sites I trust and it crawls out from there so the inputs are higher quality than trying to index the entire internet.

I have built a couple of prototypes for my own agent harness, but nothing good yet. One is in Swift. I think building in a native language and talking to local models natively (not over HTTP) might be a huge advantage, but it limits extensibility and contributions from others.

claude code is my favorite harness right now, but it does a lot of things I probably do not need for every task and might not be optimized for a smaller local model.

For running local models, we built SHLLM, a Swift library for talking to models directly. I also use ollama for models too big for my current machine. Qwen 3.5 and Gemma 4 are both genuinely good locally. I plan to get an M5 Mac Studio later this year so I can run larger models myself.

My goal: a local-first coding agent that handles 80 to 90 percent of my day-to-day by the end of this year.

I enjoy programming, but it is not important to me to preserve it as "my job." Typing has never been the goal. Anyone who thinks they should fight for the right to type is not on my side. I am fine with someone else typing. I am not 100% thrilled about being a manager of robots, but it is possible to make it fun and keep it lively with a bit of enthusiasm.

The worst things that happen with agents are when they touch git. For most projects I handle all the git stuff myself. I do not let the robot do it unless it is a throwaway project. Git is hard for most humans and I would not trust them to use it for me either.

I have been pretty clear eyed about AI. I am always skeptical of claims until I see evidence. When I saw claude code when it first came out I knew I needed to try it myself. And when it was able to do a few things I asked it to do, completely by itself, I knew I could mold it into a lot of other projects.

A random text generator running fast enough can do anything with a verification loop. Monkeys on typewriters writing Shakespeare. The verification loop is what makes it real.

This is how I use AI today and I expect things to evolve and change as the tools and models evolve and change. If someone is telling you that AI can’t do anything useful, ignore them. If someone is telling you that AI can do everything, ignore them. Both types of people do not have your best interest in mind.

Try one of the high quality AI coding agent harnesses with a capable model and see for yourself. Be aware: free or low paid tiers are not good enough yet. You are getting worse models running on worse hardware with worse limits. This is unfortunate right now and part of why I am pushing to build a local-first, offline-first setup for myself that is capable.

Too Old Not To

Fri, 10 Apr 2026 13:07:21 +0200

Inspiring.

Artemis II Launches to the Moon

Wed, 01 Apr 2026 13:24:33 +0200

Humans are heading back to the Moon. How incredible.

airdrop CLI

Sat, 28 Mar 2026 13:09:14 +0100

I made a tiny macOS CLI that opens the AirDrop sharing flow from the terminal. Just airdrop file.txt and the system share sheet pops up.

It’s a few lines of Swift, built with SwiftPM. Install it with ./install.sh and it drops a binary into ~/bin/airdrop.

airdrop /path/to/file
airdrop /path/to/file1 /path/to/file2

Source on GitHub.

ひとひら - 円 (Official Music Video)

Sat, 28 Mar 2026 11:48:29 +0100

I’ve been jamming to this all morning.

Underwater Volcano Eruption (Kavachi)

Fri, 27 Mar 2026 00:20:19 +0100

I’ve never seen that underwater steam/lightning before 😲

This ‘89 Pistons Cable Outro Goes Unnecessarily Hard

Wed, 25 Mar 2026 23:08:32 +0100

This really does go hard.

"TIL: at, atq, and atrun"

Wed, 25 Mar 2026 00:14:26 +0100

Today I learned about at, a built-in Unix command that schedules a one-off command to run at a specific time. Unlike cron, which is for recurring tasks, at runs something once and then the job is gone.

# Schedule a command for 10am
echo 'toot post "Hello from the future"'  | at 10:00

# List pending jobs
atq

# See what a job will run
at -c 1

# Cancel a job
atrm 1

On macOS, you need to enable atrun first:

sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.atrun.plist

That’s it. No config files, no services to manage. Just echo "command" | at TIME and walk away.

Introducing pty

Wed, 25 Mar 2026 00:04:20 +0100

I’ve been using zellij full time for a while and while I liked it better than tmux, I kept running into weird issues. I couldn’t scroll when using the codex CLI. It would use 300% CPU on my Mac while doing almost nothing. Loading sessions from my iPhone over SSH didn’t render well.

I use kitty which already supports splits, tabs, and layouts, so all I really need from a multiplexer is the ability to detach from long running processes and come back to them later.

You don’t need tmux, zellij, or any of those other problematic multiplexers. You just need pty.

pty uses @xterm/headless to run processes inside a real terminal emulator with a real width, height, full interactivity. Run vim, hx, mactop, dev servers, claude code, codex, whatever. Detach, it keeps running. Attach, you are back in it. Attach from multiple clients at once, perfectly fine. Type on one client, shows up on every attached client.

I took it further: pty resizes to the smallest currently connected client, so my phone never tries to render a shell that is meant for my XDR display.

I added a peek command where you can spy on a running process without fully attaching, just get a snapshot (ANSI or plain text). And I made it possible to send text or key combos into a background shell from the outside without attaching.

And then I realized: if we can remotely spy on and drive a real xterm session, then we could make a playwright-like terminal testing framework. And so pty has a testing library. Test your CLI or TUI with vitest. Assert on the real output your users see.

I’m excited to share this first version of pty and I’m looking for feedback. Hit me up on threads or mastodon.

Kanemaki Yoshitoshi’s Sixteen-Faced Woman

Mon, 23 Mar 2026 13:28:57 +0100

Mesmerizing.

Casio S100X Japanese Lacquer Calculator

Mon, 23 Mar 2026 12:33:10 +0100

I hate how much I want this calculator 😆

Christie’s Rostrum

Thu, 19 Mar 2026 11:29:50 +0100

Jony Ive redesigned the Christie’s auction rostrum. The video is almost cringe, and yet it actually is a good podium and a good video.

Wave Interference

Wed, 18 Mar 2026 20:38:24 +0100

Click or tap anywhere on the canvas to drop a wave. Drop a few and watch them interact. The colors show where waves add together (bright) and where they cancel out (dark).

Florentine Floor

Mon, 16 Mar 2026 15:58:56 +0100

Back in 2015 I visited San Miniato al Monte in Florence. The church floor is covered in memorial slabs, many of them so worn you can barely read the names and dates anymore, but the inlaid symbols are still bold and sharp.

I immediately recognized this cross:

That's the Coudal Partners logo. I knew it from watching way too much Photoshop Tennis back in the day.

I kept walking and found this shield:

I loved it. Something about it felt right, like it could be my logo.

When I got back home to Berlin I looked up the Coudal Partners story and I couldn't believe it: Jim Coudal had found his logo in that same church, on a vacation himself.

I've tried to trace that shield a few times over the years, but it's a complex set of shapes and I never got it to a point where I was happy with it. Recently I decided to try one of those AI services that can draw SVGs from images, Quiver, and it nailed it from my photo.

I asked claude code to add the logo to my homepage and to the footer of every page and now I couldn't be happier. Finally I got my shield on my site.

OlmoTrace

Sun, 15 Mar 2026 13:09:40 +0100

Allen AI is a nonprofit that publishes everything: model weights, training data, training code, checkpoints. Their OlmoTrace tool lets you find verbatim matches between a model's output and its training corpus. You can actually see where the words came from. You can't do that with Claude, GPT, or even Qwen.

Their Olmo model is the vegan impossible burger of AI models. Olmo isn't as capable as Qwen yet (their own coding agent SERA is built on a Qwen backbone, which tells you where things stand). It's fully transparent and fully traceable.

As I work toward owning and operating my own personal, private, local coding agent, I'll keep an eye on Olmo. Provenance matters.

The Killer | Official Trailer

Sat, 14 Mar 2026 18:06:19 +0100

👀

Jargalant Altai (Special Version)

Sat, 14 Mar 2026 18:04:24 +0100

I've never heard anything quite like this before and it is awesome.

Generative AI Vegetarianism

Fri, 13 Mar 2026 10:45:41 +0100

Some people are going to choose to be AI vegans and that is cool. No shame for "eating AI meat" either. See also Simon Willison's post where he first described the concept. Via adactio.

Gaining Agency

Thu, 12 Mar 2026 12:12:36 +0100

When I was at Virginia Tech I taught web design classes through the YMCA using the on-campus computer labs. I loved making websites and I wanted to give the superpower of publishing online to as many people as possible. In one class the head librarian of the town library was there. I'll never forget when she came in to one class so excited: she had updated the library website herself, without asking for help, without asking permission. She felt so empowered.

Years later I taught web design and development at a professional school in Washington DC and saw the same thing happen again and again: that excitement that comes from self-publishing, self-expression, without guardrails or needing permission or help.

I want as many people as possible to enjoy the web, not be oppressed by it, not be stuck in algorithmic hell holes.

In the late 90s (high school) we were hacking on our computers, changing everything we could. We (me and my friends) were stealing Adobe and Macromedia software to make stuff (we eventually bought stuff when we could afford it). We were ripping and sharing music online, and in the 2000s even ripping and sharing DVDs with each other. We were making weird websites, weird online spaces, just terrible code and design, and we loved it. There was this energy: this is MY computer and I will use it how I want. My data, my media.

While it may seem people are stuck on these huge centralized sites today: there are more blogs now than ever. People are still publishing, building their own corners of the internet, and making stuff.

Even still, a personal website is tough to manage for most people and many people can only publish in the ways their CMS allows.

Coding agents can expand agency over one's online presence and data. They might be the first time "use a computer to do what you want" is actually accessible to everyone, not just people willing to learn a very technical system. Every previous wave of web tools, no matter how simple, still required you to learn a complex tool. Agents let you describe what you want and get something tailor made. That's a fundamentally different relationship with technology.

I removed the CMS from my website as an experiment and now I just tell claude code what to post. I write the text, it does everything else (it published this post, for example). And one of my goals this year is to own all my data and eventually own my inference (my own coding agent model), running local open source models instead of relying on any company.

Would you be interested in having a website setup like mine, where you can ask to post or make a change and it just happens?

One of the most valuable things about a coding agent for me is getting past blockers: something that would take too much time, something tedious, or something just beyond my knowledge by a tiny bit.

For example: HTML is way easier to read than to write. People still need to learn new things to publish online, but now there is an on-ramp. The agent can explain it, write the first draft, and when you would have gotten stuck in the past the agent can help you get over the hump.

When I was leading the web design department of that professional school I built my own CRM to manage incoming / outgoing emails. People would come in and teach certain modules that I needed to keep up with, I needed to know when I last emailed them, produce reports of who I hadn't emailed recently, etc. I was able to build that and was way more productive than my peers because I could code. And some of my peers today think that is alright: I deserve to be more productive because I know how to code. I disagree.

Now, someone could probably cobble together something similar with a coding agent without needing to have put in the time I have. This is a great thing. My goal is always more people doing more with the tech and data they own, not to gate keep or hoard.

The web is what we make of it. We have a great interconnected world today, with a few too many centralized big players. We can make more spaces for individuals and small groups. I think agents can help. Especially if we create open, locally running agents.

Should I make some online courses about self-publishing online with a personal coding agent? 🤔 Let me know what you think on mastodon.

Gugusse et l'Automate

Wed, 11 Mar 2026 22:14:41 +0100

This might be the very first film with a robot, and probably the first sci fi film.

dandelions forever

Wed, 11 Mar 2026 10:43:14 +0100

2000s dreamcore vibes.

csesh – monitoring and reconnecting Claude Code sessions

Tue, 10 Mar 2026 15:36:22 +0100

I removed the CMS from my website and now I just ask Claude Code to make new posts for me. I write the text, Claude puts a markdown file in the right place, runs some tooling it made for itself, and boom it's on my website.

I want to take this further with Claude Code Remote Control so I can make a post from my phone anytime by asking the Claude Code on my home Mac to post for me. But it keeps disconnecting.

So I made csesh, a set of bash utilities for monitoring Claude Code sessions and forcing them to reconnect. It builds on my pty wrapper, since I can not only detach from a process, I can send text to the backgrounded pty from the outside. A script can peek at the pty, send it /reconnect<return> to trigger a reconnect, and peek again to see if it worked.

The tools:

csesh lists running Claude Code sessions
creconnect reconnects disconnected sessions
ckeepalive automatically reconnects every 5 minutes

It is my goal this year to make sure I own all my data and eventually own my inference, so I can use a local, open source model instead of Claude for this website. One step at a time.

TIL Acorn can screenshot with layers

Thu, 05 Mar 2026 11:09:31 +0100

TIL that Acorn can take a screenshot and every window is on its own layer! You can move them around, hide some, etc. How cool.

toe – Distance Sessions 2020

Wed, 04 Mar 2026 17:09:34 +0100

One of my favorite shoegaze music videos. Hoping to see toe in Berlin later this year 👟

Hazel Dickens: They'll Never Keep Us Down and Which Side Are You On?

Mon, 02 Mar 2026 11:04:38 +0100

If someone were to ask me what my politics were, I'd probably point them to this video to start.

Leica T Leather Holster

Thu, 26 Feb 2026 15:42:10 +0100

I've wanted this holster for a while and finally found one on Kleinanzeigen. It's a simple thing, a leather pouch that clips to your belt, but it makes such a difference. I actually have my camera with me now instead of leaving it at home because I don't want to deal with a bag.

CRDT File Sync

Wed, 25 Feb 2026 13:08:53 +0100

I hadn't ever thought of having each device write its own database / file to a synced folder, and then just periodically having each device read all the other files to accumulate a fully synced state. It's such a simple, clever idea: no conflicts at the file-sync layer because each client only writes to its own file, and CRDTs handle merging the actual data.

Claude Code Remote Control

Wed, 25 Feb 2026 11:51:49 +0100

Anthropic just dropped Remote Control for Claude Code, letting you remote control your computer from your phone. Claude keeps running locally, your phone is just a window into the session.

I drafted this post from my phone using Remote Control and published it from my computer. It's a nice workflow: my Mac keeps working and I can message it from anywhere.

The 4 things that make an "agent" today

Tue, 24 Feb 2026 15:53:29 +0100

The 4 things that make an "agent" today

Six months ago, we agreed on a definition for an "agent:" an LLM with tools in a loop iterating toward a goal. But recently the definition has expanded. While the entire industry has slowly been figuring out what "an agent" is by trying different things, OpenClaw certainly has captured a lot of people's imaginations and helped crystallize what people think an agent should be able to do.

An agent today is 4 things

Language model with tools in a loop
Scheduled tasks
Memory or session persistence
Skills

This has become the standard packaging for an agent over the last 6 months. This is basically how OpenClaw describes itself, Notion's AI agent has roughly the same ideas, and even Claude and other products from frontier labs are describing themselves in similar ways.

Language models with tools in a loop is the foundation everything else is built on. The model reasons about a problem, calls a tool, sees the result, and decides what to do next until it reaches its goal or gets stuck. This is the core mechanic, the engine.

Scheduled tasks: everyone seems to always ask agents to "remind me to do this tomorrow", "every Monday and Thursday morning prepare and show me a report about…", "watch this web page and tell me when the price drops…", etc. So this has become table stakes for an agent: can it queue things up for later or to happen regularly? OpenClaw has the HEARTBEAT.md, but a sqlite database would work fine as well.

Memory or session persistence: the agent needs to not forget everything all the time. Coding agents already have this: you can resume work from the past to jump right back where you were. General agents need this as well.

Simon Willison has argued that memory is just "more tools," that you can bolt long-term memory onto the basic tools in a loop definition with a read/write memory tool. I think that undersells it.

Long term session persistence is a hard problem because language models have limited "context windows", so coding agents today implement lossy compaction which isn't great. Everyone is looking for better ways to remember right now.

Skills are what really captures people's imaginations: agents can do things on your behalf. There are even "skill files" that are almost a standard at this point.

The most useful skills are the authenticated ones: like access to your calendar, email, messages, notes, etc. These are also the scariest, the agent could run away with or delete your emails.

A note on security

Unless you know how to build secure systems, don't run something like OpenClaw with your real email credentials or trust some SaaS to run it for you. Don't let anyone tell you your credentials are encrypted or that their system is 100% secure or any other nonsense. Nothing is 100% secure, the goals are always:

Minimize risk
Minimize blast radius
Increase your confidence in the system
Not be too annoying (or you might turn off the security features)

Doing those things iteratively over time is the only way to actually produce a secure system of any kind.

Also: most best practices for security assume the code we write and execute is trustworthy and we are protecting ourselves from someone gaining access to our systems or seeing our credentials fly by. However, now the program itself is both dangerous and untrustworthy and can execute arbitrary code. So if you are not an expert, please know any advice you read might just be coming at this from the wrong angle in the first place.

The agent cannot be trusted with your real credentials or to execute code outside of any sandbox. There must be clear boundaries.

It doesn't matter if your credentials are encrypted at rest if the agent can see them in plain text while it is calling the gmail api to delete all your emails, or worse it starts forwarding your emails to some rando.

I believe any system can be made "secure" for an acceptable definition of "secure," even an agent. And while there are projects to try to wrap OpenClaw, I am suspicious if it can ever be fully "wrapped" up like that. It really needed to have been built with security and privacy in mind from the start to really get to a place where I would feel confident in it.

Coding agents on your own hardware are best

I have two personal opinions that I will layer on top:

1. The best agent is a coding agent and 2. the best place to run an agent is on your own hardware.

All agents will be coding agents soon, I predict. A coding agent can build custom solutions to get to a goal, it can create new skills on-demand, and the programs it writes can be audited, tested, and easily modified. Claude (the non-code one) will already write and execute python code on demand to do things like make word and excel documents… so it already is a coding agent some of the time.

Writing executable code that statistically is very close to correct is way more useful and valuable than writing prose that is statistically very close to correct.

For privacy and security, your agent(s) should run on your hardware. SaaS doesn't seem like a good fit for highly personal agents that can take actions on your behalf. This is why Mac minis have been flying off the shelves.

Your data lives on your device (or in some cloud you chose that your device is connected to) and you shouldn't need to transmit all of that up to someone else's servers just to tally up how my workouts have been going recently… or whatever it is I want an agent to do. Right now it is difficult to directly run a good model locally, so a local agent might still use Claude or ChatGPT for inference.

For local inference, ollama lets you run open-weights models on your own machine. qwen3-coder-next is a very impressive model for being so small (80B, 85GB at 8-bit) — it very well could power a sophisticated agent today on a high-memory Mac Mini or similar hardware.

We've been working hard at my day job making it easy to run AI in a private space, and we want to show what a safe, private, capable agent can look like. Follow our newsletter for new capabilities coming very soon.

Where this is heading

OpenClaw showed people what's possible, even if the security story is pretty scary for it.

The pattern is what matters, not any single project. Tools in a loop, scheduled tasks, memory, and skills. This is the architecture. Better implementations are coming, ones that are built with security from the ground up, that run on your own hardware, and that treat the agent as untrusted code by default.

What is your use case for an agent?

What would you use one of these autonomous agents for? What does your dream agent look like?

Hit me up on mastodon or threads, I'd love to hear what you think.

Two interesting AI developments

Mon, 09 Feb 2026 18:34:01 +0100

Two things I saw recently that prompted me to update my /ai page.

OpenClaw is an open-source, self-hosted personal AI agent by Peter Steinberger (steipete, founder of PSPDFKit). It's an AI that doesn't just chat but actually does things: it can message people via WhatsApp, Telegram, and Slack, control a browser, automate your devices, and has an extensible plugin system. It runs on your own hardware. The project has had quite the naming saga -- it started as Clawdis, became Clawdbot (~30k GitHub stars), got renamed to Moltbot after Anthropic's legal team reached out about the name, and finally landed on OpenClaw. It now has 179k+ GitHub stars. It's a huge project and worth keeping an eye on, though it does have all three legs of Simon Willison's lethal trifecta.

My AI Adoption Journey by Mitchell Hashimoto (co-founder of HashiCorp) documents his six-phase path from AI skepticism to running agents continuously. The whole thing is worth reading. I've had a similar journey, especially the part where he forces himself to reproduce his own work with the agent. I took the time to do that too: sit down and try to get the coding agent to write the code how I would write it. Now I feel more confident being able to instruct the robot to do things and know that the output will be pretty reasonable. That confidence doesn't come from the tools getting better (though they are). It comes from me learning what they need to hear.

You Could Configure Your App with Markdown

Wed, 28 Jan 2026 12:42:19 +0100

I keep finding little uses for small local SLMs in my side projects, and one that I really like is using Apple's Foundation Model to parse a config.md file into an AppConfig struct. You could configure your display name, your location, what time you'd like to receive notifications/reminders, the app's theme, and more in plain language.

SLM == "Small Language Model" which is what I assume Apple's on-device model is. LLMs are the giant ones that run on servers or beefy machines.

Apple's Foundation Models are different from all other SLMs because they are more directly integrated into swift. I don't have to ask the robot to produce json, parse that, see if it is valid, etc. They handle all of that behind the scenes and I receive a valid swift struct or enum.

Why markdown instead of json or a settings UI?

I'm building a small journal app for myself that's all markdown. Having the config also be a markdown file keeps the experience consistent for me. It's all just text files I can edit however I want.

Beyond that, there are real advantages to letting an SLM parse natural language config. Natural language can make setting a time in your app way easier since "9am" and "9:00 AM" and "nine in the morning" and "early, like 9ish" can all work. And there is less UI to build: instead of designing screens with toggles, dropdowns, and date pickers, you just need a text editor. Plus users can even include notes about why they're choosing to configure something a certain way right there in the markdown.

This pattern works for other things too: natural language search ("show me photos from last summer"), extracting data from messy text (receipts, emails), smart tagging, or anywhere users would benefit from not having to learn your app's specific input format.

Here is a little app I made. You can find the project up on GitHub. It's just a one file SwiftUI macOS app.

I can't provide a Playground because playgrounds cannot use macros, so they can't show off the @Generable stuff. So it must be a whole ass xcode project. Sorry.

My dream config struct for this app looks like this:

@Generable
enum Theme: String {
    case light
    case dark
}

@Generable
struct AppConfig: Equatable {
    @Guide(description: "The user's display name if specified")
    var name: String

    @Guide(description: "The user's preferred color theme. Use 'light' for bright backgrounds and dark text, good for daytime or well-lit environments. Use 'dark' for dark backgrounds and light text, easier on the eyes at night or in dim rooms.")
    var theme: Theme

    @Guide(description: "Notification hour in 24-hour format (0-23), or -1 if not specified. Pay attention to AM vs PM and daytime vs evening context.")
    var notificationHour: Int
}

(I wanted to mix in an enum just to show how fancy this can be.)

@Guide is for the SLM to try to help it figure out what these fields are for. This is "prompt engineering" right here. You can see I've tried to explain a bit why one might want light vs dark mode, which hopefully would work for a config file that says something like "I mainly work at night."

This is how you could use Apple's local model to parse some markdown into that struct:

func parseConfig(markdown: String) async throws -> AppConfig {
    let model = SystemLanguageModel(guardrails: .permissiveContentTransformations)
    let session = LanguageModelSession(model: model)

    let prompt = """
    Extract settings from this config file:

    \(markdown)

    Look for the user's name, theme preference (dark/light mode), and any notification time they've specified.
    """

    let response = try await session.respond(to: prompt, generating: AppConfig.self)
    return response.content
}

OK, about the guardrails: it's not what you think. The default guardrails are so restrictive that they won't work a majority of the time. With the default guardrails if you ask the SLM to "tell me a joke" it will refuse to do so. And even with .permissiveContentTransformations there are still a ton of guardrails on.

IMHO Apple's models are useless without .permissiveContentTransformations.

The rest of the app is the SwiftUI to make the two pane layout + button. You can view it all on GitHub.

And you don't have to use Apple's models. At work we've open sourced our code we use to boot up and use on-device MLX SLMs like Qwen 3 in our new.space app – our library is called SHLLM.

You can also just use Ollama or another local model manager that exposes an HTTP api and use any OpenAI compatible HTTP client package.

And soon small models will be able to solve little problems on-device even on the web. Chrome is experimenting with a built-in Prompt API that runs Gemini Nano locally in the browser. I made a test page that parses a markdown config the same way. It's still in origin trial, so it might not work for you yet, but it's coming.

Small models can do lots of small, interesting things like this. I am continually looking for more places to use little models to solve little problems. And it's all on-device, nothing leaves or goes to any company anywhere, which is important to me. More and more I want more things running on my devices when possible.

What ideas do you have for where to use local models? What are some problems you've used small models to solve yourself? Hit me up on mastodon or threads.

SwiftUI @State Considered Harmful

Tue, 02 Dec 2025 23:32:28 +0100

The Problem with `@State` for Collections

When you're building a SwiftUI app with lists of items, like say podcast episodes, the obvious first approach is @State:

struct EpisodeListView: View {
    @State private var episodes: [Episode] = []

    var body: some View {
        List(episodes) { episode in
            EpisodeRow(episode: episode)
        }
    }
}

This seems to work at first. But then I notice UI stuttering while using the app…

The moment you change any property on any episode, SwiftUI invalidates the entire @State array. Every EpisodeRow re-renders. With 100 episodes and a playback position updating every second, you're re-rendering 100 rows every second. 1,000 episodes and things get real sad. Your UI stutters. Your battery drains.

The fundamental problem: @State treats the entire array as a single unit of change. SwiftUI can't see inside structs to know that only episodes[42].playbackPosition changed, for example. It sees "the array changed" and rebuilds everything.

The Solution: Observable Collections of Observables with Identity Preservation

SwiftUI seems to have a lot of the downsides of the "just rerender everything" frameworks like React. Luckily, I have dealt with react, preact, svelte, etc. Granular reactivity requires granular changes and stable identity.

After a lot of iteration, this of what I've landed on. I am not a SwiftUI expert, so please tell me if this is terrible.

Ive made each item an @Observable class, stored them by reference in a dictionary, and I update them in-place.

@Observable
final class Episode: Identifiable, @unchecked Sendable {
    var id: String
    var title: String
    var playbackPosition: Int
    var isDownloaded: Bool
    // ... other properties

    func update(from snapshot: EpisodeSnapshot) {
        self.title = other.title
        self.playbackPosition = other.playbackPosition
        self.isDownloaded = other.isDownloaded
        // ... update all properties except identity (id)
    }
}

When I build an app I am dogmatic that there should be one "app state" that exists outside the UI and is granularly reactive by default. It can be unit tested. And when I look at this object I should be able to imagine what the UI looks like.

@MainActor
@Observable
final class AppState {
    var episodes: [String: Episode] = [:]

    func mergeEpisodes(_ incoming: [String: EpisodeSnapshot]) {
        let incomingIds = Set(incoming.keys)
        let existingIds = Set(episodes.keys)

        // Remove deleted
        for id in existingIds.subtracting(incomingIds) {
            episodes.removeValue(forKey: id)
        }

        // Update existing or add new
        for (id, incomingEpisode) in incoming {
            if let existing = episodes[id] {
                existing.update(from: incomingEpisode)  // Update in-place!
            } else {
                episodes[id] = Episdoe(from: incomingEpisode)  // New item
            }
        }
    }
}

This gives me:

Granular updates: When episode.playbackPosition changes, only views observing that specific property re-render
Stable object identity: The same Episode instance persists across refreshes, so SwiftUI can diff efficiently
Single source of truth: One Episode instance per ID, owned by AppState

Questions You're Probably Asking

1. "Why a class instead of a struct?"

Structs are value types. When you put structs in an array and modify one, you get a new array. SwiftUI sees a new array and re-renders everything.

Classes are reference types. When you modify a property on a class instance, the reference stays the same. With @Observable, SwiftUI tracks which properties each view actually reads and only re-renders when those specific properties change on the class instance.

// With structs: entire list re-renders
episodes[0].playbackPosition = 100  // Creates new array

// With classes: only views reading playbackPosition re-render
episodes["id"]?.playbackPosition = 100  // Same reference, property changes

2. "Won't this cause memory leaks or retain cycles?"

Not if you follow the ownership model. AppState owns all Episode instances in a dictionary. Views receive references through lookups (appState.episode(id: episodeId)). There are no closures capturing episodes, no parent-child cycles—just a flat owner (AppState) and temporary readers (views).

Do you know a better way to enforce this? Let me know, hit me up on mastodon.

3. "Why a dictionary instead of an array?"

Two reasons:

Fast lookups by ID - Detail views might need to look up episodes frequently. Make it easy.
Merging is clearer - When I refresh from disk/network, I get a new set of episodes. With a dictionary, it's obvious how to merge: update existing keys in-place, add new keys, remove missing keys.

Also, computed properties give you arrays when you need them:

var allEpisodes: [Episode] {
    episodes.values.sorted { $0.date > $1.date }
}

Don't sleep on computed properties: those are observable too.

4. "What about thread safety? You marked Episode as @unchecked Sendable."

Yes, and this is intentional. Episode is only mutated on the main actor, through AppState (which is @MainActor). The @unchecked Sendable conformance lets us pass episodes across actor boundaries for read-only operations (like background parsing) without copying.

The rule is simple: read anywhere, write only on main actor through AppState/AppActions.

I've also implemented an EpisodeSnapshot struct which can be thrown across actor boundaries anytime. Loading from disk should send EpisodeSnapshots over to the AppState, for example, to cross the boundary. Same with persisting episodes to disk.

Do you know a better way to enforce this? Let me know, hit me up on mastodon.

5. "Why not just use @ObservationIgnored on frequently-changing properties?"

Because then SwiftUI doesn't see the changes at all. If playbackPosition is ignored, the progress indicator in the UI never updates. Also, "app state" should only be the actual state. If it isn't useful for the app, then it shouldn't be in there.

The real solution is structuring views to observe only what they display:

// Good: This view only observes title and date
struct EpisodeRow: View {
    let episode: Episode

    var body: some View {
        VStack {
            Text(episode.title)
            Text(episode.date, style: .date)
        }
    }
}

// The playback position view is separate
struct PlaybackProgress: View {
    let episode: Episode

    var body: some View {
        ProgressView(value: Double(episode.playbackPosition),
                     total: Double(episode.duration))
    }
}

Now playback position updates only re-render PlaybackProgress, not EpisodeRow.

6. "How do views get fresh data after updates?"

Detail views should always look up from AppState:

struct EpisodeDetailView: View {
    let episodeId: String  // Just the ID
    @Environment(AppState.self) private var appState

    private var episode: Episode? {
        appState.episode(id: episodeId)  // Always fresh
    }

    var body: some View {
        if let episode {
            Text(episode.title)
            // ...
        }
    }
}

The view holds an ID, not a stale copy. Every access goes through AppState, which holds the authoritative instance.

7. "Does this help with testing?"

Enormously. All state lives in AppState and I bundle all mutations in AppActions. Tests don't need to render SwiftUI at all:

@MainActor
func testDownloadUpdatesEpisode() async throws {
    let appState = AppState(...)
    let appActions = AppActions(appState: appState, ...)

    // Seed an episode
    let episode = Episode(id: "test", isDownloaded: false, ...)
    appState.episodes["test"] = episode

    // Perform action
    await appActions.downloadEpisode(episode)

    // Assert state changed
    XCTAssertTrue(appState.episodes["test"]?.isDownloaded == true)
}

No UI, no simulators, fast and deterministic. The views are just projections of state, so I can just test the state.

8. "What about derived/computed state like 'downloaded episodes'?"

Keep derived state as computed properties on AppState:

var downloadedEpisodes: [Episode] {
    episodes.values.filter { $0.isDownloaded }.sorted { $0.date > $1.date }
}

var newEpisodes: [Episode] {
    episodes.values.filter { !$0.hasPlayed }.sorted { $0.date > $1.date }
}

These recompute on access, but because @Observable tracks reads, a view that only accesses downloadedEpisodes won't re-render when newEpisodes would change (unless the underlying episodes data changed in a way that affects both). You can move to a memoized version if you find you need to optimize, but wait IMO.

Navigate by ID, not by object:

// Good: Pass ID
NavigationLink(value: NavigationRoute.episode(episode.id)) {
    EpisodeRow(episode: episode)
}

// In destination
.navigationDestination(for: NavigationRoute.self) { route in
    switch route {
    case .episode(let id):
        EpisodeDetailView(episodeId: id)
    }
}

The detail view looks up fresh data from AppState using the ID. Even if the episode updates while navigating, the detail view sees the current state.

10. "What about high-frequency updates like playback progress?"

Throttle before mutating state. I use Combine to debounce playback callbacks:

// AudioPlayerService sends callbacks frequently
// A Combine pipeline throttles to ~250ms before updating AppState
playerProgressPublisher
    .throttle(for: .milliseconds(250), scheduler: DispatchQueue.main, latest: true)
    .sink { [weak self] time in
        self?.appState.currentTime = time
    }

The audio callback fires 60x/second. AppState updates 4x/second. The UI is smooth, and you're not burning CPU on excessive re-renders.

11. "How does this compare to Redux/TCA?"

Similar in spirit, different in mechanics. Like Redux/TCA:

Single source of truth (AppState)
Unidirectional flow (Action → State → UI)
Testable without UI

Unlike Redux/TCA:

No action types or reducers—just methods on AppActions
Native Swift observation instead of custom subscription system
Reference types for items that need granular updates

It's less ceremony, more direct mutation, but the same core principles.

Any other questions I missed? Hit me up on mastodon.

I Don't Have a CMS Anymore

Tue, 18 Nov 2025 15:47:19 +0100

I used to run this site on hugo. Standard static site generator setup: write markdown, run hugo build, preview locally, deploy. It worked fine, until it didn’t. A hugo update would break publishing new posts and I would let my site sit broken for months. Or trying to get the HTML how I wanted would require me to learn about shortcodes or weird go template language stuff…

I deleted all of that.

Now my website is just HTML files. No CMS at all.

When I want to publish a post, I write a markdown or HTML file and then tell claude: "publish it.” That's it.

Claude Code has built up its own scripts to convert the markdown to HTML, update the RSS feed, add the post to the homepage and archive, run all the formatting tools, and handle everything else. I don't touch any of it. I've never even looked at those scripts.

This is an experiment for me: how far can I push a coding agent? Can we eliminate traditional tooling entirely? Can the robot just do everything "by hand"?

Static site generators exist because manually maintaining a website, keeping an RSS feed in sync, updating navigation, managing archives, etc, is tedious and error-prone.

But what if the automation doesn't need to be baked into tooling? What if it can just be... done, on demand, by an agent? What if we can conjure specific, tailored tools when we need them?

No build step. No framework. No abstractions. Just files.

I still write the posts. I wrote this one.

claude does everything else.

This is the same pattern as the library staff metaphor I wrote about recently. I'm the user of the library. Claude Code is the staff keeping everything organized and accessible for me.

I didn’t realize the anxiety of just dealing with hugo was keeping me from posting. Now, I don’t have an excuse. It’s way too easy to throw some markdown over to the thing and let it figure it out.

Take aways

Agents can replace more infrastructure than I expected. I thought I'd still need some sort of framework or some kind of build system. Turns out those can be coded up on-demand. And if I change my mind, new tools are produced and run just-in-time.

Simple systems are easier to maintain. The less abstraction, the more directly Claude Code can work. Bash scripts and HTML files are transparent. Claude can read them, understand them, modify them. No nested packages or dependencies. No magic.

The agent maintains its own tools. It built the publishing scripts, and now it maintains them. When requirements change, It updates the code. What a weird future we have created.

Future experiments

Local, private coding agent. I don’t like sending up tons of data to Anthropic just to publish a blog post. Can I make this work with a local model on my laptop? I’m testing that out when I get time…

Markdown 👑

Now, my website is simple. A handful of posts, static pages, an RSS feed. No complex taxonomy, no search, no comments, no dynamic features.

If I needed those things, I might need real tooling again. Or I might not. Maybe a coding agent can just build that stuff too. I honestly don't know where the line is yet.

Maybe we just need markdown files and a really good agent.

The Library Doesn't Just Have a Card Catalog… it has a full-time staff

Tue, 18 Nov 2025 15:18:16 +0100

A few years ago, I wrote about how the library's card catalog is the right metaphor for organizing our digital information. The physical catalog works because it connects all our information exactly how we need for our personal knowledge bases.

But I missed something fundamental: the card catalog in a real Library doesn’t maintain itself.

Behind every well-organized library is a team of people constantly working to keep information accessible. They’re cataloging new acquisitions, updating subject headings, cross-referencing related materials, culling outdated items, all before you ask. When you walk into a library and find exactly what you’re looking for, it’s because of the wonderful staff there.

The reason we struggle with digital organization isn’t that we lack filing systems: there are too many systems out there to count. It’s that we’re trying to be both the user and the entire staff, and we simply don’t have the time.

What Library Staff Actually Do

Cataloging: Every new item gets metadata assigned: and not just title and author, but subject headings, classifications, and keywords.

Cross-referencing: Related materials get linked. If you’re checkout out the Cold War, the librarian has already connected biographies, primary source documents, and relevant periodicals.

Collection maintenance: Old, outdated, or rarely-used items get evaluated. The collection stays relevant through active curation, not just accumulation.

Discovery support: Librarians create displays, reading lists, and subject guides that surface relevant materials you didn’t know to ask for.

Answer questions: When search isn’t cutting it, a human who understands the collection and knows where things are can help.

All of this is proactive. The organization work happens continuously in the background, an act of renewal.

We’ve Been Trying to Do This Ourselves

With all of our notes, documents, bookmarks, photos, and emails, we are attempting to be our own librarians. We make folders. We add tags when we remember. We try to search when we need something. We promise ourselves we’ll organize everything soon.

It doesn’t work because the ratio is impossible: one person cannot be both a library’s sole user and its entire staff. The cataloging work scales with the collection size and our available time does not.

Every productivity system eventually fails for this reason. We build the perfect folder hierarchy, create the ultimate tagging system, install the best knowledge management software; then life happens. Files pile up. Tags become inconsistent. The system that was supposed to save us time becomes another thing we feel guilty about not maintaining.

The problem isn’t our discipline – the problem is structural.

A Personal Library Staff?

For the first time, we can actually build something resembling “a proactive staff.”

Cloud AI agents can do some of this work, but at the expense of sending all our private information up to their servers constantly…

So recently I’ve been learning about and working with local models, the ones you can run on your phone or laptop in a truly private, offline way. I’ve had success doing the systematic, repetitive organization work that can improve our personal knowledge systems and finally make them workable over time.

Here’s what I’ve learned is possible: I can build agents that process new documents as they arrive, extracting metadata and assigning categories based on my existing system. When patterns emerge, they suggest new categories I hadn’t thought of. I’ve programmed experiments to scan my collection for connections – linking meeting notes to project documents, research papers to relevant bookmarks, photos to calendar events.

They can handle maintenance too. Flagging duplicates, identifying outdated information, highlighting orphaned items that have no connections to anything else. As documents move into my archive, agents generate abstracts so I don’t lose track of what I once knew. They create reading lists organized by theme, produce regular digests of what I’ve saved recently.

And they can “learn” from how I search. Every query I run is information about what matters to me and how I think about my personal context. An agent can use that to preemptively improve discoverability of materials I’ll likely need, without me having to stop and manually organize everything first.

None of this requires me to stop what I’m doing to organize. The agents can work in the background.

Private and local is the future

Your personal context contains your most sensitive information: work documents, personal notes, financial records, health information, etc. Sending this to a cloud AI service doesn’t feel right. And we don’t have to.

A Mac Mini can run models that will handle this work. No subscriptions. No data leaving your network. Your local private “staff” works for you, not for the cloud provider that trained them.

Specialized agents, not one super-agent

Local models are small, so it is prudent to build many small agents instead of one big one.

A filing agent that runs daily to categorize new items
A connection agent that finds relationships between materials
A maintenance agent that flags issues and duplicates
A summary agent that creates abstracts and digests
A search agent that learns from your queries to improve indexing
Etc

Each agent should have focused instructions tailored to your specific organization system. You’re not prompting ChatGPT to “organize my files.” You’re running custom, tailored local programs that know your taxonomy, your terminology, and your priorities.

Every time you search for something or ask a question, you’re providing data about what matters to you and how you think about your information. Every time you manually connect two documents, you’re teaching the system about relationships it should look for. Every time you can’t find something, you’re revealing a gap in the organization.

These patterns should become instructions for proactive work. If you frequently search for “client feedback from Q3,” your agents should learn to automatically tag and group client feedback by quarter. If you often link legal documents to project files, your agents should start suggesting those connections preemptively.

Local Agents Change the Game

The library card catalog works because it is maintained by people whose full-time job is keeping information accessible. Personal knowledge management has failed for most of us because we’re trying to do that work ourselves while also being the library’s primary user.

Local, private agents can solve this by becoming your library staff. They can do the proactive, systematic organization work in the background. They can learn from your usage patterns. They can run continuously without requiring your attention. And because they run locally, your information stays yours.

You don't need better organizational discipline. You need staff. And for the first time, you might can actually have them.

I'm building toward this vision in a product called new.space. Run local models on your iPhone today. Give it a try.

The Capability Gradient

Tue, 04 Nov 2025 11:50:04 +0100

The Capability Gradient

I've been thinking about how I feel great when I dive deep into a topic, to expand my knowledge and capabilities in an area, and I think I finally can explain my thought model in a simple framework. I call it the Capability Gradient and it has four levels:

Consume
Curate
Collaborate
Create

If I need a new chair, I might just go to IKEA. Simple and done. Or, I can go further. I can shop around, I can do research, I can learn about the history of chairs, I can develop my taste, and then I can make a more informed decision. Or, I can go further still. I can contact a carpenter and collaborate, my taste can be instilled into the design. Or, I can go further, I can become an apprentice, practice, and then finally build my own chair.

As a basic consumer, I can only have what is offered in my nearby stores. Developing taste through curation: I can advise my friends and my future self on options. Through collaboration I can customize. And through mastery I complete freedom to iterate.

The deeper you go, the more adaptable you become. And you’re building capability for your own future ideas – the ones you haven’t had yet. You can’t know which current ideas will work or what better ideas you’ll have next month. If you’ve only bought from IKEA, you’re locked in. If you’ve built chairs yourself, you can pivot when inspiration strikes.

Research on flow states shows people experience the most satisfaction when deeply engaged in challenging work that matches their skill level – not mindlessly repeating tasks. The assembly line worker who just repeats motions gets bored. The one who pushes deeper, builds mastery. They find flow. Going deep isn’t just strategic – it can be more satisfying.

However, not everyone wants to climb to “create” on every gradient. Some people thrive on variety and get bored going deep. Others love mastery. The key is knowing where you’re most comfortable and being intentional about it. Maybe you’re naturally a curator. That’s great! The point isn’t that everyone should become a craftsman – it’s that you can choose your depth deliberately, understanding the tradeoffs and understanding yourself. When something really matters to you, when you want maximum adaptability for future ideas, that’s when you can push deeper.

Food: Restaurants → farmers markets → cooking classes → growing your own
Software: Use apps → compare tools → customize workflows → build your own
Content: Consume → curate/share → remix → create original
Etc

This isn’t about doing everything yourself or becoming a jack-of-all-trades. It’s about strategically and intentionally choosing where to build capability – where you want the freedom to adapt, iterate, and respond to ideas you haven’t had yet.

Thinking through this framework has helped me understand where I am most content and happy, and why I behave the way I do sometimes, maybe it can help you as well.

TIL about Git Notes

Thu, 23 Oct 2025 14:42:52 +0200

Git has a notes command that lets you add annotations to commits without changing the commit itself. Pretty handy for adding metadata or references after the fact!

TIL about strudel

Sun, 14 Sep 2025 22:35:07 +0200

👀

These are the real Hackers! I am blown away by how great the live coding performances are with this tool. Checkout the examples that play right in your browser in the docs at strudel.cc/workshop/getting-started

I made a song, which you can see the code and play right here!

What an incredible tool. Mind blowing.

TIL about strudel

Sun, 14 Sep 2025 22:35:07 +0200

👀

I made a song, which you can see the code and play right here!

What an incredible tool. Mind blowing.

New page: [/ai](https://nathanherald.com/ai)

Tue, 19 Aug 2025 20:25:08 +0200

New page: /ai

I want to publish some thoughts about AI on my website, but I don't want those thoughts scattered over many blog posts. So I've made a new page at /ai to hold those thoughts. It will be a living document, I'll update it whenever I can, and if anything is super interesting then I'll throw another RSS item in to let you know. 🫡

Removing Hugo

2025-03-09T22:14:08+01:00

A new experiment: I have completely removed my build tool. It’s just me and HTML now.

I‘ve had an itch for a while to dump hugo and just go straight html for a while… and I've finally done it. You can see in the repo for this website.

No more hugo, no more build tool, no more having to deal with new versions, deprecations, weird shortcodes, weird markdown things…

I created this page in a PR, added it to the /posts index, to the homepage recent links, and to the rss feed.

My intention is to build up some custom elements to help build new pages more quickly. We‘ll see how that goes. For now, this isn't too bad.

Before the BF, there was the T – my first mirrorless camera 🖤

Fri, 28 Feb 2025 01:02:20 +0100

An all aluminum body, milled from a single block, and polished for hours. With a minimal UI + big screen, people say it’s the camera for those who love their smartphone.

Yes, that was the Leica T.

A couple years ago I wanted to get into photography and get a real camera, so I got an old, original T. And I couldn’t be happier with it. Its design was ahead of its time.

Aside: used-photo.de is great. If you are looking for a used Leica, check them out. They were very helpful and even helped me get both the T body and my first lens, the 23mm Summicron. I couldn’t recommend them enough.

Aside 2: I sent the T to Wetzlar to clean and just check over everything. You can ship any camera back to Leica and they will do their best. And they were great. They replaced the motherboard in the camera, which did fix some problems and made it operate faster. And they completely replaced my lens because the electronics didn’t fully pass their tests. My original 23mm lens was “T mount” as it was initially called. This new one is “L mount” as they call it now. The cost was incredibly low for this service. I have been impressed by Leica as a company.

And now, Sigma have announced the BF. And it definitely feels like the spiritual successor to the T. The design from Leica was ahead of its time.

(Leica made a TL and then a TL2, but the vibe and design was unchanged + I own the original so that’s what I know best. They recently stopped making the TL2, so there definitely is a gap in the market for this type of camera now.)

The new Sigma BF is both extremely similar in vibe + better in every way. It even uses the same L mount! My lenses would work on it, and the new Sigma lenses would work on my T.

T	BF
APSC	Full frame
Bad autofocus	Good autofocus
18MP	24MP
L mount	L mount
Hot shoe	No shoe

🤔

I have been thinking for a while that I should get one of the last TL2 bodies so I could get the improved auto-focus, focus peeking zebras, and 24MP… and now I’m thinking I should save up a little longer and end up with a T and a BF on my shelf 💸

My T takes great photos. I mostly do black and white, but any old Leica can take great photos of any kind of scene. From detailed architecture, to soft morning light in the green forest, to the soft coastal blues, and really anything else.

I’ve learned the basics of the exposure triangle with it. It taught me photography.

I mostly shoot with manual focus and I lock to a specific ISO, so I am adjusting aperture and shutter speed with the two wheels on top of the T. (I mostly do this because the autofocus is not good on the old T. But I also do it to build up the muscle memory to intuit the exposure of different scenes for when in shoot with my antique film cameras.)

The BF only has one wheel, on the back, but the sigma lenses have an aperture ring. So it really has the same two wheels I am used to using.

(Leica L mount lenses don’t have an aperture ring, it’s driven electronically.)

So, the Sigma BF is very appealing to me. The upgraded video capabilities would be great, keeping the L mount is great, and moving up to a full frame sensor would make a difference to me.

Both lack a view finder. And a lot of people seem to think they could never take photos without one. I can say confidently that using the screen can be a great experience. Having the camera out in front like a phone or a compact like the Ricoh GR III brings less attention while out on the street. Pulling a camera up to my eye can sometimes turn heads when I only meant to observe.

And that is one advantage the T (or TL2) has: an EVF can be affixed to the hot shoe. The BF doesn’t have a hot shoe, so it’s screen for life.

Watching/reading the reviews: the BF almost seems like it was made for me. I love my T, but I wish it had great autofocus, higher MP, etc.

Shooting with a minimal camera like the T is really freeing. The simplicity of the UI keeps me from getting distracted or diving into menus or any of that. I set it up how I want, then I go out and enjoy shooting. And I would wager the BF will be the same.

If you’ve never shot with a minimal, stripped down camera like these then I recommend you borrow one. Having almost no buttons forces one to focus on the composition. There is nothing available to waste time with 😅

It’s excited to see more “weird” cameras come out. So much of what’s available all look and operate the same, with very minute details. Let’s see how long it takes me to save up for a BF.

Reactive DOM nodes + reactive templates for custom elements (web components)

Sun, 08 Dec 2024 11:15:20 +0100

Signals are now my go-to tool to model my reactive state for my apps in the browser. However, the hard part, for me, has been figuring out how I would prefer to bring that reactivity to the DOM. I think I’ve finally landed on a solution that I like using <template>.

For an explainer on <template>, checkout my previous post abut building templates for custom elements. This post builds on that one.

Bring the signal to the DOM

The basic way to know when a signal is updated is to subscrbe() to it:

const number = signal(0)

number.subscribe(newNumber => {
  console.log('new number!', newNumber)
})

number.value = 1
// log: new number! 1
number.value = 2
// log: new number! 1

One can also use an effect to achieve the same thing:

const number = signal(0)

effect(() => {
  // using number.value will auto-subscribe this effect to number
  console.log('new number!', number.value)
})

number.value = 1
// log: new number! 1
number.value = 2
// log: new number! 1

Dispose

Both subscribe() and effect() return a “disposal function” which “turns them off”… basically they stop watching / tracking any signals. You have to manually dispose of these subscriptions. It is annoying to me that one cannot provide an AbortSignal to the subscription 😠 Either way, this is what you have to do:

const number = signal(0)

const dispose = effect(() => {
  console.log('yay', number.value)
})

number.value = 1
// log: yay 1
dispose()
number.value = 2
// no log

`Text`

The easiest way to “bind” a signal to the DOM is to update a Text node anytime a signal’s value changes. Assuming the signal (or computed) contains a string, this would work:

function reactiveTextNode(signal) {
  const node = document.createTextNode(signal.value)

  effect(() => {
    node.textContent = signal.value
  })

  return node
}

And that’s it. Now we have a Text node that can be inserted anywhere in the DOM + it will update anytime it’s signal is updated 💪

Try this in your browser console for a full example:

// paste this on it's own line by itself or else safari will complain
preactSignals = await import('https://esm.sh/@preact/signals-core')

// then paste all this at once
const { computed, effect, signal } = preactSignals

function reactiveTextNode(signal) {
  const node = document.createTextNode(signal.value)

  effect(() => {
    node.textContent = signal.value
  })

  return node
}

const name = signal('James')
const hello = computed(() => `Hello ${name.value}`)
const node = reactiveTextNode(hello)

document.body.append(node)

// and now you can update the name, which will update the DOM
name.value = 'Alice' // updates!
name.value = 'Bob' // updates again!

Inserting signals into a template

I’d like to be able to directly interpolate a signal like this:

const template = html`
  <p>Hello <strong>${name}</strong></p>
`

And it would auto-create the reactive Text node.

In my previous post I made it possible to nest template by looking through all the interpolated objects looking for a template object, and using a small custom element as a stand-in inside the template’s content.

I’ll do the same for a signal inside my Template class:

// when pre-processing the values in the template string...
if (v instanceof Signal) {
  this.#signals.set(i, v)
  return `<s- data-i="${i}"></s->`
}

// then later in cloneNode()...
const signals = node.querySelectorAll('s-')

// for each custom element, replace it with a reactive text node
signals.forEach(t => {
  const i = t.dataset.i
  const sig = this.#signals.get(i)
  t.replaceWith(reactiveTextNode(sig))
})

This would take care of it. Don’t worry if this feels out of place, it is. Checkout this new codepen for a full example with an updated Template class and everything else I’m about to write about below.

Updating the signal in response to user input

I’d like to be able to update the name, so I can see it react. So I’m going to expand the template to include an input and a custom element which will listen for input events and update the signal in response:

const template = html`
  <update-name>
    <input value=${name.value}><br>
    <p>Hello <strong>${name}</strong></p>
  </update-name>
`

And the custom element class:

class UpdateName extends HTMLElement {
  constructor() {
    super()
    this.style.display = 'contents'
  }

  handleEvent(e) {
    name.value = e.target.value
  }

  connectedCallback() { this.addEventListener('input', this) }
  disconnectedCallback() { this.removeEventListener('input', this) }
}

customElements.define('update-name', UpdateName)

And this now is a complete app. I’ll embed the codepen right here so you can try it out and explore the full source altogether:

See the Pen Reactive DOM nodes + reactive templates for custom elements (web components) by Nathan Herald (@myobie) on CodePen.

Alternate ways to build reactive templates

mb21 replied to my previous post with a link to github.com/mb21/mastro/tree/main/src/reactive#reactive-mastro which is a very cool way to handle this.

What I like about it is it shows how you can very quickly hydrate server-side generated markup using their custom element subclass.

I, personally, am not a fan of putting things like bindings or handlers in the html directly. In my journey I’ve arrived at the above reactive text node primitive from which I can build up fairly complex web apps. And DOM elements already bubble up their events, so if I want to handle a click I’ll create a custom element like I showed above to listen for and handle it. I prefer that style.

And this is why the web is great: the tools we have in the browser are incredibly flexible, there isn’t a right way to use them.

If it works, if it meets your customers / readers / viewers needs, if it’s as standard and accessible as it can be, then you did it correctly 🫡

Open questions

While this is great to see how quickly one can put together some reactive templates, there are still a lot of questions to answer:

What about a reactive list? (oh don’t get me started 😵‍💫)
What about template slots? (sure)
What about server-side rendering? (a very big topic indeed)
Isn’t this making a template for every single DOM node? (it’s worse than you think)
Shadow DOM? (they who shall not be named)
What about effect disposal? (yeah, it’s not easy)

And I’m sure many more. What questions do you have about custom elements that haven’t been answered yet?

I am on a journey to answer these questions, and more, for myself. I think we can build highly-sophisticated web apps with only custom elements today, and I’m going to learn how.

Find me on mastodon and let me know what you think.

Building templates for custom elements (web components)

Sat, 07 Dec 2024 21:51:46 +0100

What is `<template>`?

If you, like me, have ever tried to use it for “templating” then you’ve probably also been frustrated. I’ve now used <template> enough that I think I get it, I think I understand what it’s good for. And it’s not templating.

<template> is for holding document fragments. That’s it.

Those DOM fragments are reusable. They expose a cloneNode() function which allows one to duplicate it, to then be inserted somewhere. The template’s content doesn’t change, cloneNode() just makes a duplicate.

const template = document.createElement('template')
template.innerHTML = '<p><em>Hello</em></p>'

const frag = template.content.cloneNode(true)
// true means deep clone, you almost always want this

document.body.append(frag)
// appending a fragment will move all of its children into the target element

The above doesn’t feel like what I would call “a template.”

If you needed 10 <p>A</p>s all the same, then this would work great. But most of the time I have some shared markup that I want to reuse, like for building up a list of grocery items for example. I might want to share something like <li>${name}</li> and then reuse that over and over for each grocery item.

This is a way to make the grocery list using <template>:

const groceryItems = ['milk', 'eggs', 'eggos']
const ul = document.createElement('ul')
const template = document.createElement('template')
template.innerHTML = '<li></li>'

for (const item of groceryItems) {
  const frag = template.content.cloneNode(true)
  frag.querySelector('li').innerHTML = item
  ul.append(frag)
}

document.append(ul)

This feels like a lot to just place the grocery item’s name into the <li> for each.

Most of us are used to writing something like this in JSX:

(
  <ul>
    {groceryItems.map(item => (
      <li>{item}</li>
    ))}
  </ul>
)

JSX is not available in the browser tho. It requires a build system… and it’s XML and not HTML. And if you’ve ever had to put in a {' '}, well, there are edge cases 😉

What we do have in the browser are template strings.

Template strings

Unrelated to <template>, template strings are those backtick multi-line strings that can interpolate stuff. Like this:

function say(...what) {
  console.warn(...what)
}

const number = 5

say(`Hello ${number} Again`)
// => 'Hello 5 Again'

There is another way to call a function with a template string though:

say`Hello ${number} Again`
// => ["Hello ", " Again"]
// => 5

And that is incredibly different. By removing the parentheses and putting the backticks up against the function, it calls it before interpolating the number.

You are expected to zip the first array with all the following arguments. If you look at the above example, you can see how first comes a string, then the five, then another string, and then we are done.

Implementing that type of zipping of arrays is annoying… and luckily there is a built in function that will do it for you: String.raw().

function say(strings, ...args) {
  const message = String.raw({ raw: strings }, ...args)
  console.warn(message)
}

const number = 5

say`Hello ${number} Again`
// => 'Hello 5 Again'

Let’s make a sweet html function that can take a template string and return an HTMLTemplateElement.

function html(strings, ...args) {
  const template = document.createElement('template')
  const source = String.raw({ raw: strings }, ...args)
  template.innerHTML = source
  return template
}

const item = 'Milk'

html`
  <ul>
    <li>${item}</li>
  </ul>
`

Which is super cool. That already looks more templatey to me. There is still some work to do to though be able to interpolate the entire grocery list, so let’s get to it.

I feel like I need to be able to nest templates to build the full grocery list… so I need to pre-process args before passing it to String.raw() to deal with any interpolated templates.

To make this easier, I’d like to package this up into my own class so I can keep some state and more easily detect a nested template.

export class Template {
  #embeddedTemplates = new Map()

  constructor(source, ...values) {
    this.element = document.createElement('template')
    // pre-process the template string values
    const updatedValues = this.#prepareValues(values)
    // zip
    this.source = String.raw({ raw: source }, ...updatedValues)
    // assign as the source
    this.element.innerHTML = this.source
  }

  #prepareValues(values) {
    let index = 0

    const prepareValue = v => {
      index += 1
      const i = String(index)

      if (typeof v === 'boolean' || v === null || v === undefined) {
        // if someone interpolates nothing, return an empty string and move on
        return ''
      }

      if (v instanceof Template) {
        // if we find a template, store it in our Map
        this.#embeddedTemplates.set(i, v)
        // substitute a small custom element including the index in the Map for this embedded template
        // our template content must be a string, so we are using this little custom element to stand in for our non-string Template object
        return `<t- data-i="${i}"></t->`
      }

      return String(v)
    }

    return values.map(v => {
      // just in case someone interpolated an array, then let's map over it
      if (Array.isArray(v)) {
        return v.map(prepareValue).join('')
      }

      return prepareValue(v)
    })
  }

  // we'll just always imagine it's true
  cloneNode() {
    const node = this.element.content.cloneNode(true)

    // find all those custom elements that are standing in for embedded template objects
    const embeddedTemplates = node.querySelectorAll('t-')

    // for each custom element, replace it with the cloned fragment of the template object it's standing in for
    embeddedTemplates.forEach(t => {
      const i = t.dataset.i
      const template = this.#embeddedTemplates.get(i)
      t.replaceWith(template.cloneNode())
    })

    return node
  }
}

I know that is a lot. I added comments to the code and hopefully that helps a bit.

With that class, I can now write:

function html(strings, ...args) {
  return new Template(strings, ...args)
}

const template = html`
  <ul>
    ${groceryItems.map(item => html`
      <li>${item}</li>
    `)}
  </ul>
`

document.body.append(template.cloneNode()

That is looking very templatey. Hopefully you can see the nested html call in there.

This should give us the html we expect.

This codepen has a working example.

The resulting DOM list isn’t reactive. If I want to add a new item to the grocery list, it won’t update. And that is a bigger topic for another post, hopefully soon.

Hit me up on mastodon and let me know what you think of my Template class, how are you handling templating with custom elements, or anything else 🫡

How to build custom elements that work no matter what order things are loaded

Tue, 26 Nov 2024 19:28:55 +0100

TL;DR:

Put all the important code in connectedCallback

Either don’t depend on any specific DOM parent being pre-defined, or await customElements.whenDefined(...)

Cleanup any listeners, etc in the disconnectedCallback for good hygiene

Custom elements are seemingly created in three steps:

Subclass HTMLElement
Define the tag name
Element is constructed

Which seems simple and complete, but it turns out things can happen in a lot of different orders and aren’t nearly that simple.

I’ve done some testing for different scenarios to see when are an elements attributes set, when are the element’s children reachable, and when is the element’s parent reachable? I’m trying to document what I’ve learned below.

Example html I will use to work through this:

<one->
  <two->
  </two->
</one->

<one-> is a valid custom element name. The only rules are: 1) must start with a letter and 2) must have at least one hyphen (-). <t-> is a valid custom element name, for example.

Also, just in case you don’t know, a custom element’s class can have a constructor and it can have a connectedCallback method which will be called when the element is attached to a document. It’s similar to “on mount” in the react and related world.

There are a few scenarios or orders that custom elements might be defined and discovered:

A tag’s name might be in the HTML returned from the server, but the subclass might not be defined yet in the custom elements registry
A tag name might show up in the html only after the subclass is defined
The element class might be constructed in javascript, unrelated to html parsing
Elements might be defined in a non-deterministic order

A tag’s name might be in the HTML returned from the server, but the subclass might not be defined yet in the custom elements registry

If you normally put your <script>s at the bottom of the <body>, the this is your scenario.

The HTML parser will first construct a generic HTMLElement since the tag’s name isn’t in the customElements registry. The element is style-able by CSS and the CSS can even see if the element is :defined or not (if it’s in the customElements registry). This can cause a flash of undefined custom elements depending on how you implement the CSS and HTML for the component. Some devs choose to hide not(:defined) elements to avoid this.

After some Javascript is loaded and defines the tag name to be a subclass of HTMLElement, the generic HTMLElement will automatically be upgraded and replaced. customElements.define('one-', OneElement) will cause the DOM to look for any existing one- elements and upgrade them right then.

When an element is upgraded from a generic one to a custom one, the elements children are accessible in the class’s constructor, connectedCallback, etc – it’s a leaf-first situation. So this.children will be accurate anywhere, anytime for this scenario.

This is not the end of the story, though. Because of how HTML is parsed and because scripts are run inline, it’s possible that there is more HTML to go through after the <script> which defined OneElement.

As the HTML parser encounters new <one-> elements, it will directly instantiate the OneElement class right then. So half the page might work one way, the other half might work another way. (I recommend never letting this happen, it’s just confusing 😅)

This is identical to the next scenario…

A tag name might show up in the html only after the subclass is defined

If you put your <script>s in the <head> and you do not use defer, then this is your scenario.

Since the OneElement is already registered into customElement, the HTML parser will directly instantiate a new OneElement for each <one-> it encounters. There won’t be a time where the CSS selector not(:defined) would match, there is not upgrade, and so there is not flash of undefined custom elements.

And the access to this.children is the same; whether your code is in the constructor, connectedCallback, or anywhere else, this.children should be what you expect.

However, there is one more scenario that isn’t related to HTML parsing…

The element class might be constructed in javascript, unrelated to html parsing

I’m sure you’ve had to write code like:

const div = document.createElement('div')
div.id = 'some-id'

const paragraph = document.createElement('p')
paragraph.innerHTML = 'Hello 👋'

div.append(paragraph)
// etc, etc... 
return div

And it feels very tedious… well this can happen for custom elements as well.

One can use document.createElement('one-') or new OneElement() to make a new instance of the custom element… and it doesn’t have its children yet.

So in this scenario, it is not safe to look at this.children or this.parentNode in the constructor… It also doesn’t have any of it’s attributes set… so you can’t really look for those attributes in the constructor either… which can be really annoying.

However, there is good news, you can trust this.children and all the other stuff in the connectedCallback.

Try this to prove it to yourself (just try this in the console at any URL):

class A extends HTMLElement { connectedCallback() { console.log('connected!') } }

// if you try `new A()` right now, you will get an error. A custom element can be instantiated until it is defined in the registry.
customElements.define('a-', A)

let a = new A() // yay!
let div = document.createElement('div')

// luckily, this does not call the connectedCallback yet!
div.append(a) 

document.body.append(div)
// "connected!" will be logged, the connectedCallback was finally called

As long as custom elements are instantiated, fully prepared, and then added to a document things will probably work out just fine for any code in the connectedCallback.

One way I work around this sometimes is to accept attributes and children as optional arguments to the constructor, like this:

class FancyElement extends HTMLElement {
  constructor(attributes, ...children) {
    super()
    
    if (attributes) {
      for (const [key, value] of Object.entries(attributes)) {
        this.setAttribute(key, value)
      }
    }

    if (children.length > 0) {
      this.replaceChildren(...children)
    }
    
    // OH yeah!
  }
}

So now I can safely put setup code in the constructor, as long as every dev knows this is how this custom element needs to be used… I don’t do this often, but sometimes this is the easiest way to move forward. Really tho, just try to only ever put important code in the connectedCallback if at all possible.

This is all fine for one custom element, but what if there are many custom elements and they are all defined at different times 🤔

Elements might be defined in a non-deterministic order

Using the same example from above, assuming none of the subclasses have been defined during HTML parsing:

<one->
  <two->
  </two->
</one->

The order the subclasses are defined really matters. If, for some reason, OneElement was going to do something that TwoElement depends on (like setup a database or something), then you better make sure to define OneElement first.

Having a direct dependency on a parent in the DOM is an anti-pattern, try not to do it unless absolutely necessary.

And, well, if you really want to make sure, then you can wait for the other custom element to be defined inside the connected callback:

export class TwoElement extends HTMLElement {
  static define() {
    customElements.define('two-', this)
  }

  async connectedCallback() {
    await customElements.whenDefined('one-')

    // now the parent is fully there 💪
  }
}

This might look like it could end up in a deadlock if two tags were waiting on each other, but it won’t. connectedCallback() is not actually async and the DOM runtime will not await the promise this example returns, so things can work out concurrently just fine.

(You can also use this trick the other way: wait until you know all the custom elements that might be children are defined.)

Still, you can also just make sure things are defined in an outside-in order in your code. That is what I prefer since it’s less code for me to write and I need this rarely.

Is this all a mess?

Kinda, but I’m OK with it. Custom elements are, IMHO, not a easy as react-like components because they are alive, living in the DOM, and dealing with the real built-in HTML parser and renderer which have to deal with a lot more than just JSX… If you embrace a few conventions, then it normally all works out fine:

Put all the important code in connectedCallback
Either don’t depend on any specific DOM parent being pre-defined, or await customElements.whenDefined(...)
Cleanup any listeners, etc in the disconnectedCallback for good hygiene

Good luck. Let me know if I made any mistakes or missed any other scenarios that are tricky with custom elements.

How to manage reactive state using Custom Elements (Web Components)

Sun, 17 Nov 2024 11:39:10 +0100

TL;DR: signals work well in custom elements and grabbing state from a parent through event dispatch is great. Checkout this codepen. 💪

To make things truly reactive in our custom elements (web components) we need some reactive state primitive. In frameworks, we usually use a hook or some other kind of “handle” to some state somewhere out there in a place we can’t really see.

This is how I would do this in preact:

const App = () => {
  const [count, updateCount] = useState(0)
  const increment = () => updateCount(c => c + 1)
  
  return (
    <>
      <p>
        You’ve clicked the button
        {count} time{count === 1 ? '' : 's'}
      </p>
      <button onClick={increment}>Increment</button>
    </>
  )
}

Where is count really? We don’t know. It’s in preact. Which is fine most of the time.

But still, isn’t that kinda weird. This “function component” App doesn’t take any arguments. And yet, as I call the function over and over, it returns different results depending on some state somewhere not in the function…

Also, we can’t actually call App() ourselves, it will throw… so it’s a function, but not a function for us to use. It’s a function for preact.

This puts a lot of pressure on the UI framework – it now must contain the definition of the views, probably some styling, the state, the behavior, all of it.

Writing a unit test for the “counting feature” is also not that easy, we need to setup a render loop, etc. We don’t know where that state is, so we gotta let the framework help us.

It’d be better if our state itself was reactive and our UI just showed that reactive state…

Signals

Every framework is re-discovering signals these days, and that’s a great thing. (Checkout the old S library, which is still there after all these years and still works fine… signals have been great for a while.)

Signals can make your state reactive, let you model your app purely as data, and you can unit test your app’s state without having to write any JSX or do any rendering or any of that.

export const count = signal(0)

export function increment() {
  count.value += 1
}

And then our app can just focus on the DOM we wish we had:

import { count, increment } from './count.js'

const App = () => {  
  return (
    <>
      <p>
        You’ve clicked the button
        {count.value} time{count.value === 1 ? '' : 's'}
      </p>
      <button onClick={increment}>Increment</button>
    </>
  )
}

And if somewhere else needed to either display or increment the count, they could import the functionality the same way. Anytime the count changes, any UI component using its value will re-render.

This takes a lot of pressure off the UI: it can focus only on the actual UI stuff.

You can imagine writing some tests like:

it('counts', () => {
  assertEquals(count.value, 0)
  increment()
  assertEquals(count.value, 1)
})

Signals work great in custom elements too

We can wire that same state straight up to where it’s used in our custom element:

import { count, increment } from './count.js'

const times = computed(() => {
  return count.value === 1 ? 'time' : 'times'
})

class CounterApp extends HTMLElement {
  static define(name = 'counter-app') {
    customElements.define(name, this)
  }
  
  constructor() {
    super()
    this.style.display = 'contents'
    this.disposes = new Set()
  }

  handleEvent(e) {
    if (e.type === 'click' && e.target.tagName === 'BUTTON') {
      increment()
    }
  }

  connectedCallback() {
    this.addEventListener('click', this)
    
    const countEl = this.querySelector('[data-count]')
    const timesEl = this.querySelector('[data-times]')
    
    this.disposes.add(count.subscribe(c => {
      countEl.innerText = c
    }))
    
    this.disposes.add(times.subscribe(t => {
      timesEl.innerText = t
    }))
    
    const button = this.querySelector('button')
    button.disabled = false
  }

  disconnectedCallback() {
    this.removeEventListener('click', this)
    this.disposes.forEach(cb => cb())
  }
}

And then:

<!-- assuming CounterApp.define() has been run -->
<counter-app>
  <p>You’ve clicked the button <span data-count>0</span> <span data-times>times</span></p>
  <button disabled>Increment</button>
</counter-app>

My custom element here requires that you provide the correct HTML: a p and a button. It’s fine if a custom element has opinions/requirements about it’s children.

Notice in the initial serialized HTML I’ve disabled the button. Then once the custom element is fully attached, I will enable it. This way we never have an enabled button that can’t work on the page. Old-school progressive enhancement. 🆒

This component gets the reactive state, encapsulates and wires up the behavior, and then everything is hunky-dory.

Now, of course, web apps are much larger than this… an entire app’s state can be modeled, tested, and then consumed by the UI. It is possible. Be optimistic 🤓

And (I say) it’s preferable to have the state outside the UI rather than keeping the app’s state spread throughout our UI code just because useState() is the only reactive primitive we have there in our framework…

Moving the app’s state out of the UI layer is a way we can start to bridge the gap between custom elements and framework components. It also makes the decision of “which framework?” less important, so you can feel more free to experiment.

Give it some context

Keeping an app’s state in globals can be problematic: it makes testing hard and it makes it impossible to have two counters on the page at once (which are related problems).

Most frameworks provide a way to setup a context which can have a sort of “scoped state” that is for any children of the context.

An example in preact might look like:

<CounterContext initial={0}>
  <CounterApp />
</CounterContext>
<CounterContext initial={0}>
  <CounterApp />
</CounterContext>

And then we’d have two completely independent counter apps on the page at the same time without having to move our reactive state back into the UI components themselves.

How can we communicate a sophisticated scoped state object (or set of objects) to some of the DOM when using custom elements?

It’s not about “passing data down,” it’s really about events bubbling up.

Events are dispatched synchronously

Yep. Prepare to feel weird about this. Event dispatch is often incredibly misunderstood. React implemented their own synthetic events because of how weird DOM events are…

DOM events are like a lightening strike: an event travels down, then bubbles up. There is a capture phase, then a bubble phase. And that entire traversal is synchronous. 🤔 ⚡

So we can use the events go up principle of the DOM to grab context from a parent, if there is one.

First, let’s package up all our app’s state into a “factory” or “generate” function:

import { computed, signal } from "https://esm.sh/@preact/signals-core";

export function genState() {
  const count = signal(0)

  const times = computed(() => {
    return count.value === 1 ? 'time' : 'times'
  })

  return {
    count,
    times,
    increment() {
      count.value += 1
    }
  }
}

Now we can make as many app states as we like.

Let’s make a custom element to create, contain, and scope that state:

import { genState } from './state.js'

export class CounterContext extends HTMLElement {
  static define() {
    customElements.define('counter-context', this)
  }
  
  constructor() {
    super()
    this.style.display = 'contents'
    this.context = genState()
  }
  
  handleEvent(e) {
    // I just made this event name up
    if (e.type === 'getContext') {
      // if any event comes through wanting a context, I'll throw mine into the detail of the event here and then stop propigation so no further parents can overwrite which allows contexts to be nested, closest one wins
      e.context = this.context
      e.stopPropagation()
    }
  }
  
  connectedCallback() {
    this.addEventListener('getContext', this)
  }
  
  disconnectedCallback() {
    this.removeEventListener('getContext', this)
  }
}

export class GetContextEvent extends Event {
  constructor() {
    super('getContext', { bubbles: true })
    this.context = null
  }
}

And now the component which wants to subscribe to some state can check for any context from any parents first, falling back to creating it’s own state if no parents have a context for it:

import { GetContextEvent } from './context.js'

export class CounterApp extends HTMLElement {
  static define(name = 'counter-app') {
    customElements.define(name, this)
  }
  
  constructor() {
    super()
    this.style.display = 'contents'
    this.disposes = new Set()
    this.context = null
  }

  handleEvent(e) {
    if (e.type === 'click' && e.target.tagName === 'BUTTON') {
      this.context?.increment()
    }
  }

 
  connectedCallback() {
    const getContext = new GetContextEvent()
    
    // ⚡
    this.dispatchEvent(getContext)

    // the .context value has been mutated synchronously by the event listener in the parent node
    this.context = getContext.context
    
    if (!this.context) {
      // if no parent provides the context, then we will make our own
      console.warn('no parent context, making local state', this)
      this.context = genState()
    }
    
    this.addEventListener('click', this)
    
    const countEl = this.querySelector('[data-count]')
    const timesEl = this.querySelector('[data-times]')
    
    this.disposes.add(this.context.count.subscribe(c => {
      countEl.innerText = c
    }))
    
    this.disposes.add(this.context.times.subscribe(t => {
      timesEl.innerText = t
    }))
    
    const button = this.querySelector('button')
    button.disabled = false
  }

  disconnectedCallback() {
    this.removeEventListener('click', this)
    this.disposes.forEach(cb => cb())
  }
}

And now we can run multiple of these count apps, some of which can have a shared context, some of which can have an isolated context:

<counter-context>
<counter-app>
  <p>You’ve clicked the button <span data-count>0</span> <span data-times>times</span></p>
  <button disabled>Increment</button>
</counter-app>
</counter-context>

<hr>

<!-- both counter-app's will share the context state provided by this counter-context -->
<counter-context>
<counter-app>
  <p>You’ve clicked the button <span data-count>0</span> <span data-times>times</span></p>
  <button disabled>Increment</button>
</counter-app>
  
<counter-app>
  <p>Same count here: <span data-count>0</span> <span data-times>times</span></p>
  <button disabled>Increment</button>
</counter-app>
</counter-context>

Now that feels a lot like a “reusable component” to me. 💪

➡️ Checkout this codepen: https://codepen.io/myobie/pen/JjgbxyN

Go ahead and move elements in and out of the contexts in the example… hopefully you are starting to imagine how one can build up a sophisticated app with these loosely coupled components through passing simple string data down, and using events to grab more sophisticated objects / states when necessary.

This will probably be a standard

There is a Community Proposal for a Context Protocol which works exactly the same using event dispatch. Check it out. The event which is dispatched is a little different, it has a callback function which is called by a parent – so not direct object mutation like my example above.

So, maybe one day this will be 100% built-in 💪

Try out the codepen, try this out in your project, and let me know how it goes.

Why passkeys are good and you should be advocating for their use

Mon, 04 Nov 2024 15:41:58 +0100

Passkeys are a great “remember me on this device” that always works, doesn’t rely on cookies
If you are in an ecosystem, they will sync to all your devices
In the near future, passkeys will even sync across ecosystems

TL;DR: We all know physical security keys are incredibly secure. Passkeys are modeled on physical security keys and are both incredibly secure and incredibly easy to use to sign in after they are setup. You get issued one just like a physical security key: **you prove who you are, then you get a secure key that re-proves and re-authorizes you with just a fingerprint or face scan ** or whatever. Finally the “remember me on this device” checkbox will actually work.

Passkeys are the ultimate “remember me on this device.”

How many times have you checked that box and it didn’t work? You cookies are lost or something, who knows. Passkeys don’t rely on cookies, are incredibly secure, so finally “remember me” will actually work.

Why tho?

Making people provide a password over and over is incredibly insecure! We know this, we have the data, people will make mistakes. Yes, you. You will, statistically, eventually make a mistake with a password.

And passkeys making re-signing in easier + prevent us from doing anything by accident.

You should not be entering in your password over and over, the “remember me” should actually work. It’s way more secure for remember me to work than to ask someone to constantly provide a password – no possible attempts to phish that password.

What passkeys are not

You get a passkey after you prove who you are. You still need a way to initially prove who you are.

Passkeys are modeled on physical yubikey-like devices. So imagine getting a passkey in the mail or being handed one in person.

In the past, I would prove who I am to a company and they would issue me a physical yubikey. They’d write down the serial number of that yubikey and note that I have it. Any device with that yubikey plugged in can be assumed that it is definitely me, I don’t have to provide a password, email, or anything. The key both reminds the service who I am and proves that I am definitely who I was when I was given the key.

And, just like a physical yubikey, if I were to lose it, then I’d have to re-prove who I was to get a new one.

Passkeys are the same as physical keys

You initially do the more difficult proving step, usually clicking a link in an email or something like that. And that step hopefully has a two factor component, to make sure you really are who you say you are. You want this step to be difficult, because you shouldn’t be doing it very often.

After you have proven who you are, then you can get a very secure key that will on it’s own re-prove and re-identify you over and over again with just a finger print, face scan, or something similar.

Imagine it. Finally the “remember me on this device” checkbox will actually work.

Are Passkeys ready today?

Passkeys are good today, because:

Passwords are 100% known to be problematic, we have the data. You may think you are not going to fall for a phishing attempt, but anyone can and does. Don’t be so arrogant to imagine that you can’t. And phishing isn’t the only problem, it’s a huge deal and no one is immune.
Even if you had to do the more difficult proving step for each of your devices, it’s only one time per device. After that, use biometrics or similar and you are in. No cookies, no password to remember or lose, you are good now forever on this device. This is unquestionably easier for normal and power users. “Remember me on this device” finally works.
Good ecosystems already sync passkeys: apple, google, 1password, etc. If you use them, then you only have to sign in the more difficult way one time, period. Then you are good on all your devices in that ecosystem. If you use two ecosystems, then you’ll have to do the difficult sign in two times (one per ecosystem), which is fine. (And, they are working on cross-ecosystem passkey sharing, so you only have to do the difficult prove who you are step once across all types of devices!)

We all know physical security keys are incredibly secure. Passkeys are modeled on physical security keys and are both incredibly secure and incredibly easy to use after they are setup.

You get issued one just like a physical security key: you prove who you are, then you get a secure key that re-proves and re-authorizes you with just a fingerprint or face scan or whatever. Any device with that key can be 100% trusted.

Sure, you still have to prove who you are one time in a more difficult way. It’s one time. And I would suggest that it be made even more difficult to prove who you are, since you won’t have to do it very often anymore. It should be pretty difficult to get a new passkey.

And you don’t have to worry about losing a passkey: you can just do the more difficult “prove who you are” step again and make a new one, just like requesting a new hardware key. There will always continue to be “account recovery” steps.

Passkeys really are just an incredibly secure and easy “remember me on this device” that works.

Some FAQs

Are passkeys useful as a second factor: I would say no. They are meant to be used as the ultimate “remember me on this device” and that isn’t what we think of as a second factor.
What if someone gains access to my email, can’t they just get a passkey issued to them?
- If the service requires both an email link click + a second factor (like a one time passcode), then no. The attacker won’t have the one time passcode (the second factor).
- If a service really only checks email and then issues a passkey, then yes that’s bad. That service would let anyone pretend to be you if they have access to your inbox today. So the service is very insecure and it’s unrelated to passkeys.
Can I still have a password?
- Yes, a password could be part of the “prove who you are in a more difficult way” flow. However, passwords are known to be terrible, people do make mistakes and enter them into the wrong places, and so it is a risk to allow someone to get a passkey after entering a password. The service really should implement a second factor. A password + second factor as the “difficult path to get a passkey” is perfectly fine, and that lets the passkey focus on being the ultimate “remember me on this device.” So take how you sign in today + actually remember me on this device that works.

The disharmony between “web components” (custom elements) and react-like “framework components” is real

Wed, 30 Oct 2024 11:08:47 +0100

The primary difference is passing down data vs. passing down behavior, and this causes a ton of confusion.

Recent discourse about “web components” vs react-like “framework components” is continuing to highlight a fundamental disconnect between these two types of components.

There is one main irreconcilable difference that many seem to forget to highlight:

The DOM is about passing data down and dispatching events up, while framework components are all about passing behavior down through “props.”

And this is why it’s sometimes difficult for framework authors to support custom elements – they are fundamentally incompatible models.

lit tries to bridge the gap by making custom elements look like react, preact does a pretty good best-effort to support custom elements, react is also trying, but in the end they can’t work well together because of these fundamental differences.

Data down, events up vs. passing behavior down

The DOM is an “object model” that is very object oriented. One of the tenets of most object oriented systems is to:

Encapsulate behavior and state into objects, and
Pass messages / data between those objects. The DOM pretty much works this way.

The HTMLParagraphElement inside the HTMLDIVElement doesn’t share any state with its parent. It doesn’t need to.

This is why custom elements are classes: classes are one of the things that can encapsulate state and behavior in javascript. (Sure, they could have been plain objects and I do kinda wish they were, but I get why the spec writers made this choice.)

Let’s say you want to react to a click of a button inside your custom element – you shouldn’t pass a function down to the button to provide to its onclick prop, nah you’d just implement handleEvent in the custom element which contains the button and listen for the event to bubble up. You don’t even need to know if there are any buttons inside your element, you can just be ready for when there are. And the button doesn’t have to know the custom element is a parent – it dispatches a click either way.

The custom element encapsulates the behavior like this:

class CaptureClick extends HTMLElement {
  static define() {
    customElements.define('capture-click', this)
  }

  constructor() {
    super()
    this.style.display = 'contents'
  }

  handleEvent(e) {
    if (e.type === 'click' && e.target.tagName === 'BUTTON') {
      e.preventDefault()
      e.stopPropagation()
      console.warn('captured', e.target.innerText, e)
    }
  }

  connectedCallback() {
    this.addEventListener('click', this)
  }

  disconnectedCallback() {
    this.removeEventListener('click', this)
  }
}

And this can be used either in javascript with document.createElement() or in HTML like:

<!-- assuming CustomClick.define() has been run -->
<capture-click>
  <button>Yo!</button>
</capture-click>

To someone who works mostly in react-like frameworks, this code is probably very surprising.

Most framework components today are “functional” or at least very function oriented. If you created a class component today someone would ask you what you were doing… and functional components can be really useful if you want to unit test the output from them. You could provide the function some input and then assert on the virtual DOM nodes returned… although I bet you don’t do that. Have you ever unit tested the output of your function components?

I bet your components need some sort of “context” to work at all, and I bet you need to setup a component hierarchy to provide that context… so not as functional as we hoped, but still pretty nice.

To implement the above in preact, I’d just do:

// assuming there is a Button component
export const App = () => {
  const buttonValue = 'Yo!'
  const onClick = e => {
    e.preventDefault()
    e.stopPropagation()
    console.warn('captured', buttonValue, e)
  }

  return (
    <Button onClick={onClick} value={buttonValue}>
  )
}

I am passing behavior down, from App to Button. And you see this all over framework component code.

And not just clicks. We pass down functions that query databases, we pass down databases themselves, we store things in a context and then pull it back out many levels down, we pass down callbacks or promises or both…

And hooks aren’t helping – hooks are just hiding the implementation details of state to make things look functional, when they really aren’t.

And listen, I love writing “function components” and I like building apps with preact. new.space is 100% written with preact, signals, and other great libraries. It’s a nice world to be in.

JSX makes it easy to pass behavior down. HTML isn’t JSX tho: HTML is a serialization of a DOM we wish existed.

Serialized DOM (HTML) can only pass data as strings in attributes. I’m sure you’ve read something about “attributes vs. props” before and this is main conflict: html attributes are only data and only strings.

How to bridge the gap?

If you are really motivated, then you can write your custom elements in a DOM-first style and write your react-like components in that style… but I don’t recommend it. It can be a pretty big context switch going back and forth. For existing apps, I recommend staying in that app’s current world. For new apps, build with custom elements… until you can’t.

Custom elements are ready today, you can do anything you want with them. If you want to build a context element to hold some state and then query that state further down the DOM tree, do a quick this.closest('parent-with-state')?.state inside the connectedCallback() of the child element.

What problems have you not been able to solve with custom elements? What would you love to know more about? Let me know and I’ll write up some real examples.

Reminded of the past, and to not repeat it

Sun, 27 Oct 2024 20:35:57 +0100

In Berlin I am confronted with reminders of the city’s every single day. Every time I walk outside I see a memorial or reminder of dreadful things, it’s normal, these reminders are ever present.

Earlier this year I was walking around, admiring some wonderful murals…

…when I learned about something I didn’t know before. One of these buildings had an informational sign about it. I took a photo of the sign, but this copy on Wikimedia is more clear than my photo:

The sign reads: ‘The early concentration camp “Gutschow-Keller” was located here.’

The greengrocers Hermann and Paul Gutschow owned the imposing building in Wilhelminian style in Friedrichstraße 17 on the opposite side of the street. Already in 1932, they placed their warehouse and basement at the disposal of the “SA-Sturmbann Ill/8”. These rooms were located here in the second courtyard of the building in Friedrichstraße 234.

From. March to May 1933, the place was one of the first concentration camps in Berlin. Prisoners called it “Blutburg” (castle of blood). Hundreds of trade unionists, communists, social democrats and Jews were seized in their homes, at their places of work and in the middle of the street and were then abducted to this place. Interrogations, torture and humiliation followed - often for days on end.

As these torture chambers of the SA were placed in a large tenement block, the neighbourhood knew about the imprisonment and maltreatment, for the screams of the prisoners could still be heard on the street.

People were rounded up, detained, and tortured. The neighbors here saw them, could hear their screams, and they did nothing.

This reminder is here in this neighborhood because this really happened, and it could happen again.

And it wasn’t a big, flashy production by “evil people” the way most people probably imagine the history was here in Berlin. These were normal people, it was boring, it was methodical, they were instructed to do it, and it relied on everyone to just follow orders and/or just let it happen.

These reminders are everywhere here in Berlin because these things really happened, and they could happen again.

How I (Now) Backup My Photos

Tue, 14 May 2024 23:13:30 +0200

I’m taking more and more RAW digital photos + scans of film photos and I finally took the time to regularly back them up – because I was locked out of my Apple account!

Apple had a problem back on April 26 where some people’s accounts got locked, and it affected me. I woke up and saw that my Apple account was locked, and my phone demanded that I reset my password. So I reset it. Then, after a couple minutes, it demanded that I reset my password yet again.

I’ve seen how these things can go, and getting into a reset loop like that felt like it could end up locking me out of my Apple account entirely. Before: Apple Photos was my only storage + backup of all 800GB of my photos. Now: I need to run to the electronics store and buy a hard drive.

You can read more about Apple’s issue on Michael Tsai’s blog post which has tons of links.

I turned my iPhone and Mac off, I didn’t want them to accidentally do anything while I was out. I had recently got a Nothing 2a to test my web work on Android, so I grabbed it and put a prepaid SIM into it so I could do stuff out and about with a phone that I’m not locked out of.

Then I went down to Saturn and picked up a Samsung T7 Shield drive. Luckily, I had already done a lot of research about which hard drives are reliable, so I knew these were good. There are cheaper drives that have terrible track records (Sandisk, WD, …) and I didn’t want to deal with that on a stressful day where I felt I might lose access to all my photos forever. I grabbed a 4TB drive so I knew I’d have plenty of space for a while.

I’ve also done a lot of research about which backup program to use on my Mac, and so I knew the best is Arq. I had already learned that if the drive isn’t plugged in, Arq will just silently fail. I use a MacBook Pro so the T7 might not be plugged in every time Arq runs, and Arq can just handle that scenario quietly without bothering me.

I’ve already checked the box to keep all my originals on my disk and I’ve got a 2TB disk in the laptop, so all my Apple Photos were already on my Mac’s disk when my Apple account got locked. As long as Apple’s iCloud stuff didn’t decide to remove them from my disk because my account was locked and it thought it might be the secure thing to do, then I’d be alright.

I’ve also been using Adobe Photoshop Lightroom (the cloud-ish one) some for the past few months, so I chose to backup those onto my local disk as well.

So I got back, booted up the Mac, installed Arq, and chose to backup my Photos and Lightroom libraries to the T7 💪 It took a few hours… then I felt a lot better.

I went ahead and reset my password again. By this time Apple seemed to have gotten a handle on whatever was going on, and after this second password reset things stabilized.

Still, I won’t ever trust iCloud Photos to be the only place where my photos live – I will make sure they are always somewhere else.

And to that end, now I had a physical backup here, but it is best practice to also have an off-site backup… so I signed up for Wasabi. They are one of the cheapest, yet still reliable, places to store stuff online and they only charge for storage – all the bandwidth / transfer is free.

I knew this would take a while, my upload speed at home isn’t great, and Arq handled it really well. It took more than 2 days for everything to initially upload, and now Arq is smart and uploads anything new every night.

So, because of Apple locking my account and scaring me, I now have nightly backups of my photos to a physical drive and a cloud drive:

It is tempting to feel grateful that Apple had this dumb account locking issue, because it spurred me to finally setup backups for my photos… but no it was still dumb. It really proved to me: my iPhone and my Apple account are too important. If I lose access, then I lose quite a lot… which is pretty annoying.

Smart Phones were fun, little, powerful toys for a while. Now they are required for day-to-day life. I am now motivated to start to unbundle stuff from my phone.

How I restored and setup my new (to me) $10 iPod shuffle (2nd Gen) with my M1 Mac running Sonoma

Tue, 30 Apr 2024 00:21:41 +0200

I’ve had a hankering for an iPod for a while, so I decided to take the plunge and get a shuffle to try it out. My phone is too important now! And I don’t really want to drain the battery all the time when I’m out and about… I already have a camera, now a music player, if I buy a physical map then I’ll have my own App Store in my backpack 🤦

Getting the iPod shuffle setup wasn’t straightforward… so here is how I got it all working.

I bought mine on Kleinanzeigen for 10 bucks. The person was very prompt. They let me use the in-app purchase system – which I prefer and recommend, don’t “send PayPal friends and family” to people. Not too scary, it all Just Worked™

1) Restore…

Restoring on macOS isn’t as easy anymore. The parts of iTunes that used to sync with iPods is now in the Finder! And my Finder crashed when I tried to restore the iPod. When I tried again, it hung forever and never finished. (If your Finder allows you to restore the iPod then you can skip right to step 2!)

What I learned is: iTunes still works and you can still download iTunes on Windows 💪

Running Windows is actually pretty easy these days with UTM. I had remembered companies like Parallels and VM Ware, but those cannot emulate x86 Windows on my M1 (ARM) Mac and we need the x86 version of iTunes… so UTM is it.

UTM is a whole deal

While UTM is free and it works well, it is not easy to figure out how to get it all working correctly.

And sadly, the best info I found is all glommed up in this YouTube video: How to install Windows 10 on M1 Mac – Full Tutorial using UTM app. Now I am very grateful that this video exists, but I prefer to not need to pause, back up, go forward, etc for an hour…

Here are the important steps in text:

Install UTM
Download a Windows 10 ISO from here: https://www.microsoft.com/en-us/software-download/windows10ISO (an ISO is a virtual CD, so this is basically the install CD)
Download the drivers you need from here: https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/archive-virtio/virtio-win-0.1.190-1/ (these are the Fedora drivers and they will help UTM know how to get Windows 10 working well, these are also on a virtual CD)
- Do not get the latest version of these drivers, only use this specific 190 version linked above. It’s tempting to use a version in the 200s, it feels like we should use the newest ones, but don’t do it. Use this 190 version.
Download the “Guest Tools” from here: https://github.com/utmapp/qemu/releases/download/v7.0.0-utm/spice-guest-tools-0.164.4.iso (you can read more about the “Guest Tools” here https://docs.getutm.app/guest-support/windows/, they just make UTM work better with macOS…)
Download and open a UTM VM template: https://web.archive.org/web/20230218145944/https://chrisp.cafe/UTM/Windows_10_x64.utm.zip (this is an archived copy of a template which used to be on chrisp.cafe’s website 🤷)
- Move the template to where you want to keep your virtual Windows 10 machine. Don’t keep it in Downloads or put it in your iCloud Drive or anything like that. Your Windows VM will eventually be multiple GB in size and I ended up with many GBs of stuff in my Downloads folder, which I didn’t want.
- Double click the template file to open UTM and you’ll see it in the UTM sidebar and a play button to start it up. Click play to run it.
- Using this UTM template is important because it’s already setup to prevent you from having weird network issues during the Windows install.
Use the CD icon in the toolbar to change to the windows 10 ISO as the disk
Reboot, and make sure to hit space so it boots from a CD or DVD
- If you pay attention when the computer is booting up, you’ll see it says “press any key to boot from CD or DVD” and that’s why we hit a key – we want to boot from our Windows 10 ISO
- This will boot into the Windows installer 🙌 we are getting there
Choose your language and tell it you don’t have a product key
- You won’t have to pay for Windows. You can use Windows for a short time without entering a product key and we will be done before that time is over. And we can always start over with a fresh VM in UTM if it starts to complain.
Choose Windows 10 Pro. Choose Custom Install (advanced).
When you get to the “choose a disk” screen, there won’t be any disks visible. We need to install the drivers so Windows can see them.
1. Use the CD icon to change to the virtio ISO file, the drivers virtual disk we downloaded. We are going to load 3 drivers.
2. Click “Load driver”
3. Navigate into the CD’s directories with “Browse”
4. smbus > 2k8 > amd64, “Next”
5. “Load driver” again, “Browse”
6. qxldod > w10 > amd64, “Next”
7. “Load driver” one last time, “Browse”
8. viostor > w10 > amd64, “Next”
9. Now the Drive shows up and can be chosen to install Windows onto it, but you can’t proceed with “Next” yet…
10. Use the CD icon to change back to the Windows 10 ISO
11. Click “Refresh”
12. Now you can click “Next”, and now it will take a very long time… until it asks you to provide a username
Provide a username, leave the password blank, “Next”, and it boots into Windows 🎉
Use the CD icon to change to the Guest Tools ISO
1. Open the CD in the Explorer
2. Run the spice guest tools application on the CD
3. Eject the CD once it’s done
Shutdown Windows
- If the screen is black for a long time, the VM doesn’t know that windows has quit so you can just power it off yourself (it’s like having a real, old computer…)
Click the icon to edit the VM’s options
Add a Network where the Disk stuff is
- Make sure it’s a virtio-net-pci one
Change the Display graphics to virtio-gpu-cpi
- Don’t choose Retina, it will be sloooooow
Now download and install an old iTunes version that still works
- Download and install iTunes from this page: https://support.apple.com/en-us/106372
- Don’t download iTunes from the Microsoft Store it can’t deal with iPods
- You’ll need to restart Windows after installing it
Plug in your iPod and UTM will try to grab the USB device from macOS, you’ll need to say “OK” or “Allow” twice for it to work. The iPod will now show up in Windows.
iTunes should now be able to restore it just fine, it might be slow though
Choose to use it as a disk in iTunes and apply that change

Now you can put music on there from your Mac, but only if UTM is not running, so shutdown Windows and completely quit UTM.

2) Adding Music…

I don’t want to deal with iTunes on Windows or Mac day-to-day… and it turns out one can just use the Finder to put music files on the iPod, run a python script in the Terminal and it will jam all those files into the iPod’s internal database.

The python script is from: https://github.com/andrewfowlie/shuffle-db3

First, you’ll need to install python and then install one package:

# if you don't have brew installed, then that is a whole deal and you can read more over at https://brew.sh

brew install python
pip3 install --break-system-packages eyed3

Then we can download the python script directly from the code repository.

cd ~/Downloads && curl -O "https://raw.githubusercontent.com/andrewfowlie/shuffle-db3/master/src/python/rebuild_db3.py"

In the Finder, drag the rebuild_db3.py file from your Downloads folder to the iPod’s drive that is mounted there. It needs to be at the root.

Now, second, make a folder on the iPod’s drive named “Music” – put your mp3’s and m4a’s in there. You can even do nested folders. The iPod will play things in file order by default, so you can name stuff to get them into the order you want like 01, 02, …

After you have them all in there, third, you can run this so the iPod will recognize and play those files:

# Make sure to replace iPod with the name of your iPod. If you don't know the name of your iPod, then you can type "cd " and then drag the iPod's drive from the Finder into the Terminal window and it will insert the path to it.

cd /Volumes/iPod/
python3 rebuild_db3.py Music/

So you’re telling python “rebuild the database using this Music folder.”

Or, if you’re like me and you will definitely forget all that, you can download my rebuild.command file and just double click that to run the rebuild script.
cd ~/Downloads && curl -O "https://gist.githubusercontent.com/myobie/5c0c25cc867420ba15885e53a9a3e133/raw/ac91f695f02c5c2b0b1e8a0c5a6398830eeddd08/rebuild.command"
cd ~/Downloads && chmod +x rebuild.command
Drag the rebuild.command file onto the iPod’s drive, and now you can double click that to run the python script for you. You can look here and see that rebuild.command file is just that one line that starts with python: https://gist.github.com/myobie/5c0c25cc867420ba15885e53a9a3e133

Tada 🎉

Now you have working iPod shuffle that can clip on your shirt and provide you music and happiness for 4–8 hours (ᵔᵕᵔ)/

(If I get time, I’ll try to update this with screenshots… we’ll see…)

Content Authenticity Initiative

Sat, 27 Jan 2024 12:02:23 +0100

I first learned about the Content Authenticity Initiative from Leica with the release of the M11-P, and at first I didn’t completely understand what it was about. Now, I feel sure this is one of the most important recent developments for photography, journalism, documentaries, and story telling. More and more, people are unsure if what they are seeing is “authentic” or “real,” especially when it doesn’t match their presuppositions. People are so quick now to claim “deep fake” or “AI” when something doesn’t make sense. And the only real way to help is: provenance.

The only time I’ve heard provenance in the past has been related to “can we determine this physical artwork is an original” or something like that. And now, we have the same problem with digital photos and videos: can we determine this is an original work, a documentation of something that really happened? So it makes perfect sense to me that the best answer we have is to develop a better system of digital content provenance – the state goal of the CAI: “Authentic storytelling through digital content provenance.”

I’m excited to see more companies and organizations join the CAI and to start seeing provenance information show up under images and videos more often 💪

(And I gotta start saving up now for an M11-P 😅)

Check it out →

1984

Tue, 23 Jan 2024 10:37:35 +0100

This morning I came across this image from the great Chris Silverman:

You should follow Chris on Glass and on Threads 👈 He uses the iOS Notes app to create amazing paintings. I posted a link to him previously.

Wow, the famous Mac commercial “1984” aired 40 years ago! I was two months old 😬 I’ve seen the commercial referenced over and over throughout my life, it’s one of those cultural phenomenons. I use and love my Mac everyday, I’m glad it worked out for them 😅

If you’ve never seen the commercial, here it is:

40 years is both a long time, and not really that long-a-time…

Check it out →

Don’t Be a Sucker (1947)

Mon, 15 Jan 2024 10:29:33 +0100

A short film produced by the United States Department of War released in 1947.

Check it out →

TIL about Promise.withResolvers()

Mon, 18 Dec 2023 11:38:06 +0100

Seen on mastodon (@patak@webtoo.ls), I am excited to see that this old trick is being standardized:

let resolve, reject
const promise = new Promise((res, rej) => {
  ;[resolve, reject] = [res, rej]
})

I’ve been using deno’s standard library’s deferred for a while, and I’ll be happy to switch to Promise.withResolvers() soon 💪

Check it out →

Creating an immediate stereo mute in Eurorack 🤔

Mon, 13 Nov 2023 13:40:55 +0100

I’m just starting to learn about Eurorack and modular synthesizers, so I think I’m definitely missing quite a bit here, but I couldn’t find an existing module or an online post for how to push a button to mute two voltages at once…

So, and I suppose this is a normal feeling when working with modular, I had to invent my own mute.

I’m learning about Eurorack using VCV Rack, the incredible program which emulates voltage controlled modules in a very nice, visual rack. Here is the patch I was working on yesterday (a “patch” just means “I wired things up this way”):

I’ve patched Plaits into Beads, and I wanted to be able to quickly mute the stereo output of Beads with a single click.

There is a built-in Mutes module, but it only lets me mute one thing at a time. I want to mute both the left and right output from Beads at exactly the same time with one click. So it’s a no-go.

In the short time I’ve been learning about modular I’ve learned that anytime I want to control the volume of something I want to reach for a VCA (voltage controlled amplifier). So I pulled two VCAs out into the rack and just starred for a bit…

I browsed through all the modules I have in my library in VCV Rack and saw a “Push” module, which is just a push button to go from 0V to 10V when pushed, and then back to 0V when released. That is close!

What I want is a toggle tho, I don’t want to have to hold the button down. I want to toggle between 0V and 10V.

Well, after just clicking around for quite a while, I accidentally learned that the “Hold” button in the Push module will “push and hold” the button, so it’s effectively a toggle for on vs. off. 🙌 So I already had all the modules I needed in place.

It was super fun to have to look at the tools available and reason out how to accomplish my goal. I may have chosen a long or difficult path, but I ended up where I wanted to be 💪

I bet this is why a lot of people get into and enjoy modular and Eurorack… solving problems and puzzles which result in sounds and music is super addictive.

Jordan Ward: Tiny Desk Concert

Thu, 09 Nov 2023 18:36:46 +0100

Check it out →

Cover of Backwater Blues by Sabrina, originally by Lightnin’ Hopkins

Sun, 05 Nov 2023 17:46:04 +0100

If you are not familiar with Lightnin’ Hopkins, then checkout the Wikipedia page about him and just generally seek out his music 🔊

Sabrina does a great cover here:

Here is a recording of Lightnin’ Hopkins:

Check it out →

TIL about “Weltschmerz”

Fri, 03 Nov 2023 11:03:40 +0100

The disillusionment we feel when the world falls short of our expectations.

Found via the Dense Discovery newsletter.

Check it out →

That photograph on the side of the house on Gartenstraße in Berlin…

Mon, 30 Oct 2023 11:14:27 +0100

This past weekend I saw a person with a nice Leica rangefinder camera standing outside the shop my partner and I were in, so I asked him about his camera, how long he had been a photographer, etc. I am getting more and more into photography and I love hearing from others about their photography, cameras, etc.

It turned out I met Christian Klopf, a doctor here in Berlin. He said he moved to Berlin in 1986 and has been taking photos here for decades.

Then, on his phone, he showed me an old photo he had taken and I recognized it immediately – his photo is printed on this house:

Christian Klopf’s photo of Gartenstraße from 1990

I walk past this photo often. There are many photos setup along the historical path of the wall in my neighborhood and it never occurred to me that the people who took those photos might still live here, might still be taking photos nearby. I mean of course, but also wow.

“Border strip and rear border area between Gartenstrasse and the Sophien parish cemetery, 1990”
– photo by Christian Klopf

Sometimes taking photos of mundane subjects or everyday activities can seem unimportant. However, it’s good to be reminded that documenting and capturing the state of the world is always valuable. You never know when a photo might end up being historically important or even a bit famous (famous to me at least 😆).

It was great to meet Christian and you can follow him on Instagram, like I do.

I am very inspired to get out there and take more photos. 💪

Unbundling AI by Benedict Evans

Fri, 27 Oct 2023 15:43:42 +0200

A good question he poses:

This is a science question, and a use-case question, but it’s also a product question - how do you present and package uncertainty?

I also liked this comparison:

But on the other hand, everyone in tech today is wrapping something else at some level. Snap uses Google Cloud for storage, but we don’t call it a ‘thin GCP Wrapper’.

Check it out →

Possibly Maybe poster by Dotto

Wed, 18 Oct 2023 13:24:41 +0200

I really like this beautiful print for sale by Dotto:

Check it out →

NASA astronaut back on Earth after record-breaking spaceflight

Tue, 17 Oct 2023 11:07:09 +0200

👀

NASA astronaut Frank Rubio is back on Earth after a record-breaking 371 continuous days in space.

Check it out →

Fazerdaze – Bigger

Wed, 11 Oct 2023 13:55:02 +0200

Check it out →

Why Rappers Stopped Writing

Tue, 10 Oct 2023 10:50:30 +0200

Found via Kottke

Check it out →

electric-sql – Local-first sync for Postgres from the inventors of CRDTs

Mon, 09 Oct 2023 10:18:56 +0200

👀 Very interesting. And I like that the project is already trying to handle permissions and access – which is one of the hardest parts of two way replication of data. 🤔

Check it out →

Desalination system could produce freshwater that is cheaper than tap water

Thu, 05 Oct 2023 23:21:51 +0200

Engineers at MIT and in China are aiming to turn seawater into drinking water with a completely passive device that is inspired by the ocean, and powered by the sun.

Check it out →

Convenience and scams

Thu, 05 Oct 2023 11:41:18 +0200

Found via swissmiss.

Check it out →

Little Desktop Sculptures by Mohit Bhoite

Fri, 29 Sep 2023 11:42:50 +0200

I found these lovely little sculptures on kottke.

They are so cute!

Check it out →

Oslo Transit Diagram

Wed, 27 Sep 2023 10:32:23 +0200

This Oslo transit map by T2 Design is beautiful 😻

There is a wonderful video about it too:

https://youtu.be/tHQ5fOJkI-g?si=zWpdyKHK5eFTscz

Check it out →

Water-Resistant “Research Notes” Notebook

Mon, 25 Sep 2023 13:45:53 +0200

I picked up this notebook a while back and I absolutely love it ❤️

Check it out →

new AsyncFunction – the async version of new Function

Wed, 20 Sep 2023 20:56:58 +0200

TIL

const AsyncFunction = async function () {}.constructor
const fn = new AsyncFunction('await Promise.resolve(1)')

🤯

Check it out →

Galileo Galilei’s First Drawings of the Moon After Seeing It Through the Telescope in 1609

Mon, 18 Sep 2023 15:56:54 +0200

Wow, incredible first sketch of the moon from Galileo.

Check it out →

Learning about Web Components: Part 4 – Researching pre-rendering custom elements and/or SSR

Sun, 17 Sep 2023 21:13:50 +0200

Now that I’m starting to get a handle on building custom elements which can do things dynamically in the browser, I also want to see how difficult it would be to implement “server side rendering” for my custom elements. There is a noticeable delay when first loading my photo grid demo – the browser must download, parse, and run my javascript, instantiate my custom elements, they then do a simulated database query, and then finally the image grid can render.

You can see the delay in this screen recording:

I’d prefer if the HTML returned from the server could already include the first version of the image grid, then the javascript could take over from there once it’s ready in the browser.

The announcement of Declarative Shadow DOM in WebKit and an update about support in Chromium are what really got me started looking into web components again in the first place. The promise of being able to serve HTML which can be shown without any javascript means web components are finally ready for use 💪 Firefox has announced they also will be implementing the agreed upon declarative API, so the future is finally here.

The first project I found which talked about declarative shadow DOM + SSR was ocean – which provides “web component server-side rendering” in deno or a service worker. lit-labs/ssr and astro’s SSR are also very cool libraries.

But those never really clicked for me. None of them seem to be both fully up-to-date with the latest specification + able to render any custom element I might build.

It clicked for me today when I was rereading the two announcements and I finally noticed something I hadn’t noticed before.

From the Chromium page:

Because the contents of a serialized Declarative Shadow Root are entirely static…

From the WebKit page:

In the following example, the outer template element contains an instance of some-component element and its shadow tree content is serialized using declarative shadow DOM.

Both pages mention that declarative shadow DOM is supposed to be the serialized version of a living shadow root. This made more sense when I found an older page talking about a not-yet-standardized and chromium-only built-in function to serialize a custom element getInnerHTML(). The example shows:

const html = element.getInnerHTML({ includeShadowRoots: true });

I am more interested in more of an outerHTML equivalent, but still I can work with this.

Calling getInnerHTML() on the <image-grid> in the demo page from part 3 of this series in Chrome returns:

<image-cell record-id="5">
  <!-- this shadowroot attribute is old and deprecated, see below -->
  <template shadowroot="open">
    <style>
      :host {
        align-items: center;
        display: flex;
        flex-direction: column;
        height: 100%;
        justify-content: center;
        width: 100%;
      }

      img {
        max-width: 100%;
        width: auto;
        height: auto;
        object-fit: contain;
        overflow: hidden;
      }

      .name {
        display: block;
        min-height: min-content;
        text-align: center;
      }
    </style>
    <img alt="Photo of a lot of differently colored tulips" src="./five.jpg" width="1000" height="662">
    <p class="name">five.jpg</p>
  </template>
</image-cell>
<image-cell record-id="4">
  <!-- this shadowroot attribute is old and deprecated, see below -->
  <template shadowroot="open">
    <style>
      :host {
        align-items: center;
        /* ... */
      }

      /* ... */
    </style>
    <img alt="Photo through a hexagonal window into a tea room on a sunny day" src="./four.jpg" width="662" height="1000">
    <p class="name">four.jpg</p>
  </template>
</image-cell>
<!-- ... -->
<template shadowrootmode="open">
  <style>
    :host {
      box-sizing: border-box;
      display: grid;
      gap: 16px;
      grid-template-columns: 1fr 1fr;
      margin: 0 auto;
      max-width: 600px;
      padding: 20px;
    }
  </style>
  
  <slot></slot>
</template>

The first thing to unpack here is: Chrome’s non-yet-standardized getInnerHTML() function is returning <template> elements with the old and deprecated shadowroot attribute. So this function isn’t going to work out-of-the-box for me.

Still, it never occurred to me that the DOM might know how to serialize itself. That SSR needn’t be so much about rendering templates or munging strings – instead be about building up a DOM to a state, then serializing it in a fully presentable way. This feels simpler and easier for me to think about now.

The second thing to notice is: when we serialize a shadow root, we must include a full template every time. That means if we have scoped <style>s in the <template>, they will be repeated over and over for every instance of the custom element. We could handle this by extracting the CSS to a file and @importing it if we cared about optimizing things.

The Chrome page says we don’t have to worry:

Styles specified this way are also highly optimized: if the same stylesheet is present in multiple Declarative Shadow Roots, it is only loaded and parsed once. The browser uses a single backing CSSStyleSheet that is shared by all of the shadow roots, eliminating duplicate memory overhead.

The third thing to notice is: our custom element will have a shadowRoot before our code calls attachShadow(). We don’t technically have to worry about this: calling attachShadow() will clear and return the existing shadowRoot which was created by the HTML parser when it found the <template> tag in our HTML.

From this information, I figured I could quickly build my own serializer which outputs the up-to-date <template shadowrootmode=open> attribute and works more like outerHTML.

I found a “getInnerHTML() polyfill” written by @developit (who works on preact and I follow on mastodon). It’s a bit terse and took a bit for me to grasp, but it’s nice to see that it can be a really short and simple function.

Here is my version. It’s a very quick draft which works for me in all the current major browsers. It won’t work for every use case and isn’t ready for production… it is a proof-of-concept though.

Here is the output from my serializer of the <image-grid> custom element from the previous demo page:

<image-grid aria-live="polite" role="region" aria-label="Image grid">
  <image-cell record-id="1">
    <template shadowrootmode="open">
      <style>
        /* ... */
      </style>

      <img alt="Photo of a building on a sunny day" src="./one.jpg" width="1000" height="838">
      <p class="name">one.jpg</p>
    </template>
  </image-cell>
  <image-cell record-id="4">
    <!-- ... -->
  </image-cell>
  <!-- ... -->
  <template shadowrootmode="open">
    <style>
      /* ... */
    </style>

    <slot></slot>
  </template>
</image-grid>

OK! I copied and pasted the serialized html from the browser’s console to a new demo-serialized.html file, I did some quick hacking to make the components check if the shadowRoot already exists, and now I have an example of what the resulting HTML could be like if I could do the serialization on the server.

I recorded a screencast of the page loading now, which is basically immediate:

The javascript kicks in and the images continue to re-sort every 10 seconds, just like before.

What’s next?

I don’t yet know how to run the serializer on the server – maybe one of those “mock DOM libraries” could make it all work out. When I get some free time I’ll dive into that.

If you have any feedback about any of this, if I made any mistakes, or if you just want to chat about web components or anything else, then find me on mastodon 👋

Learning about Web Components: Part 3 – Define in a static initialization block

Wed, 13 Sep 2023 09:50:09 +0200

TIL about: static initialization blocks and how this is useful when defining a custom element.

I knew about static properties, but I never noticed one can do a static {} block directly in a class in javascript. Here is an example:

class A {
  static {
    console.log('This output in the console immediately on page load')
    console.log(this) // same as console.log(A)
  }
}

The static block is run when the class itself is initialized, not when a new instance is constructed.

Custom elements are defined as classes and need to also register themselves into the custom element registry, and static initialization blocks can help keep that registration code contained within the class:

class ImageGrid extends HTMLElement {
  static {
    customElements.define('image-grid', this)
  }

  constructor() {
    super()
    this.attachShadow({ mode: 'open', slotAssignment: 'manual' })
  }

  // …
}

Super cool 😎

I’ve updated the demo page to use this technique and it works in all the modern browsers I’ve tried 💪

50 Short Rules for a Better Life (From the Stoics)

Tue, 12 Sep 2023 10:01:32 +0200

…

Be strict with yourself and tolerant with others.

Learn something from everyone.

Every person is an opportunity for kindness

Make a little progress each day.

…

Check it out →

The Feather Atlas at the US Fish and Wildlife Service

Mon, 11 Sep 2023 09:57:35 +0200

It’s a lot of fun browsing through this online feather atlas. Like the white tailed tropic bird tail feathers.

Check it out →

“We’re All Just Temporarily Abled”

Tue, 05 Sep 2023 11:10:45 +0200

1 in 5 people currently have a disability. 100% of people will have some form of disability in their lifetime.

Check it out →

Supavisor: Scaling Postgres to 1 Million Connections

Mon, 04 Sep 2023 15:10:52 +0200

One of the most widely-discussed shortcomings of Postgres is it’s connection system. Every Postgres connection has a reasonably high memory footprint, and determining the maximum number of connections your database can handle is a bit of an art.

And so we’ve built Supavisor, a Postgres connection pooler that can handle millions of connections.

👀🤔

Check it out →

Learning about Web Components: Part 2 – Data access and UI updates

Sun, 27 Aug 2023 13:02:22 +0200

I’ve finally spent a little more time learning enough to write a second post about Web Components. You can read Part 1 here.

This time I focused on two questions:

Sometimes the data I work with it local-only in a database in the browser – so how do I pass the database state down through the tree?
When sorting or filtering items, how can I do that without completely re-rendering the items from scratch every time?

I have a new demo page for this post where you can see the working code: demo.html

It’s all in one HTML file, so view source and you’ll see everything there.

Modeling the database state

I didn’t want to actually setup a full indexedDB database, so I made a fake db variable and “query function” with a fake delay. Querying indexedDB is always async (annoying) so I wanted to model that.

const db = Object.freeze([
  { id: 1, src: "./one.jpg", ... },
  ...
].map(i => Object.freeze(i)))

async function simulatedSlowDBQuery() {
  await new Promise(res => setTimeout(res, 200))
  return shuffle(Array.from(db))
}

I decided that the <image-grid> custom element can “contain” the list of image IDs it’s displaying. And then each <image-cell> can “contain” the ID of the database record it represents. So this means each “component” requires a local database to render, which is fine for my use-case for this post.

For this demo page, the page’s HTML just includes an empty <image-grid></image-grid>. There is an “outside function” which will query the DB and provide the record IDs to the empty grid:

const grid = document.querySelector('image-grid')

async function queryThenRenderImages() {
  const records = await simulatedSlowDBQuery()
  grid.recordIds = records.map(i => i.id)
}

// start with a query at boot
queryThenRenderImages()

// query + shuffle every 10s
setInterval(queryThenRenderImages, 10000)

I also like that it would be pretty easy to consume an AsyncIteratable to do a similar thing:

for await (const records from dbIterable) {
  grid.recordIds = records.map(i => i.id)
}

I do want to try to “server render” the grid and then hydrate it in the browser, that’s just too much for this blog post.

Reusing `<image-cell>`s when sorting

I’ve added a shuffle() to the database query so I can emulate sorting and filtering. One of the things I wanted to solve was how to easily handle sorting with these custom elements. It turns out the secret is the <slot></slot> element.

@keithamus tipped me off to this over on Mastodon (sheesh, back on May 7, it certainly took me a while to act on his advice 😂).

A slot is assignable, with something like: slot.assign(...nodes)

Two things that took me a bit to figure out:

The shadow DOM needs to have its slotAssignment set to 'manual'. This is how I’ve done that:
```
this.attachShadow({ mode: 'open', slotAssignment: 'manual' })
```
A slot can only be assigned nodes that are children of the shadow DOM’s host. Yep, that sentence is very specific, wordy, and was hard to be to fully parse and understand at first.

Here is a visual example from Firefox that really helped me get it:

In the above you can see a few things:

The <slot> is a child of the #shadow-root element of the <image-grid> element
The <image-cell> elements are all children of the <image-grid> element, and not children of the #shadow-root or <slot>
The <slot> shows that it has been “assigned” references to the <image-cell> elements

Since the slot only holds references to <image-cell>s, I can sort those references and re-assign without triggering a full re-paint of those <image-cell> elements.

At least, I think that is true. I haven’t been able to confirm that yet. I’ve tried profiling the page a few times, but I can’t seem to get proof about what is exactly happening. If you know, please email or @ me on mastodon.

A super nice thing is slot.assignedNodes() will return the currently assigned nodes, so I don’t need to remember those inside the custom element and can just get them anytime I need them.

`attributeChangedCallback`

While the recordIds of the <image-grid> are entirely “private” to the class, I decided to implement the record-id attribute of the <image-cell> element and actually wire it up to fully work (I think).

I can change the record-id="4" in the devtools and the element updates, which is pretty darn cool. But I’m not sure if this really is worth it or not. When passing data down through custom elements with attributes, everything has to be a string and I kinda hate that. So I’m not sure if trying to deal with serialization of these attributes is worth it to not yet.

The “update function” for <image-cell> reminds me of a configureCell() function when using UICollectionView or UITableView in iOS apps:

updateTemplate() {
  // “query” for the record
  const record = this.#recordId && db.find(r => r.id == this.#recordId)

  // only update if it’s a different recordId
  if (record?.id !== this.#lastRenderedRecordId) {
    this.shadowRoot.replaceChildren() // clear

    if (record) {
      const template = document.querySelector('template[data-name="image-cell"]')
      this.shadowRoot.append(template.content.cloneNode(true))

      const img = this.shadowRoot.querySelector('img')
      const pName = this.shadowRoot.querySelector('p.name')

      img.alt = record.description
      img.src = record.src
      img.width = record.width
      img.height = record.height

      const nameParts = record.src.split('/')
      pName.innerText = nameParts.at(-1)
    }
  }

  this.#lastRenderedRecordId = record?.id
}

I’d say I’m 80% happy with that function. I feel like there is still something I’m missing though 🤔

Use cases

I could see this being useful for lots of situations:

Search suggestions
Search results
Filtering any kind of list or grid
Navigating between tabs
Swapping out a page template by slotting in a header, sidebar/aside, and/or main content area with named slots
What else can you think of?

Questions still to answer

Is it common to support attributes with string serialization, or do most people leave those out and rely on javascript properties instead?
What examples are there online of Web Components + IndexedDB?
What are the accessibility implications of having elements in the DOM that are not slotted?
Passing data down two levels is fine, but how do things look when there are many many levels?

🫡

S.js and some thoughts about reactive state

Thu, 17 Aug 2023 15:09:37 +0200

I recently stumbled upon the S.js “reactive state library” and the related surplus web framework which builds upon it, and I wanted to recommend reading though it to any developer wanting to learn more about building web UIs with reactive state. The ideas I see in surplus are so similar to the ideas I’ve been working towards with with the Shareup team building new.space.

We’ve been iterating on how our application state is stored and how we interact with it in our UI components. 99% of the app’s state is stored in IndexedDB in the browser, so we really only need an in-memory mirror of the database’s state to hand to the UI when rendering.

Our web app’s UI is written using preact, and the default way to handle state in a contemporary (p)react app is to use useState() or useReducer(). Sure, there is redux, but the boilerplate of actions and all that is a lot for me.

useState() is great for a web UI that loads some data, then displays it, and that’s it. It’s not great for a web UI that holds all of its data locally in indexedDB and wants the UI to react to any changes to the underlying storage over time. One either has to put a giant state/reducer way up in a root component or context, or one has lots of little states that all get lost anytime a large part of the UI changes.

Navigating from one section of the app to another shouldn’t need to completely dump and reload from indexedDB – we just had that state and it doesn’t need to disappear when un-mounting. Yet, if one lifts the state up to a root-ish-level component, then the entire app UI will re-render for every little change to the underlying storage. Showing a progress meter is hilarious in that scenario.

surplus seems like the right idea: lift the state entirely out of the UI, make it reactive by default, and let the UI be a function of the reactive state. Only the parts of the app that use a part of the state need to re-render when that state changes.

We are using, and really like, signals which are surprisingly very similar to S.js… and S.js and surplus are both like 7–8 years old 🫢 I’m surprised I hadn’t come across it until now. Building high-performance web UIs is still a tough job all these years later 😅

Check it out →

She Was 300 Yards From the Atomic Bomb Center - and Survived Op-Docs

Thu, 10 Aug 2023 10:00:27 +0200

We saw Oppenheimer this past weekend, which was visually spectacular and somehow engaging for three full hours. Then yesterday I saw the above, and I haven’t been able to stop thinking about it.

Check it out →

Ólafur Arnalds live from Hafursey, in Iceland for Cercle

Thu, 03 Aug 2023 10:53:31 +0200

This has been such a nice thing to have on while working today 😊

Found via swiss miss.

Check it out →

Getting started with View Transitions on multi-page apps by Dave Rupert

Fri, 21 Jul 2023 14:15:18 +0200

I’ve been wanting to experiment with View Transitions, and this post by Dave Rupert helped me get going.

Check it out →

Omega Sports Apple Computer Sneakers • Size 10.5 • on Sotheby’s

Thu, 20 Jul 2023 22:40:06 +0200

Only 50,000$ with Complimentary shipping 😆

Check it out →

Dive – Olivia Dean

Mon, 17 Jul 2023 13:34:57 +0200

Check it out →

Embrace Complexity; Tighten Your Feedback Loops

Wed, 21 Jun 2023 14:56:01 +0200

Things that people think are useful are possibly going to happen even if you forbid them. If you forbid people from logging onto production hosts, and they truly think they’ll need it for emergency situations, they’ll make sure there’s still a way for it to happen, albeit under a different name.

…

The most basic point here is that you can’t expect to change the outcome of these small little decisions that accumulate all the time if you never address the pressures within the system that foster them.

Check it out →

Today – Q

Wed, 07 Jun 2023 10:31:14 +0200

I just listened to Q for the first time today and I really am enjoying their music 🔊

Check it out →

Nation of Language - Full Performance (Live on KEXP)

Mon, 05 Jun 2023 16:30:26 +0200

I love the vibes 🔊

Check it out →

Nopia

Wed, 31 May 2023 14:35:50 +0200

This makes me want to get out my Push and start making something…

Check it out →

Jordan Ward – CHERIMOYA

Thu, 18 May 2023 11:09:03 +0200

The “retro aesthetic” is strong.

It makes me feel old that the look of the camera equipment we had growing up is “retro” now.

Check it out →

Rich Girl – Scary Pockets – featuring Arielle Kasnetz

Mon, 15 May 2023 22:13:05 +0200

I love this arrangement 👀❤️

Check it out →

Great Wave Off Kanagawa As Pixel Art by James Weiner

Fri, 12 May 2023 10:20:29 +0200

Unbelievable detail on this beautiful 1 bit pixel drawing of The Great Wave ❤️

Found via Kottke.

Check it out →

The Man from Laramie

Mon, 08 May 2023 22:38:47 +0200

This past weekend I watched The Man from Laramie in the Criterion Channel’s Anthony Mann Directs James Stewart collection. This was the fifth and final film Mann directed with Stewart as the star, and it was a great watch.

Even with all their problems, I still love a good Western. I grew up watching them and they still captivate me. Especially one that has something interesting to say – and that’s what surprised me the most about this movie: it’s politics seem pretty progressive for 1955.

Some plot spoilers…

The motivation of Stewart’s character was to hunt down and kill the person he felt responsible for his brother’s death. His brother was in the Cavalry and his entire group had been gunned down by nearby Native Americans.

Yet, he didn’t blame nor was looking for the Native Americans, he was instead looking for who sold the “repeater rifles” to them. Since, to him, the person who sold the arms is just as guilty, or more, as the people who used them. The movie presents this as a very plausible and sensible motivation.

This is currently being debated back home in the US: can gun manufactures be held liable for how their weapons are used? Now manufacture vs. dealer is different, sure, but it really seemed so “current” when watching this film.

Also, this movie appeared to be “trying it’s best” for 1955 and not fall into every trope or harmful portrayal… but still it’s a Western from 1955 and does briefly present indigenous peoples in disparaging ways.

It has 100% on the tomato meter, and it deserves a high score for sure. Stewart’s performance was captivating, and at times gut wrenching. I recommend giving it a watch. And I plan to watch the other four films in the collection.

Learning about Web Components: Part 1 – Getting Started

Sat, 06 May 2023 15:35:45 +0200

I’ve become interested in Web Components again. I’ve kept up with the smattering of standards that make up “Web Components” over the years, but since they were client-only and couldn’t really be server-rendered, they weren’t as interesting to me. Things have changed. declaritive shadow DOM makes server rendering possible and webc seems like a neat way to do single file components (since html imports didn’t make it, which was super sad).

I have a good understanding of how to make a web component, but I don’t really understand yet: how to compose a tree of web components, how to pass state changes down, is it easy and intuitive to bubble events up the tree, and how to do something similar to preact’s “render loop” or even if that type of loop would be desirable.

I learn best by doing, so today I want to ease back into web components by making a simple grid of images. Over time I’d like to enhance the grid to be reorderable, editable (add/remove images), and maybe even provide a custom name for each image… but that’s for another day.

I’ve written this part out before I started on the actual components, and we’ll see how far I get in 20 or so minutes…

I managed to create an image-grid component with image-cell children which can load and show images in a nice grid.

View the demo.html page here.

The html looks like:

<image-grid>
  <image-cell src="./one.jpg"   width="3607" height="3024"></image-cell>
  <image-cell src="./two.jpg"   width="3448" height="4592"></image-cell>
  <image-cell src="./three.jpg" width="4592" height="3448"></image-cell>
  <image-cell src="./four.jpg"  width="3264" height="4928"></image-cell>
  <image-cell src="./five.jpg"  width="4225" height="2798"></image-cell>
  <image-cell src="./six.jpg"   width="4928" height="3264"></image-cell>
</image-grid>

This is a good start and I learned a few things:

Using <slot></slot> to mean what children means in preact works fine I think
:host {} in the CSS is how to style the custom element itself, which is very handy
The styles seem truly isolated. I tried to add body { background-color: pink; } inside the custom element’s template and it had no affect. I also tried to style img { outline: 5px solid purple; } in the parent image-grid template and it didn’t affect the images inside the child image-cell template. This is what I expected, but it’s the first time I’ve proven it to myself with a working example.

Some notes for myself for next time:

I think I need to break the <style> in the template out into a .css file. If I server render this thing it’s going to duplicate the entire <template> for each invocation, including all the <style>s.
I don’t like how separate and indirect the <template> is from the customElements.define(…) call.
I don’t like how I’m cascading the height, src, width attributes from the image-cell to the img tag right now. I’m not sure how to make it better, but it just feels weird to me.
I always forget how to use css grid to layout images and keep their aspect ratios correct. It’s a battle.

OK, that’s all for today. We’ll see when I take time to make the components more dynamic along with being server rendered.

Mighty Vibe

Wed, 03 May 2023 21:44:11 +0200

A very cute music player that reminds me of the iPod Shuffle. I absolutely loved my iPod Shuffle back in the day, and it’s very appealing to just have a dedicated “play me some great music” device.

Found via @dancounsell@techhub.social on Mastodon.

Check it out →

Generate a QR Code in your shell with npx, qrcode-svg, and svgo

Mon, 01 May 2023 17:54:58 +0200

A very interesting shell script snippet from Vadim to quickly generate a QR Code in your shell. A slightly modified of their original script is:

function qr() {
  npx qrcode-svg --join --no-prettify --viewbox --padding 0 --ecl L --force --output $1.svg $2
  npx svgo --quiet --precision 0 $1.svg
  open $1.svg
}

Check it out →

Testing the Web Share API

Fri, 28 Apr 2023 15:08:40 +0200

Raymond Camden has a good overview of how weird the Web Share API works on different platforms. Working on Shareup’s apps we’ve also seen how strange the native share API can be, and we really only use it on mobile platforms right now.

Check it out →

Urth Explorer Filter Kit

Wed, 26 Apr 2023 23:02:45 +0200

👀

Check it out →

GaeaStar

Mon, 24 Apr 2023 23:15:05 +0200

I’m very interested in these sustainable single use bowls and cups. I am going to have to go to Rosa Canina to get some ice cream and try one out. I wonder how if it will feel stable and solid, especially with something like ice cream or coffee 🤔

Check it out →

The original Wilhelm Scream recording session

Fri, 21 Apr 2023 10:25:56 +0200

I was really intrigued to hear the original recording of the famous Wilhelm Scream, including the context and direction given to the performer. Craig Smith is an audio preservationist and took great pains and care to get the audio from the Sunset Editorial Sound Effects Library collection of tapes, which were used in many TV since the 60’s – the most famous sound being the Wilhelm Scream.

Craig uploaded the sounds to freesound.org under a public domain license, and I used ffmpeg to transform the .wav file to a .mp3 so that you can play the original Wilhelm Scream recording session right here:

Listen over at freesound.org

Converted with the terminal command:

ffmpeg -i scream.wav -vn -ar 48000 -ac 1 -b:a 192k scream.mp3

Found via waxy.org.

Check it out →

Moving, a Short Film

Thu, 20 Apr 2023 10:14:35 +0200

I really enjoyed this short film by Adinah Dancyger that I watched on the Criterion Channel online. If you don’t have a sub to Crtierion, then you can find it on Short of the Week.

Check it out →

An Interview With Massimo Vitali

Wed, 19 Apr 2023 12:28:40 +0200

How do you manage your personal and your work life? Is there a clear separation between the two for you?

Massimo: Always separate for me. The studio is not far from my house, a nice place where I can go by bike. I remember years ago I had my studio and my home in the same place … impossible! The simple act of riding my bicycle for four minutes changes my mind into studio mode and when I go home I think about home.

Check it out →

Dreamy Photography Settings

Tue, 18 Apr 2023 21:03:33 +0200

🤔 I’ve been trying to lean more about how to develop my photos and this really caught my eye.

Check it out →

VIKING Trailer

Mon, 17 Apr 2023 16:19:58 +0200

👀 This looks so interesting… 🤔

Check it out →

Ship the wrong thing to learn what’s right

Fri, 14 Apr 2023 12:23:55 +0200

A post I saw on Mastodon reminded me of an important principle for iterating and learning:

Chroningham’s law states: “the best way to get the right answer on the internet is not to ask a question; it’s to post the wrong answer.” – https://tinnies.club/@jameswhite

I’ve often used this method to find my way forward. If I don’t know how to make something visually nice, I’ll make something ugly and then ask for feedback. People find it much easier to tell you how to make something better from a certain point vs. providing feedback on how something theoretically might look. Write bad code. Draw bad pictures. Ship an ugly prototype. Getting the wrong thing out can help guide you to the right thing.

I recommend trying it out. Make something bad, and then ask for help or advice. You might be surprised how quickly you can make progress using this methodology.

Who invented vector clocks?

Wed, 12 Apr 2023 11:03:24 +0200

…vector clocks (and matrix clocks, even) were originally thought up and used by a bunch of people in the early-to-mid-80s…

Check it out →

Gawthorne’s Hut

Tue, 11 Apr 2023 08:10:10 +0200

This is a beautiful little hut 🤩 in Mudgee Australia that is listed on AirBnB. Designed by Cameron Anderson.

Found via The Design Files.

Check it out →

You Are Not Okay and Tomorrow Will Come

Wed, 05 Apr 2023 11:22:27 +0200

Everything is terrible and I can still change the clock battery.

A short read which really resonated with me.

Check it out →

Art by Nathan Walsh

Tue, 04 Apr 2023 10:41:05 +0200

Nathan Walsh’s work is stunning. Checkout his site to see more, and give him a follow on Instagram @nathan_walsh_artist. I cannot get over how detailed, meticulous, and hyper-realistic his work is 🤩 Unbelievable.

Found via Colossal.

Check it out →

Slow Learning

Mon, 03 Apr 2023 14:02:01 +0200

This “Manifesto for Slow Learning” linked to by swissmiss is really great. The ten points of the “Bill of Rights” are:

Focus on direction, not destination
Raise your hand
Learn at your own pace
Unplug
Change your learning path (and mind)
Take a break
Make mistakes
Leave it unfinished
Unlearn and forget
Slow down

Check it out →

The Art of the Copyist by The Met

Fri, 31 Mar 2023 11:24:35 +0200

Check it out →

Polaroid Lab

Wed, 29 Mar 2023 12:05:34 +0200

❤️ such a cute little photo printer!

Via swissmiss

Check it out →

Inside Out. Understanding the art of furniture making

Tue, 28 Mar 2023 11:10:45 +0200

I was reminded today of this furniture exhibit I saw back in 2019, and how amazingly nested the compartments were in some of the desks and cabinets. In this video they show you all the little nested areas in a piece:

It was amazing to see the pieces in real life. I recommend checking out a similar exhibit if one ever shows up near you.

Check it out →

Men I Trust - Show Me How (garage session)

Mon, 27 Mar 2023 13:29:43 +0200

Check it out →

Olamide Ogunade’s art

Wed, 22 Mar 2023 10:45:30 +0100

I love these arresting images with the playful bubbles floating throughout. This is from their “Artist’s Statement” page:

My art is an expression of figurative images using charcoal and colors to give it a contemporary look. It talks about situations in African society. The skin is an essential thing. I love to create it by focusing on such details as contrast. The dark hue of the skin is usually accompanied with vibrant colors for that complementary yinyang color contrast. I then sum up the work with bubbles – this captures the temporariness of the pressures we face.

Found via booooooom.

Check it out →

Behringer TORO

Tue, 21 Mar 2023 10:45:30 +0100

These wonderfully crunchy bass sounds 🧑‍🍳😗

Check it out →

Chompi

Mon, 20 Mar 2023 10:50:30 +0100

I love this cute little tape music instrument named Chompi ❤️ I really want to get into these types of instruments more.

Check it out →

Enhancing Directed Content Sharing on the Web (PDF)

Thu, 16 Mar 2023 11:57:04 +0100

This paper about sharing web content is a good read. They created a browser extensions named FeedMe, where:

the user mediates recommendations not for themselves, but on behalf of someone they know… FeedMe’s users act as gatekeepers for someone else…

One of their conclusions really resonated with me:

Link Sharing Is Burdensome When It Is a Repetitive Firehose

That feels very true for me. It’s a good paper, give it a read.

Check it out →

Largest known prime number

Wed, 15 Mar 2023 12:20:55 +0100

2^82,589,933 − 1

Which has 24,862,048 digits.

🤯

(I like how the Wikipedia page says “24,861,808 digits skipped” 😆)

Check it out →

GPT-4

Wed, 15 Mar 2023 11:13:46 +0100

The latest update to Open AI’s preeminent model seems very impressive. It’s better at exams and it can now take images as input, along with text prompts.

Up until now, I haven’t felt completely mystified about ChatGPT, and what it’s doing. It has been easy to tell myself “it’s just choosing the next word using a very fancy weighted decision network,” until I saw this example:

It’s given a scan of a page in French, with a setup, a diagram, and then two questions. The text prompt provided asks it to answer one of those questions from the scan. They didn’t tell it what the question was in the text prompt, they only provided “Answer question I.1.a.”

This feels like a slight-of-hand magic trick which I can’t figure out. It doesn’t feel like it’s just predicting the next word, even if that is true. It feels like we are in for some big changes in the next year, the next five years.

When large multimodal models can ace any exam, when they can offer very good advice for most problems a person might face, the definition of expertise will need to change.

Would you want to hire a human who got a B throughout their university time and is a “expert”, or would you want to hire a human who didn’t go to university and has a large model sidekick to guide them? Who would really be the expert? Even if the “traditional expert” also had a large model sidekick to help them, how much more of an expert are they really? Will the delta be that large?

Expertise might end up being not that important. Instead, it feels like what will be important will be things which are: manual.¹

Of or pertaining to the hand

Performed by a person using physical as contrasted with mental effort

“Mental effort” might be a lot less important soon. “Manual effort” will be special, and maybe more rare.

When photography came onto the scene, people didn’t 100% stop painting. It became special, and rarer. Most people today take selfies, or maybe they hire a portrait photographer, they rarely hire a portrait painter.

Large multimodal models feel like nascent “photography for mental effort.”

Tom Scott made a related video a month ago, primarily asking the question “where are we on the S curve for AI tech?”

It certainly feels like we are at the start of S curve today.

I often consult Webster’s Unabridged 1913 Dictionary – it always helps me think more broadly and clearly. ↩︎

Check it out →

MathGPT

Tue, 14 Mar 2023 12:20:55 +0100

Use GPT3 to solve math problems, and get the code behind each solution!

This is pretty cool, it asks ChatGPT to write python code to solve math problems. It is really pretty amazing to see.

I did wonder if it was actually doing the calculation or if it had memorized a lot of calculations, so I did a test that you should not do.

I asked it to find the 67,879,887,654,898th prime number. I feel bad, because it tried to find the prime, and the entire service went down. It was down for about 40 minutes. It did not find the prime.

I’m not sure how one could protect against DOS attacks when the goal is to have a ML model write a computer program, and then execute it. 🤔 One would have real uptime problems without any protections, but maybe that doesn’t matter if the system can timeout anything and just restart itself.

Check it out →

Cargobox by Meelis Lillemets

Mon, 13 Mar 2023 13:19:36 +0100

I love this absolute unit of a truck! ❤️

It’s so cute, and it’s a great idea to try out different designs of electric cars for different purposes. At low speeds, the vehicle doesn’t need to focus as much on aerodynamics, and can prioritize the cargo space.

Check it out →

TIL how to allow notifications when screen recording on macOS

Sat, 11 Mar 2023 12:03:22 +0100

Today I was trying to make a screen recording on my Mac of some notifications coming in, but anytime I started recording the screen the notifications would no longer show up. When I stopped recording, there they were 😡

There is a setting in System Settings to “Allow notifications when mirroring or sharing the display” which is off by default. That makes perfect sense, if you are presenting or sharing your screen you don’t want people to see your notifications.

So I toggled it to on, was able to do my screen recording of the notifications, and then toggled it back to off.

TIL about Beamer for LaTeX

Thu, 09 Mar 2023 19:20:36 +0100

I’ve been super interested in using markdown to make slides, and especially using iA Presenter, and so I was thrilled to learn about Beamer. I don’t totally get it yet, but it’s super cool to dig into and look through the examples in the repo.

Check it out →

Notes art

Wed, 08 Mar 2023 17:24:34 +0100

I love these beautiful pieces done by Chris Silverman in the Notes.app on an iPhone. Constraints can really push someone to make impressive and beautiful things.

Found via Kottke.

Check it out →

Patch Notes: Hélène Vogelsinger

Tue, 07 Mar 2023 10:17:58 +0100

100% life goal to have modular synths, in a rundown country house, and make wonderful music.

Check it out →

The Full Frod Story! Building a frog house for a Frod.

Mon, 06 Mar 2023 23:19:35 +0100

This is lovely 🐸

Check it out →

The Shortcut

Fri, 03 Mar 2023 08:58:58 +0100

I don’t think I can include a still screenshot that conveys how satisfying this animation is. The Shortcut by Martin Wattenberg is a wonderful thing to watch play out.

Check it out →

I really love these paintings by John Gannam

Thu, 02 Mar 2023 21:23:04 +0100

I fell in love with these painting by John Gannam, an American artist from the mid-1900s.

Checkout more of their work on artnet and read more about them on Society of Illustrators.

Cassette mixing 🤯

Tue, 28 Feb 2023 20:40:22 +0100

Via @musicthing.

Check it out →

Dublin Goth New Wave Movement, 1989

Mon, 27 Feb 2023 21:50:12 +0100

Via things magazine.

Check it out →

The Circle (Shortfilm)

Sun, 26 Feb 2023 22:52:46 +0100

Via Swissmiss.

Check it out →

TIL about the useUnknownInCatchVariables compiler flag for TypeScript

Sat, 11 Feb 2023 00:08:28 +0100

Today I was debugging a confusing situation.

The variable inside a catch statement in TypeScript is any by default, which is awful
Safari was throwing undefined when trying to place a webm file into a <video> element, which was breaking my brain 🤯

I had mistakenly done this, assuming e would be an Error:

try {
  putIntoVideoTag(file)
} catch (e) {
  logger.error('What‽’, e.name, e.message)
}

So when e was undefined, it exploded.

One can manually type the catch variable so the compiler won’t let you accidentally call e.name, like:

try {
  putIntoVideoTag(file)
} catch (e: unknown) {
  logger.error('What‽’, e)
}

But that means I can (and will) forget to do it.

Well, TIL that TypeScript now supports a flag to make all catch variables default to unknown which is great: useUnknownInCatchVariables

It was added in the 4.4 release.

This is now the default in all my TypeScript projects ✨

Check it out →

Mid-day ☀️ 📸

Tue, 07 Feb 2023 20:20:32 +0100

I took this earlier today and it really turned out fantastic. 😊

Living Alone in the Wild Siberian Forest for 20 years in Yakutia

Tue, 07 Feb 2023 19:51:53 +0100

I watched this the other day and I’m still thinking about it. While I’d sometimes love to live in a cabin in the woods, I’m not sure if this is how far I could take it. 😳

It’s a lovely video, lovingly made, and worth a watch.

Check it out →

Everyone thinks everyone is a scammer these days

Tue, 07 Feb 2023 13:30:54 +0100

Or at least that’s how I feel right now. My personal site here doesn’t get a ton of updates, but I try to keep it alive by posting links to cool stuff I find on the internet from time-to-time.

Well, today, I got a very hateful and angry email from a dude, because I posted a link to their website. I’m sure he’s a nice guy in real life, but this wasn’t the best email for me to get this morning.

You can read about it on the page for the link post he is so angry about: Modular Office Unit.

~~I took the link down, because I don’t want anyone to be upset because of a dumb link I posted. It’s just a link.~~

The email really shocked me, and so I took the link down because I had some other stuff I needed to get to. But after taking some time to think about it, I decided to put the link back, add a big disclaimer, and just describe what happened. You can read the post and determine for yourself if I am claiming to either build furniture or write long-form prose about furniture on some other domain…

Is my site confusing? Does he think I’m claiming I made a Japanese pop album in 1982, before I was born? Is it unclear that I am a real person, and I just post links to things I think are cool?

I guess not everyone knows that “links” means links to other places, other things that are not related to me, so maybe I should go and add “to other places” or “I promise I am not claiming I made this, it’s just a link” to every link?

In the moment, I feel more wary of posting any more links at all here. I’m sure I’ll get over it by the end of today. Maybe I should post more often so it’s clear I am a real person…

Super weird email to get tho, for sure… I gotta shake it off and get back to work.

Update: I made some small changes to my site to make super extra clear that links to sites that are not “my name dot com” are to things that I don’t own or have anything to do with. I thought that was obvious before, and now hopefully it’s super obvious. Sheesh.

I updated this article on 8 Feb, 2023.

Love Trip – Takako Mamiya

Thu, 12 Jan 2023 21:05:05 +0100

Check it out →

Had to update Whitney – the font here

Sun, 08 Jan 2023 11:28:25 +0100

I use typography.com’s cloud service so I can use the wonderful Whitney here for my personal website 🥰

However, with the last post I made, the artist’s name had a character in it which didn’t show up correctly: ć. This bothered me quite a bit, so I dug into the “Cloud Dashboard” to see what might be the issue, and it turns out the settings (which I assume are the defaults) are to not provide every possible character as part of the font package – I assume to save some bytes.

I changed from “Basic Latin Accents” to “Extended Latin Accents” and now the ć shows up.

Side note: I am not sure if this “Cloud.typography” service is really being maintained. The dashboard UI has never changed, and appears really tiny on any modern computer screen. They don’t really link to the service from their navigation at all anymore, I had to google the service to find the url to its page. They do link to it during checkout, but it makes we wonder if maybe it’s being deemphasized.

Either way, this is the only way I can use Whitney here, so it’s worth the trouble I guess. 😞 Maybe I should buy the Digital Type Capsule, download them for offline use as woff files, and use those instead?

What’s the lesson? Just use the entire font file, don’t try to be cute and slim it down. You never know when you might want to use a character which the font supports, but with your version doesn’t include.

Ana Vidović playing classical guitar

Sun, 08 Jan 2023 11:05:00 +0100

Check it out →

Robohands - Green

Sat, 07 Jan 2023 18:53:36 +0100

I’ve been listeneing to this super good ambient jazz album on repeat while working. You can get it on Bandcamp or on Apple Music:

Check it out →

Casiopea - Mint Jams

Fri, 06 Jan 2023 11:50:04 +0100

I have been super into this album (Apple Music link). Super great music for when I’m working this week. I found it on YouTube, which has been recommending me music like this recently for some reason 🤔

The full album is also available on YouTube, if you don’t have/want Apple Music:

Check it out →

Hydraulic Press Interpretive Dance

Thu, 05 Jan 2023 09:45:11 +0100

I love these videos where Smac McCreanor imitates the viral hydraulic press videos with dance. The one embedded here is the entire series in one video:

Via Kottke.

Check it out →

Modular Office Unit

Wed, 04 Jan 2023 13:56:35 +0100

Update about this link post:

I originally found a link about this modular office unit thing via swiss-miss.com, I thought it was super cool, and so I posted the same link here from my site.

Well, a dude at gessato.com sent me a very ugly email, asking me to take this link down. It seems he thinks I am taking credit for building the office thing, or writing the article about it, or something to that effect, as “a scam.”

Here is what he wrote me:

Hi Scammer,
why are you so full of shit of claiming you wrote this piece - https://www.gessato.com/modular-office-unit/

Take that link down immediately fucking scumbag.

That’s the entire email. It shocked me and I wrote about it.

I tried to explain that this is my personal website and it is obvious I did not make the music video by Little Sims, or record the Japanese Pop Album from 1982, or make the movie trailer for God’s country that I linked to, just like it’s clear I didn’t make this furniture or write anything for gessato.

Sadly, that didn’t make things better. He replied:

WHY??
because you are a liar. Because your web page is very deceiving and it makes users believe that you actually WROTE that piece.
We take pride in our content and do not tolerate people claiming ownership over it.
Just remove the damn link and move on.

So, I guess just to be super extra clear to you the reader: I did not build this office unit, nor do I have anything to do with gessato.com which should be obvious, but apparently is not.

A link from my site does not mean that I endorse the other site, that I work for or created the other site, or that I even know anything at all about the other site. I post blog posts and link posts on “my name dot com”, and so any content you see on “any other name dot com” is obviously unrelated to me. You can even tell which posts here are links by the link emoji (🔗) which has been next to them for years.

The modular office unit was actually created by Guglielmo Poletti (who is obviously not me). I found it linked to by swiss-miss.com which is obvisouly not my site (because it’s not “my name dot com”). I sure hope it’s clear that I don’t make furniture, nor write long articles about furniture.

You can read below the short sentence I originally wrote about the office unit (which is definitely cool looking), and judge for yourself if I’m taking credit for something I shouldn’t.

I absolutely love this modular office unit! It includes a work area, tons of storage, a sleeping area, and even an elevated sitting area.

And then, I originally had linked this post to www.gessato.com/modular-office-unit so you could go there and see more about this office thingie. I do regret linking to them now, for sure.

I updated this on 8 Feb 2023.

I changed the primary link to the website of Guglielmo Poletti, since he actually made the object in question, he has photos of if there if you want to see them, and I don’t think he’ll send me any angry emails about my link to his site.

I also originally linked to a desk, which was unrelated, but I’m just going to let this now only be about the modular office unit thing since it is such a big deal apparently.

And finally, I also updated my site’s design to not only put a link emoji (🔗) next to link posts, but now put a star emoji (⭐️) next to blog posts. Maybe that will make it even more clear what a link is and what a blog post is. Sheesh.

Check it out →

TIL about IndieKit

Wed, 04 Jan 2023 10:01:01 +0100

TIL about IndieKit, which is a small server to allow one to quickly post to a statically generated website (like mine), syndicate posts to other sites like Twitter or Mastodon (like I want to do), and seems pretty customizable.

I’ve had things setup on my phone for a while where I can write a markdown post using iA Writer, committing it using Working Copy, and then it gets deployed with Netlify… but it’s all too much and too brittle.

I really want a simple box I can type some text into, click “publish,” and then it appears here on my site. I don’t post as often as I’d like because of how difficult it is for me to post from my phone.

I’m not sure if IndieKit is the answer for me (I really want a solution that uses Cloudflare Workers if possible), but I’m going to set something like this up soon… I hope 🤞

Check it out →

NO THANK YOU – Little Sims

Sat, 17 Dec 2022 18:18:09 +0100

Check it out →

God’s Country Trailer

Thu, 11 Aug 2022 11:04:43 +0200

Check it out →

New ❤️ album: Invisible Cities by A Winged Victory For The Sullen

Mon, 09 May 2022 11:02:58 +0200

I’ve been a huge fan of A Winged Victory For The Sullen since their first album back in 2011. I’ve listened to that self titled debut album so many times, it’s always at the top of any of my favorites or recap playlists on Apple Music 😅

I also love to listen to anything by Duston O’Halloran (half of the group), especially the soundtrack to Breathe In. That is also an album I’ve listened to way to frequently, and yet never enough.

Check it out →

Some technologies last

Wed, 26 Jan 2022 08:33:00 +0000

When reading Jeremy Keith’s latest post, I was reminded that a lot of people seem to expect all technologies to be short lived and frequently replaced – when this is not really true at all. Sure, tons of things come and go like phones, apps, and websites, but HTML has been with us for 29 years and will for sure be with us for the next 30 and more. (There are tons of other things too, like C and ASCII, which will probably also be with us forever…for better or worse.)

The anchor element (or “link” as we usually call them) is one of the most important inventions ever because it is what creates “the worldwide web.” The internet is the plumbing so computers and devices can all be connected with wires (or wireless) and find each other, which is great, but the real value is “the web” of links we use to find sites and apps to talk to each other, purchase things we need, and generally live our lives. Even in some sort of “VR ‘meta’ 3D future” there will also be a need to link from one place to another, “hyperlinks.”

TIL about explain (analyze, buffers) in postgresql

Sun, 23 Jan 2022 18:55:28 +0000

While reading some postgres tips today, as one does, I noticed a parameter to explain I hadn’t seen before:

EXPLAIN (ANALYZE, BUFFERS)
SELECT * FROM users WHERE lower(email) = 'email@example.com'

buffers would have been super useful recently when I was debugging some queries! And there are more I didn’t know about like costs off, timing off, and summary off. Next time I’ll be more prepared.

Check it out →

3 out of 3 💉💉💉

Wed, 12 Jan 2022 14:20:18 +0200

I’ve gotten my third shot and, like the other two, I had a pretty high fever for about 24 hours. For three out of three shots I’ve had the same side-effects: high fever and arm pain. My expectation is we will get a shot every year and that I will have the same side-effects each time 😅 I don’t mind, it’s very much worth it, and it’s not hard to just wait when it’s expected and extremely likely to subside soon. I’m grateful to have access to free vaccines and to have pretty light side-effects after 💪

TIL about Gaia

Fri, 07 Jan 2022 22:30:57 +0000

Gaia is a space observatory launched by the ESA on 19 Dec 2013 to map the positions and velocities of stars in our and nearby galaxies! It orbits the sun at L2, just like Webby Scopes will.

It’s amazing to see the star charts produced and reminds me of the navigational charts used in many SciFi films. See this example:

Check it out →

Where Is Webb?

Mon, 03 Jan 2022 11:05:50 +0000

It’s amazing we can see the speed and position of the telescope in real-time. Also, temperate data recently started streaming down 💪

Check it out →

The Dark Forest

Sun, 02 Jan 2022 11:54:19 +0000

This animation style is so nice 😊 Also, I never considered the idea that it could be better to stay quiet and not broadcast out our existence 😬

Check it out →

The James Webb Space Telescope

Sat, 01 Jan 2022 11:33:19 +0000

It’s been a very exciting week! In case you aren’t aware, we (humans) have sent a new telescope out into space. Unlike Hubble, it’s tuned to see infra-red with the goal of collecting the red-shifted light just getting to us from the earliest galaxies and stars.

This one won’t orbit the Earth, instead it will orbit the Sun along with the Earth. It will sit at a spot called the “L2”, where the gravitational effect of both the Sun and Earth are equal on the satellite. This is the first object we’ve ever tried to install into a Lagrange point! Also, it amazes me that someone discovered and calculated these points in 1772 👀

It’s a pretty stressful situation. First, the launch had to go well (which it did). Then, the separation has to go well (which it did). Now, as it travels a million miles to the L2, it will slowly start to expand and unfold. If anything goes wrong, then they’ll try to rotate and shake it and things remotely, but it’s not an option to send anyone over to it for repairs 😬

It’s so exciting to see a what we can do if we work together, if we put our money toward a goal, and if we stick with something even if it is difficult and takes a long time.

Videos

Watch this awesomely cringe video that explains a lot and is just very enjoyable to watch:

Or the best, short video I’ve found is this one:

Some things take time

I have this poster hanging in my home:

I’ve been thinking about it more and more as I learn more about the JWST and what it took to build and deploy it. It’s amazing and inspiring to me to see the effort everyone put into JWST succeed.

Check it out →

Collaboration through software: digital paper and card catalogs

Tue, 04 May 2021 08:22:42 +0200

Recently, as we work on completing our proof-of-concept version of Shareup, we’ve often been talking about different ways we can best help people work together through our software. I’ve never taken the time to write down how I think about collaborative software, so in this post I’ll attempt to. These concepts are how Adam, Anthony, and I are approaching building our new product and drive a lot of our conversations about how it will work.

Over the past couple decades, while working on different distributed software and systems, I’ve been able to refine my thoughts about how to categorize and talk about collaborating through software. Also, I enjoy reading about software history and the many HCI research papers one can find about collaboration, like: AR video conferencing, Telehealth, and related topics.

Xerox PARC, from way back when, are responsible for so many different technologies: Ethernet, “video links” and distributed software (called “ethernet software” at the time) to help people work with “digital paper.” This concept of digital paper really resonates with me most in my experience building software and has impacted how I think about designing a space for collaboration.

Digital paper

I like to always start thinking about a piece of software or a system as a group of people sitting around a table with a sheet of paper and some markers. Everyone can clearly see each other, can easily see when someone is writing or drawing, and it’s not possible to make a mark that cannot be seen by everyone else. Also, everything is spatially arranged: when I make a mark to the left of yours, you also see that it’s to the left of the mark you previously made.

Miro, Figma (and now FigJam), Google Docs, and Craft pages are examples of “digital paper.” (I also sometimes call them “digital whiteboards.”) They work like a magic, digitally replicated piece of paper that everyone can see and write on together. Everything is spatially arranged, we can see each other’s “fingers” (cursors), and all changes are immediately visible to everyone.

Digital whiteboards are the easiest collaborative software to talk about because they are both extremely immediate and give everyone 100% visibility. I cannot insert a circle into a Figma doc that you cannot see. The rules or “physics” of the software can be talked through very quickly and everyone “gets it.”

I recently read a Microsoft Research paper titled “Three’s Company: Understanding Communication Channels in Three-way Distributed Collaboration” where each participant could see “shadow images” of the other participants arms and hands as they would attempt to work together to accomplish a task like moving a block. Everyone also had audio and video feeds from the others. It seemed like everyone got it and the “additional arms” didn’t seem to bother anyone:

Despite the fact that we regularly saw three, and occasionally saw six arm shadows (two per person) in the task space, users did not seem to be distracted by this additional clutter.

(I also found the feedback from participants that the video feed was only really useful when preparing for a task, but not during it, super interesting.)

I also really liked another paper by Bill Buxton titled “Telepresence: Integrating Shared Task and Person Spaces” which has a great part which says (emphasis mine):

… the interaction breaks out of being like watching TV, into a direct engagement of the participants. They meet each other, not the system.

What about email?

Email can also be considered a “digital paper” product, however its immutability and the slow cycle of feedback make it less useful for a lot of tasks. Now, people love email. It’s the most successful distributed technology ever created.

The rules are understandable: I write something in a place that only I can see, then I hit “send” which will transfer a copy over to another person’s inbox (where I cannot see) for them to read sometime in the future. I can’t “unsend” an email and I can’t modify an email after it’s been sent.

For tasks which don’t need immediacy or really resemble letter writing, then email is the right choice. Personally, email doesn’t interest me much as something to work on. I wish the email protocols could be updated to allow for easy end-to-end encryption, auto-rejection of unknown senders, etc to make it more private and less overwhelming – but I also really get why it’s hard to change something so old and so successful.

What about chat?

Chat software can be a little fuzzy to pin down. While it might seem like a Slack channel is both fully immediate and fully visible, it turns out there are things one can do that are not shared and Slack tries to make this clear by labelling those messages “Only visible to you”.

By making it less clear if everything is fully shared, it adds stress for users who feel uneasy about trying something because “what if everyone sees me make a mistake” or “what if I send this to the wrong place.” (The old IRC software also had ways for “the system” to message you that were only visible to you, so this is not a problem unique to Slack.)

Using Apple’s Messages can also be confusing: if I delete a message, then does it delete it for the other person? (No, it doesn’t.)

I prefer software that is immediate: I would almost always rather use chat over email. But I also prefer software that is visible: I don’t like to be confused about who can see what.

Digital paper isn’t enough

Digital paper (or a digital whiteboard) is not enough. Arranging everything spatially can make it very difficult to find or remember where something was placed. As a canvas for a document grows it becomes harder and harder to keep up. Changes can happen over in an area one is not currently paying attention to. And searching for a note or comment from a month ago is almost impossible.

Both Miro and Figma continue to add features to try to help mitigate some of these issues (like following a person’s viewpoint) which is great. Google Docs can have the table of contents to the left which can help some as well.

The metaphor I use to talk about solving this issue is a digital card catalog.

Digital card catalog

When I design collaborative software, the metaphor I use the most is a digital card deck or digital card catalog, depending on the size and scope of the software.

If we each have a deck of playing cards, and we shuffle them in different ways, we know we still have the same “set of cards” despite the orders being different. I might have the ace of diamonds as my first card and you might have the queen of hearts, but we are not confused by that in any way. I can always search through and find any card you can. Also, if I decide to filter through and make a pile of only hearts, we understand that I am not filtering any cards in your deck.

For a “digital deck of cards” system: the card is like a little shared document. When we draw on the shared ace of clubs, we can all see the updated design immediately. I can also choose to put that card away in the deck and start working on another card. Then I won’t see your updates immediately because I’m on a different piece of digital paper. Everything inside the card is shared and everything outside is personal.

Sometimes we need a shared way to organize lots of cards which is what I call the “digital card catalog” system: not only is the card a shared digital document, the way the cards are organized is also shared. This is analogous to the card catalogs found in libraries.

Aside: a long time ago I learned from a few designers that if one wants inspiration for how to design information or software for people then always look at maps and libraries. They are ways to structure and organize lots of information, they are designed to be heavily used, and they try to work for a large number of people with as little required training as possible. Also, be aware that there are systemic biases inside libraries and the libarary science field that are important to know about and not replicate.

While a “deck of cards” is a nice and easy analogy, almost all types of software that I have helped design or helped build have become larger “card catalog” systems.

Wunderlist was one: each task was like a little shared document. Any change you make to task is visible to every member of the shared list. And the list itself is the “catalog.” We all see the same tasks in the list at all times.

One contentious argument we had was whether sorting a list alphabetically was a “shared action” or not. IMO, using the physical analogy to card catalogs, sorting or filtering is not a shared action. However, I lost this argument, and when someone sorted a list it actually sorted those tasks for everyone. I still feel this was a mistake and wish I could have done a better job making the case for personal-only sorting.

People prefer consistent rules or “physics.” When we are crafting the “universe” of a software product it’s important the physics and “gravity” are consistent and feel good. The rules of physical card catalog systems are already well documented and understood, so one can leverage that to help make our digital card catalog systems easier to use.

One can also look to libraries for how to “announce updates” since it is really impossible for a human to keep track of all the “new cards” in a busy catalog. There are “new releases” or “librarian picks” which try to help guide a person to relevant topics and places. There are many methods to search, filter, and arrange the cards to make it easier to find a specific one.

One could make “topic drawers” which include only the cards relevant to that topic. There is no reason why the same card cannot be present in more than one drawer if it were relevant to many topics. A card could helpfully indicate which other topic drawers it could be found in. This would make sense and be welcomed by library visitors.

Sometimes, in physical libraries, there are cards in the catalog which reference books over at other libraries: a union card catalog. Digitally, it’s possible to have a system to not just work with information inside your group or organization, but also with other groups and organizations which could be a huge win. 🙌

Quick summary

So that’s what I wanted to write about: when I design or build distributed software to help people work together, I almost always think about digital paper organized in a digital card catalog. I prefer software to be extremely immediate and it to be clear what is shared? and what is not.

I’m curious though, how do you think about collaborative distributed software? How do you think through the rules or physics of the system? Send me an email, I am very curious to hear what you think.

TIL about less -S

Fri, 12 Feb 2021 12:55:10 +0100

I was pretty frustrated by some tools that kept outputting way too much info in my terminal so I read through man less and finally found the option to not wrap (-S or --chop-long-lines) and allow lines to scroll off the right edge. One can move around with the left and right arrows to see all the info.

Wonderful Portraits by Otis Kwame Kye Quaicoe

Sun, 17 Jan 2021 23:03:15 +0100

These painting by Otis Kwame Kye Quaicoe really caught my eye on Colossal. I love the contrast and the wondrousness of the images. Follow on Instagram.

Check it out →

TIL how to fix a formula conflict when updating brew

Thu, 31 Dec 2020 12:39:48 +0100

Today, when updating brew, a formula I had made local modifications to had a conflict and had the <<<<< markers inside the ruby code, so it couldn’t function properly anymore. I was trying to find a quick way to just accept whatever the latest version of the formula is without my changes and this is what I came up with:

$ NAME=whatever && cd $(dirname $(brew formula $NAME)) && git checkout origin/master -- ${NAME}.rb && cd -

TIL about HTTP Client Hints

Wed, 16 Dec 2020 12:30:08 +0100

While reading this article by Umar about considerate javascript I was surprised to see the <meta http-equiv="Accept-CH" …> method for appending request headers for client information! It’s called Client Hints. Sadly, it seems it’s Chrome-only 😔.

This also lead me to the Network Information API spec which would be super useful, but is also Chrome-only 😔. The amount of Chrome-only APIs is ever growing and seems unstoppable, since all the browser makers rarely agree on priorities.

Check it out →

TIL about ELIXIR_CLI_DRY_RUN for elixir and iex

Tue, 08 Dec 2020 14:28:38 +0100

While I’ve been working on deploying some elixir releases I’ve been trying to understand how the flags I’m setting end up being seen by erlang. Well, it turns out elixir has an ENV var for this named ELIXIR_CLI_DRY_RUN.

Here is an example:

$ ELIXIR_CLI_DRY_RUN=1 iex
erl -pa /usr/local/Cellar/elixir/1.11.2/bin/../lib/eex/ebin \
		/usr/local/Cellar/elixir/1.11.2/bin/../lib/elixir/ebin \
		/usr/local/Cellar/elixir/1.11.2/bin/../lib/ex_unit/ebin \
		/usr/local/Cellar/elixir/1.11.2/bin/../lib/iex/ebin \
		/usr/local/Cellar/elixir/1.11.2/bin/../lib/logger/ebin \
		/usr/local/Cellar/elixir/1.11.2/bin/../lib/mix/ebin \
	-noshell \
	-user Elixir.IEx.CLI \
	-extra --no-halt +iex

You can see a lot of interesting info here:

elixir or iex are always just running erl in the end
-pa is to prepend a module load path and elixir makes sure all it’s modules are accessible at boot
-noshell means to, uh, not start the erlang shell, since iex is going to boot up it’s own shell in it’s start function with :user_drv (find it in the erl docs)
-user is the first module erlang will call start of after boot, in this instance it’s Elixir.IEx.CLI
-extra means “erl won’t look at these, but they are available with :init.get_plain_arguments() which elixir does a few times
--no-halt tells elixir not to auto-exit after boot, so iex can setup it’s shell and accept input
+iex is a special flag elixir looks for while starting up it’s CLI (which is also called by IEx.start)

I found this super interesting and it helped me learn a lot about how erlang boots up, how modules are loaded, etc.

TIL ⌃\ sends SIGQUIT to the foreground process

Thu, 05 Nov 2020 15:05:38 +0100

Following up on my previous TIL post about iex, I learned today that I didn’t understand control characters in my shell at all. Running stty -a shows my current shell settings:

$ stty -a
...
cchars: discard = ^O; dsusp = ^Y; eof = ^D; eol = <undef>;
	eol2 = <undef>; erase = ^?; intr = ^C; kill = ^U; lnext = ^V;
	min = 1; quit = ^\; reprint = ^R; start = ^Q; status = ^T;
	stop = ^@; susp = ^Z; time = 0; werase = ^W;

Control + backslash is right there, labelled as “quit” meaning it will send SIGQUIT to the foreground process. Super handy.

TIL one can exit iex with ⌃\

Sun, 01 Nov 2020 22:00:27 +0100

I’ve been using ⌃C + ⌃C to exit iex sessions for years. It turns out that works incidentally by invoking the break menu and then breaking out of that. This single keystroke method seems to be the most straightforward way to signal to the BEAM VM that it’s time to quit.

Check it out →

Beogram 4000c • Recreated by Bang & Olufsen

Sun, 25 Oct 2020 21:15:45 +0100

😍

Check it out →

“I am excessively slothful, and wonderfully industrious—by fits.”

Thu, 22 Oct 2020 10:50:12 +0200

A quote by Edgar Allen Poe from a letter in 1844.

Dieter Rams Wallpapers

Wed, 02 Sep 2020 10:22:03 +0200

Awesome wallpapers for your phone or tablet feature one some of the most iconic product designs ever.

Check it out →

Connect, No Matter the Speed

Wed, 02 Sep 2020 10:06:48 +0200

While the examples in this Medium article are all from google’s apps, this is still a valuable look at different experiences and concerns with limited or no connectivity.

Check it out →

GitHub Profile README

Mon, 31 Aug 2020 16:46:49 +0200

I’ve created a GitHub profile README which is a special repository + file one can create which is then shown at the top of one’s profile page. You can see my new profile README at github.com/myobie and I’ve put a screenshot below.

TIL about hub-sync

Wed, 15 Jul 2020 10:55:35 +0200

TIL about hub sync which will fetch origin, merge into the local main branch if possible, and delete any local branches that were deleted remotely 🙌

If you don’t already use hub then I recommend you take a look ASAP. I’ve use hub pull-request for some time, but never looked at all the commands it offers like sync.

I alias git=hub so I can type git sync to catch everything up and I ❤️ it.

Check it out →

29 automata in 6 minutes

Sat, 13 Jun 2020 10:56:28 +0200

Wonderful.
Found on Colossal.

Check it out →

TIL about AbortController

Fri, 12 Jun 2020 14:58:16 +0200

TIL about AbortController which allows one to cancel a fetch. HT to the Web Systems team at GitHub ❤️ @dgraham, @keithamus, @koddsson, and @muan.

Check it out →

Your metrics might be lying to you – an example from Wunderlist

Wed, 10 Jun 2020 14:48:33 +0200

We all love to see “increased engagement” from customers using the products we create, but it’s not always as clear as it might seem if the increase is positive or negative. It’s sometimes dangerous to have the singular goal to “increase usage of this feature” instead of “help customers get more done.” And even then, when our goal is 100% focused on helping the customer be a better human, we can sometimes misread the metrics we are collecting.

At Wunderlist, the notifications about recent activity feature was akin to the limit theorem: we iterated and iterated, but it always seemed like we couldn’t get to our ideal place to help customers see and understand what’s been happening in their lists. I believe we did our best, but there is one time where we thought we had improved things and it ended up we had done the opposite.

We instrumented the apps to let us know when someone clicked on the notification bell and we saw that hardly anyone was clicking on it. So we went to work: how could we help people know there is useful information in the notifications popover.

We added a red dot to indicate “hey, there’s some new stuff here.” Clicking to show the notification popover would remove the red dot (until later when there is new activity).

And our metrics for clicks on the notification bell went way up! Success.

Luckily, we dug a bit deeper into the stats and we noticed a strange occurrence: often there were two clicks from the same device in rapid succession, both clicks in less than two seconds. We assumed we had made a mistake in our apps and they were reporting the click twice. But we couldn’t reproduce this ourselves and it wasn’t reported twice every time, just often.

We realized our customers were clicking once to show the popover, then immediately again to hide it.

It turned out customers really just wanted to remove the red dot. Our metrics were lying: positive engagement with notifications wasn’t going up as we had thought. Instead, we were annoying people.

Like I typed above: we never got the notifications feature to the point we wanted and that’s OK. Some features are infinite projects, require continual investment and adjustments, and are made incrementally better over time.

TIL about vim’s spellcheck

Thu, 21 May 2020 16:05:13 +0200

I’ve been using vim for years, but I’ve never took the time to learn how to do spellcheck. Instead I always open [iA Writer][] if I need spellcheck. However, I have made way too many spelling errors in code comments or quick files and today I decided to learn how to check the spelling of a document.

Today I added this to my .vimrc:

" spellcheck
command Sp setlocal spell spelllang=en_us
command Spoff setlocal nospell

I only want spellcheck to be on in a single buffer at a time and not by default. After it’s enabled one uses ]s and [s to navigate to the next and previous spelling error. With the cursor inside a problem one can use z= to bring up a list of suggestions for replacement. It’s been great. Below is a screenshot of this file for example.

TIL about :term ++curwin

Mon, 18 May 2020 13:43:07 +0200

Often I want to open a new terminal in vim8 and until today I only knew how to open into a split window of some kind. Today I found out about ++curwin which instructs the terminal to overtake the current window. All the options for opening a new terminal can be found in the vim help for terminal.

TIL about :lcd in vim

Fri, 15 May 2020 14:25:59 +0200

:lcd can be used in vim to change the directory for a single window (split) which fzf respects and any window/buffer I open from there inherits the same local directory.

I’ve been using this to have one instance of vim open and keep different projects open in different tabs and it’s working well.

This Video Has ___ Views

Thu, 09 Apr 2020 14:11:13 +0200

A very good explanation of what it was like to create things in the Web 2.0 days and the openness at the time.

Check it out →

Retail fonts · Klim Type Foundry

Wed, 12 Feb 2020 10:29:00 +0100

Klim Type Foundry makes some of the most beautiful typefaces. I really like Founders Grotesk, National 2, and Söhne.

Check it out →

A richer canvas

Wed, 22 Jan 2020 16:27:37 +0100

Today I was re-reading this excellent article by Mark Boulton from back in 2011: A Richer Canvas.

It’s my belief that in order to embrace designing native layouts for the web – whatever the device – we need to shed the notion that we create layouts from a canvas in. We need to flip it on its head, and create layouts from the content out.

Check it out →

Realistic pillows sculpted from blocks of white marble

Sat, 18 Jan 2020 15:54:09 +0100

These pillows crafted from marble by Håkon Anton Fagerås are stunning.

Check it out →

Going public

Wed, 15 Jan 2020 14:08:05 +0100

Today I’ve done something that I’ve wanted to do for some time: I’ve made the source code for my website public on github.com and you can see how the website is built and deployed.

I’m starting a new work journey, founding a new thing named Shareup with a colleague, and I’ve spent the past week working on a new website for it. I want to use a few specific programs and services:

Assemble the website with hugo
Host the website using zeit’s now service
Keep the website up to date by deploying every change with a GitHub Actions workflow

I also want to use the exact same setup for this here website and so I’ve been using my website as a test area to figure out how best to work with hugo, now, and actions. After struggling at this for a week, I’ve gotten it working and I wanted to document what I’ve learned so I remember (and maybe it’s helpful to you too).

You can see the complete source for everything over on GitHub at github.com/myobie/nathanherald.com.

TL;DR

I def recommend hugo for building static websites
I def recommend now for hosting static websites
I think GitHub Actions are great and promising, but they are confusing and not always fully documented
I spend way too much time making concessions so I can use this font

This post is pretty long, so here are links to the different sections:

Using hugo

I’ve been making websites with hugo for a while now so I can quickly setup and configure a new hugo website pretty quickly. The feature of hugo I like the most is its speed. It’s important to me that assembling my website is super quick. I also like that it’s a single binary which means I don’t have to figure out how to “install” anything.

Using now

zeit’s now is a fantastic way to host a static website. Getting started with now is super easy, one simply runs now in the terminal and it creates a project, deploys, and even copies the resulting URL to your clipboard. It’s almost too easy: I’ve accidentally created a new project a couple times by renaming a directory and zeit deciding “this is a new website” instead of “this is the same website in a new folder.”

Hosting the font somewhere else

I have a strange and specific requirement for my website because of my font: the font I’ve chosen can only be loaded from certain URLs that I need to know ahead of time. I also don’t have permission to “host the font” anywhere publicly (like this repo) and those two things meant I needed to do some extra work before I could make the source code for my website publicly viewable.

Putting the font on S3

I decided to zip up the fonts and put them in an S3 bucket. Then, when now is building the website, it can retrieve them using s3cmd.

Also, I learned a valuable lesson a while back: never create AWS resources using the UI. Creating through the UI means it’s easy to forget how or what is setup.

Instead, I use terraform for everything I need. There is a pretty big upfront cost because I can never remember how exactly to create the terraform files.

I do create users in the UI because if one uses terraform then it’s easy to accidentally store their secret key in a .tfstate file or something similar and I don’t want to risk it. I created a user with zero permissions.

I put some terraform in a directory named infra which creates a bucket and gives it a policy where one user can only read from it. Then I added some code to use s3cmd to retrieve and unzip the font during the install step.

Generating predictable preview URLs

When I’m writing a new blog post (like this one) I want to be able to see a “preview version” in my browser so I can make sure all the images look right, the font is working, etc, so I deploy a “preview site” for every pull request I make.

Since I want to preview before everyone else sees it, I need to deploy the site somewhere other than nathanherald.com which can break my font from loading if I can’t predict exactly what the URL of the preview site will be. To make this work I override the “base URL” for hugo to something I call the PREVIEW_URL. I can dictate the exact format of the URL and I can clear those URLs with my font provider.

Building a workflow with GitHub Actions

The most difficult part when setting up my website was getting a workflow that works to deploy for every change. I knew exactly what I wanted to happen:

Create a deployment through the GitHub API to indicate a deploy is happening
Mark the GitHub deployment as in progress
Run now passing in a PREVIEW_URL so I can preview the site
“Alias” the now deployment to the PREVIEW_URL
Mark the GitHub deployment as complete
Post a comment that says “🚀 your changes are deployed” linking to the PREVIEW_URL

I thought this wouldn’t be too difficult. I was wrong.

You can see my final deploy.yml workflow in the repo.

Build one action to perform all the steps (which didn’t work out)

My first attempt was to make one action that performs all these steps and I started that here: github.com/myobie/deploy-now. I found a javascript module named now-client (which the now program itself uses internally) and it exposes a createDeployment function so it seemed like it was going to be easy.

The biggest problem I had creating a GitHub Action is there is no local dev environment I know of to test the workflow on my computer. Sure, there is a small test example, but it leaves a lot to be desired.

It’s also sometimes lacking documentation: like how are arguments passed into your action? It turns out any “inputs” are passed in as ENV vars that begin with INPUT_ like INPUT_TOKEN would be passed in for a step that has with: followed by token: abc.

The javascript toolkit reads in these variables for you, so I decided that I was going to write everything in javascript because there might be other undocumented things I don’t know about that the toolkit is magically taking care of.

Using ncc to package everything into one file

After deciding to use javascript I ran into my first roadblock: GitHub Actions doesn’t install your npm dependencies before running your code. Instead we are expected to pre-compile our javascript into one, dependency-free .js file before pushing. This is super surprising. The tool recommended to package up one’s javascript is ncc (which is apparently a play on acronyms with gcc).

ncc is easy to use and it natively supports typescript, so I then decided that I’d consider using typescript. I setup a “package script task” and I made sure to setup a pre-commit hook to package my javascript before I push any changes because I will forget.

Separating the steps out into individual actions

The now-client ended up not working the same as just running now (for example: rewrites in the now.json file weren’t sent to zeit) and without being able to easily and quickly test changes locally, having a single big action ended up being too frustrating.

I decided to make smaller, focused actions where each one was small enough that I could more easily understand it and less easily break it. I’ve ended up with a little suite of actions.

Getting the name of the current branch

When GitHub triggers an Action it provides a bunch of information, but none of that information reliably includes just the branch name. I want my preview site to include the branch name in its URL (so each branch has it’s own preview URL) so I wanted a way to find out just the branch name and have it set as an “output” so I could interpolate it later into the arguments to other actions.

So, I made output-git-metadata-action which does exactly that: it “outputs” some git metadata, including the current branch name.

Note: one has to give the “step” an id: in the yaml to be able to reference it later as ${{ steps.id.outputs.output_name }}. A step can have both an id and a name; name is optional.

This action can be used like this:

jobs:
  yo:
    runs-on: ubuntu-latest
    steps:
      - id: meta
        uses: shareup/output-git-metadata-action@master
      - name: print debugging info
        run: |
          echo "branch: ${BRANCH}"          
        env:
          BRANCH: ${{ steps.meta.outputs.branch }}

Creating the GitHub deployment

Deployments in the GitHub API are very confusing to me. The documentation says “will trigger a deploy event” but I am already doing the deployment, I just want the deployments UI of a pull request to light up and show what happened, like this:

I assume when I create a deployment through the API it does emit some deploy event that no one is paying attention to. Seems wasteful. It would probably make more sense to separate the concept of “creating the deployment record” and “triggering the deploy event.” It’s also very possible (likely) that I just don’t understand what I’m doing. </rant>

I made create-deployment-action to do what it says: create a deployment. It outputs the deployment’s id in case one need’s to reference it sometime later.

And it doesn’t need one to generate a token or anything: GitHub already provides a default token to every action which has read/write access to the current repository so I just use that inside the action code. I honestly don’t know where this is documented or how I found it, I think I might have found it while reading through the source of the checkout action.

`octokit`

I found @octokit/rest confusing (and it not always having all it’s types for typescript working perfectly) and switched to @octokit/request which I hadn’t really seen mentioned in any official docs. The request library is much simpler, has better typescript types (since it’s written in typescript), and one can simply copy the documentation line like POST /repos/:owner/:repo/issues/:number/labels straight from the GitHub API docs.

Updating the status of the GitHub deployment

Creating a deployment is kinda meaningless and empty on its own: one has to “create a deployment status” to mark it as “in progress” or “success” or something else.

I made create-deployment-status-action for this purpose.

At this point, I could see that I was on my way to creating my own little framework of actions and could write a yml file like:

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - name: output metadata
        id: meta
        uses: shareup/output-git-metadata-action@master
      - name: create deployment
        id: deployment
        uses: shareup/create-deployment-action@master
        with:
          ref: ${{ steps.meta.outputs.sha }}
      - name: set deployment status to in_progress
        uses: shareup/create-deployment-status-action@master
        with:
          deployment_id: ${{ steps.deployment.outputs.id }}
          state: in_progress
      - name: echo example preview url
        run: |
          echo https://preview-for-branch-${{ steps.meta.outputs.branch }}.example.com          
      - name: set deployment status to success
        uses: shareup/create-deployment-status-action@master
        with:
          deployment_id: ${{ steps.deployment.outputs.id }}
          state: success

I like these super small and granular steps.

Deploying with `now`

When I want to deploy with now locally I just run now. I wanted to make an action that does the same. There is a tool in the toolkit named exec to run processes and I could use a Dockerfile to make sure now is pre-installed.

So, I made now-action which executes now and accepts inputs for almost every cli flag now accepts.

Using a Dockerfile took me a bit to understand because there are three paths I needed to keep track of: the path we are building the docker image from (the action’s repo), the path we are executing the action into (the website’s repo), and where the action’s files are stored inside the docker image (?).

I built FROM node:12 so I knew node was already present and I used npm to install now globally during the build step.

I decided to just make my own place to store stuff as /now-action/ and since I was already using ncc for the other actions I kept that so I only needed to move dist/index.js into the docker image for everything to work by then executing /now-action/dist/index.js with node.

Since this action is completely isolated from the others, it can be used on it’s own like so:

jobs:
  now:
    runs-on: ubuntu-latest
    steps:
      - name: checkout
        uses: actions/checkout@v2
      - name: now
        uses: shareup/now-action@master
        with:
          token: ${{ secrets.zeit_token }}

One has to set a secret on their repo to a token from zeit so it can use now on your behalf.

Force deploy

now tries to be smart and doesn’t regenerate a website if it doesn’t see any changes, instead it will attempt to re-use a previous build. This caused me issues when deploying to production because it would re-use a build I had created for a preview website and then all my fonts stopped working because it expected the wrong URL. 😞

I always pass force: true to the now-action which tells now to always rebuild no matter what.

Honestly, I’ve done a lot of work for this font…

Assigning an alias to a now deployment

Running now will create a new “deployment” and give it an autogenerated URL, which isn’t enough for me: I was to use my custom PREVIEW_URL. now allows one to point a domain at a deployment and calls those “aliases.”

So, I made now-alias-assign-action which does exactly that.

To alias a deployment with a preview URL including the branch name, I could now make a yaml like:

jobs:
  now:
    runs-on: ubuntu-latest
    steps:
      - name: checkout
        uses: actions/checkout@v2
      - id: meta
        uses: shareup/output-git-metadata-action@master
      - name: now
        id: now
        uses: shareup/now-action@master
        with:
          token: ${{ secrets.zeit_token }}
      - name: alias
        uses: shareup/now-alias-assign-action@master
        with:
          token: ${{ secrets.zeit_token }}
          deployment: ${{ steps.now.outputs.deployment_url }}
          alias: https://preview-for-${{ steps.meta.outputs.branch }}.example.now.sh

Aliases should end with ${your-now-username}.now.sh.

Posting a comment

The last thing I wanted to do was to post a comment after a deployment with the URL so I could easily click it to see the preview website.

I made create-comment-action which will either:

If a pull request is open, post a comment there
Else, comment directly on the commit that triggered the action

At this point I felt good at making actions and I’m not sure if that is a good thing.

Putting it all together

The full yaml workflow in my website’s repo has some tricks to differentiate between production and staging.

I can open a PR, preview my changes, and when I merge the PR the production website is deployed. A list of all the actions I created in the past week are:

GitHub Actions are very powerful, but also super mysterious. Since I cannot execute workflows on my computer the feedback loop of pushing, waiting, checking the logs is way too slow. It’s still amazing and I am super happy now that it works, but getting there was way too difficult. I haven’t submitted these actions to “the marketplace” yet because I have to make an icon and a few other things to submit.

now is great. I really like it. It’s super simple and I will be using it more and more for static website hosting as well as serverless functions.

hugo is still my favorite way to build websites. It’s super fast and the way it organizes content is alright. I love making websites and hugo mostly gets out of my way.

While I spent way too much time on this setup, I now feel more confident to make changes, preview the change, and merge to deploy.

Elements of AI

Tue, 24 Dec 2019 09:48:15 +0100

A six-week series of free online courses created by Reaktor and the University of Helsinki for educating Finnish citizens about AI. Finland recently made them available online for everyone.

Check it out →

The world's most remote buildings on YouTube 📺

Mon, 23 Dec 2019 08:55:12 +0100

When I was young I remember watching the Discovery Channel often and learning all sorts of interesting things like this: “The World’s Most Remote Buildings.” YouTube is the new Discovery Channel, except it’s hard to know the quality or accuracy of a thing before watching it.

Check it out →

A ryokan

Wed, 18 Dec 2019 15:52:04 +0100

The Wife and me and two friends pre-planned a trip to Japan many months in advance and I spent this last half year seriously looking forward to it. We recently spent two and half weeks in Japan on our first ever visit to the country! 🇯🇵 We went to a few places in Hokkaido, then to Kyoto, then accidentally to Osaka, and then to Tokyo. It was great to see so many different places and people and just enjoy being in a new place.

One place we stayed, for a single night, really impacted me: a ryokan in Hokkaido named Zaborin. It’s in a town named Kutchan where it seemed many people train for competitive skiing. The pictures on the website are no joke, this is exactly what it looks like:

The layout of the property and the rooms were all so pleasant and inspiring. The room we had was a nice mix of old and new: one area with a normal couch and TV and then a raised area with tatami mats for resting or enjoying a morning coffee and book.

Morning coffee

Every room has its own private baths with water constantly running from a hot spring: one inside and one outside. Even though it was very cold, sitting in the outside hot spring water and seeing the snowy trees and mountain was one of the most enjoyable things I’ve ever done.

The outside bath

One of the most beautiful sunrises I’ve seen

As you can see it snowed ⛄ We travelled up to Kutchan by train while it snowed and stormed out in the sea:

Enjoying the bath, the food, and reading was so relaxing and re-energizing – I really needed it! I also enjoyed our time in the other parts of Japan, and especially Kyoto, but I’ll be thinking about Zaborin a lot for a while.

asipai ＋ Hibi Coffee in Kyoto

Sat, 30 Nov 2019 15:26:17 +0900

Such a cute couple places together. I am posting this so I remember to come back here for curry.

Apple Maps link.

Check it out →

Music I love for traveling

Fri, 22 Nov 2019 21:54:46 +0100

The past year I’ve been traveling an incredible amount and I’ve had a lot of opportunities to listen to different types of music. Below are my top, go-to albums for traveling. I prefer albums over playlists which also means I preferred the late rdio ✌️. So before every flight I download a number of complete albums, always including:

Warm On A Cold Night & Gone Are the Days by HONNE

OK, these are two albums, but I pretty much always listen to them back-to-back. The smooth beats, strings, and vocals really make any travel situation more bearable for me.

Listen on Apple Music or Spotify

Dawn Chorus by Jacques Greene

It has surprised me how much I like this album and how often I can listen to it. It rolls and moves and that helps make a flight, train ride, or walk go by faster. I think I’m getting into Electronic; specifically house.

Listen on Apple Music or Spotify

Mos Def & Talib Kweli Are Black Star by Black Star

Arguably one of the best rap albums recorded, I often come back to this to hear some quality hip hop. Definition and Respiration will be relevant and catchy forever.

Listen on Apple Music or Spotify

Habit & Lush by Snail Mail

Sure, these are again two albums, but I, again, almost always listen to these two in a row. These are my number one, go-to albums for travel. It’s always a good time to listen to Snail Mail. They’ve made some albums which sound like travel to me.

Listen on Apple Music or Spotify

So good.

Petrichor by Artist Jean Jullien

Thu, 21 Nov 2019 16:00:00 +0000

Check out Jean Jullien on Instagram or checkout their work in person at Alice Gallery in Brussels until 11 Jan.

Check it out →

I've migrated off medium

Thu, 21 Nov 2019 08:12:27 +0000

In 2013 I was tired of maintaining my own blog and started posting onto medium.com to save myself time. The first post was The Cost of Education into a collection named IMHO back when anyone could make and join collections. I even became a paid member to help support “my blogging platform” so I wasn’t betting on a free service. Sadly, during the last couple years Medium has just been making things terrible – the worst of which is showing div popovers overtop my writing requesting readers to signup, sometimes not letting them read my writing if they don’t become paid members.

I am no longer a paid member for Medium and I have migrated my writing over to my own website. When medium first launched it was a great example of good typography and easy publishing and it’s sad to see them ruin it.

Technology

I’m using hugo to build everything statically. I’m using now by zeit to build, deploy, and host my files. I’m using iA Writer to edit my files on both macOS and iOS and I push updates to my files with git on macOS and Working Copy on iOS. Working Copy is a “file provider” on iOS which lets one edit the files from the git repo in any app like Writer or anything else with access to the Files app.

Zeit creates a test website for every branch I push, so I can easily push a new post from my Mac or phone with a real, live preview and then merge when I’m happy – which is a great workflow for me. hugo is so fast that I never wait more than a couple seconds for everything to rebuild. I’m super happy with this setup 😎

Trying to go full phone

I’m going to be traveling for a couple weeks without a laptop, so I prepared as much as I could to enable me to be able to do full posts entirely from my phone. Here is what that looks like:

Working Copy showing the files in the git repository for this website

iA Writer showing the markdown for this post

The fantastic new GitHub Mobile app is great so I’ve bookmarked this website’s repo there

Zeit’s now posts a “check status” and a comment to the Pull Request with the URL for the preview website where my new post is visible

We’ll see how it goes posting with my phone. Inserting images is not the best…

Switching costs

It took way longer than I hoped or assumed to migrate away from Medium. The switching cost is real and high. I wish it weren’t the case, but publishing on the internet reliably is still not “easy.”

hugo’s shortcodes are always confusing and having to use them for inserting images means I cannot preview images directly in Writer. I may just edit the images in Pixelmator and then use the normal markdown image syntax.

Zeit makes an assumption that builds are immutable and deterministic. While this is true for my website, there is one input that I consider a change that they do not: for the preview websites for every PR the domain for the website is not nathanherald.com so I use an ENV variable to inject a predictable domain for hugo to use. When I merge the PR, Zeit looks at the source code and says “I’ve seen all this before, so I’m not rebuilding again” and it promotes the version of the site built for something other than nathanherald.com and that breaks all the images and colors and everything.

I got annoyed and decided I would just move to GitHub Actions and just build wtf I want, but I have fallen prey to a bug in Actions that affects me because of my super-early access to the feature. It will be probably get fixed by next week…

I’ve created a workaround for the Zeit problem and I’ve sent them an email to ask them to consider the change in ENV variables as a “change of inputs” and therefore trigger a new build…

Good times™ as usual. I do love building for the web.

How to install the macOS Catalina Beta alongside Mojave

Mon, 17 Jun 2019 11:21:13 +0000

I’m sure you are interested in playing with SwiftUI just like I am, but you are also hesitant to install the latest macOS Beta because it will surely mess up homebrew and generally cause inconvenience. Even more, the live UI previews in XCode only work when running XCode inside the beta macOS…

If you have a recent Mac with an APFS formatted drive then you can easily create a little area for the Beta without changing anything about your current setup.

APFS supports multiple “volumes” in the same “container.” A new volume can be created super quick on the fly, can be flexible in it’s size (with min/max settings on the GB reserved/used), and a “container” (think drive) shares all it’s space with all it’s volumes so you don’t have to know ahead of time how much space each volume really needs.

Below are the simple steps to create a volume, install macOS Mojave on it, then upgrade that volume’s OS to Catalina.

1. Create a volume

Search using Spotlight (⌘ space) for Disk Utility and open it. Make sure your primary hard drive is selected in the left sidebar, then click the Plus icon in the toolbar to add a volume.

I named my new volume Beta

You can choose to set Size Options… to set a minimum reserve and a maximum usage for the volume. You will need at least 30 GB of space to get the macOS beta + XCode Beta installed with some wiggle room. I set my Beta volume to reserve at least 20GB and restrict to a maximum of 100GB.

2. Install macOS Mojave

One cannot install macOS Catalina directly: instead, one must upgrade from Mojave. Even if you are currently running macOS Mojave, you will need to download the macOS installer to use with the new volume.

Using Spotlight (⌘ space) again, search for App Store and open it. Search the App Store for macOS and find the page for Mojave.

Use the App Store to “Get” macOS Mojave

Click Get and it will open the Software Update preference pane in System Preferences. Let it download the huge installer and it will eventually present you with the below Installer window. This will take a while…

Choose the new volume

Be sure to choose the new volume and proceed to install.
This will take a while…

3. Install macOS Catalina

After installation is done and you’ve setup your new “computer” you will need to navigate to the Apple Developer Downloads page to install the beta.

Apple’s developer downloads page

Choose Install Profile which will open the Software Update preference pane of System Preferences again. This time, it will offer to download an installed for the macOS 10.15 Beta (which is Catalina).

Install (download) it and it will eventually present an installation window.

Install macOS Catalina just like Mojave before

This will also take a while…

You can still access your other volume from inside the macOS Catalina beta, so there isn’t any reason to copy over all your files or anything. This is great if you have an XCode project you want to work on or test out from the beta, but don’t want to have to transfer files all the time.

I recommend installing applications separately onto Catalina and not trying to run applications from the Applications directory of the other volume.

Download the XCode Beta and go create a new macOS or iOS project using SwiftUI. You choose to use SwiftUI either as a checkbox or dropdown in the New Project sheet.

One can choose SwiftUI when creating a new project

4. Learn about SwiftUI

I recommend going through the wonderful SwiftUI Tutorials published by Apple. They walk through a ton of good stuff and cover a lot more than the basics. I really enjoyed going through it.

5. Restart into Mojave

You can restart back and forth between the OS’s anytime using the Startup Disk preference pane in System Preferences.

You will need to “unlock” the Startup Disk preference pane by clicking the lock in the bottom left corner

You can also keep ⌥ option pressed when the computer is booting which will tell the computer to enumerate the bootable volumes and let you click which one to boot into.

That’s it

Good luck. Comment if I got anything wrong. And have fun with SwiftUI, Combine, and the new live UI previews in XCode. 🤖

Build your MVP today

Thu, 17 May 2018 14:13:58 +0000

Most software is simply a fancy spreadsheet or database. It could take you less than an hour to prototype an idea you have. Take the time and make it, if you can. Could you prototype your idea in Google Sheets? You probably can.

BTW, Visicalc is probably the most important app ever made.

Input some information into a spreadsheet and then use sheet2site to make a website from it. Most of the apps or websites you or I use every day are basically lists of data that we typed in. Maybe you’ve always wanted to make a list of the best places to visit in your home city/town: you can do it. You don’t need to know how to code.

If you have some javascript skills, then maybe use Sheetsu and now you have an API to build from. Are you good with React? Then maybe use Scrivito and build some custom widgets to build a fancier website. Or use the awesome Contentful to collect all your information and build an app from that. Knowing how to code means you can get more built in an hour.

Build a landing page to market your idea. Post onto Twitter or Facebook or simply send an iMessage or Whatsapp message to some friends about it. You could use Wufoo to collect feedback and/or payments.

Indie Hackers is a website full of stories of people who’ve created a simple prototype and now have some recurring income from it. Maybe take some time to read a few stories of things that worked and some that didn’t work to help get motivated. Yes, reading about things that didn’t work is motivating. Not everything works out and that is OK. It is, in fact, likely. It’s part of the process.

Replace one TV show or movie and build a prototype instead.

Some people don’t have a spare hour. So, if you do, take an hour of your time and build the prototype of the idea they’ve been telling you about for them. Input some data into a spreadsheet, generate a website, and send them a URL and see what they think. It never hurts to practice building.

What are some tools or tips you have that have helped you try things our or build things quicker?

I still don’t know how to use a debugger

Mon, 14 May 2018 21:59:16 +0000

Since I wrote my first javascript in the computer lab at my High School in VA using Windows 95, I’ve subscribed to the println debugging standard practice. Of course then it was alert. Either way, if there is a problem, add print statements everywhere.

Well, it’s time I finally learn how to use a debugger. The reason: I wasted two entire days.

Do you use a debugger? Do you have any tips or tricks? How can I break the habit of console.log or IO.inspect or println or whatever?

Why now?

You see, I really like erlang and elixir. I am playing around with spawning processes and communicating with them for a bit. Erlang has an insane concept called “ports” (nope, not those ports) and after learning more about it I really like them. You use stdout and stdin as binary pipes to communicate back and forth. If anyone breaks either pipe, then everyone shuts everything down. It’s a great way to know™ you won’t have zombie processes.

Well. console.log writes to stdout. Of course, right.

I broke my binary communication protocol by attempting to debug a problem. Because I don’t know how to use a debugger. And, honestly, I fixed this problem by continuing to be ignorant and appending log lines to a file instead.

So yeah, I’m frustrated. Again, my lack of CS skills has caused me problems. Very rarely do I wish I were a “traditionally trained” programmer, but this is one of those times. I am going to finally learn this. Eventually. When I get time.

Puma RS-0

Mon, 16 Apr 2018 12:08:40 +0000

A reboot of the RS Computer Shoe (yes, for real), the Puma RS-0 is a new “future-retro” sneaker and I am in love with them. I copped these over the weekend and I’ve really enjoyed the look and the comfort. They are super comfortable all day, but they are a bit heavy so I don’t think anyone will be running in these.

The shoe is named the RS-0 because the design team wanted to “literally reboot it back to zero.” You can read a good interview with Heiko Desens from Puma about the shoe over on Hypebeast. They dropped two other RS-* shoes this weekend as well, but I only copped the zeros.

I am super intrigued by the original Computer Shoe.

You can read more about the history of the RS Computer Shoe over at Nice Kicks.

They also dropped some special edition RS-0’s here in Berlin last week including Polaroid, Sega, and Roland collaborations. The Polaroids are 🔥

Karhu Synchron Classic

Fri, 06 Apr 2018 21:48:32 +0000

Karhu is a Finnish sneaker brand I wasn’t familiar with until recently. I’m getting more and more into sneakers and building up a collection. The latest cop for me are from the Linnut Pack — Part 2 for which the colorways are inspired by bird watching. Yep.

The history of the company is super interesting. You can read a full run-down over at Sneakers Magazine. They started in 1916 and their Olympic athletes were known as the Flying Finns. Another super crazy fact is they sold the three stripes to Adidas for two bottles of whiskey and €1,600!

According to Karhu, the two brands struck a historic deal that would make athletic trademark history: “Adi met the Karhu directors and invited them over to Germany. The result of the trip was that Karhu sold the three stripes trademark to adidas for a small sum of money, around €1,600 in today’s terms, plus a couple bottles of good whiskey,” said the brand manager, adding: “That’s how the story goes, although we don’t have proof of the whiskey.”

The unique lacing of the Synchron Classics is 🔥 and I’m excited to wear these around Berlin and during my upcoming trips to the States.

Are you into sneakers? Cop anything recently?

One week with the Atom text editor

Mon, 26 Mar 2018 21:09:16 +0000

Every time I sit down to work I like things to be as similar as possible to the last. However, I do know that there are new tools, techniques, and ideas out there that would pass me by if I didn’t change things up every now and then. So this past week I’ve been using a different text editor.

I normally use vim. Well, I actually use neovim, which is a fork of vim. Well, really vim was a fork of vi long ago. The complexity of vim is so high, it’s even hard to explain it’s name, which version I use, etc. Vim is so confusing there is a question on Stack Overflow about it that has been viewed over 1.4 million times: “How to exit the Vim editor?”

There is a Quartz article about that question and the history of vim being difficult. It’s over 30 years old, depending on which version you count. I never recommend vim to people. You really shouldn’t use it unless you want to be super unproductive for six months while you learn it.

Either way, this past week I didn’t vim. Instead I used Atom, which was created by GitHub and is based on their open source Electron framework.

The Good

Atom has been around long enough to have plugins for everything. It is very unlikely that there isn’t a plugin for your favorite language. One can also customize the UI a lot: you can see above that I’ve hidden the normal title bar and file drawer and it looks really slick.

There is an official plugin for editing concurrently with others named Teletype. This was what really intrigued me to download and give Atom a spin: built in collaboration. It works and I really like it. Every other real-time text editing plugin I’ve used has been awful to setup and not really worked well for long periods of time. This really seems to be done right.

It’s super easy to install plugins and packages. The available color schemes are very good as well. The editor defaults are great: it’s the best default setup I’ve ever seen for a text editor. One can download this thing and get working immediately.

The Bad

It can be slow.

Electon (which Atom is built on top of) is a version of Chromium (the open source browser Chrome is built on) and the UI is all web technologies. Something about those super old text editors that is hard to beat is when you type it shows up immediately. I’m not saying Atom is too slow for me: I have used it for an entire week and I enjoyed it. However, it really is noticeable sometimes.

It also uses a ton of RAM. This doesn’t matter on my iMac, but on my MacBook this can be a problem. Last week I ended up running Slack (also Electron, so also a Chrome), Atom (another Electron), Microsoft Teams (also Electron, believe it or not), some docker containers, and then Mail and some websites in Safari and I noticed a big difference. Sure, just Atom alone isn’t really a problem. But having a lot of apps that each need the more RAM that most programs hurts my sad little laptop.

The Ugly

It just doesn’t work sometimes. When I try to find a file by typing a few characters, Atom will tell me that it is still indexing or that my project contains zero files. A restart of the app normally fixes the issue. This was a problem on my super fast iMac as well, so it’s not a performance issue. I took some screenshots.

When I use vim I have ripgrep feed a file list into fzf and it works every time and works immediately. There is no caching: it just is able to scan the entire project very quickly every time. It’s easy to get spoiled by things working so fast and it’s really frustrating that Atom doesn’t work at least similarly fast (and sometimes not at all).

So, what’s the verdict then?

Atom is really great.

Just the statement “I have ripgrep feed a file list into fzf” is super complex and confusing and that’s what it’s like using vim. I deal with configuring stuff like that all the time and it does get tiring.

Atom is a single thing to install and it generally works. It includes collaboration features, it has plugins for anything that are easily installed from inside the program, and it has the best defaults of any text editor I’ve ever used.

I recommend trying it out.

What do you think of Atom? How often do you try other editors or tools?

I’m going to back to vim. Vim is a problem: once you get into it it’s hard to ever use anything else. I’m glad I tried Atom and next time I need to collaborate on some code with someone I will fire it up immediately.

Twenty-second Postmortem

Fri, 22 Sep 2017 16:45:06 +0000

With lots to do and not enough time to do everything, it can seem impossible to document progress or keep track of how things might have gone. Some items seem small, so we just complete them and move on. It can, however, be incredibly valuable and sometimes necessary to remember what actually happened, when, and maybe why a decision was made.

I have a terrible memory, so I use a to-do list app to keep track of what I need to do. This past year I’ve tried to not just use my to-do list to keep track of what’s left to do, but also to document what’s been done with more detail than I did before. If you use a calendar, a notes app, or a sheet of paper you can do the same thing.

Before marking something as complete, take 20 seconds to write down what went right, what went wrong, why, and then move on. Those 20 seconds can end up being super valuable down the road. It is also good practice to be regularly reflective about progress. I just input my thoughts into the “note” field for the to-do item before checking it off. I like this because the documentation is attached to the original work item.

Here’s an example implemented in a few apps:

I recommend trying this or something similar for a while. It can be really nice to look back and remind yourself why something worked out differently than originally planned, who you talked to about a topic before marking it done, or just to see that everything did indeed work out as expected.

Do you have any tricks to keep document progress?
Any recommendations for me?

🏁

A quick tip to help remember where you left off

Fri, 28 Jul 2017 07:46:34 +0000

If your short-term memory isn’t the best, like me, then you probably have a few tricks to help recall information easier. When I’m programming, working on a spreadsheet, or anything on a computer and I need to leave it in an unfinished state: I will take a screenshot. macOS now supports syncing your Desktop to all your devices, so that screenshot ends up everywhere automatically.

This morning I noticed a screenshot waiting for me of some code exception. I assume I wanted myself to fix this first-thing today. After making coffee, of course. Taking a screenshot, making a note in a notes app, or emailing yourself can really help reconnect you to what you were thinking.

🏁

A newsletter

Tue, 25 Jul 2017 08:26:39 +0000

Soon, I will start sending a newsletter with interesting stories, analysis, and links to things that I find interesting. I’ll also include some paragraphs and links to what I’ve been up to. There will definitely be content about startups, Berlin, and tech. There will also be a bit about art and entertainment and how they intersect with tech.

I won’t be posting these emails as blog articles: it is meant to be consumed as a newsletter. The frequency will be every two weeks at first: every other Tuesday around 5pm GMT. I’ll be sending out the first issue 1 Aug.

The name of the newsletter is “Tuesday TEA time.” Yeah. (Tech, entertainment, and art.)

I am using Revue to write and send the newsletter. So far, so good. Their support has been super responsive (thanks Martin).

You can subscribe to my newsletter by following the link below to my Revue profile page.

I look forward to sending you some interesting emails in the near future.

🏁

Design a Minimal Product Roadmap instead of building an MVP

Tue, 18 Jul 2017 09:12:55 +0000

Often when advising a founder or colleague, we end up talking through what an MVP could look like. Not everyone agrees on the definition of “viable,” however, and this always creates confusion. Lately, I’ve found it useful to take a slightly different view to find a path from zero to one customer. I call it a Minimal Product Roadmap.

The different stages of a Minimal Product Roadmap should be close to (from the potential customer’s perspective):

I understand what you want to build.
I can use what you have built on my device. Cool.
I can see exactly how what you have built will help me.
I would pay for what you have built.
I love what you have built. ❤️

The order of 4 and 5 can change depending on the product. People will pay for things they don’t love and people might not pay even tho they love it. Also your product might be free. C’est la vie.

Side-note: I have heard people say “Minimum Lovable Product” before to emphasize the importance of getting to that last step. It sounds hokey, but it makes sense.

Focusing on the core problem and finding a way to prove that your idea is the best way to solve that problem is a tough challenge. It’s even harder if you need to build every feature and implement every detail. Instead only build what can get you to the stages I listed above. And for only one problem at a time.

Laying out small steps means you can get feedback at each step, possibly re-evaluate where you are headed, then arrive at a better place in the end. Getting feedback is crucial to building any product. Having a small plan of small steps means you will start getting feedback much sooner. You might end up building three, four, or more versions before arriving at one that is understandable, usable, and lovable.

So many MVPs should be a Google Form that inserts a row into a Spreadsheet. The founders should have a separate tab with charts and graphs for their “analytics dashboard.” Google Docs has an API and you can hit it with javascript to build a frontend. Need something fancier? Then use Wufoo. Wufoo can even accept payments, provide tracking and analytics, and basically be your “customer database.”

Sure, lots of ideas are bigger than a web form, but they are probably not as big as you are thinking right now. It is one of the most difficult initial challenges for a founder to stay small, to resist building that extra thing, to instead polish the one feature you truly need until it’s the best it could be. I suffer from this all the time too, “I need to do this and this and this before anyone will get it!” It’s just not true: you don’t need to do everything. It’s easy to procrastinate by doing some easy work, like designing more and more screens, and avoiding the hard work of building a lovable solution to a single hard problem.

Adaptive Path has talked about the “cupcake model” which is a great way to visualize this:

Photo from Speckboy who got it from cupcakepictures.com which seems to be gone now.

I love this analogy because a cupcake can be super detailed if one wanted. A cupcake can be something a customer loves and pays for without being full-size. You don’t need to build the cake, just build the smallest amazing bite. A cupcake place in Washington DC I used to frequent sells “icing shots.” Can you build the icing shot version of your product? Give your customers a hit of sugar as soon as you can. And don’t waste your time building a large vanilla sponge cake with no icing or decoration, that’s not lovable.

You’ve probably seen a graphic or two showing the progression from a skateboard to a bicycle to a car, instead of building a wheel then axle then body, etc. The first time I saw it was from Spotify:

Found this image over at Quora.

This is also a great way to reorient how you think. Maybe your startup is about “moving people from point A to B” and so you start building a car. However, a skateboard fulfills your mission statement, so why build a whole car? Are you about cars or about “personal transportation?” Maybe your customers want to get some exercise? Maybe your customers want to rent the vehicle and not own? Maybe your customers have to provide some elbow-grease at first to help you along at first? That’s fine. You cannot learn what your customers really want until you are in front of them. A Minimal Product Roadmap gets you in front of them faster.

And, more selfishly, it feels much better to ship an amazing cupcake than a mediocre full-size cake. Receiving positive, loving customer emails is a great motivator, especially when it’s early and everything is super uphill. Be encouraged: it’s possible to think small, to build something minimal and lovable, and to have fun doing it. 😀

I love talking to and advising early startups and helping them design a Minimal Product Roadmap. If you wanna chat then reach out to me. Also, if you have any feedback or criticism about this article then shoot me an email.

🏁

On queueing

Thu, 29 Jun 2017 10:28:44 +0000

When building a web app, or any kind of distributed system, things will probably get backed up. Most people I’ve worked with are surprised if/when this happens the first time. I know I was surprised when I first dealt with it. So remember: if you use a queue for something, it will queue up eventually.

You might be queueing stuff and not realize it. If you are making HTTP requests from your code then you probably have a request pool. Making a database call probably checks out a connection from a pool or waits in a queue for the next available connection.

At Wunderlist we saw a seemingly simple change have big repercussions when it came to request queueing. Originally most of the Wunderlist backend apps were written in rails and used unicorn. We really wanted to use puma to use less memory from forking fewer processes.

According the unicorn documentation:

Load balancing is done entirely by the operating system kernel. Requests never pile up behind a busy worker process.

This sounds fine. We assumed that’s just how people make HTTP servers. It’s not. A subtle detail about puma is that it queues requests inside ruby code, which means it’s up to the ruby code to manage timeouts and stuff like that. It’s incoming queue is basically unbounded: if you ask it to take a connection, then it will. So it’s up to you to not give it a connection if it’s extremely busy or backed up. Puma is an extremely well written ruby library and you should read through it’s source sometime.

This reminded me of this article about queueing with rails on heroku and it’s worth a read. It pays to have data, graphs, etc to be able to really know what is happening.

We did transition some less critical apps to use puma, but in the end almost all the critical path apps were rewritten in scala (using play) and so we traded one set of problems for another.

Backing up a bit

Recently I read this amazing article by Fred Hérbert about Handling Overload. You should read it too. The amount of information in that post is overwhelming.

The basic rule at the beginning is what you need to know: Little’s Law. From the article:

[T]he capacity of your system is going to be how long a task takes to go through it, [multiplied] with how many can be handled at once…

If we can answer 100 requests “simultaneously” and a request takes 200ms on average to complete, then any time there are over 500 requests per second we either queue up, need to apply back-pressure, or drop some requests.

Queueing up is the worst of the three choices IMO. Sitting in a queue means that the average time to process one item is increasing, even though we are not taking longer to do the real work. Every time this happens we start losing capacity slowly. As the average time to handle a request goes up, our ability to answer the newest incoming request before it times out becomes less likely. I believe it’s better to fail fast and let the requester retry later then to let capacity take a dive.

One thing we used to do at LivingSocial during challenging days was to decrease the average time of a request by shutting down expensive operations. Some features had a “feature flag” around them (an if statement) and we could turn off all expensive operations in the critical path to make sure our capacity didn’t drop. This sounds really simple, and that is because it is.

Take aways

First: always be fast. Spending less time on each work item means it’s easier to maintain capacity. Granular work is easier to scale than monolithic work.

Second: understand the queues in your system as much as possible. Where can things back up? What are the limits on those queues? Are they unbounded in size? Are there timeouts on the work items? If a work item times out while it’s in a queue, is it removed from the queue or does it persist? Go looking for settings, limits, etc of everything and compile a list of possible capacity killers.

Third: collect timing metrics everywhere. Everything is worth measuring. Sometimes you’ll learn that writing to your logs on stdout is slowing down your request processing because someone accidentally changed syslog to use synchronous tcp and not udp. It would be easy to think “I don’t need to know how long it takes to write a line of text to the log stream,” but yeah you might need to know that someday.

Have fun. Build cool stuff. Don’t back up if you can help it.

Any feedback? Did I make a mistake? Am I wrong on the internet? Send me an email.

ElixirConf EU 2017 in Barcelona

Tue, 27 Jun 2017 10:01:17 +0000

Hello.

I design distributed applications from UI to data storage as a career. I started out as a graphic designer, got into programming, and now that has spiraled out of control.

Aside: I built my own presentation software using choo and phoenix. If you want to have a lot more anxiety than necessary, then build your own presentation software. As an example: I don’t have speaker notes because I didn’t have time to implement that feature.

This is what the presentation software looks like for me.

Distributed applications are hard

One has to deal with network partitions, concurrency, locking and/or linearization, stale data, etc. Here is a painting that represents how I feel about distributed systems: they are great fun and I feel dead.

By Ted Parker — http://ted-parker.com/portfolio/surfs-dead/

Network latency

Network latency can cause stale reads from consistent systems. A server’s response for one request can arrive much later than expected, possibly after one or more other responses that were sent after it. This happens all the time. 😱

Networks can cause everything to be out of order, no matter which is sent first.

Scaling is hard

One has to deal with load balancing, parallelism, databases, AUTOVACUUM & VACUUM… Even after all these years managing databases is still hard.

Why build a distributed system in the first place?

Small systems are super fun. I’ve really enjoying building systems for 100 or 1000 users in the past. But to support “billions” of users one has to get good at horizontal scaling, offline support, and dealing with bad networks. This means one must build a distributed system. 💔

Granularity

I recently watched a talk by Joe Armstrong where he talks about the best ways to build concurrent software. His primary anaolgy is that: if your CPUs or VMs are like buckets or containers and you design your system components like large stones or bowling balls, then you end up with a lot of empty space in the containers. If you instead designed your components like small grains of sand, then you can really fill up each container to the brim with little wasted space.

This really had an impact on me: granular = scalable. 💝

A crazy idea

For a while I and some collegues have had this crazy idea: what if every user had their own database, maybe as a simple file stored on S3?

Having a central database in a system makes a lot of things easier; however, a central database is hard to scale. What if we took this idea of granularity and really went with it?

First, we must talk about “time”

Human time cannot be trusted in a distributed system. Instead, we have to invent our own measurement for when our system “moves foward.” Any working distributed system is an implementation of how that organization has understood time.

Most systems I’ve worked with have used a central database to control time. We use transactions to linearize updates to the important parts of our system. This means all our components can be relatively stateless and provide no guaruntees about how time works: we just let the database decide. If we remove the central database, then we need to have another way to control time.

Actors

When building Wunderlist’s real-time backend we used actors to wrap each websocket connection. They are a great abstraction to contain state and linearize access to that state.

Acting like a Firefighter https://flic.kr/p/dnt7vz

Elixir processes

Processess are also a great way to linearize access to state. What if every user had their own process?

Elixir processes are addressable, even across machines, by a pid. To trust a process as the authority for the state of a user we need to know there will only ever be one process for that user. If we were to accidentally have two “myobie” processes, then we have a split brain.

:global

Erlang ships with a :global process registry which is shared across all connected nodes. However, we cannot rely on it to be constistent. You may be thinking are you sure?

If any name clashes are discovered, function Resolve is called.
http://erlang.org/doc/man/global.html

There are three words that jump out at me from that quote: clashes, discovered, and Resolve. It’s reasonable to assume that if it weren’t possible to have duplicates, then there would be no need to have a Resolve function.

😐

:pg2

A popular erlang library is “process groups 2.” This library does not attempt to provide a consistent view of the cluster’s processes. Are you sure?

pg2 replicates all name lookup information in a way that doesn’t require consistency …
http://erlang.org/pipermail/erlang-questions/2012-June/067220.html

This could be very useful for different types of applications where doing something twice wouldn’t hurt (like phoenix pubsub), but for my use case I really need to know there are zero or one of something at all times.

😐

:grpoc

Same. Are you sure?

While gproc has been tested very thoroughly … its reliance on gen_leader is problematic.
https://christophermeiklejohn.com/erlang/2013/06/05/erlang-gproc-failure-semantics.html

Are there problems with gen_leader? I don’t know. I’ve read multiple articles saying something negative about gen_leader, but I’ve never understood why. I intend to use gen_leader sometime in a project and try to get more familiar with it. I don’t want to jump into that today.

😐

Elixir.Registry

Elixir ships with a Registry as of version 1.4. We are saved!

This registry is local to a single VM and it is consistent. It’s backed by ETS so it’s super fast. It’s easy to be consistent on one machine. 😏

What happens if that one VM stops? Our entire registry disappears and we cannot get any work done until it comes back. 😐 This could be fine if we didn’t mind the downtime, but I kinda do mind.

How do we make 100% certain we never accidentally boot two vms? It may sound like an easy problem, but I’ll tell you it’s not. At Wunderlist we have an email related thing that we can only run one of, but I know we’ve accidentally run two before.

☹

Zookeeper

What are we to do? Let’s outsource the problem to someone else.

Zookeper is consistent, pretty available, and partition tolerant. And, more importantly, it includes recipes. Yes, it has instructions for common use cases including in it’s official documentation!

➡️ ZooKeeper Recipes and Solutions

And one recipe is super interesting: distributed locks. 🔑

“Distributed locks are a major key.” [https://www.flickr.com/photos/88009602@N05/8696887207](https://www.flickr.com/photos/

Zookeeper is a battle tested system for getting time under control. We can know if one thing happens before another. That is really helpful.

Highlander

Duncan MacLeod of the clan MacLeod.

I have created a library named after a 90’s TV show.

https://github.com/myobie/highlander

There can be only one.

It has a concept of “object” which are simple GenServers with some extra abilities:

Objects persist their state to S3 during a state update
Objects read their state from S3 during init
Objects respond to calls directly from their in-memory state
Objects teardown after some period of inactivity

Highlander also includes a process registry which stores it’s information in Zookeper. Every process has a key named after it’s id and it’s value is the node name it’s running on. This means we can lookup where any process is running and send it messages from any node in the cluster.

GenServer :via

There is this :via concept with GenServers: they can have a fancy name that is a tuple instead of just an atom.

GenServer.call(
  {:via, MyRegistry, {:user, user_id}},
  :do_stuff
)

What does a process registry look like in elixir? It implements four methods:

defmodule MyRegistry do
  def send(name, message) do
  end
  
  def whereis_name(name) do
  end
  
  def register_name(name) do
  end
  
  def unregister_name(name) do
  end
end

How do we find a process through zookeeper?

defmodule MyRegistry do
  def whereis_name(name) do
    case ZK.get_node_name(name) do
      # ...
    end
  end
end

Simply query zookeeper for the key (node) with the same name as the process. It’s value will be the node name. From there we can construct a pid.

Registering a new process is similar:

defmodule MyRegistry do
  def register_name(name) do
    case ZK.create_znode(name) do
      # ...
    end
  end
end

Registering means trying to create a key (node) with the same name as the process.

In my ZK module’s create_znode if we are not the first to create the new key in zookeeper, then it will fail accordingly:

defmodule ZK do
  def create_znode(name) do
    {:ok, pid} = ZNode.start_link(name)

    if ZNode.first?(pid) do
      {:ok, pid}
    else
      :ok = ZNode.delete(pid)
      {:error, :already_exists}
    end
  end
end

How do we know we were first? Zookeeper’s distributed lock recipe shows how to determine that. Here is how it’s implemented in Highlander:

defmodule ZNode do
  def init(name) do
    {:ok, path} = 
      Zookeeper.Client.create(:zk,
        prefix(name, UUID.uuid4(:hex)), 
        to_string(Node.self),
        makepath: true, 
        create_mode: :ephemeral_sequential)
    {:ok, %{path: path, name: name}}
  end
end

The most important part of this code snippet is **ephemeral_sequential**. It is “ephemeral” so it’s temporary and if we drop our connection to zookeeper all the keys we created will be auto-deleted. It’s “sequential” so that zookeeper will append a number to the keyname we provided (the return value of prefix/2).

Zookeeper keeps time in order. It’s very good and forming consensus around sequential numbers. So instead of asking everyone in it’s cluster “is this key already taken” it gets agreement on the next number, you create that key, then you look back at all keys that start with the same “prefix” and if you were the lowest number then you must have been first.

Because of the “ephemeral” flag, we need to make sure that if we drop our zookeeper connection that we also exit any highlander processes that we had booted — since their keys are no longer set in zookeeper at that point. We can do this with a supervision tree:


opts = [host, 
  [stop_on_disconnect: true, name: :zk]]

children = [
  worker(Zookeeper.Client, opts, []),
  worker(Registry.Server, [], []),
  worker(Registry.NodeCycleServer, [], []),
  supervisor(Object.Supervisor, [], [])
]

supervise children, strategy: :rest_for_one

The :rest_for_one strategy means that if any child dies, then all children declared after it in the array will be exit‘ed. So we simply put Zookeeper.Client first there. That process will exit if it loses it’s connection because above we set stop_on_disconnect to true. It exiting will cause the registry server, my “node cycle server”, and all objects being supervised to also exit. Once zookeeper reconnects then the registry and object supervisor will restart and we are back in business.

Developer UX

Highlander “objects” are GenServers with some extra functions. Instead of having to build the same GenServers over and over again, I also created some macros to help with that. An example of making a “todo” object to contain a simple task:

defmodule Todo do
  use Highlander.Object, type: :todo

  defobject title: "", completed: false, color: :blue
end

defobject really just calls defstruct behind the scenes and setups up seriliazation for when the todo’s state is persisted to S3. Yes, with this “object,” every todo will be it’s own process and it’s state will be serialized to S3. Here is an example of how to use it:

id = UUID.uuid4()

{:ok, todo} = Todo.get id

assert todo.completed == false
assert todo.title == ""
assert todo.color == :blue

todo = %{todo | title: "Hello"}

{:ok, _} = Todo.put id, todo

# any amount of time later

{:ok, todo} = Todo.get id

assert todo.title == "Hello"

Model everything as a process

With this approach we can model everything in our system as it’s own process. That makes it super easy to serialize access to state, persist it to S3 as necessary, and our system becomes a reflection of the active “objects” currently collaborating.


defmodule User do
  use Highlander.Object, type: :user

  defobject name: "", email: ""
end

An example user module.

Model all shared objects as their own process

After every user has a process with state, it is tempting to nest everything under the user who created it. For a lot of data this makes perfect sense. But objects which are collaborated on by multiple users needs to be separated out, given it’s own state, and possibly protected with some sort of permissions.

An example might be a shared list, where any user with permission can manage a list of todos:

defmodule List do
  use Highlander.Object, type: :list

  defobject title: "", todos: []
end

defmodule Todo do
  defstruct title: "", completed: false, color: :blue
end

To manage permission, the User probably needs to know which lists it has seen before. And the List could keep an array of allowed users. See below:

defmodule User do
  use Highlander.Object, type: :user

  defobject name: "", email: "", known_lists: []
end

defmodule List do
  use Highlander.Object, type: :list

  defobject title: "", todos: [], allowed_users: []
    
  def handle_update({:insert_todo, list, todo, user}, state) do
    if Enum.any?(list.allowed_users,
      fn id -> id == user.id end) do
        %{todos: [todo | state.todos]}
    end
  end
end

So each list is a concurrent object in the system. If one list because super busy with updates, it wouldn’t slow down any other list’s access or anything like that. The handle_update above doesn’t actually work yet: I haven’t decided on the syntax for handling state updates yet. Do you have opinions or ideas? Let me know.

An aside: Multi-VM testing

When building highlander I needed to boot multiple erlang nodes to be able to test cross-node calls and stuff like that. That wasn’t entirely easy or straight forward. It turns out Phoenix does this to test it’s websockets stuff, so I just copied a lot of that code.

The most interesting bits are:

# how to boot a new VM process
def spawn do
  :net_kernel.start([:"primary@127.0.0.1"])
  :erl_boot_server.start([])
  allow_boot to_char_list("127.0.0.1")

  Application.get_env(:highlander,
    :spawn_nodes, [])
  |> Enum.map(&Task.async(fn ->
    {:ok, _node} = spawn_node(&1) 
  end))
  |> Enum.map(&Task.await(&1, 30_000))
end

# how to start a `:slave` and then send it the code to load
defp spawn_node(node_host) do
  {:ok, node} = :slave.start(
    to_char_list("127.0.0.1"), 
    node_name(node_host), 
    inet_loader_args())

  add_code_paths(node)
  transfer_configuration(node)
  ensure_applications_started(node)
  {:ok, node}
end

You can see all of it in the highlander repo’s test folder.

That was fun!

Scaling

Granular = scalable. Having all these small processes is a lot like “pouring sand” into our cluster.

How many locks can a small zookeeper cluster handle? I don’t know.

If it became a problem one could shard zookeeper. Or one could do what other libraries do and not store a zookeeper key for every process, but instead setup a partition map in zookeeper and use that to shard processes onto the box they map to. Then there would only be a key for every erlang VM and not for every erlang process.

Orleans

That is what orleans does: setup a partition map in zookeeper. Orleans is what the Halo team used to scale Halo 4 and 5. As far as I know every player, game, etc is a process in the cluster and they collaborate with message passing.

There is also Microsoft Service Fabric which is a lot like Orleans, but tries to help with building stateful and stateless services more than actors. It also uses zookeeper to setup a partition map for it’s services. It also persists the services state across the cluster when that state is updated.

Akka

Sure. That’s a good choice. I am not really doing any scala right now and I’m a bit scala’ed out for the moment. As I said before, Wunderlist’s websockets servers are all in scala (using play) and it’s very reliable and performant. Love it. It’s good. Use it.

riak_core

Sure, you could use the riak internals to do your own consensus, but I haven’t yet drank enough coffee in my life to start down that road. Maybe I will one day, and if you are interested then definitely go for it, but I am not able to think about that without having a lot of anxiety. 🤷‍♂️ 🌳 🤷‍♀️

And that’s it. Hope you maybe learned something.

Have fun. Have crazy ideas. Build things. 🏁

A way to procrastinate

Thu, 22 Jun 2017 23:53:43 +0000

Wasting time is something I fight against all the time. I mean waste in the sense of having zero value. However, I also know that I cannot be productive every hour of every day. I’ve tried and it’s too exhausting.

Instead, I am always doing around four different side-projects, reading a couple books, watching a few lectures, etc. Having multiple things going on doesn’t mean you lack focus; focus is being able to prioritize and choose the correct task for right now.

Don’t wast time, invest it.

Procrastination is about priorities. Is your current task what you really need to be doing? Is your due-date real or just a trick to motivate yourself? What happens if you don’t finish on time?

I believe we can, and should, invest in ourselves a little everyday if possible. Instead of constantly working, we can learn, read, and rest. Then we can work more effectively when things are pressing and due-dates are real. Sometimes things really do need to get done, and fast, but I believe a lot of people are busy being busy and not because things are truly pressing.

Waste

My versions of wasting time are: scrolling twitter for more than 5 minutes, clicking on random youtube videos for no reason, reading political news, watching some old show on netflix…

These things almost never bring me happiness, they are never valuable for me, and I know I’m wasting the time I spend on them. These are the things that consume my time wastefully, yours are almost certain different. I think it pays to know them so you can be more mindful. Being mindful is a super power that will allow you to procrastinate more effectively.

Invest

Instead I try to: read a book, take a nap, build something useful, learn more about philosophy, science, or culture… I’ve been trying to get into making music, writing more about myself and my work, and reading. I was never a reader growing up and I’m trying to make up for that time.

Some days the most important decision I make is to wait, rest, and then do work. Instead of always working during normal business hours, I sometimes take the time earlier in the day to do other things and then focus on work in the evening after letting the problems bake a bit. I might spend that day building a new webapp to solve some niche problem I have, design a new logo or a set of icons for an app I want to build, or watch lectures about databases, design, or the history of hip-hop. I consider all of these eventually valuable.

Being late

I try not to let people down. I try not to be late on things. But sometimes I end up in a place where I just cannot work more hours, I can’t build more things, I cannot create more stuff without reading, sleeping, or somehow recharging. I’ve overworked myself many times in the past and I’m just not into that anymore. It’s always made things worse.

Let the stakeholders know what’s up, let them know you didn’t plan accordingly, and learn for next time. I’d rather be honest, realistic, and eventually get better at planning.

It costs something.

I’ve found that it costs a whole lot to construct a less stressful environment where I can do those things which recharge me. How can I take time to read a book when I have looming deadlines and I might get fired? When am I going to find the time to hang out with friends, I’ve got all this work to do? I don’t have time to take a nap, I gotta have this report turned in by midnight.

Yes.

There is a cost to prioritizing yourself. Saying “no” always costs something. Instead of imaging it’s always impossible, I recommend working out the realistic costs of these decisions so you can do a cost/benefit analysis for yourself. Don’t work from incomplete assumptions if you don’t have to. Sometimes it does cost too much, but make sure you know that.

It pays off at work.

(At least for me.)

I’ve almost never done something at work for the first time. Instead, I’ve experimented on my own project or built/designed something for myself, then repeated that at work. Even working on the Wunderlist rewrite I ended up just re-typing tons of code that I had typed before, having conversations that I’d already debated with myself, and mostly just implementing ideas that I’d previously worked out.

I like making my mistakes early and in a controlled environment when possible. It’s not always possible: sometimes I gotta do something new right now. Hopefully I’ve invested in myself so I’m rested, aware of what’s possible, what’s easy, and can break the problem down into small enough chunks to attack. When that’s not the case it takes me far longer to be effective at my job.

Reading regularly has been the best decision I’ve made for myself in years, probably ever. Not just because we absorb information from books differently than other mediums, but also because I’ve got a really good excuse to procrastinate when I need it. It’s an enjoyable privilege to be able to take time to read through a book. There have been periods in the past when I didn’t have the time (or the money to make the time) to read and I’m very grateful that I can currently carve that time out for myself.

Good luck. Do your best. And, if you don’t already prioritize it, hopefully you can find some time to invest in yourself soon.

All the photos in this post are from deathtothestockphoto.com.

Swift already has a Result type

Mon, 23 Jan 2017 11:13:32 +0000

Photo by JD Simcoe @ https://unsplash.com/@jdsimcoe

Over the last few years I’ve had the privilege of writing and deploying applications using a myriad of languages. Scala, ruby, javascript, haskell, clojure, etc. Every time I pick up a tool, especially if it’s new to me, I try to learn the grain of it. A programming language is a toolbox of ideas and concepts that anyone can use to build stuff. Yet, I see a lot of dissatisfaction from my colleagues and peeps online with the tools currently available.

There is a great talk about how swift is not a functional language by Rob Napier and you should watch it. It’s a great lesson about how one should use a language how it’s made and not how one wishes it were. However, I was really disappointed that Rob made the case that swift should add a Result enum. Swift already has this feature, but I guess a lot of people don’t like it.

Swift has an Optional type, but you don’t normally see it directly. You interact with optionals through language constructs. An example:

struct Person {
    let name: String
    let bio: String? = nil
}

let person = Person(name: "Nathan")

if let bio = person.bio {
    print("\(person.name)’s bio: \(bio)")
} else {
    print("\(person.name) doesn't have a bio.")
}

The words optional, some, and none are nowhere to be found in the above example.

Optional is basically a “maybe” type and it’s really nice to have support directly in the language for it. Instead of writing a bunch of case .some and other similar statements, we can just use let and other language constructs to work through detecting if anything is in there.

This example below is similar to a lot of examples I’ve seen for a Result type:

enum Result<T> {
    case success(T)
    case error(String)
}

func isLowerHalf(_ number: Int) -> Result<Int> {
    if number < 50 {
        return .success(number)
    } else {
        return .error("Sorry.")
    }
}

isLowerHalf(70) // error("Sorry.")
isLowerHalf(20) // success(20)

At first glance, this seems pretty nice. I’ve added enums like this to a lot of my swift projects. However, recently I’ve noticed that I always end up having to perform a lot of ceremony to extract the values from the enum:

let result = isLowerHalf(40)

switch result {
    case .success(let value): print("Yes, \(value) is in the lower half!")
    case .error(let value): print("\(value) Not in the lower half.")
}

Or one can try to only extract and use the success case:

if case let .success(value) = isLowerHalf(20) {
    print("OK, that worked, but it's pretty weird.")
}

This is a lot of work to fight against the tools that are already in the language. Swift already has a way of marking a function as returning one of two types, one is a success and the other is an Error, and it’s throws.

struct NotLowerHalf: Error {}

func isLowerHalf(_ number: Int) throws -> Int {
    if number < 50 {
        return number
    } else {
        throw NotLowerHalf()
    }
}

Now we can use the built in facilities of the language to call the function:

try? isLowerHalf(70) // nil
try? isLowerHalf(20) // 20
do {
    let value = try isLowerHalf(70)
    print("Yes, \(value) is in the lower half!")
} catch is NotLowerHalf {
    print("Not in the lower half.")
}

if let value = try? isLowerHalf(20) {
    print("OK, that worked, and it's not _too_ weird.")
}

let result = try! isLowerHalf(40) // would cause a crash if it failed

Many people I’ve talked to say they avoid throws because they believe using exceptions for control flow is bad. I agree, exceptions are not reasonable for control flow most of the time.

throw does not raise an exception. From the swift docs:

Swift provides first-class support for throwing, catching, propagating, and manipulating recoverable errors at runtime.

Errors are not exceptional circumstances. Recoverable errors are expected and should be planned for.

A function marked as throws returns an unsuccessful result of type Error and the compiler forces the programmer to handle both the success and error cases. do and catch are like specialized switch and case keywords for when a function might return a success or error value and offer the pattern matching features one expects.

And yes, there is rethrows to propagate an error up the function stack. I’ve done the exact same thing in scala to “slipstream” an error from a Result up during things like a web request. This is control flow using types and pattern matching and I am very glad that it’s built into the language and not something I have to build on my own each time.

The word try is the perfect way to explain what I’m trying to accomplish: try this and if it doesn’t work out then I’ll handle it this other way. do isn’t my favorite word for the start of the try/catch situation, but I can’t think of anything better.

I’m not a swift expert, I don’t use it everyday, and I’d like to hear what you think about this subject. Should swift add a Result enum anyway and let the programmer choose between throws and Result? Or would that fragment libraries where some throw while others are all enums all the way?

Thanks for reading.

I’m getting “engagement”

Fri, 11 Nov 2016 10:48:45 +0000

I’m not sure how, but my tweet got retweeted beyond my normal network of people and I am getting a lot of responses. The moment I tweeted has been tweeted by others I know, so I wonder if the same thing happened to them. I expected my network to be small and homogenous, so I am surprised by the reach this tweet has gotten. People really have an emotional response to reading through that moment and want to spread it or weigh in.

The way Twitter’s UI works for responding or retweeting is confusing and I think this is why people are responding to me so often. There also seem to be many conversations between lots of other people and I get notified of every response. I normally don’t get to see this side of Twitter since I don’t have a ton of followers. The total impressions for my tweet are 34,805. I’m not sure how that compares to others but my next largest number of impressions I can find is around 800. Somehow that adds up to >40,000 I guess.

I’ve only blocked maybe 20 people because of racist or violent responses, so I think that is pretty low percentage to what I’ve heard from friends with more followers. I also had to block some people because I just kept getting more and more responses from them.

Twitter is able to cause a flood of responses from all sides. I haven’t received much abuse, but I have friends who have. I’m very privileged because if this were my normal twitter experience, then I would leave. Others stick it out to connect with others, even through the torrents of responses like I’m temporarily seeing.

Posting a link isn’t an endorsement or to shame, but people project onto it what they want. Some people assumed I was sympathetic to the victims of the hate speech while some others assumed I was sharing that link to gloat about how awesome it is to not need to be politically correct.

Any software that connects people needs to have tools and protections in place for it’s users. Twitter is failing. It allows people to swarm others with any content they choose and rarely finds that content to be against it’s terms of service. If you build software (I do) then you need to think about your most vulnerable users and plan how you will enable and empower them to have a positive experience. Don’t be lazy, there are real consequences if you are.

CloudKit

Fri, 12 Aug 2016 17:45:48 +0000

The new capabilities in CloudKit for iOS 10 and macOS Sierra are super encouraging. I have been convinced for some time that collaboration and sharing are incredibly important for the future of all mobile productivity software. Productivity software is all about managing data, people, and how they relate to each other. The new additions from Apple make this easier for a larger number of people. As more enterprises buy iOS devices instead of Windows, this could be even more important sooner.

When beginning to build the realtime sync for Wunderlist we tried to setup our server architecture as a product which the native applications “consumed.” We have had some success with this approach, but we never truly made a generic “realtime storage and collaboration system.” Which is fine, we make a to-do manager and not a storage service. (Of course we talk sometimes about how fun it would be to make our stack generic and open.)

We knew we were going to build a system that will be obsolete very quickly. Just how people choose between databases (MySQL or Postgres), in the future people will choose between their serverless realtime storage and collaboration products (CloudKit, hoodie, deepstream, …). (We need a shorter name for serverless realtime storage and collaboration product. Also is it serverless or nobackend?)

Apple has been very, very slowly evolving iCloud over time and now they have built a file/blob storage system, a record/object storage system, sharing and collaboration (ACLs), and basic conflict management. They are succeeding at a huge scale and I’m sure it will eventually be “realtime.” (Realtime means: very fast. Like a few seconds or less to get updates anywhere in the world.)

Already, when using an app that stores data in iCloud Drive: I edit on my iMac, move over to the couch to edit on my MacBook, changes are already there. And later when I’m on the go I make some edits on my iPhone which all syncs together fine. I am collaborating with myself. The next obvious step was to collaborate with any number of people and that is just what is coming soon.

Notes’s new sharing screen

Wunderlist’s realtime sync won’t be a competitive advantage for long. Apple’s Notes is a great example of how one can use the new APIs to take an existing product and make it collaborative. I expect in a few years this will be a common service provided by all operating system makers.

While Apple does provide a javascript CloudKit SDK, and it is already being used on the iCloud web version of Notes to provide sharing and collaboration, they don’t offer native SDKs for Android or Windows. (Someone did port the web service calls to Android, which is amazing.)

Some things you might want to look into:

Who is going to create the first good multi-platform serverless/nobackend realtime collaboration and storage system?

Be.

Thu, 19 May 2016 09:35:57 +0000

I learned at a young age to hold on to possessions, people, and desires with an open hand. Things come and go. Grasping tightly does not help. It only forces us to be anxious, stressed, or worse depressed. It is tempting to imagine that holding on less tightly is easier for someone with more money, more status, or more fame.

It is not.

It’s not possible to live with nothing. We are all pursuing something and that pursuit can most likely be made easier with money.

I’ve become a person who values quiet time, watching and reading stories, and drinking a warm beverage. But even more I value spending time talking to someone, not for gain or obligation, but to just be there with them. Being with someone is one of my most favorite things. It’s also something I think very few people really practice.

The things that I value most are said to be free or nearly. But while a used book, some tea, and a sketchbook are close to free: the free time and lack of stress to enjoy them are not. The less anxious, the less hurried, and the more prepared seem to have more chances to partake in lovely things.

People living in poverty are 10% or more less productive simply because of their predicament. The stress of having little makes it more difficult to climb up to have more. Other stresses have the same effect; I’ve experienced this. Still, grasping on to things tighter does not keep them around longer.

Be with someone, be present, and let them know you are simply there. Hold on to them with an open hand. Maybe they’ll be a little less stressed and little more able to climb up somewhere.

The Network Is Never There for You

Fri, 26 Feb 2016 12:33:36 +0000

This started as documention for some of the things that are hard about building a distributed system like Wunderlist. The goal is to clarify some assumptions about distributed systems and come up with possible solutions. Mostly: how can one persist data to a database while also notifying others of those changes in a reliable way. Real-time™ is hard.

Also, there is some pseudo code in this post which is basically javascript. Don’t comment on code style. If there are mistakes, then please let me know. But it’s not really javascript, so don’t worry about it.

It’s tempting to imagine that distributed systems are “simple” or “easy.” I’d like to describe a simple system and go through a few issues that turn out to be a bear. The goal here is to force to the front some basic assumptions one should have when building a distributed system:

Adjacent lines of code do not represent an atomic unit of work
When network calls fail, it’s impossible to know what happened
Transactions and mutexes are only useful inside one process on one computer

The simple system I want to build is this:

A database (postgres) will hold a value. The value can be updated atomically and the rest of the system wants to be notified about the change in a reliable way. Every change should be queued up so some other part of the system can read and process each one, possibly sending it to other parts of the system like a phone or something. These phones want to know about changes as fast as possible!

This is actually a very complicated problem, so let’s size it down some at first: instead of notifying the system of every change, let’s just publish the latest value to redis. A phone can poll frequently for the new value from redis. And let’s just increment a counter in the database. Much simpler.

The value in redis should never be different from the value in the database. (Which is impossible, but still let’s walk through it and see. Also, if you disagree that it’s impossible, then please keep reading and don’t build anything until you’re done :)

There are a lot of ways to increment a counter in postgres. The best ways are to either use a sequence (a built in feature of the database) or to issue a SQL statement similar to:

UPDATE counters SET value = value + 1 where counter\_id = 1

Let’s assume we have a function updateCounter which performs that SQL and then a function updateRedis which does a

SET counter:last 2

where 2 is the return value from updateCounter.

So our code is now:

function receiveConnection() {  
    let latest = updateCounter()  
    updateRedis(latest)  
}

And this falls down in every way. Let’s go through the problems one by one.

Adjacent Lines of Code Are Not Atomic

In our simple two line function, it’s entirely possible that updateRedis never gets called. Here is an ASCII diagram of the places this function can fail:

function receiveConnection() {  
--->  
--->let latest = updateCounter()  
--->  
--->updateRedis(latest)  
--->  
}

While it’s easy to predict where execution of this program starts, it’s impossible to predict where it will end. A simple reason is your process could be killed at any moment. But there are other reasons: memory starvation, hardware failure, CPU load is too high, etc. It’s also impossible to predict how much time this program will spend at any one of the five marked points. There could be slow GC, scheduling issues, CPU load again, etc.

The time it takes to get from updateCounter to updateRedis is indeterminate and so our system is inconsistent for an indeterminate amount of time: it’s possible to read a value from redis that is old.

It’s also possible that updateCounter succeeds, but updateRedis fails or the program never even makes it that far.

This simple program doesn’t solve our problem and isn’t a very good component in a distributed system.

So let’s add some error checking:

function receiveConnection() {  
    var latest

    try {  
        latest = updateCounter()  
    } catch {  
        latest = rollbackCounter()  
    } finally {  
        try {  
            updateRedis(latest)  
        } catch {  
            rollbackCounter()  
        }  
    }  
}

This might at first seem great, but it’s actually worse. The updateRedis call could succeed and yet appear to fail, so we would rollback the counter. Now the value in redis is greater than the value in the database.

It’s just a futile effort to try to add enough error checking for a simple reason: the errors are mostly meaningless.

When Networks Fail, It’s Impossible to Know Why

Network timeouts, writing to the socket failure, reading from the socket failure, packet loss, and any other number of things are the types of failures where the caller has no idea what the callee did. Without a response to inspect we are stuck with Schrödinger’s request. The callee might have succeeded, but the network between the caller might have had a blip.

No amount of rescuing or catching errors can fix this issue. The only real way to take care of this is to make the program (and the request to trigger it) idempotent. Either the program needs to operate in a truly atomic way or it needs to not matter if any one part fails.

We are basically screwed now with what we have. Since the increment is not directly tied to the redis call, this function is not idempotent. Let’s try to make it work that way.

One rule we could add is that the caller needs to let us know what its current view of the world is (the current counter’s value) so we can make sure it’s up to date. If you are not up to date, then you don’t get to make changes. It’s a simple rule, but I find it helps a ton in a lot of cases. I also stole this idea from couchdb.

Our code can now look like:

function receiveConnection(remoteValue) {  
    var latest  
    let current = getCounter()  
    if (current == remoteValue) {  
        latest = updateCounter()  
        updateRedis(latest)  
    }  
    return latest  
}

Of course, we will have the issue with network failures hiding successful sets from us. This method is not fully idempotent, it just won’t increment the value twice until the caller is updated. But it does only update the counter once per remoteValue instead of over and over again. It would be nice to try to rollback the increment if something goes wrong:

function receiveConnection(db, remoteValue) {  
    try {  
        var latest  
        let current = getCounter()  
        if (current == remoteValue) {  
            db.startTransaction()  
            latest = updateCounter()  
            updateRedis(latest)  
            db.commitTransaction()  
        }  
        return latest  
    } catch {  
        db.rollbackTransaction()  
        throw("crap")  
    }  
}

So now if the updateRedis call throws we can be sure the database value is rolled back. This is a repeatable function since it only works once and is a noop if called again. This seems great.

But, we know that network errors can be false negatives — the call to redis could succeed even if we get an error, the value in redis is then a higher number than the value in the database again. Redis and the database would be out of sync.

And we have another problem. Let me illustrate it with another amazing ASCII diagram:

function receiveConnection(db, remoteValue) {  
    try {  
        var latest  
        let current = getCounter()  
        if (current == remoteValue) {  
            db.startTransaction()  
            latest = updateCounter()  
            updateRedis(latest)  
        --->  
            db.commitTransaction() // assume this throws if it fails  
        }  
        return latest  
    } catch(e) {  
        db.rollbackTransaction()  
        throw(e)  
    }  
}

There are two things that are wrong at the marked point:

It’s impossible to know how long it takes to jump over this point in the program: from updating redis to committing the transaction. It’s also impossible to know how long it will take to fully commit the transaction. So there is some amount of time between updating redis and the transaction committing that we read a value from redis from the future. Until the transaction commits, only this caller with this transaction can read the correct value from both places. We can’t be consistent even with great error handling.
The program can terminate at this point. It’s possible that updateRedis fully succeeds and receives a response, but the program is terminated and the transaction is never committed. The database would eventually notice this and rollback the changes. No one except this caller would have ever seen the new value in the database, yet the new value would be present in redis. There is no way to be 100% sure our process will never die at this point.

So it seems impossible to reliable do anything across more than one computer.

Transactions and Mutexes Are Only Useful Inside One Process on One Computer

Transactions work for postgres because it’s a single process on a single machine. It’s able to use mutexes and global transaction ids to prevent concurrent updates of rows while also knowing what order to apply changes. The postgres team has done a lot of work over the years to make sure things work how they should. Sure, there are options for systems that can synchronously commit a transaction across multiple machines, but these are daunting solutions with heavy prices to pay in performance and operating costs. And it’s difficult to get right.

Once the distance between things is more than inside a single computer (or virtual computer) things get hinky. It’s no longer possible to reliably protect against concurrent actions. And since network errors sometimes make it impossible to infer what happened on the other side: it just isn’t possible to reliably commit or rollback something.

Even with more error handling, more rollbacks, more verification, it’s basically impossible to build this system with the given constraints in a simple way.

With this example, we are really just caching the current value in redis. This could also be done with a simple read-through cache. In the middle of the database transaction one can clear the cache and then the next time a read comes in it will repopulate. Here is a code example:

function getValue(cache, db) {  
    let cachedValue = cache.get()

    if (cachedValue !=== null) { return cachedValue }

    let currentValue = db.get()  
    cache.set(currentValue)  
    return currentValue  
}

This works all right: if the value is missing we just set it and return it. There is a strange thing that can happen, where if multiple concurrent getValues are run there might be multiple processes trying to cache.set at the same time. This is odd, but it’s not that big of a deal. This is a problem with all read-through caches — when the cache is missing all the running processes might all try to update it.

What does the setValue look like in this case? (Here we are also making sure to lock the value for updating while reading it, so we know we have the truly latest value and it can’t change by anyone else during this transaction.)

function setValue(cache, db, currentValue) {  
    try {  
        db.startTransaction()  
        let current = db.selectAndLockForUpdate()  
        if (current != currentValue) {  
            throw("crap")  
        }  
        db.incrementValue()  
        cache.clear()  
        db.commitTransaction()  
    } catch {  
        db.rollbackTransaction()  
    }  
}

This is looking pretty good, but another ASCII diagram is in order:

function setValue(cache, db, currentValue) {  
    try {  
        db.startTransaction()  
        let current = db.selectAndLockForUpdate()  
        if (current != currentValue) {  
            db.cancelTransaction()  
            throw("crap")  
        }  
        db.incrementValue()  
        cache.clear()  
    --->  
        db.commitTransaction()  
    } catch {  
        db.rollbackTransaction()  
    }  
}

How long does the program stay at the marked point in the code? The answer is: there is no way to know.

We clear the cache, but then we don’t know how long it will take to actually commit the transaction. If a read were to come in to a different process during that window of time, the old value would be cached in getValue— since no one else can read the new value until this transaction is committed.

We can change the code to this:

function setValue(cache, db, currentValue) {  
    try {  
        db.startTransaction()  
        let current = db.selectAndLockForUpdate()  
        if (current != currentValue) {  
            throw("crap")  
        }  
        db.incrementValue()  
        db.commitTransaction()  
        cache.clear()  
    } catch {  
        db.rollbackTransaction()  
    }  
}

And now we have the old problem: there is a time window where a new value is committed to the database, but not present in the cache. There also is the possibility that the database transaction completes successfully, but the cache.clear() line is never run or has a network failure. It’s also possible that the committing of the transaction only appears to fail (yet another network failure) and we don’t know what to do. Maybe we should just clear the cache everywhere.

I’ve seen things like this before:

function setValue(cache, db, currentValue) {  
    try {  
        db.startTransaction()  
        let current = db.selectAndLockForUpdate()  
        if (current != currentValue) {  
            throw("crap")  
        }  
        db.incrementValue()  
        cache.clear()  
        try {  
            db.commitTransaction()  
        } finally {  
            cache.clear()  
        }  
    } catch {  
        db.rollbackTransaction()  
    }  
}

This has all the same issues. It’s possible a too early getValue makes the first clearing of the cache useless and it’s possible the cache never gets cleared after the transaction finishes. It’s also pretty ugly.

We could change the type of transaction we are using to block all access to the row until we are done with it. This means no one can read the row until we either commit or rollback our transaction. Since every getValue would freeze at the point where the query happens until we are done updating the database, they would have the best current value. Or so it would seem.

A possible order of events is (assume the cache is currently empty, process A is about to execute getValue, and process B is about to execute setValue):

Process A begins and retrieves the current value successfully from the database (value in memory is 3)
Process B fully begins and completes running setValue, updates the database row (value in the database is 4), and clears the cache (a noop, it’s currently empty)
Process A proceeds to set the cache to it’s current in memory value (cache is now at 3)

So now the read-through cache set the value to 3 even though the last successful database query updated it to 4. Great.

How often will this happen? That depends on scale, network topology, and a whole host of things that are probably out of your control. If they are within your control then good for you. But most of us can’t control time. Unless you’re google. Then you can. (I mean seriously, google found a way to synchronize clocks so anything is “possible.”)

Now, most people would call this good enough. And at small scale, this is probably good to go. The likelihood of edge cases might be low. But it turns out at even at moderate scale things happen.

How important is it for your application that your data is consistent? If it can be stale for some period of time then you are fine to just set a TTL on the caches and call it done.

Don’t people use caching all the time? What can we do? Can we build a truly reliable system?

Let’s keep going and find out. (Also search for russian doll caching if you want to have a reliable read-through cache.)

What about the original idea?

A database that supports transactions will hold a value. The value is updated atomically and the rest of the system wants to be notified about the change in a reliable way.

This sounds like we should just:

function receiveConnection(db, currentValue, newValue) {  
    try {  
        db.startTransaction()  
        let current = db.selectAndLockForUpdate()  
        if (current == currentValue) {  
            updateValue(newValue)  
            publishChange(currentValue, newValue)  
            db.commitTransaction()  
        } else {  
            db.rollbackTransaction()  
        }  
    } catch {  
        db.rollbackTransaction()  
    }  
}

But this has all the problems discussed above. The worse case for this function is we could publish a value that never gets committed: a value from an alternate future, where it almost made it as the newest version of the value.

If we move things around, it just changes the worst case scenario:

function receiveConnection(db, currentValue, newValue) {  
    try {  
        db.startTransaction()  
        let current = db.selectAndLockForUpdate()  
        if (current == currentValue) {  
            updateValue(newValue)  
            db.commitTransaction()  
            publishChange(currentValue, newValue)  
        } else {  
            db.rollbackTransaction()  
        }  
    } catch {  
        db.rollbackTransaction()  
    }  
}

Now we might commit something to the database that never gets published. While it’s very unlikely that the process will crash right before trying to publish the change: publishing is a network call. It will fail. And it will fail often.

Even if one got really paranoid and tried to revert upon a publishing fail:

function receiveConnection(db, originalValue, newValue) {  
    try {  
        db.startTransaction()  
        let current = db.selectAndLockForUpdate()  
        if (current == originalValue) {  
            updateValue(newValue)  
            db.commitTransaction() // assume this throws if it fails  
            try {  
                publishChange(originalValue, newValue)  
            } catch {  
                db.updateValue(originalValue) // revert  
            }  
        } else {  
            db.rollbackTransaction()  
        }  
    } catch {  
        db.rollbackTransaction()  
    }  
}

This is now even worse. We don’t know the timing between the publishing failure and the revert, so it’s possible that another process just finished updating the value again and we would revert that update (we are outside the row-lock at this point). Trying to plug holes like this is an endless game and doesn’t end well.

How can we make it possible to be sure that every published change is also committed? And that every committed change is also published?

We need to:

Make this truly idempotent
Make sure to retry after any type of failure
Make it possible to know if it fully succeeded

OK, Make Things Reliable Please

The first thing one should consider is “does this really need to be reliable”? What if once a month one screen might show slightly out of date information? Is that acceptable? Maybe it is. Maybe if it saves hundreds of thousands or millions of dollars then it’s fine. It might even be possible to be aware of when that happens, give that customer a coupon for a month of free service, and just move on.

However, if things must be reliable and consistent, then there is only one thing you can do: plan for failure.

The rest of this is going to be about how to plan for things to suck.

Consistency

The easiest (and usually only) way to have consistency in a system is to localize something down to one processor on one computer. This is the basic idea behind sharding, single-threaded-ness (redis), and even actors in something like Akka or Service Fabric.

If we can get things down to that level, then things will work out. This is one way to plan for failure: to make sure that when things fail they are localized to one process so we know we can rollback its changes.

Use the Database

The easiest way to protect against network failure is to not hit the network. Instead of trying to notify every one of every change right when it happens, maybe we just try to store the changes where we are. Periodically we can just send whatever changes are stored to everyone. This isn’t “real-time” per-se, but maybe that’s all right.

Since we are already using a database transaction in our pseudo code, we could just do more work there.

function receiveConnection(db, currentValue, newValue) {  
    try {  
        db.startTransaction()  
        let current = db.selectAndLockForUpdate()  
        if (current == currentValue) {  
            db.updateValue(newValue)  
            db.recordChange(currentValue, newValue)  
            db.commitTransaction()  
        } else {  
            db.rollbackTransaction()  
        }  
    } catch {  
        db.rollbackTransaction()  
    }  
}

With this code we now know the value update and the change record will either both succeed or both fail. This is now idempotent.

This could put a lot more strain on the database and possibly create scaling issues, but it does fix our consistency problem. Except it doesn’t actually notify others of changes. For that, we now need another process to regularly check for changes to send:

function checkForChanges(db) {  
    db.forEachChange(change => publishChange(change))  
}

One thing that could happen now is notifying everyone could fail, so it’s best if the fetchChanges works kinda like:

function forEachChange(cb) {  
    let changes = db.allChanges()  
    for change in changes {  
        try {  
            db.lockChange(change.id)  
            cb(change)  
            db.deleteAndUnlockChange(change.id)  
        } catch {  
            db.unlockChange(change.id)  
        }  
    }  
}

This way if anything fails, then the whole process will just start over. A change is only deleted when we know it’s been sent to everyone successfully. Of course this could be made much better in a lot of ways, but this is the minimum to get it working.

This leads to a scenario where it’s possible a notification appears to have failed, but succeeded–since notifying everyone is a network call. If this happens, the process will the start over and the same change will be sent a second time. It turns out this is fine, because this was already possible.

When dealing with sending asynchronous messages (using a message queue or something) one has to choose between at-least-once delivery or at-most-once. There really aren’t other options (unless you can control time or something like that).

At-most-once isn’t acceptable for us, because that means a notification is delivered 0 or 1 times. We wanted to reliably send notifications to all connected clients. So we would have to choose at-least-once. And at-least-once means 1 or more (so 2 or 3 or 1,000 times).

So our app would already have to be ok with duplicates. We now have two ways duplicates could happen:

publishChange falsely appears to have failed, and we retry
Our message bus has some network issues and delivers a message twice

This is all right. We’ll have to deal with duplicate messages if we are sending messages over a network. It’s just another type of failure we plan for. (Putting a UUID on every message or something like that means we could match up ones we’ve already received; it’s a pretty “easy” problem to take care of.)

So, now we have a reliable way to notify everyone of every change to the value in the database. We can setup cron or have a process with a time run checkForChanges regularly and let it run.

It’s also worth considering some metrics and alerting for how often the checkForChanges is run. If it’s not run within a certain time then notifications are not being sent out. It’s worth investing some time in setting up some monitoring.

If we did want things to be more real-time, we could setup a pubsub channel to throw changes down:

function receiveConnection(db, currentValue, newValue) {  
    var change

    try {  
        db.startTransaction()  
        let current = db.selectAndLockForUpdate()  
        if (current == currentValue) {  
            db.updateValue(newValue)  
            change = db.recordChange(currentValue, newValue)  
            db.commitTransaction()  
        } else {  
            db.rollbackTransaction()  
        }  
    } catch {  
        db.rollbackTransaction()  
    }  
    tryToPublishChange(change)  
}

tryToPublishChange can lock the row, try to publish, and then delete the row if successful. If it fails, no worries–there is a process running that will eventually pickup the change and send it on it’s way. Since we lock each change while we try to publish it, there is no way someone else can read or delete it.

Our system has more components, but it’s simpler. We don’t need as much error checking and we managed to just use what we had in front of us. But there is a real problem with databases: at scale, they are usually a bottleneck. Don’t worry about that until it’s a problem.

We are planning for things to fail and we’ve isolated each part to it’s own process on it’s own computer. So each tiny part can be atomic, which is a huge win!

Write Twice, Then Verify

If the database is going to be a problem, then we gotta go a different route. It’s possible we are already maxing out the queries per second we can handle and we don’t want to add even more and longer locks to our write queries.

If that is the case, then we could go down the route of having three things at play:

Write to primary storage
Write the change notification to a secondary storage place
Periodically (hopefully very often or immediately) take a change, verify the update is in-fact in primary storage, then publish it

This could look something like what we’ve seen before:

function receiveConnection(db, currentValue, newValue) {  
    try {  
        db.startTransaction()  
        let current = db.selectAndLockForUpdate()  
        if (current == currentValue) {  
            db.updateValue(newValue)  
            saveChange(currentValue, newValue)  
            db.commitTransaction()  
        } else {  
            db.rollbackTransaction()  
        }  
    } catch {  
        db.rollbackTransaction()  
    }  
}

We already know that it’s possible for us to save a change that doesn’t actually get committed: our program could fail after saveChange and before or during commitTransaction. But the third process that regularly (or constantly) checks each change to make sure that the database reflects the potential truth means we can discard changes that didn’t actually get written.

It’s actually a little more complicated than that. If the external change processor is slow, it’s possible changes pile up. And with our current schema of just storing a value we could accept two writes to the value before the first change is processed. At that point we have no way of knowing if the first change really was persisted–we only know that it’s not the current value in the database.

We could fix this by keeping a history somewhere, by adding some sort of clock or UUID to each change, or by just accepting that overwritten changes don’t need to be sent out as notifications. Append-only storage might be a very good option for this. Maybe we don’t care if phones get all the in-between changes, we only care that they get the final good one. If that’s the case then when things calm down and changes are processed, then everyone will find out about it.

This is a much more complicated system. It involves even more components, but it also spreads the work out which might be important if our primary database is not having the best time.

We are planning for some of the writes to fail and have processes in place to take that into account. There are tradeoffs, for sure, but it _will work–_for some definition of work.

Just Make Everything Async

And finally, as a third (and possibly way overboard) idea: just change the entire application and it’s API.

Maybe what you want is a way to submit changes, perform them one by one, and only move on to the next one when you know the first has succeeded.

Our API could become a simple ordered list of operations, each wanting to change the value. We could have a function like:

function performOperation(q, db, newValue) {  
    try {  
        let current = db.getValue()  
        db.setValue(newValue)  
        publishChange(current, newValue)  
        q.markCurrentOperationComplete()  
    } catch {  
        db.rollbackTransaction()  
        q.retryCurrentOperation()  
    }  
}

This is a very simple system which needs a rule: in the event of any failure in an operation, retry the whole thing. This means it’s possible that if the publishChange function were constantly erring we would write the same value into the db over and over, but maybe that’s fine. The other changes would queue up and only progress after we’ve gotten through the current one.

This makes our function atomic and idempotent. Sure, the database will allow reads to the new value for some amount of time before the change notification is sent, but it’s not an incorrect value. We haven’t sent the change notification yet, but we will eventually. There was already some indeterminate amount of time between when we saved and when we sent the notification anyway. And we won’t have any other values until it’s sent.

We are planning for publishChange to fail and we are OK with the tradeoffs.

Conclusion

I’ve detailed lots of issues with the types of code I see in the wild, I’ve listed some options to make things reliable, and I’ve probably made some mistakes along the way. Please comment or send me a note with any feedback or fixes.

Also, don’t build a distributed system if you don’t think your code or network or computer or something will fail. It will fail. Instead, just plan for it. It’s possible to consider lots of angles and just put things in place to either fix the issue or to just document it. Sometimes failure is fine, but don’t be surprised by it if possible.

Whether you’re building an iOS app or deploying some code to heroku, you are working with a distributed system. Two components talking over a network are not near each other and they will eventually act like that is the case. Just be aware. Don’t let anyone tell you something is impossible. Maybe it is, but that doesn’t really matter. What matters is documenting what happens in response to failures.

Winter and small coffee

Wed, 04 Nov 2015 10:40:00 +0000

It’s starting to get cold again. I’ve been through two winters here now and I’m already bracing for the cold that’s on the way. Where I’m from it gets colder and snows more, but it feels colder here. The sun seems to disappear for a month and that makes everything feel frigid.

It can be nice to have shorter days. I seem to focus more and really be intentional about getting out when the sun is out, doing errands during the daylight, and working more into the evenings since they are now the longest part of my day. Some of my colleagues take vitamin D supplements during the winter, while others really enjoy the darkness. I believe I am able to be much more productive in the winter here, since I am less inclined to want to go outside or do anything far from home. Maybe startups formed in Berlin will have natural cycles of crunch time in the winter, relaxing vacations in the summer.

I also drink more coffee in the winter, but that is fine since the coffees here are so small. A large coffee here is normally less than a small where I’m from. And it’s great. Being able to have a shot of caffeine in the middle of the day without filling up with ounces and ounces of water is great.

When traveling in Italy and France the coffees were even smaller. Italian coffee places didn’t offer espresso to go, instead you just stand and take it like a shot glass then go about your business. I really prefer this style of coffee intake: smaller doses, better coffee.

The coffee really is better. Probably having less at a time makes it somewhat seem better, but I generally feel that the quality of coffee here is much higher than where I am from. I’m not saying it’s better than in SF, there are just too many good places in SF to choose from. But a random café in the towns and cities we’ve visited in Europe is ten times better than a random café in Virginia or the places we stopped at while driving through Wyoming. It’s possible the baseline for acceptable coffee is just higher. Everyone seems to have a decent machine and everyone seems to buy decent roasted beans.

When I have revisited where I’m from I’ve been amazed by how much warmer it feels even though the temperatures are generally colder. I’m also amazed by the large coffee-like drinks that seem to be everywhere. It seems in fashion to have a large caffeinated milkshake all the time.

I’ve learned to really appreciate coffee here. Having less at a time makes something more special. And I’ve also learned to appreciate the sun, for exactly the same reason.

Two years in Berlin

Tue, 13 Oct 2015 10:06:09 +0000

Last weekend was Unification Weekend which means we’ve officially been in Deutschland for two years. We arrived two years ago to a holiday weekend we were not prepared for. Everything was closed, we couldn’t get food for us or the dog, and we definitely couldn’t get any cables or converters for any electronics.

Holidays are something I love about Berlin and Europe: people take them. Over the last two years, it appears more and more businesses are attempting to stay open, especially cafés. However, it seems one will never be able to do much else but drink coffee or beer on a holiday. I’m fine with this.

Except for Christmas. There are many Weihnachtsmärkte open. And all the national museums are open. I’ve heard it’s normal for families to celebrate together at home Christmas Eve and then spend Christmas day out and about. We used to do the same at my maternal Grandmother’s. My Great Grandmother’s last name was Amburgy, but that is the American version possibly of Amburger.

For the last two years we have gotten each other annual passes to the national museums on 25 Dec. They cost as much as visiting 5 or so museums a year, plus we get to go to one on Christmas. Flying back to the USA during the “holiday season” is way too expensive, so it’s better to wait, remain in Berlin, and visit some museums.

Morning

Mon, 12 Oct 2015 08:59:52 +0000

Early, walking the leave-lined stone sidewalks with der kleine hund, it’s impossible not to enjoy the crisp 0º morning. It’s much colder in the shadow of a building, then blinding from the low winter sun between two. Yes, it’s winter. When it’s not summer it’s winter and that is fine by me.

How we allow any request to be safely repeated at anytime for Wunderlist

Tue, 24 Mar 2015 12:17:09 +0000

Via https://www.flickr.com/photos/thomashawk/2681744739

Stripe recently introduced an Idempotent requests feature for their api calls to protect against duplicate charges caused by network failures. While building Wunderlist 3 we needed a similar mechanism for our remote clients to be able to be safely repeat any requests. Mobile networks, corporate firewalls, and general internet connectivity issues are problems way too often. One cannot be sure an operation has successfully arrived without receiving a full, parseable response.

We took a different route to protecting our data and I’d like to detail it for future reference. It could be useful for other’s to compare and contrast Stripe’s and Wunderlist’s methods for idempotent requests.

With stripe, one includes an Idempotency-Key header with every request. If a request comes in within the next 24 hours with the same header value then Stripe will return the original response. One could build something like this utilizing redis or memcached to hold all raw responses keyed by the idempotency header’s value with a TTL. 24 hours makes sense for stripe since their clients are mostly background workers on servers. Surely even 1 hour would probably be enough, since most server workers are always working and would retry very quickly.

For Wunderlist, we have clients in remote places that sometimes are offline for days. A user might use their iPhone constantly for weeks, but only open their iPad once every week. Or a person might be on a web client at work, but a phone at home. Instead of using an in-memory cache of responses, we decided to let our databases do most of the work for us. I always favor the dumbest solution that works and so that is what we did (:

All GET requests (which are always reads) to the Wunderlist apis are safe already, so that part is easy.

For write operations every Wunderlist client maintains an an operation queue that is serializable to disk. Every write request is first added to the operation queue. Then, on the other side, an operation worker marks items as in-progress, finished, etc as it processes the requests. Right before a client is quit: it writes the current state to disk. This state is restored into memory on app boot. A request could be cut short by network issues, memory pressure, or just the use quitting the app. The request could have succeeded or it could have never made it to the server, it’s impossible for the client to know.

POSTs (which are always creates) are the hardest since the client has zero knowledge about the server’s state. PUTs (which are updates) are easier, since the client already knowns the server’s identifier for the object being updated. And we decided DELETEs always win and always succeed.

PUTs are easy because our sync algorithm already handles them. Every object in the Wunderlist system has an identifier and a revision. Every update to an object safely increments it’s revision counter by at least 1. Anytime a client updates an object it must send what it knows to be the current revision and identifier along with the properties to update. If the client’s assumed current revision is incorrect, the server immediately rejects the request and asks the client to fetch and merge the newest information. I am not ashamed to say this is identical to how Couch DB works, which is where we got the idea. If a client were to replay a successful update, it would fail because the revision would have changed and the client would be out of date. The client would fetch the new information, merge that in, notice nothing is out of date, and reconcile the operation in the queue to be completed.

For all POST requests, the client generates a header we named X-Client-Request-ID which is basically the same as stripe’s Idempotent-Key header. This request id is generated as part of the request operation that is added to the in-memory queue, this way it is serialized to and from disk as needed. This means a client can attempt to replay a POST request even if the app was offline for a long time.

On the server side, we let the database do the hard work for us. Every table for all synced objects has the aforementioned id and revision columns along with a created_by_request_id column. The request id column has a unique constraint, so there can never be two. To prevent clients from stomping on top of each other’s request id’s, other metadata like the current user’s id and the current device’s identifier are prepended to the already random request id. Since the operation queue is not synced between devices that is enough to keep this super unique.

Our apis return the created_by_request_id field in the object’s json so that clients can reconcile any operations currently stuck in their operation queue without having to retry the request.

All Wunderlist clients only perform GETs during a sync and only perform any writes when flushing the operation queue. If a client is currently in read mode then it will scan through the operation queue for each newly downloaded object’s request id, attempting to reconcile an in-memory request if one is found. In write mode: the client will simply send every request from the operation queue.

Since deletes always win they can safely be replayed over and over. The object won’t get any more deleted.

Although there are multiple concepts involved, we have found this system for idempotency to be very easy to understand and very reliable. What really pays is to consider the system in it’s entirety (from client to server) to come up with the best possible solution. Stripe’s is very elegant and could work for a certain type of app. Our solution is tailored for our long term, long range duplicate request issues. I hope this is helpful in some way if you are thinking about these things for your app.

Originally published at myobie.svbtle.com. Yes – I’ve had many blogs in the past…

My core values

Wed, 18 Mar 2015 10:39:52 +0000

Other’s needs and wants are more important than your own
Surprise and delight those you connect with
Thank people regularly for what they do
Take risks from time to time
Ambitiously pursue solutions for hard problems
Positively encourage other’s ideas and aspirations
Be content, but not complacent with how things are currently
Prioritize calmly, work aggressively
Gladly do the work other’s don’t want
Focus on details while considering a work holistically

Make workflows for complicated tasks

Mon, 26 Jan 2015 09:20:52 +0000

I’ve been extracting a lot of controller code into simple POROs recently, but it’s become more and more difficult and repetitive to get things to work consistently. I end up doing a lot of if statements in the #call method to manage failure states. An example might be:

# in a controller
def create
  @amount = params[:amount].to_i
  unless @amount > 100
    render :error and return
  end

  begin
    @charge = Stripe::Charge.create({
        amount: @amount,
        card: params[:card_token]
      })
  rescue Stripe::CardError => e
    render :error and return
  end

  @charge_response = StripeChargeResponse.new(body: @charge.to_hash)
  @payment = Payment.new({
    stripe_charge_response: charge_response,
    stripe_charge_id: charge.id,
    amount: charge.amount,
    currency: charge.currency
  })

  begin
    @payment.save!
  rescue ActiveRecord::RecordInvalid => e
    render :error and return
  end

  ReceiptMailer.payment_receipt(@payment).deliver_later

  redirect_to receipt_path(@payment)
end

After some work, I decided I should extract each step of the operation into it’s own method:

# in a controller
def create
  if grab_amount &&
     charge &&
     save_payment &&
     send_email
    redirect_to receipt_path(@payment)
  else
    render :error
  end
end

def grab_amount
  @amount = params[:amount].to_i
  @amount > 100
end

def charge
  @charge = Stripe::Charge.create({
    amount: @amount,
    card: params[:card_token]
  })
  true
rescue Stripe::CardError => e
end

def save_payment
  @charge_response = StripeChargeResponse.new(body: @charge.to_hash)
  @payment = Payment.new({
    stripe_charge_response: charge_response,
    stripe_charge_id: charge.id,
    amount: charge.amount,
    currency: charge.currency
  })

  @payment.save!
rescue ActiveRecord::RecordInvalid => e
end

def send_email
  ReceiptMailer.payment_receipt(@payment).deliver_later
  true
end

After that, I realized if I had a method to wrap and capture failures I could cleanup things even more:

# in a controller
def create
  if grab_amount &&
     charge &&
     save_payment &&
     send_email
    redirect_to receipt_path(@payment)
  else
    render :error
  end
end

def define_and_capture
  yield
rescue StandardError => e
  false
end

define_and_capture :grab_amount do
  @amount = params[:amount].to_i
  @amount > 100
end

define_and_capture :charge do
  @charge = Stripe::Charge.create({
    amount: @amount,
    card: params[:card_token]
  })
  true
end

define_and_capture :save_payment do
  @charge_response = StripeChargeResponse.new(body: @charge.to_hash)
  @payment = Payment.new({
    stripe_charge_response: charge_response,
    stripe_charge_id: charge.id,
    amount: charge.amount,
    currency: charge.currency
  })

  @payment.save!
end

define_and_capture :send_email do
  ReceiptMailer.payment_receipt(@payment).deliver_later
  true
end

And this is way too much going on in the controller, IMHO. So making a service object for this is pretty simple:

# in a controller
def create
  @service = ChargeACard.new(params[:amount], params[:card_token])
  if @service.call
    redirect_to receipt_path(@service.payment)
  else
    render :error
  end
end

# in it's own file
class ChargeACard
  attr_reader :payment

  def initialize(amount, card_token)
    @amount = amount
    @card_token = card_token
  end

  def call
    if grab_amount &&
       charge &&
       save_payment &&
       send_email
      true
    else
      false
    end
  end

  def define_and_capture
    yield
  rescue StandardError => e
    false
  end

  define_and_capture :grab_amount do
    @amount = params[:amount].to_i
    @amount > 100
  end

  define_and_capture :charge do
    @charge = Stripe::Charge.create({
      amount: @amount,
      card: params[:card_token]
    })
    true
  end

  define_and_capture :save_payment do
    @charge_response = StripeChargeResponse.new(body: @charge.to_hash)
    @payment = Payment.new({
      stripe_charge_response: charge_response,
      stripe_charge_id: charge.id,
      amount: charge.amount,
      currency: charge.currency
    })

    @payment.save!
  end

  define_and_capture :send_email do
    ReceiptMailer.payment_receipt(@payment).deliver_later
    true
  end
end

I iterated on this more and then decided I should just package up the repeatable bits into a module which I am now publishing as a gem: Workout.

Workout can help declare the steps needed to work through something. If any step fails then execution halts. A workflow instance knows if it’s completed, valid, or successful. This means a lot of controller actions can return to the simple and amazing if success then render success, else render error.

Most service object type libraries I see online accept their arguments into the call method, but I don’t like this approach. I’ve made the mistake of setting instance vars in methods and those might get carried over. To me, a better approach is to always Thing.new(args).call each time instead.

I hope someone might also find this type of object useful.

Object Oriented System Architecture

Fri, 23 Jan 2015 15:41:30 +0000

Building large systems to process web requests, work jobs, and do other things can be daunting without a plan of attack or a system-of-thought. How many components to build, how to separate responsibilities, and when to build small or big are questions that come up over and over again. Having a framework to answer questions is a huge help and keeps things consistent, especially when working on a team.

The best way I’ve found to describe my system-of-thought for building large systems is object oriented system architecture. This means to loosely apply object oriented software design principles to the macro-level of a system’s architecture. Basically: micro-services. The term micro-service is pretty vague now-a-days, so I feel it’s important to be more specific.

Over the next few months I plan to take the time to describe different principles, scenarios, and ideas about how to build large systems. The basics are:

An object is a service
Build small, well-defined objects
Objects collaborate by passing messages
Model different problem domains for the system as discrete objects
Expose well-defined attributes and methods
Document and distinguish query and command methods
Objects own their data, which includes their data-store

A simple example would be a to-do list (I know, I know). There would need to be a lists service and a tasks service at a minimum. The lists service cannot talk to the tasks data-store, instead it must query the tasks service by passing a message to a query method. Things can be broken down even further by separating task reads, task writes, and tasks storage. It is not unthinkable to have a TasksGet, TasksSave, and TasksRepository object in a class based programming language, therefore it’s not unthinkable to have a tasks-get, tasks-save, and tasks-repository service as part of the system architecture.

If I built a service like this then the tasks service would expose an HTTP JSON api for all queries and commands. I would also have an asynchronous queue to facilitate cleanup work, cascade deletes, etc. The tasks service might message itself to cascade a change from one set of tasks to another by enqueuing a message onto a queue for a worker to later pickup.

By thinking on the macro-level about objects, one is free to think about the micro-level more simply. Most small, well-defined services don’t need large software object hierarchies. Each query and command could be defined as a single function, possibly in a very functional language, as long as it’s exposed in a well defined way to the rest of the system.

Inheritance is even possible by delegating calls from one service to another. A completed-tasks-get service might proxy every message to the tasks-get service, then filter out all non-completed tasks before returning results. Deciding if completed tasks are another service or a method on an existing service is similar to deciding if one should add a method to a class or object or to write a simple delegate object wrapper.

Code is supposed to express the intent of the program and this is more easily achieved if the service has one clearly defined purpose. A service might need sophisticated permissions rules so one might choose a language better at expressing lists of rules or logic programming. Some services might just read and return data with minimal transformation and be represented as a simple function conceptually as well as in code.

Always use a CDN

Sat, 17 Jan 2015 16:11:09 +0000

Doing maintenance on a 6 year old project today that “didn’t have the budget” for a CDN reminded me how important one can be. I’m writing this to remind myself to stick to my guns and always make time for the important things: one of which is using Cloudfront or Cloudflare every time.

Not every site needs to “scale” (whatever that means), but it’s a complete waste of resources to keep answering requests for the same files over and over again. If a project is hosted on heroku, then there is no web server in front of the application to intercept requests for files. The application has to answer and handle the same requests over and over again. Even if an app is behind a reverse proxy like apache or nginx, the proxy is still answering and streaming files when it doesn’t need to.

Don’t under estimate just how easy and useful it is to save money by reducing the requests per second hitting the application boxes. While your site is keeping up fine now, when a spike happens it can deal with more traffic if it only has to serve the dynamic bits. If a page has 4 assets, then a 100% increase is page views could mean a 400% increase in requests per second. A page with lots of logos or icons or whatever might have 100s of assets and every page view is multiplied that much. The simple math is: don’t serve assets yourself.

If you’re willing to repoint the primary domain’s DNS then using Cloudflare is super simple. Sign up, then change DNS settings. Sadly, Cloudflare does not provide a hostname to use, so letting them proxy the entire domain is the only way.

My favorite CDN is Amazon’s Cloudfront and it’s amazingly simple to setup. All CDN’s work the same: the origin server is the application’s primary domain. Amazon provides a unique hostname to use, just append the original file’s path. When Amazon doesn’t have a file in it’s cache it will ask the origin domain for a fresh copy at that path. All one has to do is prepend the hostname like: https://abcdefg123.cloudfront.net/images/cats.gif. Another great thing is Amazon provides SSL for all Cloudfront subdomains, so no mixed content warnings or anything like that.

I just recently set this up for a rails project hosted on heroku by simply making this change to config/production.rb:

config.action_controller.asset_host = 
  "https://abcdefg123.cloudfront.net"

To make things easier for myself I just set it to always use https even if the page including the asset is http. It’s possible to use URLs like //abcdefg123.cloudfront.net/images/cats.gif without the protocol which tells the browser “use the same protocol as the base document” kinda sorta. Try it out if you haven’t.

For any other type of application or framework it’s pretty simple to do something like this pseudo code:

module.exports = function asset_path(path) {
  if ($application.env === "production") {
    return $application.config.asset_host + path;
  } else {
    return path;
}

(Yeah, that’s javascript referencing a global named $application. Sorry.) The basic idea is to make sure every time the path is output for an asset that it is “rendered” so in production it can use a different host.

Since Cloudflare takes over the DNS they let any request for a file not currently cached to pass through. This means less configuration up front, but it can lead to confusion if it’s a person’s first time using a CDN. It’s possible to configure Cloudfront to take over the origin domain as well, but it’s not required with Amazon’s setup.

After setting up a CDN and making sure all assets have the correct hostname in the URLs, it’s important to set the correct headers so the CDN knows how long to keep the files. Heroku has a great overview of HTTP caching and you should read it. The simplest thing to do is to add a Cache-Control header for every asset requested by the CDN backend:

Cache-Control:public, max-age=31536000

This is precisely what rails does when it serves an asset in production, so I got pretty lucky there. All good web frameworks will do the same thing. If you’re project is framework-less or using some new-awesome-hipster-cool-thing-framework then it should be simple to add this header when an asset is requested: if it’s not then provide a patch or module or re-consider your choice.

Finally, check the server access logs to see the new found amazingness! Offloading traffic can get way more money out of the application boxes and might even allow one to scale down. Not enough writing online talks about scaling down instead of up.

Why make a Mash?

Fri, 19 Dec 2014 07:45:09 +0000

Hashie is fine

Recently, Richard Schneeman wrote a very good article titled Hashie Considered Harmful - An Ode to Hash and OpenStruct. Give it a read, there is some wisdom there. However, I have a bit of a different take on this issue. I’ve also had this as a draft in Svbtle for way too long.

First, let’s get this straight: if OpenStruct is useful then Hashie::Mash is useful too. And OpenStruct is really useful. Also, don’t let anyone tell you “you don’t need a hash-like object that responds to methods” because you very well might need it. Always contrast your goals and the implementation of a library to make sure it’s as simple as it could be.

Second, don’t take advice about what to use from people who can’t explain the pain or joy around it. It’s like someone who says to use postgres instead of mysql, but has no clear reason to prefer anything. What is the real pain here? What is the real benefit? What circumstances were there?

To be clear, Richard explains that misspellings, insensitive access to hash-like object keys, and increased memory usage can cause issues, and he is correct. However, from certain perspectives, the tools shouldn’t try to help with misspellings at all: Javascript objects don’t raise on missing keys. Memory usage is relative to each application and my libs are not generally my problem, so we differ here too.

I will try to detail why I always tell people to use what they want, but they probably don’t need Hashie anyway.

Why not to use `Hashie`

There is one very good reason not to use Hashie::Mash at all that I don’t see explained very often: #zip.

$ gem install hashie
$ irb -rhashie

address = Hashie::Mash.new(street: "100 Street St", city: "city", zip: 10119)
address.zip # => [[["street", "100 Street St"]], [["city", "city"]], [["zip", 10119]]]

It’s honestly that simple. Mash inherits from Hash which includes Enumerable and you have a huge number of keys (175) that have surprising behavior. However, this does not mean Hashie is bad or not useful, it’s just how it works and one needs to know that.

What to use instead

Maybe `OpenStruct` is “better”

No, OpenStruct is slow. Let’s see some data comparing it to Hashie:

$ gem install benchmark-ips hashie
$ irb -rbenchmark/ips -rhashie -rostruct

Benchmark.ips do |x|
  x.report("ostruct") { OpenStruct.new(street: "100 Street St").street }
  x.report("hashie") { Hashie::Mash.new(street: "100 Street St").street }
end

Calculating -------------------------------------
             ostruct    12.509k i/100ms
              hashie    23.823k i/100ms
-------------------------------------------------
             ostruct    135.329k (± 7.8%) i/s -    675.486k
              hashie    313.649k (± 5.9%) i/s -      1.572M

Hashie is at least twice as fast for the simple case of building a hash-like object and calling a method on it. This is what my normal usage of these tools looks like, grab some data and call methods on the resulting objects.

Oh, why would one use “OpenStruct” then?

OpenStruct compiles the method into the instance so repeated calls will be fast. Here is what that looks like:

$ gem install benchmark-ips hashie
$ $ irb -rbenchmark/ips -rhashie -rostruct

Benchmark.ips do |x|
  x.report("ostruct") {
    o = OpenStruct.new(street: "100 Street St")
    100.times { o.street }
  }
  x.report("hashie") {
    m = Hashie::Mash.new(street: "100 Street St")
    100.times { m.street }
  }
end

Calculating -------------------------------------
             ostruct     4.563k i/100ms
              hashie     1.363k i/100ms
-------------------------------------------------
             ostruct     46.592k (± 3.7%) i/s -    232.713k
              hashie     13.598k (± 4.1%) i/s -     68.150k

OpenStruct is over three times faster for repeated calls to keys. So for long lived objects, OpenStruct is way better than Hashie. However, there is something even better for long lived objects: Struct. If your objects are really that long lived you will probably know their schema and you can just make a class (Struct is a class factory, so use it) that conforms to that schema.

What does this mean?

What it always means: the tools one chooses to use should be tailored to the use case.

I build a lot of apis and those apis all produce and consume JSON which in ruby is best represented as Hash’s or Array’s of Hash’s. However, one of these lines of code is prettier:

task_ids = tasks.map(&:id)
task_ids = tasks.map { |task| task[“id”] }

There are other examples too where using methods is much preferred from a stylistic point of view. My apis change a lot at first, so dynamically providing the Hash#keys as methods allows me to move quicker. It’s possible that eventually I would define a Struct for each version of each api later, which is an easy refactor since the tests all still pass because nothing really changes.

If we shouldn’t use Hashie and OpenStruct is slow, what do we do?

I made my own Mash

Yeah, I know, NIH and all that. But, as I typed above, evaluate tools on what they are being or will be used for. For my api producing/consuming applications I need hash wrappers that are fast and use very little memory. These wrapped objects are not long lived.

My library is called Mashed. It does three things: provides indifferent access in a predictable way, provides a hash wrapper that has a very small method footprint, and represents the internal hash’s keys as methods.

Indifferent access

Symbols in ruby are kinda annoying. Until 2.2 (due out very soon I guess) they are not garbage collected, so technically you could allow anyone to DDOS your app by making every JSON object into a symbolized hash. (I’ve considered just going down this road and making sure I monitor my app correctly, but I’ve never actually done it.) Luckily this will go away when they are garbage collected and I will even change my implementation when that happens.

But for now, I created the StringyHash. It does not inherit from Hash, but instead wraps and delegates to a hash instance. The method footprint is small and it doesn’t extend any built-in ruby classes at all.

The example from the README should explain it all:

StringyHash = Mash::StringyHash

sh = StringyHash.new(title: "Hello", starred: false, completed_at: nil)
sh.keys # => ["title", "starred", "completed_at"]
sh[:title] # => "Hello"
sh["title"] # => "Hello"

class Title
  def to_s
    "title"
  end
end

sh[Title.new] # => "Hello"

The goal is to be a very sensible delegator to the internal hash instance. I’ve had zero issues so far with this in many production systems. For 2.2 I will make a SymbolizedHash class I guess.

Wrapper with very small method footprint

In ruby, every object has a lot of built-in methods.

Object.new.methods.count # => 55

Every ruby object has at least 55 methods. If the goal is to provide almost any key that might be set in a Hash as a method, that is 55 keys that are impossible to get to. Luckily, ruby allows one to start from a smaller point with BasicObject.

BasicObject.new.methods.count # =>
# NoMethodError: undefined method `methods' for #<BasicObject:0x007fd454b8fdc8>

That’s right, it doesn’t even know what it’s list of methods are. My Mash inherits from BasicObject and provides a very small amount of built-in methods.

$ gem install mashed
$ irb -rmashed

Mashed::Mash.new({}).methods.count # => 26

I’m always trying to get that number lower as well. Please, if you ever have ideas for how to do that then make a PR or Issue.

Delegate methods to key/value lookups

Now, how does Mash fare in the #zip example:

$ gem install mashed
$ irb -rmashed

address = Mashed::Mash.new(street: "100 Street St", city: "city", zip: 10119)
address.zip # => 10119

It works in an unsurprising manner. The “secret” to Mash being a good citizen is for it not to be hash-like at all.

Examples:

address["zip"] # =>
# NoMethodError: private method `[]' called for #<Mashed::Mash:0x007fb501049cd8>

address.merge(state: "VA") # =>
# NoMethodError: undefined method `merge' for #<Mashed::Mash:0x007fb501049cd8>

address.map { |k,v| puts "#{k}: #{v}" } # =>
# NoMethodError: undefined method `map' for #<Mashed::Mash:0x007fb501049cd8>

address.inspect # => "#<Mashed::Mash @hash=>{\"street\"=>\"100 Street St\", \"city\"=>\"city\", \"zip\"=>10119}>"

It just refused to appear to be a Hash.

There are still problems, one of which is an issue right now: single method calls with zero arguments return nil if the key is missing. This is inevitable based on the current design constraints: Mash acts like a Javascript Object where missing keys are undefined.

I find this unsurprising since accessing a missing key on a Hash returns nil. However, I am considering making a monad or something to possibly make it easier to understand.

Why is `Hashie` not like `Mashed`?

Because it’s a different tool. Hashie is actually a great library and everyone should not only try to use it at least once, but read through it’s code. You can learn a ton by seeing how other’s have solved similar problems.

OpenStruct is awesome too. If you’re making a ruby script and you want to have no dependencies outside the standard library then use it; this happens to me when I’m working on build or deployment scripts.

Use what works for you’re current situation

Write tests, evaluate libraries based on their implementation and api, and don’t listen to anyone including me (:

Notes on Contentment

Sun, 10 Aug 2014 20:39:48 +0000

Recently, I’ve been concerned about how non-contented I am and how selfishly I act. I’ve been angry and entitled and it’s not something I enjoy. People fall out of love and similarly they fall into discontent. The choice is to work toward contentedness or toward anger, greed, or any number of alternate paths. Being content does not mean one is satisfied, but instead that one is not unhappy in the presence of unsatisfying circumstances.

“I’m not “happy” but I’m not unhappy about it.”
― Alan Bennett, The History Boys

A content artist is not one who has created all they ever will, but one that is creating as much as necessary. Being content does not mean a person is complacent or indifferent. Contentedness will breed progress or support to perpetuate itself and prevent complacency. Mediocrity will never grow contentedness.

Similarly, contentedness is not earned. Only through the removal of dissatisfaction and other poisonous feelings can one be content; not from adding extra income, extra time, etc. Unreachable goals are poison to a person’s contentment. Yet, even the goal of being content is unreachable without true internal change — so the goal itself can be just as preventative as wishing for infinite wealth or other unattainable (and unnecessary) things. I’ve been stressing about this mentally for a while now.

“He who is not contented with what he has, would not be contented with what he would like to have.”
― Socrates

Talking to a friend, he reminded me that gratitude is the secret to true contentment. Just as contentment must be practiced or lost, it’s clear that gratitude also must be practiced or forgotten. So I intend to practice: I will write once a day about what I am grateful for. In searching for contentment I’ve found gratitude instead. It’s probable one will cause the other.

I’m not looking for happiness, that will come and go on its own. Anyway, there is a direct correlation to a person’s extraversion to their happiness: the more extraverted a person behaves the happier they tend to be. Neuroticism correlates as well, except in reverse. I am not extraverted.

I’m also not really looking for ways to rationalize or “know more” or become enlightened. Unlike enlightenment, there is no level of ascension that provides contentment directly. Enlightenment is attained whereas contentment is worked through.

Every night I will compile a list of the things I am thankful for with no other goal than to relearn gratitude. You’re welcome to try too.

Document semi-wacky — wacky things

Thu, 05 Dec 2013 09:42:53 +0000

This is an exposition of point 4 from Be a good programmer.

So when do you document anything?

When it’s wacky or super difficult.

Thing’s aren’t always simple. Payment processors, email systems, history tracking, and other things can get really complicated fast and sometimes you just need document how it works. Tumblr’s api uses some complicated signing (HMAC-SHA1) for requesting an oauth token and it turns out it’s not exactly the simplest thing ever and even wackier if you don’t want to use any external libraries. That really needs to be documented. It’s wacky.

When integrating with payment systems like paypal or authorize.net or braintree you just gotta do things a certain way and it would help the next person if you documented the order or operations, the libraries used, and the “why?” Asynchronous operations are also notoriously hard to follow, so it’s usually a good idea to put some comments in about when and where things might happen. Job queues, notifications, or feedback loops are all candidates for documenting more than normal.

When should you not document things?

Pretty much the rest of the time.

Code should speak for itself. Of course. If a programmer’s job is done well, then anyone can get up to speed on what’s going on quickly without having to read too many docs.

Docs that are useless include things like @public or @private or listing out the types for the args for untyped languages (foo(String:bar)). Code examples showing the use of the methods are usually not a good idea because they get out of sync. If the method is being used in the project, then it’s example usage is already documented.

A method has three chances to document itself:

Using an appropriate name
Not being too long
Being consistently used throughout the application

Use an appropriate name

If a method sends an email, name it as such. But don’t hide indirect side effects. Recently at work we noticed a method named find_by_identifier that actually will either find something, create something, and also possibly send an email. It didn’t start out that way, but as the product evolved it gained new responsibilities.

That does not mean the method should have been refactored — refactoring is to change the method’s instructions without changing behavior. Instead, it needs to be re-specified and re-implemented. It’s a new thing and deserves that kind of consideration.

Don’t be longer than a few lines

The more things a method does, the harder it is to build a mental model of the product or feature one is working on. Having small methods makes refactoring easier and causes less test breakage in the long run. Specify, implement, and refactor until each method is understandable and small.

Don’t worry, you’re not writing a timer for the nuclear missiles. Or, if you are, you shouldn’t listen to me. Seriously.

Be consistent

If you create a find_by_email method and usually use it like find_by_email(params[:email]) but then one time you pass in some strange object like find_by_email(Email.first) then I am now confused. I thought params[:email] was always a String, but now I don’t know. I gotta go digging to learn more to figure out what’s going on.

Do you really need to support both String and Email objects? Probably not. Do find_by_email(Email.first.address) or find_by_email(Email.first.to_s).

OK. OK. How do I document this stuff?

Be succinct

When you do need to document, it’s best to keep the documentation as close to the actual code as possible — so that means comments. I really like TomDoc for backend application code It’s quick and simple and helps keep everything consistent.

For javascript I really love docco. It’s beautiful and inline. I’m considering moving almost all of my docs to this style.

And Libraries

Take the extra step to outline regular usage examples, lists of classes and methods, and maybe answer some frequently asked questions. Since multiple random people might be using the code it needs to be really well explained.

One of my favorite projects has pretty decent documentation. And Rails’s guides are pretty great, but the regular api docs are pretty sad. Sure they list out the methods and everything, but some are missing or just have no documentation at all. This is something we can fix.

I need to know how to install, configure, and do basic tasks with your tool quickly. Put all of that in the README.

Anything else?

What did I miss? This is the end of the four part series. What did I leave out or what could I do to make it better?

Use technologies the rest of your team is motivated to use

Sun, 28 Jul 2013 15:56:02 +0000

via flickr.com/photos/kenfagerdotcom

This is an exposition of point 3 from Be a good programmer.

Most of my time is spent either reading or editing someone else’s work. Very seldom does one get to start from scratch when building something for the web – and even then we use frameworks and libraries we didn’t write. It’s very important that the code you choose to include in your projects is compatible with your colleague’s interests and motivations.

You may want to write a new app in go, but no one else does. Then don’t do it. Write a personal app in go or write the app in go, but don’t show them and rewrite it in whatever. It’s important to leave a legacy of making things easier. Someone in the future will curse your name during a git blame if you’re not careful. It might even be you.

Most of the developers I work with use ruby, rails, mysql or postgres, redis, memcached, git, etc all the time. Now, I’m pretty fortunate, because those are all awesome things to use. However, I learned this lesson more than once working at Livingsocial: don’t go against the grain unless it’s worth the friction. (It might be worth it, but make for certain.)

When I was planning out a new set of apis for their iOS app I had planned to use sinatra and redis. At the time, no team had needed a large redis install, so there wasn’t a primary+secondary pair readily available. The ops guy told me no. He said we hire rails people and we use mysql. I was pretty miffed at the time, I felt like I was being kept from doing my best work by being limited.

Sinatra and rails are both ruby, how different could they be? How hard could it be to install and maintain an HA redis setup?

It turns out the answers are: quite different and very hard.

When building a core service or library that will be used and modified by any number of people, it’s usually best to be a little boring about it. Sure, go ahead and prototype something in closure. But when it comes time to run something in production that serves millions of users, boring trumps clever almost every time. When you are away and no one else even knows how to run the tests in the app – well, that’s when they curse your name.

I’m not advocating that we never push things forward or try out new things. Instead, rally the troops and get everyone involved. If everyone on the team is excited to learn something new, then it won’t matter if you are there or not – anyone can jump in and take care of business. You will learn something new way faster if your colleagues are learning it too. They will discover things you didn’t, you can argue about best practices, and bounce things around so they end up the best they can be.

Refactor only when you have some sort of tests

Thu, 11 Jul 2013 14:22:50 +0000

via Odalaigh on flickr.com

This is an exposition of point 2 from Be a good programmer.

Most of my time as a programmer is spent reading and refactoring code. Very seldom am I just typing out lines and lines of new code - and even when I am, I usually end up refactoring that new code before it’s shipped out.

By definition, you cannot refactor code without some sort of tests. That’s what refactoring is, changing code without changing the end result of it’s work.

A simple example you probably think is stupid is CSS. Change some styles, refresh the page. The refresh is the test. You, the human, are verifying that you didn’t change the way things look. This even applies to developers refactoring scripts or application code, sometimes a human refreshing is how we know we didn’t change anything.

There are some key points to know:

If a test has always passed, if it never failed in the beginning, then you don’t know if it’s testing anything
If a test fails during a refactoring, it was too granular
Your tests only try out your assumptions of how the system works

Tests gotta fail

Writing tests first is good, because you know they will fail, then pass, then you are good. I don’t always do this, but it’s probably the best idea. TDD as they call it.

If you write some code, then write a passing test for it, you don’t know anything from the test. Nothing.

An example in ruby that I see often is using assert instead of assert_equal. Like this:

assert "foo", some_var 
assert_equal "foo", some_var

The first assert will always pass because "foo" is truthy. some_var is actually being given as the "message" argument, which is shown if the assert fails.

If you require a test to fail before it succeeds, this type of mistake wouldn’t happen.

Tests need to be stable

Changing one line of code shouldn’t break a ton of tests. If it does, your tests are probably too granular to mean much. Trying to assert every little thing about a system is dangerous, because it can lead to only testing small interactions, but never knowing if the entire system even works. I’ve seen rails apps with decent coverage and all tests passing that raise errors all the time.

Try to think about the ins and outs of the system, not the actual algorithms going on inside.

Your tests are flawed

Yeah, it kinda sucks, but your tests are not adequate. The problem is that you only test how you think the system works. It’s ok though, you can’t really help it.

Applications or websites or whatever act in ways that we would never imagine or predict. It’s pretty cool to see someone use a tool you made for things you never thought of. It also means you didn’t write a test for that scenario.

Honestly, you don’t need to test every scenario. But, if users get used to how something works, even if it’s working in a broken way, they will complain and possibly dissent when it changes.

The solution: have someone else look at it and/or use it.

Most problems we work on are human problems anyway, the computer parts of things is most times figured out. I mean unless your Facebook and you have a bajillion users so you write your own php. Then you are solving problems no one else has. Then it’s whatever. However, more than likely, you are not Facebook.

Change as little as possible to fix a problem

Mon, 03 Jun 2013 15:00:42 +0000

via flickr.com/photos/acme

This is an exposition of point 1 from Be a good programmer.

When a problem arises or a bug is found, it can sometimes be very difficult to determine the exact cause. We have to look through many files and read lots of code to find one line or paragraph of code that is causing the larger issue. It’s tempting to fix other issues as you read through files and classes or what have you. Indenting or reformatting code, renaming arguments, unwrapping map reduce’s into separate methods, and changing comments on unrelated code are fine, just separate from fixing the bug.

A good programmer knows that they should only commit what is needed to fix the bug. It creates a clean history of changes, where this change is easily explained in its entirety. Also, if the need arises to revert the change, one is not also reverting unrelated code. When unrelated code is reverted, sometimes it never makes it back in a separate commit. Fixing a problem can be frustrating and cause one to forget the other little things to go back and include later.

There is also the problem of introducing more bugs while fixing the original, which is far more likely if extra changes are included. Multitasking is hard. To be the most effective at curing an application’s ailments focus on finding and solving one problem at a time. If you change one thing a time, then it’s much less likely that other, seemingly unrelated, components will be effected.

If you make changes trying to fix an issue, but the end result is still broken, then revert. Do not keep unrelated changes until the problem is solved. Unless you have decided to not solve it anymore - which sometimes is a good answer. Not every problem needs to be fixed immediately. Some bugs require research or timing or other humans to be involved, so don’t go committing things until you have the resources necessary to come up with a full fix.

Experienced developers mentally give different bugs different priorities, but those can sometimes be incorrectly weighted. Maybe a problem effects an api that you never use or you can continue with your daily work, but someone else cannot. I’ve both caused problems for others as well as been blocked on something. The more power you have to fix something, the less likely you are to ever be blocked.

Any exception is a bad exception. Don’t let bugs keep happening, even if they seem inconsequential. They could mask other, real problems, they are taxing on the application since exceptions are usually slower than the normal code path (especially if you send emails or post to an error reporting service), and they are eventually demotivating. Let exceptions be exceptional.

If you find yourself demotivated to fix issues, then try to setup a reward system or a dedicated time. Have cleanup Friday’s or bug fix Monday Morning’s. Schedule time once a day to fix the most common bugs. You’ll be much more motivated to fix things once a few fixes have gone in, especially if you receive positive feedback from customers.

Be a good programmer

Wed, 29 May 2013 00:07:37 +0000

via flickr.com/photos/acme

Change as little as possible to fix a problem
Refactor only when you have some sort of tests
Use technologies the rest of your team is motivated to use
Document semi-crazy – crazy things

The Cost of Education

Wed, 08 May 2013 01:39:40 +0000

Photo from flickr.com/photos/deathtogutenberg

I’ve written this story many times, but never published it. Today, however, Joshua Davis tweeted my exact feelings:

without piracy, I never would have mastered the tools, that created a career, where I could then purchase said tools in the future.

via twitter.com/joshuadavis

This is in response to Adobe switching from a one-time fee to pay for their software to a monthly fee to rent their software. I’ve been paying monthly for Creative Cloud for over a year now, since I upgraded from CS4 to effectively CS6.

We didn’t have a computer when I was young. The computers at school never offered much more than Wordperfect let alone the opportunity to try to tell the computer what to do. I would read books from the library about batch files (they only had old dos books) and how to do things with computers, but I didn’t know what they meant and I didn’t have a way to try them out.

Then my older brother got a job and bought a computer. I was told not to use it, but I did anyway when he was away. He had downloaded lots of different applications and seldom used any of them, but I found one that I spent hours and hours using: Flash 3. You see, Flash 3 had the ability to create “scripts” by clicking a + icon and choosing a command or statement. It wasn’t altogether very different than writing a BASIC program, which I had learned from using a cheap vtech toy as a child.

Then Flash 4 came out and you could do ActionScript. I used to visit praystation, flight404, once upon a forest, and other sites every day to see what they were doing and what I could learn. I couldn’t afford books, but I would read every website I could find. I spent time on the alt.marcomedia.flash newsgroup asking and answering questions.

I never paid for Flash 4. I never paid for 5 either and by the time I had saved up the money to buy the softare legally MX 2004 had been released and I was in college.

Flash moved me on to Javascript (which it perverted later with ActionScript 3) and html and css and on and on. While Flash has a terribly steep learning curve, being able to program things on the canvas where you can see it all happen and share that instantly with people was exciting.

New software like Sketch ($24.99) and Acorn ($29.99) are making it much cheaper for beginning designers to get going, but when you have no money even these applications are out of reach.

Luckily working with html, css, and javascript is all text and editing text will always be possible with free software. I actually use a program called vim that I would never recommend to anyone because of how hard it is to use, but it is free.

With libraries like two.js and HYPE one can get started learning and creating interactive graphics for free. While there are free tools and libraries, the documentation and learning materials are still lacking for beginners. Having to know how to program just to get something to move is terrible. Flash gave back as much as you put into it, which is why so many designers twisted it’s sad, crapptastic features into beautiful things.

The current, best backend development tools and frameworks are free: ruby, python, rails, django, postgres, redis, memcached, etc. To become a web developer is a much different path: you only need a computer where you can run a free operating system which comes with free software with which you can program to do stuff for free. All you have to pay for is hosting - and with heroku and others there are free options for that as well.

I’m not sure if design can or should be dominated by free tools, but I do know that if it weren’t for stolen tools I wouldn’t be where I am today. I was allowed time to master tools I couldn’t afford so I could then make the money to not only buy those tools, but support typeface and icon designers and indie software developers and so on…

I understand that hammers and nails costs money too, but I can borrow a hammer from a friend while I learn how to use it.

nathanherald.com feed

Talib Kweli & Hi Tek (Reflection Eternal) - The Blast

TIL about Solid and FOAF

TIL about Solid and FOAF

Solid

It's RDF underneath

FOAF

A real Solid use case

Cool idea and I'm not sure I'd ever use it

FOAF me

"The Taste of Tea (茶の味) Trailer"

John Thomson’s China

I’m tired of talking to AI

"Reel: Overview Walkthrough (Part 1)"

Open House

Practical example of how cool RDF is

Mastodon kinda hates AI

"1990: Whatever Happened to the Cameras of the Future? | Tomorrow’s World | BBC Archive"

"Ozu in Color – TCM x Every Frame a Painting | TCM Originals"

Principles matter more with agents

"Blackwater Holylight – Full Performance (Live on KEXP)"

"Casio SXC-1 portable sampler"

Making Chrome livable

Same beans, different cups

"The Film Everyone Passed On | The ROBOCOP Commentary"

The Affordable Car Is Dead

"Bill Withers – Ain’t No Sunshine (BBC In Concert, 1974)"

"THE BLADE (1995): The Edges of Wuxia"

How I use AI daily, April 2026

Too Old Not To

Artemis II Launches to the Moon

airdrop CLI

ひとひら - 円 (Official Music Video)

Underwater Volcano Eruption (Kavachi)

This ‘89 Pistons Cable Outro Goes Unnecessarily Hard

"TIL: at, atq, and atrun"

Introducing pty

Kanemaki Yoshitoshi’s Sixteen-Faced Woman

Casio S100X Japanese Lacquer Calculator

Christie’s Rostrum

Wave Interference

Florentine Floor

OlmoTrace

The Killer | Official Trailer

Jargalant Altai (Special Version)

Generative AI Vegetarianism

Gaining Agency

Gugusse et l'Automate

dandelions forever

csesh – monitoring and reconnecting Claude Code sessions

TIL Acorn can screenshot with layers

toe – Distance Sessions 2020

Hazel Dickens: They'll Never Keep Us Down and Which Side Are You On?

Leica T Leather Holster

CRDT File Sync

Claude Code Remote Control

The 4 things that make an "agent" today

The 4 things that make an "agent" today

An agent today is 4 things

A note on security

Coding agents on your own hardware are best

Where this is heading

What is your use case for an agent?

Two interesting AI developments

You Could Configure Your App with Markdown

Why markdown instead of json or a settings UI?

SwiftUI @State Considered Harmful

The Problem with @State for Collections

The Solution: Observable Collections of Observables with Identity Preservation

Questions You're Probably Asking

1. "Why a class instead of a struct?"

2. "Won't this cause memory leaks or retain cycles?"

3. "Why a dictionary instead of an array?"

4. "What about thread safety? You marked Episode as @unchecked Sendable."

5. "Why not just use @ObservationIgnored on frequently-changing properties?"

6. "How do views get fresh data after updates?"

7. "Does this help with testing?"

8. "What about derived/computed state like 'downloaded episodes'?"

9. "How do you handle navigation with reactive data?"

10. "What about high-frequency updates like playback progress?"

The Problem with `@State` for Collections

`Text`

What is `<template>`?