Ryan Spletzer

How to Disable OTP on Your YubiKey

Ryan Spletzer — Sat, 18 Apr 2026 00:00:00 +0000

Pieter Bruegel the Elder, The Tower of Babel, c. 1563. Kunsthistorisches Museum, Vienna. Public domain, via Wikimedia Commons

If you’ve spent any amount of time in a Slack workspace where people use YubiKeys, you’ve seen it: suddenly a message appears in a channel that looks something like this:

ccccccgkgdkvvvjvfblbbdihehndrvjkutduvftrgubfc

If you’re not familiar with YubiKeys, this looks like gibberish—maybe someone’s cat walked across the keyboard, or they fell asleep with their face on the keys. But if you are familiar with YubiKeys, you know exactly what happened: they bumped the sensor on their key, and the OTP (One-Time Password) slot fired a Yubico OTP string directly into whatever text field had focus at the time.

I have personally witnessed this more times than I can count. My response is always the same—I react to the message with the Yubico emoji.¹ Some people see my reaction and immediately know what happened. Others are genuinely confused, both by the mystery string they just broadcasted and by my cryptic emoji response. It’s a near-universal YubiKey user experience, and it’s entirely preventable.

If you’re here, you probably want to stop this from happening. Good news: it takes about 30 seconds.

Quick Reference
- Via YubiKey Manager (GUI)
- Via ykman (CLI)
Disabling OTP vs. Removing the Credential
What This Doesn’t Affect
Re-Enabling OTP
What OTP Actually Is (And Why You Probably Don’t Need It)
Conclusion

Quick Reference

If you just want the fix and don’t need the backstory, here you go. The GUI app approach is first because that’s what most people will reach for—if you’re not at home in a terminal, clicking a checkbox is the friendlier path. If you do live in a terminal, scroll past it to the ykman approach below.

Via YubiKey Manager (GUI)

Yubico provides a graphical application called YubiKey Manager. It runs on macOS, Windows, and Linux, and you can also install it through package managers like Homebrew and Chocolatey.

Open YubiKey Manager and insert your YubiKey
Click Interfaces in the left sidebar
Under USB, uncheck OTP
If your YubiKey supports NFC, under NFC, uncheck OTP as well
Click Save Interfaces

Done. The application will confirm the change, and your YubiKey will no longer emit OTP strings on touch.

Via ykman (CLI)

If you’d rather use the command line, first install the YubiKey Manager CLI if you haven’t already:

# macOS
brew install ykman

# Ubuntu / Debian
sudo apt install -y yubikey-manager

# Windows (Chocolatey)
choco install yubikey-manager -y

# Windows (winget)
winget install Yubico.YubiKeyManager

Then, with your YubiKey inserted, disable OTP on the USB interface:

$ ykman config usb --disable OTP
USB configuration changes:
  Disable Yubico OTP
  The YubiKey will reboot
Proceed? [y/N]: y
USB application configuration updated.

And if your YubiKey has NFC, disable OTP there too:

$ ykman config nfc --disable OTP
NFC configuration changes:
  Disable Yubico OTP
  The YubiKey will reboot
Proceed? [y/N]: y
NFC application configuration updated.

That’s it. No more accidental OTP strings. Read on for more context, and to understand what you just disabled.

Disabling OTP vs. Removing the Credential

There are actually two different things you can do here to remove OTP, and it’s worth understanding the distinction.

Disabling the OTP application is what the Quick Reference section above does. This turns off the entire OTP interface at the transport level—USB and/or NFC. The YubiKey no longer presents an OTP interface to the host at all. The credential that was in the slot still exists on the key, but it’s completely inaccessible. This is the recommended approach if you don’t use OTP for anything.

Removing the slot credential is a more surgical option. Your YubiKey has two OTP slots: Slot 1 (short touch, 1–2.5 seconds) and Slot 2 (long touch, 3–5 seconds). The accidental-OTP problem is almost always Slot 1 firing on a brief touch. You can remove just that credential while leaving the OTP application enabled:

$ ykman otp info
Slot 1: programmed
Slot 2: empty

$ ykman otp delete 1
Do you really want to delete the configuration of slot 1? [y/N]: y

After this, touching the sensor briefly does nothing—Slot 1 is empty, so there’s no credential to fire. But the OTP application itself is still active, which means you could program a new credential into either slot later if you wanted to.

You can check the current state of your slots at any time with ykman otp info.

When to use which: if you never use OTP for anything—and most people don’t—disable the entire application. If you want to keep one slot active (say, a static password in Slot 2) but stop the accidental Slot 1 misfires, just delete the Slot 1 credential.

What This Doesn’t Affect

This is the part where I reassure you that you’re not breaking your YubiKey.

Disabling OTP has zero effect on any of the other applications on your key. YubiKey applications are independent modules, and turning one off doesn’t touch the others:

FIDO2/WebAuthn — passkeys, hardware security keys for web authentication. This is what the industry has moved to, and it has nothing to do with OTP.
OpenPGP — GPG commit signing, encryption, and authentication. If you followed my GPG commit signing guide, none of that is affected.
PIV — smart card authentication, certificate-based auth. Unrelated to OTP.
OATH — TOTP and HOTP via the Yubico Authenticator app. This is a common source of confusion—the OATH application is a completely separate module from the OTP application, even though the names sound similar. Disabling OTP does not affect your TOTP codes in Yubico Authenticator.

Re-Enabling OTP

Disabling OTP is fully reversible.

If you ever need OTP back—and I’m struggling to imagine why you would, but who am I to judge—the process is straightforward.

Via the GUI app: same steps as before in the Quick Reference, but check the OTP box instead of unchecking it.

Via ykman:

# Re-enable OTP over USB
ykman config usb --enable OTP

# Re-enable OTP over NFC
ykman config nfc --enable OTP

If you went the slot-credential route and deleted the Yubico OTP credential from Slot 1, you can reprogram it:

$ ykman otp yubiotp 1 --serial-public-id --generate-private-id --generate-key
Using YubiKey serial as public ID: vvcccbblbgjr
Using a randomly generated private ID: 7d795b01b50c
Using a randomly generated secret key: f857d5634a0cf143b7bf88dd0ee6af92
Program a YubiOTP credential in slot 1? [y/N]: y

This generates a new Yubico OTP credential and programs it into Slot 1. Note that this is a new credential—the old one is gone—so you would need to re-register it with any services that relied on the previous Yubico OTP credential.²

What OTP Actually Is (And Why You Probably Don’t Need It)

If you’re still reading, you might be curious about what exactly you just turned off, and why it existed in the first place. The OTP application on a YubiKey supports several different credential types, and the terminology can be confusing because “OTP” gets used to mean different things in different contexts.

Yubico OTP

Every YubiKey ships with a Yubico OTP credential pre-programmed into Slot 1, and it’s the thing that fires when you accidentally touch the sensor.

When the slot fires, the YubiKey types a 44-character string that encodes the key’s identity and a counter, encrypted with a symmetric key that’s shared with Yubico’s validation servers (YubiCloud). A service that wants to verify the OTP sends it to YubiCloud, which decrypts it, checks the counter to prevent replay attacks, and responds with a pass or fail.³

That pre-programmed Slot 1 credential is registered with YubiCloud at the factory—which is what makes Yubico OTP work out of the box, but also means the symmetric key exists on Yubico’s servers from day one. This is fine for the threat model the protocol was designed around back in the day, but it’s another reason the industry has converged on public-key approaches like FIDO2 where no copy of your secret lives anywhere but on the key itself.

This was genuinely useful in the pre-FIDO2 era. Before WebAuthn became a standard, Yubico OTP was one of the best options available for hardware-backed two-factor authentication. Some services still support it—Yubico’s own forums being a notable example—but the list has been shrinking for years as the industry has converged on FIDO2/WebAuthn.

The name “YubiKey” itself is a reference to this feature—”Yubi” comes from “ubiquitous,” and the original product, released in 2008, was an OTP token only.

Static Password

A YubiKey OTP slot can also be configured to hold a static password—a fixed string that gets typed every time you touch the sensor.⁴ This is essentially a hardware-stored password.

It’s the simplest credential type: no cloud validation, no counters, no encryption. Just a string that gets typed. Some people use this as a component of a passphrase—the YubiKey types a long random portion, and they type a memorized prefix or suffix. In practice, this is rarely used, and FIDO2/WebAuthn is a far better solution for authentication.

OATH HOTP

OTP slots can also hold HMAC-based One-Time Password (HOTP) credentials, as defined in RFC 4226. These are counter-based: each touch generates the next code in a sequence.

This is distinct from the OATH application on the YubiKey, which is a separate module that can store many TOTP and HOTP credentials and is accessed via the Yubico Authenticator app rather than through the OTP slot. If you have an HOTP credential in an OTP slot, disabling the OTP application will disable it—but the OATH application remains unaffected.

OATH TOTP (A Common Source of Confusion)

Time-based One-Time Passwords (RFC 6238) cannot be generated via the OTP slots. The YubiKey lacks a real-time clock, so it can’t compute time-based codes on its own.

TOTP is instead handled by the separate OATH application on the YubiKey. The YubiKey stores the TOTP secrets, and the Yubico Authenticator app on your computer or phone provides the clock and displays the codes. This is an entirely different module from OTP—disabling the OTP application has no effect on your TOTP codes.

This is probably the single biggest point of confusion around YubiKey OTP. People hear “I’m disabling OTP” and panic about their authenticator codes. The OATH application and the OTP application have nothing to do with each other.

Why FIDO2/WebAuthn Replaced All of This

FIDO2/WebAuthn is the modern standard for hardware-backed authentication, and it fixes the problems that every OTP-based approach has:

Phishing-resistant by design — the credential is origin-bound, meaning the browser will not submit it to a lookalike domain that an attacker may have set up. OTP credentials have no concept of origin—they’re just strings that get pasted wherever you’re typing, which is, incidentally, the entire problem this post is about.
No shared secrets — FIDO2 uses public-key cryptography. The private key never leaves the hardware token. Yubico OTP requires a symmetric key shared between the YubiKey and YubiCloud, which means there’s a copy of the secret on Yubico’s servers.
No third-party validation service — FIDO2 verification happens directly between the service you’re logging into and the authenticator. Yubico OTP requires the service to send the OTP to YubiCloud for validation.
Built into everything — every major browser, operating system, and a rapidly growing number of services support WebAuthn natively.

Conclusion

The OTP application on your YubiKey is a legacy feature. It was useful in its time, but that time has largely passed. Disabling it doesn’t remove any functionality you’re likely using—it just stops the key from doing the one thing that annoys you, and spares your peers from seeing (and making fun of) your gibberish strings.

Footnotes

If your Slack workspace doesn’t have a Yubico emoji, I highly recommend adding one. It’s the most efficient way to acknowledge what just happened without having to explain it. ↩
Regenerating the Yubico OTP credential produces fresh secrets—a new private ID and a new AES key. Any service that was validating the old credential won’t recognize the new one, so you would need to re-register the YubiKey with those services. This is one more reason to prefer disabling the OTP application over deleting the slot credential—if you ever need the credential again, re-enabling the application restores the original credential since it was never deleted from the key. ↩
Yubico OTP validation requires network access to YubiCloud (or a self-hosted validation server). This is another reason the protocol has fallen out of favor—it requires the service to call out to Yubico just to verify the credential. Yubico does provide open-source validation server software for self-hosting, but very few organizations go that route. ↩
And you thought emitting a random string into a Slack channel was bad, imagine doing that with your password. Yikes. ↩

A No-Nonsense Guide to GPG Commit Signing with a YubiKey

Ryan Spletzer — Sat, 11 Apr 2026 00:00:00 +0000

William Orpen, The Signing of Peace in the Hall of Mirrors, 1919. Imperial War Museum, London. Public domain, via Wikimedia Commons

Anyone who knows me well knows that I nerd out about some specific things, like OAuth and its adjacent specs like OpenID Connect, JWT, etc., and other auth specs like FIDO2/WebAuthn and SPIFFE/SPIRE, Git itself,¹ CI/CD, Zero Trust, certain fancy words like “idempotency,” and in recent years AI and data engineering practices (just to name a few).

I’m also a big believer in the promise and evolution of the secure software supply chain, and am of the firm belief that a secure software supply chain starts with you, on your local machine.

Therefore it should come as no surprise that I nerd out quite heavily about GPG-signed Git commits—cryptographic proof that code actually came from you (possibly in tandem with an AI agent helping you)—and about taking that an extra step further by storing your signing key on a YubiKey² so that the private key never persists on your filesystem.

This guide walks through how to set up GPG commit signing with a YubiKey on macOS, Windows, and Ubuntu.

Now, you may look at this post and think: “Ryan, this is really long, and it even has a table of contents… is this truly ‘No-Nonsense’?”

Trust me when I say this: the nonsense is as minimized as possible here. Going through GPG / YubiKey setups has traditionally been not well-explained, and not for the faint of heart (hence why many people don’t do it!). Because I’ve been doing this for many years, I have thought beyond initial setup and further to the many scenarios you will run into along the way: for example, not just how to set up your key initially, but what you have to do for setting up a second YubiKey, what you have to do if you need to re-key, and more.

So while this post covers the initial setup, it also serves as a reference you can come back to when those additional scenarios inevitably arise. Feel free to jump to the section(s) that are relevant for your situation. I’ll keep adding to this as I think of more scenarios to address, so you can refer back to this post when needed, and so I can, too!³

Why Bother?
Overview / TL;DR
Planning Your Key Identity
Prerequisites
Step 1: Generate Your GPG Key Pair
Step 2: Move the Signing Subkey to Your YubiKey
Step 3: Configure Git for GPG Signing
Step 4: Upload Your Public Key to GitHub
- Making Your Public Key Easy to Find Later
Step 5: Test It
Setting Up on a New Machine
Re-Keying: Updating Your Email Addresses
Revoking a Compromised Subkey
Starting Over: Complete Re-Key (Fresh Start or Master Key Compromise)
Extending Key Expiration
Troubleshooting
Bonus: Using Your YubiKey for SSH Authentication
- Setup
- Verifying It Works
Closing Thoughts

Why Bother?

If you’ve ever looked at at someone’s activity on GitHub and noticed the green “Verified” badge across all the commits on their PR, that’s GPG commit signing at work. Without it, anyone can set user.name and user.email in their Git config to whatever they want—though it would be coming from a different GitHub account, there’s really nothing stopping someone from “committing as you” with this commit metadata set in their Git config.⁴

Signing commits with GPG attaches a cryptographic proof to each commit asserting that it came from the holder of a specific private key.⁵ (And it also personally gives me a large dopamine hit when I see the green “Verified” badge.)

If that private key lives on a hardware token like a YubiKey, it can’t be exfiltrated by malware or accidentally copied—the signing operation happens on the YubiKey itself, and you confirm it by entering your YubiKey PIN for the first signing. After that, the YubiKey stays unlocked for subsequent signings until you log out, shutdown/reboot, or remove the YubiKey.⁶

I would be remiss in talking about software supply chain if I didn’t mention that GPG isn’t the only way to sign commits. GitHub also shows the green “Verified” badge for commits signed with SSH or S/MIME, and it’s worth being aware of how Sigstore’s Gitsign uses OpenID Connect to create keyless signatures that are strongly tied to your identity without requiring long-lived keys at all. Certain organizations may be at the level of maturity where Sigstore + OIDC is the right fit for showing provenance, but everyone is on their own software supply chain journey and organizations and people are at different points in that journey. Notably, GitHub doesn’t currently show the green “Verified” label for Sigstore signatures—which my dopamine would be sad about—but Sigstore does provide strong cryptographic assurances through its transparency log. It’s also worth noting that Git only supports one signing method at a time, so you have to choose between GPG, SSH, S/MIME, or Sigstore—you can’t layer them. In the absence of other options being available to you, I recommend setting up GPG commit signing yourself, because it is entirely within your control and is also what the Linux and Git open source projects use for signing themselves. GPG signing is better than nothing. If you are someone who prefers using SSH with GitHub, then SSH signing may be more appealing for you.⁷

Overview / TL;DR

The setup involves three parts:

Generating a GPG key pair – generating a master key (certify) with subkeys for signing, encryption, and optionally authentication
Moving to the YubiKey – moving the signing subkey gets onto the hardware token so it doesn’t persist on disk
Configuring Git – telling Git to use GPG and point it at your signing subkey

The end result: every git commit triggers a signing operation on your YubiKey, and the commit gets a cryptographic signature that GitHub (or any verifier) can check against your public key.

I’ll break things down as we go between macOS, Linux (Ubuntu)⁸, and Windows.

Planning Your Key Identity

Before you generate anything, it’s worth understanding a constraint that will shape your setup: a YubiKey’s OpenPGP applet holds exactly one key slot each for signing, encryption, and authentication. That’s one identity per YubiKey—you can’t load a second, separate GPG key alongside the first.

If you use multiple email addresses—say, a personal email address and a work email address—you have two options:

One key, multiple UIDs (recommended): Add all your email addresses as UIDs on a single GPG key. Git and GitHub match the commit’s author email against the UIDs on your key to decide whether to show the “Verified” badge, so as long as every address you commit with is listed as a UID, you’re covered.⁹ This is what I do—one GPG key with both my personal and work emails attached, loaded onto one pair of YubiKeys (primary + secondary). It’s simpler to manage, and you only need one set of hardware tokens.

Separate keys per identity: Generate a completely independent GPG key for each email/identity, each on its own YubiKey (ideally a pair of YubiKeys per GPG identity, for redundancy). This gives you full isolation between identities—revoking your work key doesn’t touch your personal one—but it means more hardware to buy and manage, separate Git signing configs per repo or directory or machine, and separate public keys to upload to GitHub.

For most people, the single-key approach is the pragmatic choice. The separate-keys approach makes more sense if your organization absolutely requires dedicated hardware tokens, or if you just personally want an absolute firewall between identities.

If you ever need to change an email address down the road—a new job, a domain change, etc.—you add the new UID to your existing key, optionally revoke the old one, and re-upload your public key to GitHub. I cover that process in Re-Keying: Updating Your Email Addresses below.

Prerequisites

Before diving in, you’ll need:

At least one (but very preferably two) YubiKey(s) that supports OpenPGP (YubiKey 5 series or newer is recommended, but older YubiKey 4 works too; I use a pair of YubiKey 5C FIPS models)¹⁰
A computer with a USB port (which depends on your YubiKey model, but on modern computers I’d opt for USB Type-C if you can)
Some comfort with the command line¹¹

Step 1: Generate Your GPG Key Pair

If you don’t already have a GPG key pair, you’ll need to generate one. If you already have one, you can skip ahead to Step 2: Move the Signing Subkey to Your YubiKey, but you may want to keep reading to compare your generation method with this one.

The recommended approach is to generate a master key with Certify capability only, then add separate subkeys for Sign, Encrypt, and optionally Authenticate. This way, if a subkey is ever compromised, you can revoke just that subkey without losing your entire identity.

This guide recommends ed25519 for all keys. It’s the modern default—smaller keys, faster signing, and a simpler implementation with fewer knobs to misconfigure compared to RSA. YubiKey 5 series supports ed25519 natively.¹²

macOS – Key Generation

# Install GnuPG via Homebrew
brew install gnupg

# Or build from source: https://www.gnupg.org/download/

# Generate a master key with ed25519
# The interactive prompts will walk you through this
gpg --full-generate-key --expert

During the interactive prompts, you’ll see a numbered menu. There are two paths:

Quick path – option (9) “ECC and ECC”: creates a master key that can certify and sign, plus an encryption subkey, all in one step.

Clean path – option (11) “ECC (set your own capabilities)”: lets you toggle capabilities individually so you can create a certify-only master key, then add dedicated Sign, Encrypt, and Auth subkeys afterward.

I recommend the clean path (option 11)¹³, but option 9 is perfectly fine if you want to get going quickly.

Whichever you choose, the remaining prompts are the same:

Select Curve 25519
Set an expiration—I recommend 2 years for subkeys and 5–10 years for the master key. Shorter subkey expirations act as a safety net (if you lose access, the key auto-expires rather than lingering forever), and you can always extend the expiration later without generating new keys
Enter your name and email address—use the same email address you commit with in Git (i.e. the one in git config user.email)
Set a strong passphrase—make it long (I’d say 20+ characters), unique, and stored in your password manager. This passphrase protects your key backups; if someone obtains your exported key files, the passphrase is the only thing standing between them and your identity. Day-to-day signing uses your YubiKey PIN, not this passphrase, so you won’t be typing it often—only during key management operations

If you chose option (11), add your signing subkey now:

# Replace YOUR_MASTER_KEY_ID with your master key ID from the output above
gpg --expert --edit-key YOUR_MASTER_KEY_ID

# In the GPG prompt:
# gpg> addkey
# Choose (10) ECC (sign only), select Curve 25519, set expiration
# gpg> save

You can also add encryption and authentication subkeys the same way if you want those capabilities on your YubiKey.

Ubuntu – Key Generation

# Install GnuPG
sudo apt update && sudo apt install -y gnupg2 scdaemon pcscd

# Generate keys the same way as macOS
gpg --full-generate-key --expert

The process is identical to macOS from here (see above)—add subkeys with gpg --expert --edit-key.

Windows – Key Generation

# Install GPG4Win via Chocolatey
# Or download from https://www.gpg4win.org/
choco install gpg4win -y

# Generate keys from a terminal (Git Bash, PowerShell, or cmd)
gpg --full-generate-key --expert

Same interactive process as macOS above for key generation and subkey creation.

Step 2: Move the Signing Subkey to Your YubiKey

This is the critical step—once you move a subkey to the YubiKey, it is removed from your local keyring. The local keyring will retain a “stub” that points to the YubiKey, but the actual private key material only exists on the hardware token.

Back up your keys first. Seriously.

# Export your full key (public + private) to a safe location
gpg --export-secret-keys --armor YOUR_MASTER_KEY_ID > ~/gpg-export/master-secret-key.asc
gpg --export-secret-subkeys --armor YOUR_MASTER_KEY_ID > ~/gpg-export/secret-subkeys.asc

Also generate a revocation certificate now, while you have easy access to the master key:

gpg --gen-revoke YOUR_MASTER_KEY_ID > ~/gpg-export/revocation-certificate.asc

A revocation certificate is a pre-signed statement that says “this key is no longer valid.” If you ever lose access to your master key entirely—lose the secure backup, forget the passphrase, or worse, a third party obtains control of the key and passphrase—this certificate is your only way to tell the world the key is dead. It’s tied to your key’s fingerprint, not its expiration date, so it never needs to be regenerated (not when you extend expiration, add UIDs, or make any other key changes).

Note that GnuPG 2.1+ actually generates a revocation certificate automatically during key creation and stores it in ~/.gnupg/openpgp-revocs.d/, but you should copy it into your secure backup location alongside your key exports rather than relying on it being on a single machine.

Move these exported files into a secure storage location, then delete the local ~/gpg-export/ directory. These files should not persist on your filesystem—the whole point is that the private key material lives on your YubiKey, not on disk.

Some practical storage options:

As file attachments in a password manager like 1Password or Bitwarden (the keys are already passphrase-protected, and you likely trust your password manager with everything else already)
An encrypted USB drive stored in a fireproof safe or safety deposit box
An encrypted archive in a cloud drive

You’ll need these backups if your YubiKey is ever lost or broken.

Change Your YubiKey PINs

(Thanks to Samuel Imfeld for pointing out to me that I should add a section on changing the PINs!)

Before any key material lands on the card, insert your YubiKey and lock down its default PINs. Out of the box, your YubiKey’s OpenPGP applet ships with two default PINs:

User PIN (123456) — entered during everyday operations like commit signing, SSH authentication, or decryption. Minimum 6 characters on modern YubiKeys; 6 digits is typical.
Admin PIN (12345678) — entered for administrative operations like keytocard, changing the user PIN, changing card metadata, or unblocking a locked user PIN. Minimum 8 characters.

Setting your own PINs first means the keytocard step below is authorized with a credential only you know, not the factory-default 12345678.

A third optional credential, the Reset Code, lets you unblock the user PIN without needing the admin PIN, but most people don’t use it—the admin PIN can also unblock the user PIN, so I skip setting a reset code.

I use an 8-digit admin PIN and a 6-digit user PIN, both stored in my password manager. Numeric PINs are the most common choice and what most YubiKey-aware tools and docs assume; alphanumeric PINs are supported on recent YubiKey 5 series firmware, but sticking with numeric PINs keeps you compatible with any hardware pinpad you might encounter later.

These PINs are separate from your FIDO2/WebAuthn PIN. The OpenPGP applet, the FIDO2 applet, and the PIV applet each have their own independent PIN slots on the YubiKey. Changing your OpenPGP PIN does not affect your FIDO2 PIN used for passkeys,¹⁴ and vice versa.

PIN retry counters matter here. Entering the wrong user PIN 3 times in a row locks the user PIN—you’ll then need the admin PIN to unlock it. Entering the wrong admin PIN 3 times in a row locks the admin PIN too, at which point the only recovery path is a full factory reset of the OpenPGP applet (ykman openpgp reset), which wipes all keys on the card. This is why I recommend storing both PINs in a password manager the moment you set them—forgetting them means starting over with a fresh keytocard operation (assuming you still have your master key backup; if not, it means generating entirely new keys).

Changing PINs with `ykman` (Recommended)

The most reliable way to change your PINs is with ykman, which talks directly to the YubiKey’s smart-card interface without routing through gpg-agent or pinentry:

# Install ykman if you haven't already
# macOS:
brew install ykman
# Ubuntu:
sudo apt install -y yubikey-manager
# Windows:
choco install yubikey-manager -y

# Change the user PIN (prompts for current PIN, then new PIN twice)
ykman openpgp access change-pin

# Change the admin PIN (prompts for current admin PIN, then new admin PIN twice)
ykman openpgp access change-admin-pin

ykman prompts directly in the terminal it’s invoked from, so it works consistently across Ghostty, Apple Terminal, iTerm2, Windows Terminal, etc., without any pinentry configuration to fight with. For anyone who’s already configured a GUI pinentry like pinentry-mac for day-to-day commit signing, using ykman for PIN changes sidesteps the quirks covered below.

Changing PINs with `gpg --card-edit` (Alternative)

You can also change PINs through GPG directly:

gpg --card-edit

# At the gpg/card> prompt:
# gpg/card> admin
# gpg/card> passwd
# Choose 1 to change the User PIN
# Choose 3 to change the Admin PIN
# Choose Q to quit the passwd menu
# gpg/card> quit

This works, but how the prompts get displayed depends on whichever pinentry program you have configured—and that’s where the quirks come in.

Ghostty Quirk: “Screen or Window Too Small”

If you’re using Ghostty as your terminal on macOS and try to run gpg --card-edit → passwd (or any other operation that invokes pinentry-curses, such as keytocard itself), you may see:

gpg: KEYTOCARD failed: Screen or window too small

This appears to be a bug where Ghostty doesn’t fully propagate its window size to the spawned pinentry process, so pinentry-curses refuses to draw its dialog. The simplest workaround is to run the affected command from a terminal that reports its size correctly—I’ve had no trouble using stock Apple Terminal or iTerm2 for these one-off operations, then returning to Ghostty for day-to-day work. Ghostty itself works fine for commit signing once a GUI pinentry (like pinentry-mac) is in place; the issue only surfaces with terminal-resident pinentry variants.

You can also skip the issue entirely by using ykman openpgp access change-pin / change-admin-pin, which don’t invoke pinentry at all.

GUI Pinentry Quirk: “Sorry, No Terminal at All Requested”

If you’ve already configured pinentry-program in ~/.gnupg/gpg-agent.conf to point at a GUI pinentry (like pinentry-mac or pinentry-gnome3) and set no-tty in ~/.gnupg/gpg.conf, then try to change your PIN with gpg --card-edit → admin → passwd, you may see something like:

gpg: OpenPGP card no. D27600012401... detected
gpg: Sorry, no terminal at all requested - can't get input

This happens because the card-edit passwd flow expects to display its own “choose 1/3/Q” menu in the terminal (separate from the PIN entry prompt itself), and no-tty combined with a GUI-only pinentry leaves GPG with nowhere to show that menu.

Two ways around it:

Use ykman instead — ykman openpgp access change-pin and ykman openpgp access change-admin-pin don’t rely on pinentry or GPG’s interactive card-edit, so they’re unaffected by this. This is my preferred path.
Temporarily revert your pinentry settings — comment out pinentry-program in ~/.gnupg/gpg-agent.conf, comment out no-tty in ~/.gnupg/gpg.conf, restart the agent with gpgconf --kill gpg-agent, perform the PIN change, then restore the settings and restart the agent again.

The upshot: your GUI pinentry configuration is oriented around signing—where you want a dialog that works regardless of which process triggered the commit—but PIN-management flows in gpg --card-edit expect a real TTY to display their own interactive menu. Decoupling PIN management from GPG entirely, via ykman, avoids having to choose between the two configurations.

Move the Subkey onto the Card

With your PINs locked down, move the signing subkey onto the card:

# Edit your key
gpg --edit-key YOUR_MASTER_KEY_ID

# Select the signing subkey (check the index -- usually key 1 or 2)
# gpg> key 1
# gpg> keytocard
# Choose (1) Signature key
# gpg> save

You can verify the key is on the card:

gpg --card-status

You should see your signing key fingerprint listed under “Signature key” and your subkey listing should show ssb> (the > indicates the key is on a card).

Loading the Same Key onto a Second YubiKey

I strongly recommend having two YubiKeys: a primary YubiKey that you carry and a secondary YubiKey stored somewhere safe (I keep mine in a fireproof safe at home). This is good practice for FIDO2/WebAuthn¹⁵ as well (where you’d pre-register both keys with your services), and it applies equally to GPG signing.

When my primary YubiKey was stolen (along with my backpack—more post-mortem learnings from that to come in a future blog post!), I was very glad I had a secondary ready to go.¹⁶ Without it, I would have left hanging while I waited for a new YubiKey to arrive (not to mention since I rely on it for sign-in for many services, beyond just the GPG use case), and further without secure backups of the GPG master and sub keys I would have to generate entirely new keys, re-upload them to GitHub, and update every machine I use.

The simplest approach is to load the same signing subkey onto both YubiKeys. This way your .gitconfig doesn’t change regardless of which key is plugged in—Git just sees the same subkey ID either way. The tradeoff is that you can’t revoke one without revoking the other (since it’s the same subkey), but since the private key can’t be extracted from a stolen YubiKey, this is unlikely to matter in practice.

To do this, before you discard your local key backup from the step above, move the subkey to your second YubiKey:

# After moving the subkey to your first YubiKey and saving,
# restore the local private key from your backup
gpg --delete-secret-keys YOUR_MASTER_KEY_ID
gpg --import ~/gpg-export/master-secret-key.asc

# Insert your second YubiKey and move the subkey onto it
gpg --edit-key YOUR_MASTER_KEY_ID
# gpg> key 1
# gpg> keytocard
# Choose (1) Signature key
# gpg> save

After this, both YubiKeys hold the same signing subkey, and you can swap between them seamlessly. Just remember that when you plug in a different YubiKey than the one GPG last saw, you may need to run gpg --card-status so GPG re-discovers the key stub on the new card.

Removing the Master Key from Your Machine

Once your subkeys are on your YubiKey(s) and your backups are safely stored offline, there’s no reason to keep the master key’s private portion on your local keyring. You only need the master key for infrequent key management tasks—adding a new UID, extending the expiration date, revoking a subkey, or signing someone else’s key. For day-to-day commit signing, the subkey on your YubiKey is all you need. Assuming you performed all the correct secure backup steps for your master and subkeys mentioned during key generation above, you are free to safely remove the master key from your keyring.

# Remove the secret master key from your local keyring
gpg --delete-secret-keys YOUR_MASTER_KEY_ID

# Re-import the public key so GPG still knows about your key
gpg --import ~/gpg-export/public-key.asc

# Plug in your YubiKey so GPG re-discovers the subkey stubs
gpg --card-status

After this, gpg --list-secret-keys will show sec# instead of sec—the # indicates the master key’s private portion is absent. Your signing subkey on the YubiKey still works normally.

When you eventually need to do key management, temporarily import your master key from your secure backup, do the work, then delete it again. This is elaborated on in several scenarios described later in this blog post.

Step 3: Configure Git for GPG Signing

macOS – Git Config

The macOS setup involves a few pieces:

# Install pinentry-mac for the PIN prompt dialog
# Or build from source: https://github.com/GPGTools/pinentry-mac
brew install pinentry-mac

Configure GPG to use pinentry-mac for PIN entry—this gives you a native macOS dialog instead of a terminal prompt:

# ~/.gnupg/gpg-agent.conf
pinentry-program /opt/homebrew/bin/pinentry-mac

If you’re on an Intel Mac, the path would be /usr/local/bin/pinentry-mac instead.

This is worth calling out: using a GUI pinentry like pinentry-mac instead of the text-based pinentry-curses or pinentry-tty isn’t just a cosmetic preference—it’s a practical requirement if you use AI coding tools like Claude Code that execute Git commands on your behalf. I used to use the text-based pinentry in my terminal, but when Claude Code runs git commit it spawns the process in a way where the terminal-based pinentry can’t grab the TTY to prompt you for your PIN. With pinentry-mac, the PIN dialog pops open as a native macOS window regardless of which process triggered the commit, so signing works seamlessly whether you’re committing manually or through an AI assistant.

You may also want to set no-tty in your GPG config (which I personally do), which further ensures GPG doesn’t try to interact with the terminal directly—this complements the GUI pinentry approach:

# ~/.gnupg/gpg.conf
no-tty

If you’re using the newer keyboxd (GnuPG 2.4+), add this (which I also personally do):

# ~/.gnupg/common.conf
use-keyboxd

Set the GPG_TTY environment variable in your shell RC file (needed even with pinentry-mac as a fallback):

# Add to ~/.zshrc or ~/.bashrc
export GPG_TTY=$(tty)

# Or for fish, add to ~/.config/fish/config.fish
# set -x GPG_TTY (tty)

# Or for PowerShell, add to your $PROFILE
# $env:GPG_TTY = (tty)

Now configure Git:

# Point Git at your GPG binary, homebrew path shown below
# If you installed via a different method this may be a different path
git config --global gpg.program /opt/homebrew/bin/gpg

# Set your signing key (use the signing *subkey* ID, not the master key)
git config --global user.signingKey YOUR_SIGNING_SUBKEY_ID

# Enable signing for all commits
git config --global commit.gpgsign true

Ubuntu – Git Config

# Install a GUI pinentry (recommended) or terminal pinentry
sudo apt install -y pinentry-gnome3
# Or for KDE: sudo apt install -y pinentry-qt
# Or terminal-only: sudo apt install -y pinentry-curses

# Add to ~/.bashrc or ~/.zshrc
export GPG_TTY=$(tty)

# Or for fish, add to ~/.config/fish/config.fish
# set -x GPG_TTY (tty)

# Configure Git
git config --global gpg.program gpg
git config --global user.signingKey YOUR_SIGNING_SUBKEY_ID
git config --global commit.gpgsign true

For the gpg-agent.conf pinentry line, I recommend a GUI pinentry on desktop Ubuntu for the same reason as pinentry-mac on macOS—if you use AI coding tools like Claude Code that run git commit on your behalf, a terminal-based pinentry like pinentry-curses can’t grab the TTY to prompt for your PIN. A GUI pinentry pops up a dialog window regardless of which process triggered the commit.

# ~/.gnupg/gpg-agent.conf
pinentry-program /usr/bin/pinentry-gnome3

Or for KDE:

pinentry-program /usr/bin/pinentry-qt

If you’re on a headless server with no desktop environment, pinentry-curses is your only option:

pinentry-program /usr/bin/pinentry-curses

You may also want to set no-tty in your GPG config, just like on macOS, to ensure GPG doesn’t try to interact with the terminal directly:

# ~/.gnupg/gpg.conf
no-tty

If you’re using GnuPG 2.4+ with keyboxd:

# ~/.gnupg/common.conf
use-keyboxd

Windows – Git Config

# If using GPG4Win, Kleopatra handles PIN entry automatically

# Configure Git (in Git Bash or PowerShell)
git config --global gpg.program "C:\Program Files (x86)\GnuPG\bin\gpg.exe"
git config --global user.signingKey YOUR_SIGNING_SUBKEY_ID
git config --global commit.gpgsign true

If you installed GPG4Win, the Kleopatra application manages PIN entry via a GUI dialog. This works well with AI coding tools like Claude Code for the same reason as pinentry-mac on macOS and pinentry-gnome3 on Ubuntu—the PIN dialog pops up as a native window regardless of which process triggered the commit. If you installed GPG standalone via Chocolatey, you may need to configure pinentry separately.

Step 4: Upload Your Public Key to GitHub

For GitHub to show the “Verified” badge, it needs your public key:

# Export your public key
gpg --armor --export YOUR_MASTER_KEY_ID

Copy the output (including the -----BEGIN PGP PUBLIC KEY BLOCK----- and -----END PGP PUBLIC KEY BLOCK----- lines) and add it in GitHub under Settings > SSH and GPG keys > New GPG key.

This step is the same regardless of your operating system.

Making Your Public Key Easy to Find Later

While you’re generating your public key, this is a good time to make sure you can easily retrieve it when setting up a new machine down the road.

You can optionally upload it to a public keyserver:¹⁷

gpg --keyserver keys.openpgp.org --send-keys YOUR_MASTER_KEY_ID

Then on any new machine, you can import it directly:

gpg --keyserver keys.openpgp.org --recv-keys YOUR_MASTER_KEY_ID

Alternatively, keep your public key (not your private key!) in a place you can easily access—a private GitHub gist, a cloud drive, or even committed to your dotfiles repo (potentially), but honestly what I do is just keep this in my password manager, too, for convenience.

Step 5: Test It

Make a test commit and verify it:

# Create a test commit
git commit --allow-empty -m "Test GPG signing"

# Verify the signature
git log --show-signature -1

You should see output indicating a good signature from your key. If your YubiKey is plugged in, you’ll be prompted for your PIN.

If your YubiKey is not plugged in, the commit will fail and/or the pinentry terminal prompt or GUI will indicate you need to insert your card—this is the intended behavior, because the private key only exists on the hardware token.

Setting Up on a New Machine

Getting your initial key generated and moved onto the YubiKey is a one-time thing. But when you get a new machine—or reinstall your OS—you need to get the new machine to recognize the key that’s already on your YubiKey. This is the part that tripped me up the first time, because the steps are different from the initial setup and not always well-documented.

The core idea is the same on every platform: install GPG, import your public key, plug in the YubiKey so GPG discovers the private key stubs, set trust, and configure Git and pinentry.

One step you’ll notice here that wasn’t needed during initial setup is setting trust. When you generate a key, GPG automatically assigns “ultimate” trust to it because it knows you created it yourself. When you import a key on a new machine, GPG doesn’t automatically know it’s yours—it just sees an imported public key—so you need to explicitly tell GPG to trust it. Without this, GPG will warn you on every signing operation.

If you uploaded your public key to a keyserver earlier, importing it on the new machine is one command:

gpg --keyserver keys.openpgp.org --recv-keys YOUR_MASTER_KEY_ID

Otherwise, import it from a file (downloaded from your password manager, cloud drive, etc.) or export it from another machine that already has it:

# From a file
gpg --import ~/gpg-export/public-key.asc

macOS – New Machine

# Install GnuPG and pinentry-mac
# Or build from source: https://www.gnupg.org/download/
# Or build from source: https://github.com/GPGTools/pinentry-mac
brew install gnupg pinentry-mac

# Import your public key (from a backup, a keyserver, or export from another machine)
gpg --import ~/gpg-export/public-key.asc

# Plug in your YubiKey and tell GPG to discover the private key stubs on the card
gpg --card-status

# Trust your own key (otherwise GPG will warn on every signature)
gpg --edit-key YOUR_MASTER_KEY_ID
# gpg> trust
# Choose 5 (ultimate)
# gpg> save

Then set up the same config files as in Step 3:

# ~/.gnupg/gpg-agent.conf
pinentry-program /opt/homebrew/bin/pinentry-mac

# ~/.gnupg/gpg.conf
no-tty

# ~/.gnupg/common.conf (GnuPG 2.4+)
use-keyboxd

# Add to ~/.zshrc or ~/.bashrc
export GPG_TTY=$(tty)

# Or for fish, add to ~/.config/fish/config.fish
# set -x GPG_TTY (tty)

# Or for PowerShell, add to your $PROFILE
# $env:GPG_TTY = (tty)

# Configure Git
# If you installed via a different method this may be a different path
git config --global gpg.program /opt/homebrew/bin/gpg
git config --global user.signingKey YOUR_SIGNING_SUBKEY_ID
git config --global commit.gpgsign true

Ubuntu – New Machine

# Install GnuPG and smartcard support with a GUI pinentry
sudo apt update && sudo apt install -y gnupg2 scdaemon pcscd pinentry-gnome3
# Or for KDE: replace pinentry-gnome3 with pinentry-qt
# Or headless: replace pinentry-gnome3 with pinentry-curses

# Import your public key
gpg --import ~/gpg-export/public-key.asc

# Plug in your YubiKey and discover the private key stubs
gpg --card-status

# Trust your own key
gpg --edit-key YOUR_MASTER_KEY_ID
# gpg> trust
# Choose 5 (ultimate)
# gpg> save

Then configure the same config files as in Step 3:

# ~/.gnupg/gpg-agent.conf
pinentry-program /usr/bin/pinentry-gnome3

# ~/.gnupg/gpg.conf
no-tty

# ~/.gnupg/common.conf (GnuPG 2.4+)
use-keyboxd

# Add to ~/.bashrc or ~/.zshrc
export GPG_TTY=$(tty)

# Or for fish, add to ~/.config/fish/config.fish
# set -x GPG_TTY (tty)

# Configure Git
git config --global gpg.program gpg
git config --global user.signingKey YOUR_SIGNING_SUBKEY_ID
git config --global commit.gpgsign true

Windows – New Machine

# Install GPG4Win
# Or download from https://www.gpg4win.org/
choco install gpg4win -y

# Import your public key (from PowerShell, Git Bash, or cmd)
gpg --import ~/gpg-export/public-key.asc

# Plug in your YubiKey and discover the private key stubs
gpg --card-status

# Trust your own key
gpg --edit-key YOUR_MASTER_KEY_ID
# gpg> trust
# Choose 5 (ultimate)
# gpg> save

# Configure Git
git config --global gpg.program "C:\Program Files (x86)\GnuPG\bin\gpg.exe"
git config --global user.signingKey YOUR_SIGNING_SUBKEY_ID
git config --global commit.gpgsign true

Kleopatra (included with GPG4Win) handles PIN entry automatically on Windows, so no separate pinentry configuration is needed.

Re-Keying: Updating Your Email Addresses

At some point you’ll likely need to update the email addresses on your GPG key—you switch jobs, your company changes its domain, or you retire an old personal address. Because your YubiKey holds your subkeys (signing, encryption, authentication) and UIDs live on the master key, the good news is that re-keying doesn’t require touching the YubiKey at all. It’s purely a master-key operation.

You’ll need your master key’s private portion for this, so temporarily import it from your secure backup (from your password manager or wherever you’ve securely stored it):

# Import the master key from your secure backup
gpg --import ~/gpg-export/master-secret-key.asc

Adding a New UID

gpg --edit-key YOUR_MASTER_KEY_ID

# In the GPG prompt:
# gpg> adduid
# Enter your new name and email address
# gpg> save

Revoking an Old UID (Optional)

If the old address is no longer valid and you don’t want it associated with your key, you can revoke it. A revoked UID stays visible on the key (GPG doesn’t truly delete UIDs) but is marked as no longer valid:

gpg --edit-key YOUR_MASTER_KEY_ID

# Select the UID to revoke (UIDs are numbered starting at 1)
# gpg> uid 2
# gpg> revuid
# Confirm the revocation
# gpg> save

If you’d rather keep the old UID active—maybe the address still works as an alias—that’s fine too. Either way, historical commits aren’t affected: GitHub stores verification records at push time, so commits that were already verified keep their “Verified” badge regardless of UID revocations.

Re-Uploading Your Public Key

After modifying UIDs, you need to re-export and re-upload your public key so verifiers (like GitHub) know about the change:

# Export the updated public key
gpg --armor --export YOUR_MASTER_KEY_ID > ~/gpg-export/public-key.asc

Then replace the key in GitHub under Settings > SSH and GPG keys—add the new export first, then delete the old entry. GitHub stores a verification record at push time, so previously-verified commits keep their “Verified” badge even after you rotate or revoke keys.

If you use a keyserver, push the update there too:

gpg --keyserver keys.openpgp.org --send-keys YOUR_MASTER_KEY_ID

Cleaning Up

Once you’re done, remove the master key’s private portion from your local keyring again (just like in Removing the master key from your machine):

gpg --delete-secret-keys YOUR_MASTER_KEY_ID
gpg --import ~/gpg-export/public-key.asc
gpg --card-status

And don’t forget to update your secure backup with the new export, since the master key now has updated UIDs.

Updating Git Config

If your new email address is the one you want to commit with going forward, update your Git config:

git config --global user.email "your-new-email@example.com"

No change is needed for user.signingKey—that points to your signing subkey, which hasn’t changed.

Revoking a Compromised Subkey

To be clear: if your YubiKey is lost or stolen, your signing subkey is almost certainly not compromised. The private key cannot be extracted from the hardware token, and the PIN retry counter locks the card after 3 failed attempts. A lost YubiKey is not analogous to a leaked private key file.

That said, if you believe a subkey was compromised—for example, your subkey backup was obtained by a third party along with the passphrase—you can revoke just that subkey and generate a new one while keeping your master key and identity intact.

If the compromise extends to your master key (your secure backup was exposed along with the passphrase), use the revocation certificate you generated in Step 2 and follow the Starting Over: Complete Re-Key (Fresh Start or Master Key Compromise) process instead.

Revoke the Subkey

Import your master key from your secure backup, then revoke the compromised signing subkey:

# Import the master key
gpg --import ~/gpg-export/master-secret-key.asc

# Edit the key and revoke the signing subkey
gpg --edit-key YOUR_MASTER_KEY_ID

# Select the compromised subkey (check the index)
# gpg> key 1
# gpg> revkey
# Confirm the revocation and provide a reason
# gpg> save

Publish the Revocation

The revocation needs to reach anyone who might verify your signatures:

# Export the updated public key (now containing the revocation)
gpg --armor --export YOUR_MASTER_KEY_ID > ~/gpg-export/public-key.asc

# Push to keyserver if you use one
gpg --keyserver keys.openpgp.org --send-keys YOUR_MASTER_KEY_ID

On GitHub, go to Settings > SSH and GPG keys, add the updated public key export (which includes the revocation metadata), then remove the old entry. Because GitHub stores verification records at push time, your previously-verified commits keep their “Verified” badge—the revocation prevents new signatures from being verified under the old subkey, but doesn’t retroactively invalidate past ones.

Generate a New Signing Subkey

# Still in edit mode with the master key imported
gpg --expert --edit-key YOUR_MASTER_KEY_ID

# gpg> addkey
# Choose (10) ECC (sign only), select Curve 25519, set expiration
# gpg> save

Load the New Subkey onto Your YubiKey(s)

Follow the same process as Step 2: Move the Signing Subkey to Your YubiKey—move the new subkey to your primary YubiKey, then restore from backup and move to your secondary if you have one.

Update Git Config and GitHub

# Point Git at the new signing subkey ID
git config --global user.signingKey YOUR_NEW_SIGNING_SUBKEY_ID

Upload the new public key export to GitHub (the one containing both the revoked old subkey and the new active one). Future commits will be signed with the new subkey and show “Verified.”

Clean Up

First, export your updated public key (which now contains the revoked old subkey and the new active one) and update your secure backup:

# Export the updated public key
gpg --armor --export YOUR_MASTER_KEY_ID > ~/gpg-export/public-key.asc

Then remove the master key’s private portion from your local keyring. The --delete-secret-keys command removes all secret key material, so you re-import the public key afterward so GPG still knows your key exists (and the YubiKey stubs get re-associated when you run --card-status):

gpg --delete-secret-keys YOUR_MASTER_KEY_ID
gpg --import ~/gpg-export/public-key.asc
gpg --card-status

# Test a signed commit
git commit --allow-empty -m "Test new signing subkey"
git log --show-signature -1

Starting Over: Complete Re-Key (Fresh Start or Master Key Compromise)

The sections above cover lighter-weight changes—updating UIDs or revoking a single subkey while keeping the same master key. Sometimes you need to start from scratch with an entirely new GPG identity.

Scenarios where a complete re-key makes sense:

Your master key was compromised (not just a subkey)—practically this means someone obtained your key material along with the passphrase, or obtained your key material and your passphrase was weak enough to be guessed
You want to switch algorithms (e.g. RSA 4096 to ed25519)
You’ve lost both YubiKeys and your secure backup—you have no way to recover the old key
Your key has expired and you’d rather start fresh than extend it

Revoke the Old Key (if You Still Have Access)

If you generated a revocation certificate during Step 2 (or GnuPG generated one for you in ~/.gnupg/openpgp-revocs.d/), this is the fastest path—you don’t need the master key’s private portion at all:

# Import the pre-generated revocation certificate
gpg --import ~/gpg-export/revocation-certificate.asc

# Publish the revoked key to keyserver
gpg --keyserver keys.openpgp.org --send-keys YOUR_OLD_MASTER_KEY_ID

Alternatively, if you have the master key but not the revocation certificate, you can generate one now:

# Import the master key from your secure backup
gpg --import ~/gpg-export/master-secret-key.asc

# Generate a revocation certificate
gpg --gen-revoke YOUR_OLD_MASTER_KEY_ID > revocation-certificate.asc

# Import the revocation into your keyring
gpg --import revocation-certificate.asc

# Publish the revoked key to keyserver
gpg --keyserver keys.openpgp.org --send-keys YOUR_OLD_MASTER_KEY_ID

On GitHub, you can leave the old (now-revoked) key in place or remove it. Previously-verified commits keep their “Verified” badge regardless.

If you’ve lost access to both the master key and the revocation certificate, you can’t revoke it—just remove the old public key from GitHub and move on. Previously-verified commits still keep their “Verified” badge, and the old key will eventually expire on its own (assuming you set an expiration date, which is one more reason to always set one).

Generate New Keys and Set Up from Scratch

From here, the process is the same as a first-time setup. Walk through each step with your new key:

Generate your new GPG key pair
Move the signing subkey to your YubiKey(s)
Configure Git—update user.signingKey to point to your new signing subkey ID
Upload your new public key to GitHub
Test it

Don’t forget to store your new master key and subkey backups securely, just like the first time around.

Updating Other Machines

Any machine that was configured with the old key will need to be updated. Follow the Setting Up on a New Machine steps, importing the new public key instead of the old one.

You’ll also want to clean out the old key from those machines:

# Remove the old key from your keyring
gpg --delete-keys YOUR_OLD_MASTER_KEY_ID

Extending Key Expiration

Expiration dates on GPG keys aren’t permanent—they’re metadata that can be updated at any time, as long as you have the master key’s private portion. This applies to both the master key and subkeys.

This is one of the reasons shorter subkey expirations (like 2 years) are practical: you’re not committing to a hard deadline, just setting a safety net that you periodically push forward.

Extending Your Master Key’s Expiration

# Import your master key from your secure backup
gpg --import ~/gpg-export/master-secret-key.asc

# Edit the key (the master key is selected by default)
gpg --edit-key YOUR_MASTER_KEY_ID

# gpg> expire
# Enter the new expiration period (e.g. 5y for 5 years from today)
# gpg> save

Extending a Subkey’s Expiration

gpg --edit-key YOUR_MASTER_KEY_ID

# Select the subkey you want to extend (check the index)
# gpg> key 1
# gpg> expire
# Enter the new expiration period (e.g. 2y for 2 years from today)
# gpg> save

You can extend multiple subkeys in one session—select each one with key N, run expire, then save when you’re done.

After Extending

Once you’ve updated expiration dates, re-export and re-upload your public key so that verifiers know about the new dates:

# Export the updated public key
gpg --armor --export YOUR_MASTER_KEY_ID > ~/gpg-export/public-key.asc

# Push to keyserver if you use one
gpg --keyserver keys.openpgp.org --send-keys YOUR_MASTER_KEY_ID

On GitHub, add the updated export under Settings > SSH and GPG keys, then remove the old entry.

Then clean up: remove the master key’s private portion from your local keyring, and update your secure backup with the new export.

gpg --delete-secret-keys YOUR_MASTER_KEY_ID
gpg --import ~/gpg-export/public-key.asc
gpg --card-status

No changes are needed to your YubiKey, Git config, or signing subkey ID—the subkeys themselves haven’t changed, just their expiration metadata.

Troubleshooting

A few common issues and fixes:

“gpg: signing failed: No secret key”

This usually means GPG can’t find the key stub pointing to your YubiKey. Try:

# Restart the GPG agent
gpgconf --kill gpg-agent

# Re-read the card
gpg --card-status

“gpg: signing failed: Inappropriate ioctl for device”

This means GPG_TTY isn’t set. Add export GPG_TTY=$(tty) to your shell RC file and restart your terminal.

PIN Entry Dialog Doesn’t Appear on macOS

Make sure pinentry-mac is installed and configured in ~/.gnupg/gpg-agent.conf. Then restart the agent:

gpgconf --kill gpg-agent

GPG Keeps Prompting for PIN Even With the YubiKey Inserted

(Thanks again to Samuel Imfeld for pointing out the gpg-agent cache-ttl mitigation below!)

The YubiKey only treats the User PIN as “verified” for the duration of the current card session, and that session can drop without you ever unplugging the key. Common causes:

Sleep/wake — on setups where system suspend disrupts USB or forces re-enumeration on wake, the card session ends and the next signing operation opens a fresh one. Whether your particular machine does this depends on OS, hardware, and power-management config.
PIV use deselects OpenPGP — OpenPGP and PIV share the YubiKey’s CCID smart-card interface, and only one applet is “selected” at a time. An SSH client using PIV via PKCS#11, or a ykman piv command, will switch the selected applet to PIV, which clears OpenPGP’s PIN-verified flag per the OpenPGP card spec.
pcscd conflicts on Linux — see scdaemon and pcscd Conflict (Linux) below.
gpg-agent restart — manual gpgconf --kill gpg-agent, a system update that swaps the binary, or a crash wipes the agent’s state.

The mitigation is to extend gpg-agent’s PIN cache window so you only enter the PIN once per work session. gpg-agent caches the User PIN alongside passphrases and silently re-supplies it whenever the card asks for re-verification—both when the card session has reset and, on YubiKeys with forcesig turned on, on every individual signing operation. The defaults are 10 minutes (default-cache-ttl) and 2 hours (max-cache-ttl). You can bump them:

# ~/.gnupg/gpg-agent.conf
default-cache-ttl 28800
max-cache-ttl 86400

That’s 8 hours and 24 hours respectively—roughly “one PIN per workday.” Apply and reload:

gpgconf --kill gpg-agent

A note on why some readers never hit this symptom in the first place: if your YubiKey has forcesig off (which lets PW1 stay verified across signatures until the OpenPGP applet is deselected) and your card session is staying alive between signs, the card simply isn’t asking for re-verification, so the cache TTL is moot. You’ll still be prompted after unplugging the YubiKey or restarting the machine, but not in between—regardless of what default-cache-ttl is set to. You can check your own setting in gpg --card-status—the Signature PIN line reads either forced or not forced. Separately on macOS, pinentry-mac has a “Save in Keychain” checkbox that persists the User PIN in your login keychain across reboots—a different security tradeoff than the in-memory cache, worth knowing about but distinct from gpg-agent’s TTL.

The tradeoff for bumping cache-ttl: an unlocked, unattended machine can sign on your behalf for as long as the cache is warm. If you’ve enabled touch-to-sign,⁶ that concern goes away—every signature still needs a physical YubiKey tap regardless of cache state.

GPG Can’t Find Keys After Upgrading to GnuPG 2.4+

If GPG stops finding your keys after an upgrade (e.g. gpg --list-keys shows nothing, or signing fails with “No secret key”), you may need to enable keyboxd. GnuPG 2.4+ uses a new key storage backend (keyboxd) that replaces the older pubring.kbx file, but it won’t activate unless you opt in:

# ~/.gnupg/common.conf
use-keyboxd

After adding this, restart the agent and re-import your public key:

gpgconf --kill gpg-agent
gpg --import ~/gpg-export/public-key.asc
gpg --card-status

“Please Insert Card with Serial Number…”

This often means your YubiKey isn’t plugged in.

However it can also mean or you’ve swapped to a different YubiKey than the one GPG last saw, perhaps your secondary one. GPG remembers the serial number of the last card and asks for that specific one. If you’ve swapped Yubikeys, force GPG to re-learn the current card:

gpg-connect-agent "scd serialno" "learn --force" /bye

YubiKey Not Detected After Removing and Reinserting

If you unplug and replug your YubiKey and GPG stops recognizing it (signing fails or gpg --card-status errors), the scdaemon process has likely cached the old connection and doesn’t notice the card came back. Kill it and let GPG restart it automatically:

gpgconf --kill scdaemon
gpg --card-status

`scdaemon` and `pcscd` Conflict (Linux)

On Linux, both scdaemon’s built-in CCID driver and the pcscd service try to claim exclusive access to the YubiKey. Symptoms include “card not found” errors or intermittent failures. The fix is to tell scdaemon to back off and let pcscd handle the hardware:

# ~/.gnupg/scdaemon.conf
disable-ccid

Then restart:

gpgconf --kill scdaemon
gpg --card-status

Wrong `gpg` Binary in Git

If signing fails silently or Git seems to use a different keyring than expected, Git may be pointing at the wrong GPG binary. This is common on Windows (where Git ships its own GPG 2.2.x alongside GPG4Win) and on macOS (where multiple installations can coexist via Homebrew, MacPorts, or GPG Suite).

Check what Git is using:

git config --global gpg.program

Then verify it’s the right one:

# Should show your keys and YubiKey stubs
$(git config --global gpg.program) --list-secret-keys

If it shows nothing or the wrong keyring, update gpg.program to point at the correct binary (see the platform-specific sections in Step 3).

Accidental OTP Output When Touching YubiKey

If you accidentally touch your YubiKey’s sensor and a long string of characters gets typed into your terminal or editor, that’s the OTP (One-Time Password) slot firing. It’s harmless but annoying. You can disable OTP mode entirely if you don’t use it. This requires ykman (YubiKey Manager CLI):

# Install ykman
# macOS:
brew install ykman
# Ubuntu:
sudo apt install -y yubikey-manager
# Windows:
choco install yubikey-manager -y

# Disable OTP
ykman config usb --disable OTP

This leaves the other interfaces (FIDO2, OpenPGP, PIV) unaffected.

“Unusable Secret Key” (Expired Subkey)

If GPG gives a generic “unusable secret key” error during signing, your signing subkey may have expired. GPG doesn’t always clearly indicate that expiration is the issue.

Check your key’s expiration dates:

gpg --list-keys --with-colons YOUR_MASTER_KEY_ID

Look for exp (expired) in the output. If your signing subkey has expired, follow the Extending Key Expiration section to renew it from your master key.

Bonus: Using Your YubiKey for SSH Authentication

If you added an authentication subkey to your YubiKey during key generation, you can use that same YubiKey for SSH authentication—meaning one hardware token handles both GPG commit signing and SSH access.

The way this works is that the GPG agent can act as an SSH agent. When you ssh into a server or git push over SSH, the GPG agent intercepts the request and uses the authentication subkey on your YubiKey to perform the handshake. The SSH private key never exists as a file on disk, just like your signing key.

Setup

The steps below work on macOS, Linux, and Git Bash on Windows. The native Windows ssh.exe doesn’t use SSH_AUTH_SOCK—it communicates via a named pipe (\\.\pipe\openssh-ssh-agent) instead. Bridging gpg-agent’s SSH interface to that named pipe is possible (tools like gpg-bridge exist), but the ecosystem here is pretty immature. If you primarily use Windows, you may find it simpler to use a separate SSH key on your YubiKey via FIDO2/resident keys rather than routing SSH through GPG.

First, tell the GPG agent to offer SSH support. Add this to your gpg-agent.conf:

# ~/.gnupg/gpg-agent.conf
enable-ssh-support

Then point your shell’s SSH_AUTH_SOCK at the GPG agent’s socket instead of the default SSH agent:

# Add to ~/.zshrc or ~/.bashrc
export SSH_AUTH_SOCK=$(gpgconf --list-dirs agent-ssh-socket)

# Or for fish, add to ~/.config/fish/config.fish
# set -x SSH_AUTH_SOCK (gpgconf --list-dirs agent-ssh-socket)

Next, tell the GPG agent which key to offer for SSH. Each GPG key has a “keygrip”—a hash that identifies it independently of the key ID. You need to add your authentication subkey’s keygrip to ~/.gnupg/sshcontrol:

# Find the keygrips for your key
gpg --list-keys --with-keygrip YOUR_MASTER_KEY_ID

# Look for the keygrip on the line after your [A] (authentication) subkey
# and add it to sshcontrol, for zsh, bash, fish:
echo "YOUR_AUTH_KEYGRIP" >> ~/.gnupg/sshcontrol

# Or for PowerShell on Windows, but see caveat about gpg-bridge above as this is not mature yet:
# Add-Content -Path "$env:APPDATA\gnupg\sshcontrol" -Value "YOUR_AUTH_KEYGRIP"

Restart the GPG agent for changes to take effect:

gpgconf --kill gpg-agent

Verifying It Works

With your YubiKey plugged in, you should now see your GPG-backed SSH key:

ssh-add -L

This outputs a public key in SSH format that you can add to GitHub (Settings > SSH and GPG keys > New SSH key), paste into a server’s ~/.ssh/authorized_keys, or use anywhere else you’d use an SSH key.

The difference is that when you actually authenticate, the private key operation happens on the YubiKey—you’ll see the PIN prompt (or touch, if you enabled it) just like with commit signing.

Closing Thoughts

This setup has served me well for years. The day-to-day experience is simple: plug in the YubiKey, commit code, enter the PIN when prompted. The security benefit is significant—your signing key never persists as a file that could be stolen or accidentally leaked.

The initial setup is admittedly a bit involved, especially the key generation and subkey-to-card transfer steps, but it’s a one-time cost that pays dividends in the form of verified commits and peace of mind (and dopamine hits from green “Verified” commit labels in GitHub—gets me every time).

Footnotes

I highly recommend Pro Git by Scott Chacon and Ben Straub. It’s free to read online, and it’s the most thorough resource on Git internals and workflows I’ve come across. If you want to understand how Git actually works under the hood rather than just memorizing commands, this is the book. ↩
I mention YubiKey explicitly here because that’s what I use and it seems to be the most popular, but other brands of hardware security keys can be used, like NitroKey, OnlyKey, Token2, and Feitean. Notably, while Google’s Titan security key supports FIDO2/WebAuthn, Titan keys do not support GPG commit signing. Many of the steps outlined here for YubiKey will be similar for other keys. ↩
For the diligent and eagle-eyed readers out there, if you think of a scenario that I haven’t addressed and would like me to add it, feel free to message me on LinkedIn. ↩
It can happen, and has happened to some well-known folks out there. Also credit to Scott Hanselman whose aforementioned linked post originally got me into all this; it’s a great guide for Windows. In his case he uses a key he created from Keybase, which is an awesome service that I’ve also used, but has been a bit dormant since Zoom acquired them. ↩
Technically, GPG signing uses a hash of the commit content, which is then signed with your private key. The verifier re-hashes the commit and checks the signature against your public key. ↩
You can optionally enable touch-to-sign via ykman openpgp keys set-touch sig on for an extra layer of physical confirmation for each and every signing operation, but it’s off by default. I don’t personally use this because I consider the initial PIN unlock to be good enough security for my circumstances, and further when I have coding agents working on things on my machine I want them to be able to do signed commits without constant intervention. ↩ ↩²
And given you know how to get SSH going with GitHub, I assume you can figure out how to set up commit signing with that as well, and further can figure out how to use an SSH key on a YubiKey—just ask a couple of questions to your favorite AI and you’ll have that done. ↩
I’ll assume that if you’re well-versed in Linux that you can figure this out for your distro of choice. While Ubuntu is the most popular, and hence why I focus on it in this post and others, I myself tend to lean more towards Fedora these days. ↩
GitHub matches the commit’s author email against the UIDs on your GPG key to decide whether to show the “Verified” badge. If you commit with multiple email addresses (e.g. personal and work), you can add additional UIDs to the same key with gpg --edit-key YOUR_MASTER_KEY_ID then adduid. ↩
A couple of practical accessories I use: I keep a pair of magnetic USB-C adapters on each of my machines so I can pull the YubiKey off MagSafe-style when moving between them—this also alleviates the fear of snapping a YubiKey off in a USB port. And since YubiKeys are small and easy to misplace, I attached an AirTag on a key ring (I like the Apple Finewoven AirTag key rings) along with a small hand strap to make it easier to keep track of. ↩
On Windows, all commands in this post work in PowerShell and Git Bash. If you use cmd, you’ll need to substitute %USERPROFILE% for ~ and use backslashes for paths—but I’d recommend using PowerShell or Git Bash instead. ↩
My own keys are RSA 4096 from 2018, which is still perfectly secure, but if I were starting fresh today I’d go with ed25519. If you’re wondering about quantum computing cracking this in the future: neither RSA nor ed25519 (nor any ECC) is post-quantum encryption—Shor’s algorithm could theoretically break both. (I say theoretically because I believe there are reasons to be skeptical about how far along we really are with quantum computing.) When post-quantum algorithms land in GnuPG, everyone needs to re-key regardless. In the meantime, the “harvest now, decrypt later” concern applies primarily to encrypted data, not signatures. For commit signing, the threat from a recovered private key is forward-looking (forging future signatures as you), not retroactive—your past signed commits are already baked into Git’s hash chain and can’t be altered in-place regardless of what happens to the key. ↩
This is recommended for the same reason you don’t use your root CA to sign leaf certs—if a subkey is compromised, you revoke just that subkey without losing your entire identity. ↩
The FIDO2 PIN on a YubiKey has its own defaults, minimum lengths, and retry counter, managed separately: ykman fido access change-pin to change the FIDO2 PIN, ykman fido access set-pin if no PIN is set yet, and ykman fido info to see the current state. ↩
FIDO2 and WebAuthn are the underlying standards behind what most people now know as passkeys. A YubiKey can act as a hardware-bound passkey for logging into services like GitHub, Google, and Microsoft—separate from GPG signing, but using the same physical device. ↩
You might wonder whether I rotated my signing key after the YubiKey was stolen. I didn’t—the private key cannot be extracted from a YubiKey, and the PIN retry counter locks the card after 3 failed attempts, so the practical risk was near zero. If you want to be extra cautious in a similar situation, you should revoke the subkey and generate a new one from your backed-up master key. ↩
I recommend keys.openpgp.org specifically because it requires email verification before publishing your identity. When you upload a key, the server strips all UIDs (email addresses) and sends a verification email to each address on the key. Only after you click the confirmation link does the server associate that email with your key and make it searchable by email address. This means no one can upload a key claiming to be you without access to your inbox. Traditional SKS keyservers (like pgp.mit.edu) have no such verification, which led to issues like the 2019 certificate flooding attack. ↩

The Time I Accidentally Dropped a Database

Ryan Spletzer — Sat, 04 Apr 2026 00:00:00 +0000

“Landscape with the Fall of Icarus,” after Pieter Bruegel the Elder, c. 1560s. Royal Museums of Fine Arts of Belgium. Via Wikimedia Commons.

Over a decade ago, I was working on a system that allowed users to work with images of exterior paint test panels from exposure stations at what we called “the paint farm.” The backend that helped to track all of this was an ASP.NET Web API with Entity Framework and SQL Server, and I was working on that part of the stack, iterating on the database schema and its OData-based API. (Sidebar: OData is awesome, an actual standard for REST which should have had much more adoption than it did. Sigh. But I digress…)

If you’ve worked with Entity Framework code-first migrations, you know the drill. You’re iterating on your data model in C# model classes, running migrations, and occasionally it’s helpful to just blow away your LocalDB and let EF recreate it from scratch. Clean slate, fresh seed data (where it’s usually helpful to add one or more records to each table for testing), move on with your day.

The thing about SQL Server Management Studio is that it’s very easy to be connected to more than one server at a time. LocalDB sits right there in your Object Explorer, and so does your shared dev instance, the one that a large handful of other developers rely on for testing their own applications against your API.

I can’t recall every detail of exactly how it happened. But the part that lodged itself into my amygdala—where it remains to this day—was the moment I realized I had just dropped the dev database.

Not my LocalDB. The shared dev database.

The Aftermath

I noticed immediately. That particular flavor of dread is unmistakable, the kind where your stomach drops faster than the database did.

I went hat in hand to our DBA, explained what happened, and asked for a restore. Fortunately, he had hourly incremental backups running, and we were able to get the database back up quickly. Nobody gave me a hard time about it.

Nobody except myself.

The Corrective Action

Here’s the part of the story that surprised even the DBA.

I went back to him and said: “Can you revoke my permission to drop databases on the dev server?”

He was taken aback. Even for a dev environment, voluntarily asking for fewer permissions isn’t something people typically do.

But I thought about it, and the answer was obvious: I literally never needed to drop that database. Not once. Not ever. I could still connect, run queries, migrate schemas, do everything my actual work required. The ability to drop the database was a permission I held but had zero legitimate use for. So why keep it?

This was very much an SRE mindset. When an incident happens, even a low-stakes one, you take corrective action so it doesn’t happen again. You don’t just say “I’ll be more careful next time.” You change the system.

It Happened Again (Sort Of)

Fast forward to more recent times. My team and I had stood up an Azure OpenAI sandbox that many folks across the organization were using day-to-day for proofs of concept.

The Azure Portal has an undesirable UI paradigm where the “Delete resource group” button lives dangerously close to the “Delete resource” button. You can probably see where this is going.

I clicked the wrong one at the wrong screen. And immediately realized what I’d done.

(You may be asking why we weren’t using IaC—well, we were, but in those early days of Azure OpenAI things could get quite finnicky, and sometimes we’d need to intervene manually, and while IaC is helpful for standing up infrastructure, tearing it down doesn’t get exercised quite as often.)

Luckily, we were able to recreate the resource group and restore the exact same resources inside it with some scripting (and there’s nothing like scripting under pressure) before it caused too much disruption. But the outcome was the same as before: we didn’t just move on and hope it wouldn’t happen again. We introduced resource locks on the resource group and resources in our Azure OpenAI sandbox to prevent accidental deletion going forward (because, really, how often do you ever need to be able to delete a whole resource group, even a non-prod one; and even when you do, you can take a moment to remove the locks).

So what does this have to do with AI?

I keep thinking about these experiences now that we’re handing AI agents the keys to real systems.

The principle of least privilege isn’t new. Since I first started working with AWS, we’ve always stressed crafting IAM roles with only the permissions a service actually needs. But there’s something worth stating plainly: if there’s a permission you don’t ever want an AI agent to use, then simply don’t grant it.

That’s it. Same lesson I learned the hard way in 2014.

What’s different now is that AI is actually good at the tedious part: writing tightly scoped IAM policies in infrastructure as code, the kind of work people skip because it’s boring. And it’s going to matter a lot more when the services behind those roles are themselves AI agents.

The industry is still in the throes of discussing sandboxing for AI, but even the most sandboxed agent, when given a permission it doesn’t need, in the absence of additional checks on commands it would run, still has the potential to do bad things, like drop a database… So the principle is the same one I stumbled into when I, as a human, dropped a dev database: don’t give yourself, or your agents, permissions you don’t really need.

Jevons Paradox and the Future of Software Engineering

Ryan Spletzer — Sat, 28 Mar 2026 00:00:00 +0000

William Stanley Jevons (via University of Manchester Libraries), CC BY-SA 4.0, via Wikimedia Commons.

Here’s something that shouldn’t be true but is: when things get cheaper, we often spend more on them.

Think about lemons. 🍋

In the 1800s, a lemon was an exotic luxury, a tropical fruit shipped from far away at great expense. Only the wealthy could afford them regularly. As shipping and agriculture improved, the price per lemon plummeted.

And did we buy fewer lemons? Did we spend less on them?

Of course not. We put them in everything. Lemonade, lemon meringue pie, lemon zest on fish, lemon in our water, lemon-scented cleaning products. The cheaper lemons got, the more uses we found for them, and the more we spent on them in aggregate.

This is Jevons Paradox, and it’s one of those ideas in economics that sounds wrong until you see it everywhere. Most people who invoke it do so in passing, a quick reference to sound clever in a meeting. But the man behind it and his original argument deserve a deeper look, because the pattern he identified in 1865 is playing out right now with AI and software engineering.

The Man Behind the Paradox

William Stanley Jevons was born in Liverpool in 1835, the ninth of eleven children in a family of iron merchants and hardware manufacturers. He was, by any measure, a polymath.

Jevons studied chemistry and mathematics at University College London, then spent five years in Sydney, Australia as an assayer at the Royal Mint, testing the purity of gold during the Australian gold rush. He returned to England and became one of the founders of the marginalist revolution in economics, fundamentally changing how economists think about value. He also built a mechanical reasoning machine called the “logic piano”, essentially an early mechanical computer, decades before anything resembling modern computing.

Jevons died in 1882 at the age of 46, drowning while swimming near Hastings. He left behind a body of work that economists and logicians still reference today.

But the thing most people know him for, if they know him at all, is a book he published in 1865 at the age of 29.

The Coal Question

In 1865, Jevons published The Coal Question: An Inquiry Concerning the Progress of the Nation, and the Probable Exhaustion of Our Coal Mines.

The prevailing wisdom at the time was reassuring: James Watt’s far more efficient steam engine (yes, the watt is named after him) would reduce Britain’s coal consumption. After all, if each engine uses less coal per unit of work, the nation should need less coal overall. Simple math, right?

Jevons saw it differently.

He argued that Watt’s improvements made coal-powered industry so much more economical that it opened up entirely new applications. Before the efficient steam engine, coal powered a narrow set of uses, mostly pumping water out of mines. After Watt, coal powered factories, locomotives, steamships, and heating. The efficiency gains didn’t reduce demand, they exploded it.

As Jevons wrote in his book:

“It is wholly a confusion of ideas to suppose that the economical use of fuel is equivalent to a diminished consumption. The very contrary is the truth.”

This is the paradox: technological improvements that increase the efficiency of resource use tend to increase the total consumption of that resource, not decrease it. The more efficiently you can use something, the more uses you find for it, and the more of it you consume.

Watt’s steam engine wasn’t a conservation device. It was a device that unlocked more value.

The Cloud Proved Him Right

I’ve watched this paradox play out firsthand over a decade of working with cloud computing.

Earlier in my career, I had a manager who would side-eye me about the cost of a single thickly provisioned API gateway. It was a meaningful line item, the kind of thing that showed up in budget reviews and required justification.

A few years later, that same offering became serverless: consumption-based, pay-per-call, scaling to actual demand. The cost per unit fell so far that the side-eye went away.

And what could I do with that newfound freedom?

I wouldn’t run one API gateway more cheaply. I would run a hundred of them for different use cases, produce far more value, spend roughly the same amount of money, and nobody would bat an eye.

This is Jevons Paradox in the real world.

The broader cloud story follows the same pattern. Year after year, the cost per compute unit has dropped. Serverless architectures and managed services emerged that scale to demand (you only pay for what you use). Spot instances, reserved capacity, autoscaling. All of these drove the unit economics down.

And yet, cloud bills across the industry went up, not down.

Not because companies were being wasteful (although some were), but because cheaper compute unlocked use cases that weren’t viable before. Workloads that couldn’t justify the infrastructure cost at $X per hour suddenly made perfect sense at $X/10 per hour. So companies ran more workloads. And more. And more.

The cloud got cheaper. We used more of it. Jevons was right.

AI Is the Next Steam Engine

I see the same pattern emerging with AI and software engineering, and I want to take a clear stance here:

AI will increase the demand for software engineers, not decrease it.

I know that’s not the dominant narrative right now. The headlines are full of layoffs and hiring slowdowns, with companies citing AI-driven efficiency as the reason. But look closer at these stories and the picture gets murkier. It’s hard to separate “AI made us more efficient” from “we needed to cut costs anyway and AI is a convenient narrative.”

Correlation is not causation, and a convenient story is not evidence.

There will be an initial period (we may be in it now) where some companies look at AI-assisted engineering and see an opportunity to do the same work with fewer people. And some of those companies will be right, for a time.

But businesses are always insatiable. They always have a backlog that stretches to the horizon, features they can’t build, markets they can’t enter, technical debt they can’t address, experiments they can’t run. The bottleneck has always been the cost and availability of engineering talent.

When AI makes each engineer significantly more productive, when it turns a task that took a week into one that takes a day, businesses won’t say “great, we can cut 80% of our engineers.” They’ll say “great, now we can finally build that thing we’ve been putting off for three years.”

Just like Watt’s steam engine didn’t make Britain say “wonderful, we need less coal.” It made Britain say “what else can we power with this?”

(I’ve also seen AI compared to electricity and the printing press. Pick your poison / analogy.)

I’ve Seen This in My Own Work

I wrote recently about acceleration and compound engineering, how a dev machine setup practice I’ve carried across jobs for years took me months to modernize with GitHub Copilot last summer, but this year, Claude Code wrapped it in CI in ninety minutes and added a new Linux distro in under an hour.

Here’s the thing: I didn’t take those efficiency gains and go sit on a beach (although that sounds really nice). I immediately turned around and did more. More distros. More CI. More polish. Things that weren’t worth the effort before became trivially achievable, so I did them.

My backlog didn’t get smaller. It got different. Projects that I’d mentally filed under “someday when I have a free month” became “I can knock that out this afternoon.” And when those were done, I found new things to build that I hadn’t even considered before, because the cost of attempting them had dropped below the threshold where they were worth thinking about.

This is exactly what Jevons described. The efficiency of the steam engine didn’t reduce coal consumption. It unlocked new applications that nobody had previously considered viable.

The Counterarguments

It’s worth noting that Jevons Paradox doesn’t always hold. Economists have identified cases where efficiency improvements really do reduce total consumption.

The key variable is elasticity of demand. If demand for a resource is relatively inelastic, if people don’t actually want much more of it even when it’s cheap, then efficiency gains can reduce total consumption. LED lightbulbs appear to have actually reduced total electricity used for lighting, because there’s only so much light people want in their homes.

But zoom out from lighting and look at electricity as a whole. Demand for power is surging, driven largely by data centers that didn’t exist at this scale a decade ago. LEDs saved watts in the living room; AI is consuming megawatts in the server farm. Same resource, opposite outcomes, depending on where the demand is elastic.

And software? Software demand is about as elastic as it gets. Every business wants more of it. Every industry is being reshaped by it. The backlog of software that the world wants built is effectively infinite.

When the constraint is demand, efficiency gains can reduce consumption. When the constraint is supply, efficiency gains blow the doors off. Software engineering is firmly in the latter camp.

Token Budgets Are the New Cloud Bills

Here’s my prediction for how this plays out:

The engineers who learn to wield AI effectively, who build the compound practices (testing, CI, structured prompting, agentic workflows) and discover entirely new ways of delivering software, will be the ones everyone wants to hire. There will be more to build than ever before.

And businesses won’t spend less on software engineering. They’ll spend differently. Where today the cost is primarily headcount, a growing share is already token budgets, the cost of the AI compute that amplifies each engineer’s output. We saw the same shift in cloud computing: the engineers who learned to automate infrastructure were the ones everyone wanted.

Just like cloud bills replaced data center CapEx and then grew beyond what most people expected, token budgets will become a major line item that grows year over year. As the cost per token comes down, our consumption will go up.

William Stanley Jevons figured this out about coal over 160 years ago. The cloud proved him right. AI will prove him right again.

The paradox endures: when something gets cheaper, we don’t use less. We use more.

This is not a threat to software engineers, but there’s an uncomfortable flip side. If the cost per token keeps dropping and engineers around you are multiplying their output with these tools, then choosing not to use them makes you the expensive resource per unit of output. You don’t want to be the one still rationing coal while everyone else is building modern steam engines. We have the greatest of power tools now, and it’s up to us to use them.

Curating My Own Algorithm

Ryan Spletzer — Sat, 21 Mar 2026 00:00:00 +0000

“Woman Reading” — Édouard Manet, public domain, via Wikimedia Commons.

In a previous post, Your Information Diet in the Age of AI, I talked broadly about why it matters to be intentional with what you read, watch, and listen to. This time I want to get specific. Here is what my information diet actually looks like in practice, and how I use it to cut through the noise and the slop.

I spent many years on Twitter. (To give you a sense of how long that was, I was there in the very early days when the way you tweeted was by sending an SMS text with your tweet to the phone number 40404.) When I finally left, I realized something: I had given a ton of my time, attention, energy, and thoughts to that platform when I could have been making and owning my own content—like writing blog posts.

That stuck with me. Today I only have a handful of social media accounts: LinkedIn, Bluesky, and Mastodon. I only post to them when I share something I’ve written here. For the little social media I do have, I don’t have any of those apps on my phone. That one change alone has done more for my mental health and focus than I expected, and I find myself with more time and energy for programming and writing and the finer things in life.

The Problem with Algorithms

Algorithms in social media are a mixed bag. Some are so good at figuring out what you want that they become addicting. TikTok’s algorithm, for example, is eerily good at this. It has a level of randomness that takes you into interesting adjacent content you never would have sought out yourself. I don’t use it anymore for exactly that reason: it’s too good at keeping you glued to the screen.

Then there are algorithms that are just bad. YouTube, which I otherwise love as a platform, has a recommendation engine with a tunnel vision problem. Watch one cat video and it assumes the next ten things you want to see are more cat videos. It lacks the serendipity that makes discovery interesting. LinkedIn’s feed has its own issues—you still see good things from people you respect, but let’s be honest: those good things are floating around in a sea of slop.

Bluesky and Mastodon don’t feel nearly as algorithm-driven, which is refreshing.

RSS: My Personal Feed

My solution to a lot of this is RSS, a simple open protocol that’s been around since the late ’90s. It lets websites publish a feed of their content, and you subscribe to whichever feeds you want in a reader app. No account required, no algorithm deciding what you see, no company in the middle. RSS and its sibling protocol Atom are also what power podcasts under the hood—every podcast app is really just a specialized feed reader that fetches audio and video files instead of articles.

Now on to my modern RSS feed setup: I heard about Feedbin and NetNewsWire from Leo Laporte on the TWiT network, and the setup has been great.

Feedbin is a hosted RSS service with a small subscription fee, but what you get for it is worth it: it syncs your read/unread state across any RSS client you connect to it. That keeps you portable. You can use NetNewsWire on your Mac, Flipboard on an iPad, or try out a completely different reader—your state follows you everywhere. If you remember Google Reader, this kind of setup gets you back to a similar experience. (We all miss Google Reader.)

I currently subscribe to about 125 feeds (and counting—this number will be out-of-date quickly), organized into two categories: news and tech. I strive to keep it simple.

You would think that the sheer number of feeds would be overwhelming, but honestly besides the official news/tech publishing outfits (whose headlines you can often gloss over), the posts that come from individuals and even many organizations like companies’ engineering blogs are more of a trickle, and since they are more well-thought-out blog posts from people, it is not the same torrent as an incessant feed of quick thoughts from people coming through social feeds. The social media companies are always going to feed you more, whereas with this RSS approach I feel like I control the firehose.

My news category is deliberately lean—just a handful of sources. I started with more, but when you add major news outlets into a drama-free RSS reader, you suddenly realize from the headlines alone how much of the content is either garbage or just there to sensationalize and stoke fear. The clean, linear presentation of RSS strips away the layout and design that masks the fear-mongering. It becomes a litmus test for quality. I won’t name names on the ones I cut, but I will give a shout-out to the BBC—they consistently get it right.

My tech category is where the bulk of my feeds live. It’s a mix of individual blogs from people I respect, tech publications, product and leadership voices, company engineering blogs, AI-focused sources, and web development resources. I won’t list them all here—the whole point is that you should curate your own. Everyone is at a different point in their journey, and what I find useful might not be what you need.

As an interesting example of curation, I recently made a deliberate decision to add more product management thought leaders into my mix of feeds. That’s one of the nice things about this approach—you can intentionally expand into new areas rather than waiting for an algorithm to maybe surface something relevant.

LinkedIn as Discovery, RSS as Delivery

One pattern I’ve settled into: I appreciate when people post on LinkedIn, but on LinkedIn you may miss things, and what you see might only be snippets rather than the full picture. So when I find someone who consistently writes good stuff, I go find their actual website and subscribe via RSS. LinkedIn is how I find people; RSS is how I actually read them.

A handy thing worth mentioning: you can subscribe to people on Medium and Substack via RSS too. You don’t have to use their apps or get their emails. Just grab the RSS feed URL by appending /feed to Substack URLs or inserting /feed/username for Medium and add it to your reader.

For example, I recently followed The Airbnb Tech Blog on Medium. To get the feed URL, I took their URL https://medium.com/airbnb-engineering and inserted feed to get https://medium.com/feed/airbnb-engineering. If the Medium blog uses a vanity domain name, just appending /feed does the trick. In very rare cases there are “blogs” I want to follow that come through subreddits—there’s a trick for subscribing to those with RSS as well, by appending .rss to the subreddit name in the URL.

(Or, you know, just ask your favorite AI “What is the RSS feed for XYZ blog” and it will probably give you the answer.)

Several of the feeds I follow are Medium or Substack authors consumed entirely through Feedbin (with NetNewWire atop).

Now, unlike the good ol’ days, you may not be able to read the full length of every article in the reader app, but that isn’t really the end of the world—the point is to have a feed of high quality content, and if you have to pop out to the browser to read something, that is fine; people need to eat, and they often need the click-throughs for ad revenue, etc.

What It Feels Like

What I’m really doing with all of this is curating my own algorithm. Admittedly, maybe calling it an “algorithm” is a stretch, because it’s very simple and linear, and it’s only filled with things I consider to be high quality.

My routine is a morning and evening scan. I can scroll through headlines and be in and out in a couple of minutes. If there’s a great post from someone I respect, I take the time to read it, either in the moment or by popping it out into my browser for later. But by and large, the whole experience takes far less time than social media and is far less agitating.

Compare that to reaching for a social media app—you open it for “just a minute” and get sucked into a vortex. With RSS there’s no infinite scroll designed to trap you, no outrage-bait wedged between the posts you actually care about. You read what’s there and you’re done.

It’s healthier. I feel less anxiety, less fear from sensationalized headlines, and I have more bandwidth for the things that matter to me—writing, building things, and thinking clearly.

How You Can Do This Too

If any of this resonates, here’s the practical version:

Pick a sync service and a reader. I use Feedbin with NetNewsWire, but there are plenty of options. The key is a service that syncs state across devices so you’re not tied to one app.
Start subscribing. Think about the people whose writing you value—bloggers, newsletter authors, journalists. Find their RSS feeds. Most blogs have one; Medium and Substack do, too.
Organize into categories that make sense to you. Mine are simple: news and tech. Yours might be different.
Prune ruthlessly. Add sources liberally at first, then cut anything that consistently disappoints. Headlines in a clean reader will tell you a lot about a source’s real quality.
Remove social apps from your phone. This was a big one for me. You can still use those platforms on a computer when you choose to, but removing the temptation to reach for them in idle moments makes a real difference.
Give it time. The payoff isn’t instant, but after a few weeks you’ll notice you feel better informed with less effort and less anxiety. Especially after removing social media apps from the phone, you’ll find your thumb instinctively reaching for where those apps used to be in your phone’s app grid—I found out that this unconscious reflex subsides after a few days to a week.

That’s my information diet. Fewer social algorithms, no doomscrolling. Just a feed I built myself, filled with things worth reading.

And reading is good. Just ask LeVar Burton. 🌈

Shedding Dead Context

Ryan Spletzer — Sat, 14 Mar 2026 00:00:00 +0000

Edvard Munch, Anxiety, 1894. Munch Museum, Oslo. Public domain, via Wikimedia Commons

I have an oh-my-posh segment in my Claude Code status line that shows its context window usage as a little gauge—five bars that tick down as the session fills up.

The other day I opened a fresh session, typed one simple prompt, and watched two of the five bars vanish instantly.

40% of my 200K context window—gone—before I’d done any real work.

That was the moment I realized I had a problem.

What’s Eating Your Context Window

If you use Claude Code (or any AI coding tool with a plugin ecosystem), you’ve probably done what I did: installed every promising MCP server, enabled a bunch of skills, added a global CLAUDE.md packed with instructions, and bolted on project-level configs on top of that.

Each one felt like a small addition. Together, they were a tax I was paying on every single session.

Every plugin, every skill, every line in your CLAUDE.md occupies space in the context window. Even “lazy-loaded” tools that aren’t fully active still carry a footprint—the model needs to keep a manifest of what’s available, their descriptions, their trigger conditions, their tool schemas. None of that is free.

I call this dead context: instructions, tool definitions, and metadata sitting in the context window that aren’t contributing to the task at hand.

Your Context Window Is RAM

The analogy that made this click for me is simple.

The context window is like memory on a classical computer—RAM, specifically. It’s the finite working space where the model holds everything it needs to reason about your problem.

And plugins are like running programs.

One VS Code window with all your extensions (I checked mine recently—105) is fine on its own. The extension host process balloons a bit, language servers spawn, but modern hardware handles it. Now open five or six of those windows, each with their own extension host and language servers, add five or six Claude Code terminals, and a browser with a hundred-plus tabs organized into tab groups just to keep yourself sane—and suddenly your machine is sluggish, spending more resources managing the tools than doing the work.

The same thing happens to an LLM. Every plugin and every instruction competes for the same finite resource the model needs to actually think about your code.

Dex Horthy of HumanLayer has talked about what he calls the “dumb zone,” that middle 40-60% of a large context window where model reasoning starts to degrade. Information placed there is more likely to be ignored or misinterpreted. The model drifts, forgets its own instructions, or, *gasp*, hallucinates. (I don’t like to anthropomorphize models, so instead I’d just like to say it starts to lose details and relevant context and generates output without the relevant grounding and background. Hallucinations are reserved for those people taking things like peyote.)

If 40% of your context is consumed by dead weight before you start, you’re beginning every session in the “dumb zone.”

Bit Flips in the Context Window

The memory analogy goes further than just running out of space.

Think about rowhammer attacks on non-ECC memory: repeatedly accessing one row of physical memory causes electrical interference that flips bits in adjacent rows. Whether a bit flips from an unlikely rowhammer attack, or just from flaky RAM, the data doesn’t just disappear—it corrupts. Values that were correct become wrong, and the system doesn’t know it happened. Or, the system does what happened and it causes system instability… This is perhaps fine for your gaming PC—oh no, you crashed during your last Cyberpunk 2077 session, no big deal—but it’s a lot more concerning for critical workloads. You don’t want to lose your video project mid-render, or as Linus Torvalds notes, you don’t want to mistake flaky hardware for a kernel bug.

Something similar occurs when you overload a context window. The dead context doesn’t just sit there inertly, taking up space. It dilutes the signal. The model’s attention is spread across everything in the window, so the more irrelevant context you pack in, the less weight the relevant context carries. Instructions get misapplied, tool descriptions bleed into each other, and the model can confidently act on the wrong context without any indication that something went sideways.

It’s not just forgetting. It’s degradation—and like a bad DIMM, you might not realize it’s happening until the output is already wrong.

The Bigger-Window Trap

Now much like Police Chief Brody in Jaws, you might be thinking:

“You’re gonna need a bigger ~~boat~~ context window.”

Opus with 1M tokens is available now. Things are moving so fast that between my first draft and publication, Opus 1M went from extra-usage pricing to standard pricing in Claude plans. GPT-5.3-Codex landed too, with a 400K context window—a sweet spot, if you ask me—and GPT-5.4 pushed to 1.05M context; because that extra .05 makes a huge difference, right? 😉

So with a larger context window, this problem is solved, yes?

Maybe.

A bigger context window is analogous to more RAM on a machine with a memory leak. It delays the symptoms without fixing the cause. And worse, it removes the pressure to be disciplined.

With 200K you’re forced to be thoughtful about what you load. With 1M you can be sloppy, and it appears to work—until it doesn’t. When it fails with a million tokens of context, the failures are harder to diagnose because you can’t easily pinpoint which of your million tokens caused the corruption.

There is a well-documented phenomenon of LLMs losing coherence in very large context windows. Research like “Lost in the Middle” (Liu et al.) shows that models struggle to use information placed in the middle of long contexts, and a 2025 EMNLP finding demonstrated that context length alone hurts performance even when the extra context is relevant. More capacity does not mean better reasoning. Sometimes I wish I had something like 300K—a modest buffer beyond 200K. I’m interested to play around with Opus 1M in a really long conversation and see what may or may not break down before compaction.

Between the research and people’s own hands-on experience, the vibe is that the “dumb zone” is real, and the vibe is that longer context windows may not necessarily yield better results—at least not at this point in history. That may change, but discipline is a better bet than hope, and regardless of window size, why not just spend less tokens…

Three Layers of the Same Problem

When I stepped back, I realized I was fighting the same battle on three fronts.

It started with my editor. I had VS Code loaded with dozens of extensions—linters, formatters, language packs, themes, tools I tried once and forgot about. Each one adds overhead to the extension host process, and many spawn their own language server processes on top of that. I was carrying weight for projects I wasn’t even working on, and when you have many VS Code windows open alongside Claude sessions, after a while you start to feel it.

Then there was the AI harness itself. My global CLAUDE.md had grown into a sprawling instruction manual. I had plugins enabled globally that only mattered for specific projects. Skills had accumulated without pruning.

But the uncomfortable one was me. I had five Claude Code instances running simultaneously, five VS Code windows open, and a browser with a mountain of tabs. I was doing to my own brain exactly what I was doing to the model: overloading my own context window with competing demands, and wondering why I feel like I needed to take a nap. Steve Yegge has recently written about this vampiric effect of using so much AI and functioning at a high cognitive level and context switching to such a degree that it saps your energy.

The Plugin Paradox

Before you go uninstalling everything, there’s an important nuance.

Not all plugins are dead context. Some farm out what would otherwise be a token-intensive task to deterministic tools, and further some actually reduce overall token consumption by fetching precisely what the model needs without dumping everything into the window.

Think of it this way: a plugin that cat’s an entire file into context is expensive. A plugin that searches for the relevant function and returns just that? It saves context.

The question isn’t “how many plugins do I have?” It’s “what’s the ROI of each one?”

Some plugins add a small overhead to the context but prevent the model from doing expensive, wasteful exploration on its own. Those are keepers. The ones sitting there occupying space for a capability you use once a month? Dead context.

Memory Management Strategies

If the context window is RAM, then optimizing it is memory management. Here are the strategies I’ve landed on.

Scope your editor per project

I have a code wrapper function in my .zshrc here and in my other terminal’s dot files that launches VS Code with only the extensions listed in a project’s .vscode/extensions.json. Everything else gets disabled for that session.

You don’t need to copy mine. Ask Claude to generate one for your shell and your preferences. (I’m likely to iterate on my own code wrapper based on new conveniences I want to add to it over time.) The point is: each project only loads what it actually needs.

I’ve also started looking at editors with a lower memory footprint altogether—Neovim (via LazyVim), Emacs, and Zed. They’re leaner by default than VS Code, which matters when you have multiple editor windows open alongside multiple Claude Code sessions. Zed still has an extension ecosystem you need to be mindful of, but the baseline overhead is smaller. And with Claude Code, the barrier to exploring these editors is lower than it’s ever been. Neovim and Emacs configs used to be a rite of passage you suffered through alone. Now you can ask Claude to set up your LazyVim config, get your Emacs exactly where you want it, or dial in Zed’s themes, settings, and extensions—all without spending hours and days on end reading and setting up configs by hand.

Audit your CLAUDE.md

Your global and project-level CLAUDE.md files are prime candidates for dead context. Instructions that made sense three months ago might be irrelevant now.

I periodically ask Claude itself to audit my config with something like: “Review my CLAUDE.md and identify anything that could be removed, consolidated, or moved to a skill that only loads when needed.”

Skills are the equivalent of swapping to disk. The full instructions only load when invoked, instead of sitting in memory on every session.

Scope your plugins per project

Not every project needs every MCP server. I set project-level .claude/settings.json files that enable only the plugins relevant to that codebase. A Jekyll blog doesn’t need a database plugin. An API project doesn’t need a Playwright plugin.

Revisit regularly

This isn’t a one-time cleanup. Context cruft accumulates the way clutter accumulates in a house. You install a new skill, try a new MCP server, add a line to your CLAUDE.md. Each one is small. Over time they add up, and one day you notice 40% of your context is gone before you’ve said “hello.”

And More

Even after I wrote this blog post, I found more tips out there for reducing dead and unnecessary context. Do some research and keep an eye on people out there experimenting, since this space is always evolving and changing with new discoveries.

The Compaction Dread

If you’ve used Claude Code in a long session, you know the feeling.

You’re watching the status bar tick down. Each prompt costs tokens. Each response costs more. You start doing mental math—can I fit one more big ask in, or will it push me over?

You start rationing. You shorten your prompts. You avoid follow-up questions you’d otherwise ask. You stop kicking off ambitious tasks because you know there isn’t enough room for the model to do them well.

Eventually the session compacts—the system compresses prior messages to free up space—and you lose fidelity. The thread of what you were building together gets thinner.

Every token of dead context in your session is a token stolen from this budget. All those plugin manifests and stale instructions are crowding out the space you need for the actual back-and-forth of getting work done.

The irony is that the dead context was supposed to help.

Use AI to Fix Your AI

This is the part I find satisfying: the best tool for shedding dead context is the very AI you’re trying to optimize.

Ask Claude to generate your .vscode/extensions.json for a project. Ask it to review your CLAUDE.md and suggest what to cut or optimize or push into a skill that can be loaded on-demand (or in some cases explicitly disables when you don’t need it). Ask it to set up a scoped .claude/settings.json with only the plugins you need. Ask it to create skills for instructions that don’t need to be loaded every session.

There’s a nice recursion to it: the model, operating within its own constrained context window, helping you make that context window less constrained for the next session.

It won’t tell you to install fewer things. You have to bring the philosophy. But once you know what you want to shed, it’s remarkably good at doing the shedding.

Less Is More

It’s easy to think that having the most sophisticated setup—the most plugins, the most context, the most tools at your disposal.

However, being a power user means understanding the constraints of the system you’re working with and operating within them deliberately. It means knowing that a 200K context window is a budget, and that every token of dead context is a token you can’t spend on the work that matters.

Shed the dead context. Your AI will think more clearly. You’ll ration your prompts less. The status bar won’t give you as much anxiety. And when you do hit compaction, it’ll be because you did real work—not because your plugins got there first.

And you need to manage your own context window as a human, too. Please everyone on this AI roller coaster, give yourself some grace and space, and go take a nap.

A New No-Nonsense Guide to Setting Up Python Environments

Ryan Spletzer — Sat, 07 Mar 2026 00:00:00 +0000

Winslow Homer, “The Veteran in a New Field,” 1865. Oil on canvas. The Metropolitan Museum of Art, New York. Public domain.

Almost two years ago I wrote A No-Nonsense Guide to Setting Up Python Environments and near the top of it I said:

I even reserve the right to change my mind later as my own practices and opinions evolve.

Well, I’m cashing in that reservation.

What Changed

The short answer is uv.

uv is a Python package and project manager written in Rust by Astral, the same folks behind the Ruff linter and formatter. It is absurdly fast, and it replaces pyenv, pyenv-virtualenv, pip, pip-tools, pipx, poetry, virtualenv, and arguably conda for most use cases, all in one tool.

My old guide had you installing pyenv for version management, then pyenv-virtualenv for virtual environments, then configuring shell init scripts, and on Windows the whole thing was—I’ll be generous here—unpleasant.

uv collapses all of that into a single binary that works the same way on every platform.

Overview / TL;DR

Install uv. Use uv for everything. That’s it.

No, really. There is no step two. But I’ll walk through the details below because some of the ergonomics are worth knowing about.

Installing uv

macOS

# Via Homebrew
brew install uv

Or via the standalone installer:

curl -LsSf https://astral.sh/uv/install.sh | sh

Linux

pipx install uv

Or via the standalone installer:

curl -LsSf https://astral.sh/uv/install.sh | sh

Windows

# Via Chocolatey
choco install uv

# Via WinGet
winget install --id=astral-sh.uv -e

# Via Scoop
scoop install main/uv

Or via the standalone installer:

powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

That’s the entire platform-specific section of this guide. Compare that to the original post where macOS, Linux, and Windows each had their own lengthy section.

Managing Python Versions

uv can install and manage Python versions directly—no pyenv needed:

# Install a specific Python version
uv python install 3.13

# Install multiple versions
uv python install 3.11 3.12 3.13

# List installed versions
uv python list

# Pin a version for the current directory
uv python pin 3.13

That last command creates a .python-version file, which uv (and other tools) will respect automatically.

Creating Virtual Environments

# Create a virtual environment in the current directory
uv venv

# Create one with a specific Python version
uv venv --python 3.12

# Activate it (if you want to — more on this below)
source .venv/bin/activate  # macOS/Linux
.venv\Scripts\activate     # Windows

But here’s the thing: you often don’t even need to activate the virtual environment. uv will automatically use the .venv in the current directory when you run commands through it:

# No need to activate first — uv finds the .venv automatically
uv run python my_script.py
uv run pytest

Installing Packages

# Install a package into the current virtual environment
uv pip install requests

# Install from a requirements file
uv pip install -r requirements.txt

# Compile and sync a lockfile (for reproducible builds)
uv pip compile requirements.in -o requirements.txt
uv pip sync requirements.txt

Project Workflows

If you’re starting a new Python project, uv also handles project management:

# Initialize a new project
uv init my-project
cd my-project

# Add dependencies (creates/updates pyproject.toml and uv.lock)
uv add requests httpx

# Add dev dependencies
uv add --dev pytest ruff

# Run a command in the project's virtual environment
uv run python main.py
uv run pytest

# Sync the environment to match the lockfile
uv sync

uv run is the one I use the most—it makes sure the virtual environment exists, is up to date with your lockfile, and runs your command in it. No more “did I activate the venv?” moments.

Running and Installing Standalone Tools

In my old guide I mentioned pipx for installing standalone CLI tools. uv covers this use case too, though the mapping isn’t a direct 1:1—uvx is shorthand for uv tool run and uv tool install handles persistent installs:

# Run a tool without installing it (similar to pipx run)
uvx ruff check .
uvx black --check .

# Install a tool persistently (similar to pipx install)
uv tool install ruff
uv tool install httpie

The Windows Story

I want to spend a moment on this because the Windows section of my old guide was, to put it mildly, rough.

The old approach involved:

Installing Chocolatey
Installing pyenv-win as admin
Installing pyenv-win-venv via a separate script
Setting environment variables manually
Creating PowerShell profiles
Disabling App Execution Aliases for python.exe and python3.exe
Dealing with auto-activation that only worked when opening a shell in the directory via Explorer (not via cd)
A PR I submitted to fix a bug in the install script

With uv:

# Or install with your preferred package manager / approach
choco install uv
uv python install 3.13
uv venv
uv pip install requests

That’s it. Same commands and same behavior as macOS and Linux. No admin privileges. No special environment variables. No App Execution Alias dance. No separate tools for version management versus virtual environments.

Windows being on equal footing instead of an afterthought with a pile of caveats is reason enough to make the switch.

What About pyenv?

pyenv is a great tool and served me well. If it’s working for you, there’s no emergency—keep using it. But if you’re setting up a new machine, helping a colleague get started, or just tired of the ceremony, uv is the simpler path.

I still have pyenv installed on my machine and it continues to work fine alongside uv. The two aren’t in conflict. But new projects and new environments? I reach for uv every time now.

What About Dev Containers?

In my original post I mentioned Dev Containers as an alternative approach, and that’s still a great option—especially if your target deployment is container-based or you want fully reproducible environments across a team.

uv works great inside containers too. Astral publishes official Docker images and uv is designed to be friendly in CI and containerized workflows with its deterministic lockfile and fast installs.

When I wrote the original guide in 2024, you needed a different tool for every job, a different set of tools on every OS, and Windows was always the weird one.

uv showed up and collapsed all of that into one tool. It’s 10-100x faster for package resolution and installation (those aren’t my numbers, those are from Astral’s benchmarks), and it works the same on every OS.

I reserved the right to change my mind, and I’m glad I did!

Bonus tip: update your CLAUDE.md / AGENTS.md and hooks for whatever for your various AI coding tools are to use uv instead of pyenv and native pip, etc.—you’ll be happy you did. I often see these AI tools try to pip install things using system Python, which just emits an error and a reference to PEP668, and then they try 2-3 more steps before they know they have to spin up a virtual Python environment. These tools/models haven’t exactly caught up to the fact that we’ve moved beyond this, but if however you use instructions and hooks to steer them towards uv, it becomes a lot simpler with fewer false starts and errors for them to get going.

A Tale of Acceleration and Compound Engineering

Ryan Spletzer — Sat, 28 Feb 2026 00:00:00 +0000

“Johnson, the First Rider on the Pedestrian Hobbyhorse” — published 1819 by R. Ackermann, London. Public domain.

Last summer I spent about a week or two (or three? Or four? I can’t remember, but the git history knows, though, and turns out it was a few months that I iterated on it, funny how the brain does that) creating new dev machine setup scripts / approaches based on a practice I have done for years in my career, and which I wanted to modernize with AI assistance and put out there for others to use (and selfishly, for myself to use as well).

Quick story time: at my old company, in research every summer there would be like 15-20 interns that would show up (maybe I’m exaggerating, it’s hard to recall, again brains are weird!), and what I noticed the first summer is that they would take a couple of days to set up their freshly re-imaged Alienware dev box.

Now, when you’re in an internship, every day is precious, and interns spending two days on this mundane task always felt a bit silly to me.

PowerShell to the rescue.

At that time I coded something by hand (raw PowerShell, no frills) that would automate the setup of their machines, and as a practice when they joined I’d walk around with a thumb drive (yes, you heard that right) and plug it into the intern’s machine, dump the script off, and run it while we got to have coffee and got to know the newest group of folks coming in that summer.

We got a lot of leverage out of this little script, and it was a practice that I took with me in my career when I wrote a new (and enhanced) version at my new company (again, by hand!) and this practice has endured to this day.

Now, I wanted to have a version of this that I could create as open source for not just me to use but others out there. There were also things that could be improved upon like a YAML config file to better parameterize the setup. But the inertia usually always held me back from doing that.

However once AI hit and I got access to GitHub Copilot, it was time to try my hand at this.

Now keep in mind, last summer, 2025, even with GitHub Copilot in hand, I still had to iterate on this for a while. In hindsight I should have set up CI from the get-go, but in the initial pass I really needed to be “close to the metal” to run things in virtual machines to work out repeatable setups for macOS, Windows, and Ubuntu and get a feel for what the developer experience is really like running this these dev machine setups from scratch multiple times. (Pro tip for folks in enterprise who design developer experiences: make sure you dog food these processes yourself to build true empathy.)

Contrast that with now: With Claude Code I was able to wrap the whole thing in CI with GitHub Actions in an hour and a half, and further add Fedora support in an hour.

This last week, I added Debian support in an hour (or less—who’s really counting at this point).

Now, one might say:

“Well Ryan, you’ve been compounding your engineering practices over time.”

And yes, yes I have.

I think this is one of those things that makes developer productivity gains so hard to measure: depending on where you come in and start measuring, you may not have a great reference point of having done something in the past, but further as you go along you build compounding wins beyond just using AI for coding, by using certain techniques and practices like rigorous testing and linting that allow you to move so much faster.

It turns out the people advocating for more linting, testing, TDD and more over the years were right—the difference now is it’s easier than ever to get these practices and workflows going.

In fact compound engineering practices are so powerful that they have been codified into a Claude Code plugin that I encourage people to check out. This is certainly not the only way to experience these benefits—there are thousands of ways to to approach this from, and they very well will vary by type of workload you’re building—but the important thing to do is start.

The tools and the models have gotten better, but it’s hard to account for practices that an AI tool helps you put in place to go even faster.

It’s not all just about code throughput—more often than not, it’s about the practices you set up around the code.

I will add this as well: You really need a platform like GitHub where CI is right there and is easy to get going for fast iteration and the pit of success is very easy to fall into. The virtuous cycle of feedback with these tools being able to read out results from builds in real time is real, and if your CI is clunky / disjointed / cumbersome / slow / unreliable / inaccessible from being read by local tools and doesn’t allow for this virtuous cycle, it’s simple to understand why it’s less ideal: you’re stretching out the developer loop into adjacent tools that don’t provide as seamless of an experience. This is not to say that you can’t accomplish great DX with adjacent tools—you just have to work harder to provide the means for developers to consume them in fast, iterative loops.

It truly feels like the future—like when Doc Brown said, “I’m sure in 1985 plutonium is available at every corner drug store.”

I feel like I have plutonium and I’m blowing up all of my backlog. I’m going to have no retirement projects when I’m old now. I might have to pick up fishing.

The Angel in the Marble

Ryan Spletzer — Sat, 21 Feb 2026 00:00:00 +0000

Michelangelo Buonarroti, Angel, 1494-5. San Domenico, Bologna

I saw the angel in the marble and I carved until I set him free.

- Michelangelo

This quote has lived rent-free in my head for many years.

Though in researching this post it turns out it may be dubiously attributed to him, but this one is definitely a thing:

The block already contains the form, and the artist’s hand reveals it.

- Michelangelo

Even before AI coding assistance, I resonated with this saying when it came to software engineering.

Many people look at a blank project and see, well, nothing.

But that’s not what I see.

I see a block of marble.

And each file added chips away at that block and gets you closer to something resembling a finished work.

Shaping Sound

Many may not know this, but I grew up in a musical household. I’m proficient at guitar and know enough music theory to sit down at a piano and play something pleasant.

A lot of the software engineers I’ve worked with over the years turn out to be musicians, too. I don’t think that’s a coincidence.

Writing music and writing software scratch the same itch for me. You start with silence—or a blank file—and you make deliberate choices about what belongs and what doesn’t, shaping raw possibility into something with structure.

A wrong note in a chord voicing is obvious to a trained ear. A wrong abstraction in a codebase is obvious to a trained eye. Both are a matter of pattern recognition and accumulated taste.

Craft and Art

Software is craft, but I’ve always felt it is also an art.

Given the same problem, you can produce several solutions. One will be nasty—it works, but it fights you at every turn. Another will be coherent, with a structure that others can read and reason about.

The difference between them isn’t just skill. It’s taste.

I strive to find that structure, that particular arrangement where the code feels like it wants to be that way, as if there was no other reasonable shape it could take—like the form was always in the marble, waiting.

The Glass Blowing Shop

AI-assisted coding is undeniable now, and the metaphor I keep coming back to is glass blowing.

In a traditional glass blowing shop, the master glassblower doesn’t work alone. There are assistants who help gather the molten glass, keep the blowpipe turning, hand over the right tools at the right moment. The assistants are essential—you physically cannot do it by yourself.

But the assistants don’t decide what you’re making.

The master decides the shape, the color, the moment to stop adding material, the moment to let it cool. That’s taste. That’s years of practice and learning.

AI coding tools are the best assistants I’ve ever had in the shop. They gather material fast, they keep the pipe turning, they hand me tools before I ask.

But when I let them decide the shape?

I end up with meaningless globs on the floor—or worse, something that looks finished but shatters the moment someone picks it up.

The hard work didn’t go away. It shifted.

Instead of spending hours hand-gathering every piece of glass, I spend that time on the decisions that matter: what to make, what to cut, when to stop.¹

The Solo Act

In the past it took a few people to make a band. Now, with a few machines, someone can play solo.

But a solo act with a loop pedal and a drum machine still needs to know music.

The gear isn’t the talent. The gear amplifies talent—or its absence.

I think we’re living through that moment with software.

The barrier to entry just dropped to nearly zero, and a lot of people are going to make a lot of noise.

Some of it will be good. Most of it won’t.

The people who studied their instrument—who learned why certain chord progressions resolve and why certain architectures hold up under load—will still be the ones making things worth listening to.

I haven’t had much time to play guitar lately, due to busyness at work and in industry in general, but I’m starting to get to the point with these tools where I am accelerating fast, and maybe, just maybe, I’ll use some of that freed up time to pick up ye ‘ol axe again.

The angel was always in the marble.

The chisel just got faster.

But it’s still your hands, your eye, and your years of learning that decide whether what emerges is a masterpiece or a mess.

Footnotes

Knowing when to stop might be the hardest part. With AI tools, the temptation is to keep chipping because it’s so cheap to try one more thing. But every sculptor knows that the marble you don’t remove is just as important as the marble you do. Also, these tools are just straight-up addicting. ↩

Fear, Paranoia, and Vibe Risk Management

Ryan Spletzer — Sun, 15 Feb 2026 00:00:00 +0000

“What if aliens arrived and grabbed our documents from this repository?”

I wish I could tell you this wasn’t a real question from a real professional in industry.

But I woke up one morning while drafting this blog post and this core memory came back to me from over a decade ago.

Now, I love the X-Files just as much as the next person, and I hope this individual is someday able to get in touch with Mulder and Scully, because “The truth is out there.”

Maybe we could afford this type of paranoia in a previous era of inefficiency.

But there’s no place for it anymore.

In the last month, I refactored my entire blog all within the bounds of a single day of casual work with Claude Code while I perambulated around the house getting other things done and only occasionally checking in on the progress.

In an even smaller time span, I wrote an elaborate PowerShell script with unit tests, integration tests, and documentation for gathering information from deployed Azure OpenAI model deployments so my team could plan for model retirements.¹

Just to touch on a couple of examples.

Both of these happened in my spare time. Not dedicated sprints, not hackathon weeks—rather, in the margins. The kind of time you get between meetings or after the kids go to bed.

So when someone asks whether we can accept the risks associated with these tools, I posit a different question: can we afford not to?

Unknown Does Not Equal Unsafe

I frequently observe people equate “things I don’t understand” with “risky.”

That’s not risk. That’s fear.

Call it what it is: vibe risk management—the organizational equivalent of “vibe coding,” where gut feeling replaces rigor and nobody can point to what the control actually reduces. Or in some cases, no one can point to the reason why we shouldn’t pursue a use case besides, “I’m scared.”

Now, fear is a perfectly valid human emotion. I get it—new things are uncomfortable, especially when they touch code, data, and credentials. But feelings aren’t controls, and “what if something bad happens?” isn’t a risk assessment.

Let me offer a framework:

Fake risk = fear + unfamiliarity + hand-wavy “what ifs” that can’t be tested or measured
Real risk = a specific failure mode with measurable controls and verifiable mitigations
Business risk = slowing down so much you lose the market while your competitors ship

I keep thinking about that old line:

“A system that has not been specified cannot be incorrect; it can only be surprising.”

- Garfinkel & Stuart, “Sharpening Your Tools”, Comm. of the ACM (Aug 2023)

When organizations refuse to actually articulate a concrete risk—when the objection is just “but what if?”—then any behavior from the agent is difficult to designate as “risky.”

A recent (semi-absurd) example of fake risk I’ve encountered: refusing to connect AI tools to content sources because of fear of the agent finding things. Things that are already accessible to the humans using those same systems. The data is already there. The risk exists with or without the agent. Blocking the tool doesn’t reduce the exposure; it just makes it harder to do the work.

It is, in a very real way, “security by obscurity.”

To quote one of my favorite movies:

“Brand, God put that rock there for a purpose, and um, I’m not so sure you should, um, move it.”

- Stef, The Goonies

The rocks we put in place in the enterprise need to be moved. You need to let the bats fly out of the hole, so we can deal with them. There’s a ship full of treasure (and skeletons) at the end of that hole, and some spare rare jewels from the captain’s hoard are going to save the town from some jerk 1980s country club developers who want to turn the neighborhood into a golf course.

I want to be direct about this: the fear, the paranoia, the anxiety about things we don’t fully understand—those feelings are real and I don’t dismiss them. But they don’t belong in your risk model.

The Controls That Actually Matter

This is where vibe risk management does its worst damage: teams obsess over friction-heavy hardening theater while ignoring foundational controls that actually reduce blast radius, many of which are abstracted from the user to the point where they would never notice, or in some cases, even provide a better user experience!

A sampling of real controls that reduce blast radius, likelihood, or detection time:

Phishing-resistant auth (YubiKeys)—nearly eliminates phishing², and gives people a better experience without passwords.
Managed/compliant device gating at sign-in—if an attacker throws your device out of compliance (say, by disabling your antivirus), you lose access to important services. That’s the point. The system reacts to compromise signals.
EDR (endpoint detection and response) on every endpoint, monitored, with alerting
Least privilege + short-lived tokens—the blast radius of a compromised credential should be bounded by scope and time
Egress controls + audit logs—know what’s leaving your network and be able to reconstruct what happened after the fact
PAW (Privileged Access Workstation) strategy for production environments—separate the workstation that browses the internet from the one that touches prod³
Diff-based code review + protected branches + mandatory human approval—the agent writes the code, a human reviews and merges it.⁴

To be clear: controls with measurable justification or binding regulatory requirements aren’t theater. What follows is aimed at the rest.

Theater controls that increase friction without improving security:

Arbitrary blanket blocks on entire categories of tools
Allowlisting strategies that block everything by default⁵
Endless review rituals that don’t change outcomes and exist to produce a paper trail⁶
“Secure by inconvenience”—the belief that if something is hard to use, it must be safe
Hiding behind VPNs with no other controls⁷

You often hear the phrase: identity is the new perimeter, and it is true, and we learned that a long time ago. But one of the other things almost no one has learned is this: the perimeter is not how you approach a real Zero Trust model. You need to work backwards from your most sensitive data, not outward from the perimeter.⁸ This is good old defense in depth, but starting at the riskiest points, not arbitrary endpoints.

If your security posture assumes that client endpoint compromise is catastrophic—that one breached laptop means game over—you don’t have a “developer problem.” You have a layering problem.

The default posture should be facilitation, not blocking. Not every use case is valuable, but common productivity tools—webcam software, window managers, keyboard shortcut utilities—shouldn’t require a service ticket. Nobody should be opening a request to install something that helps them move windows around on a screen.

You might assume the biggest companies are the worst offenders, but Microsoft—over 200,000 people—lets developers bring their own home-built towers into their ecosystem. You accept the MDM, EDR, and compliance that comes with it, but the pathway exists. They don’t lock down client endpoints into oblivion because they know engineers need leeway to be creative.⁹

A wise head architect once put it simply: be loose on the development side to enable experimentation and innovation, but once you cross the operational divide into production, layer on the automated scans and controls.

Focus your energy on enabling the valuable use cases and collaboration and innovation. And build a value framework to size the “opportunities” that come to you for review, rather than treating every inbound request as a threat.

A Practical Risk Model for Agents

Vibe risk management asks “what if something bad happens?” Real risk modeling gives you a mechanical way to reason about what an agent can actually do:

Capability risk: What can the agent touch? File writes, command execution, network egress. These are measurable, sandbox-able, auditable.
Credential risk: What secrets does it have access to? Scope and duration matter here.¹⁰
Business impact: What happens if it goes wrong? Money movement, data poisoning, irreversible state changes.

And within business impact, there’s a hierarchy of bad outcomes: deleting data is bad.¹¹ Poisoning data—changing it subtly so nobody notices—is worse. Manipulating monetary values is perhaps among the worst. Each level requires progressively stronger gates.

Context poisoning is a legitimate concern, and it’s probably the best argument I’ve heard against broad agent adoption, even though I haven’t actually seen it exploited at scale. But here’s the thing: people have been opening malicious scripts from the internet for as long as the internet has existed. VBA macros in Excel attachments have been weaponized since the ’90s. The mitigation principles—sandboxing, least privilege, don’t trust input—carry over from that same playbook. But I’ll give the skeptics this: prompt injection is genuinely different from traditional code injection. The instruction/data boundary in an LLM is blurred in ways that classic sandboxing alone doesn’t fully address, and the mitigations are still maturing.¹² That said, “still maturing” is not the same as “impossible.” The surface area can be measured and bounded—it just requires treating it as a new class of problem rather than pretending the old playbook covers it completely or throwing your hands up and blocking everything.

Anthropic published a thoughtful piece on sandboxing for Claude Code, using OS-level primitives like Linux bubblewrap and macOS Seatbelt to enforce filesystem and network isolation.¹³ This is exactly the kind of measured, testable control that belongs in a risk model—not “we’re worried about it” but “here’s the boundary, here’s what it enforces, here’s how we verified it.”

And here’s a valid policy that funnels developers and users correctly without blocking them: don’t create home-concocted MCP servers for third-party services when the vendor ships their own. The vendor will always do auth and security trimming better because they own their identity model. That’s a practical, defensible policy. It doesn’t say “no agents”—it says “use the right integration.” That’s the kind of thinking that actually helps.¹⁴

Where Agents Shouldn’t Go

This isn’t a YOLO manifesto. There are real red lines, and being honest about them is what separates a practical position from a reckless one.

Here are some examples.

Agents shouldn’t have a credit card. Financial instruments that involve real money movement need deterministic gates and human approval, full stop. Maybe one day I’ll eat these words, but in the year of our Lord 2026, I believe this is sound advice. (Note, this is different from heuristics or rules, like people putting in a sell order if a stock hits $330 per share.)

Agents shouldn’t decide where to put secrets. I found this out firsthand: I was looking into hooking up the GitHub MCP server to Claude Code, and the agent suggested sticking a plaintext PAT in my .zshrc. IN MY FREAKING .zshrc! macOS keychain exists, and has existed for a very long time.¹⁵

Agents shouldn’t make management decisions. There’s a slide from a 1979 IBM training manual that says it better than I ever could:

“A computer can never be held accountable, therefore a computer must never make a management decision.”

Forty-seven years old and still exactly right. Machines have been making management decisions of various kinds for a long time. The nuance (in my opinion) is management decisions that impact humans. If you’re going to lay off 10% of your workforce, that decision needs human(s) behind it.

Agents shouldn’t be put in a position to do something potentially illegal. Accountability, again. See the Workday class action lawsuit, where AI-based hiring tools are alleged to have discriminated on the basis of race, age, and disability. Machines can’t be sued. The people and companies who deployed them in unthoughtful ways can, and as much as I don’t like it, according to the law, corporations are people, too.¹⁶

Agents shouldn’t book your travel. You don’t want to end up in the wrong town or accidentally book one fewer night at the hotel than you intended.

Agents shouldn’t destroy or modify critical data—at least not without deterministic gates and controls that a human has approved. But if you’re using an agent to gather lunch orders for the office? Probably fine.¹⁷

Agents shouldn’t handle intensely personal data. Not your name or email—that ship has sailed—but prescription drugs, medical history, the deeply personal stuff. Because who knows, in 10 or 20 years, where all these chat logs will end up.¹⁸

Agents shouldn’t handle deterministic tasks that demand exact values. Refunding a customer $9.99 is not the same as refunding $10. This is not the domain for probabilistic reasoning. The agent should offload that to a deterministic tool.

Agents shouldn’t touch cryptography code, especially in subtle scenarios where constant-time operations matter. Side-channel attacks are real, and an LLM doesn’t necessarily understand timing guarantees, unless you yourself are a professional cryptographer who is weaving the right instructions into the proceedings.¹⁹

The through line: agents do the work, humans hold the accountability.

The Credential Model We Need

This should be table stakes at every organization:

macOS: Keychain
Windows: Windows Credential Manager
Linux: pass + GPG, or Secret Service / libsecret
Individual Storage/Team sharing: password managers
Infrastructure: cloud secret managers (AWS Secrets Manager, Azure Key Vault)—unless you’re Netflix and built your own global geo-distributed secret service²⁰
Workload identity: everyone should be looking at SPIFFE/SPIRE

No plaintext tokens in shell configs. Ever.

That’s it. That’s the credential model. It’s not complicated. The tooling exists on every platform and at every layer of the stack. (Well SPIFFE/SPIRE is a bit of a lift, but I digress.) The problem isn’t technical—it’s that people don’t know these tools exist, and agents aren’t going to teach them. (In fact, as we established, agents may cheerfully suggest the wrong approach, thus deepening the fear of the paranoid person who asks.)

The Bottleneck Moved

The punchline that people keep missing: coding is no longer the bottleneck.

People can move at the speed of thought now. I wrote an elaborate PowerShell script with tests and documentation in the margins of my regular workday. I modernized an entire website in an afternoon. This kind of velocity was simply not possible two years ago.

So the bottleneck shifted. It’s now decision throughput: how fast can you make decisions, get approval for initiatives, cut through reviews that don’t change outcomes?

Security teams, legal teams, risk management teams—your job isn’t to block. It’s to enable safely.

And the villain here isn’t individuals. In many cases, people simply lack familiarity with certain systems, and that’s fixable with education and exposure. The real villain is misaligned incentives. App owners have concerns. Security has concerns. Legal has concerns. But there needs to be a shared mantra: help the business run. NOT “running in place”—rather, excelling; running towards a destination.

I’ve been in rooms where someone says, “But what if the security trimming doesn’t work?”

And the answer is: “We can verify that in two seconds. Anything else?”

Fear dissolves the moment you test it. I wrote a whole post about this—the power of a proof-of-concept to collapse months of hand-wringing into a single afternoon of verified facts. If you’re spending more time debating a risk than it would take to verify whether it’s real, you’ve already lost the thread.

Get Down to the Bits

If you’re at a cautious org and want to start somewhere, here’s the smallest innovation sandbox that no reasonable policy committee should object to: VMs and containers, locally.

A boundary without blocking anything. Full isolation, full control. Many seasoned developers already do this. They’ve set up local VMs and containers for themselves, without anyone asking, because they know it’s the right way to experiment safely. If you can’t get past that hurdle, the problem isn’t technical risk—it’s organizational willingness.

I said earlier that fear and paranoia are real emotions, and I meant it. But the antidote isn’t more gatekeeping—it’s better understanding. Those who invest in learning will reduce their fear and arrive at better outcomes.

So here’s my dare to security, legal, and risk teams: show me the control. Show me the measured risk reduction. If the control has a measurable justification, I’m with you—keep it, strengthen it, fund it. But if you can’t point to what it reduces, admit that it’s vibe risk management—and drop it before it costs you more than the risk ever would.

Because the business will be perfectly “safe”…

… right up until it stops being competitive enough to exist.

Footnotes

I also wrote an even more elaborate setup over the course of a few days where I had a local MCP server and several types of MCP clients as well as a mock Entra ID identity provider / token issuer (or you could bring your own Entra ID tenant) to show how the MCP ecosystem interacted and used OAuth with an identity provider in an enterprise setting. You can build very real things with these tools, not just scripts or refactors. ↩
But not smishing. Sigh. But again, if you have no passwords anymore with your identity provider, that’s one less thing for attackers to phish/smish for. ↩
I often have idealists tell me “Well at Google they just have one identity and one device.” To which I would say: Google is a very different company from yours, and I can almost guarantee you are nowhere near the level of control and discipline and expertise that they are at to even entertain this idea. Skeletons are in your closets and bats will fly out of your caves. ↩
Yes, we’re moving away from manual code review with orchestrators like Gas Town and patterns that will soon be commoditized into the tools themselves. But as you take your hand off the wheel, the question becomes: what are your patterns and practices for verification? Tests of all kinds are becoming easier to generate for free, so there’s no excuse not to have them. Cruise control and lane assist alone will not prevent you from getting flattened by the semi merging into your lane. ↩
Instead of blocking everything by default and making people justify each tool (except for the obvious stuff—you want to install the Tor browser at work? Come on), figure out how to facilitate the use case. The default posture should be allow, with specific, justified restrictions. macOS natively does this well: going to run something that is not notarized from the internet? We’re going to make you take like 3 steps to do so. This may be annoying for hobbyists, but it’s a very reasonable set of sanity checks. ↩
Regulation is often invoked as the reason these rituals exist, but it’s worth checking the actual clauses. Frameworks like HIPAA, PCI-DSS, SOX, and FedRAMP regulate data handling, access controls, and audit trails—they don’t prohibit developers from using AI coding tools on their machines. When someone cites “regulatory requirements” to block developer productivity tools, ask them to point to the specific clause. More often than not, the regulation doesn’t say what they think it says—which actually strengthens the theater argument. ↩
Trust me on this one, if you consider VPN your only protection, you’re asking for a cyber punch in the face. To my prior point in this post, VPNs alone are merely an inconvenience for threat actors, not a full control unto itself. Ask me how I know… ↩
In fact, the perimeter approach is the way we looked at security before Zero Trust, and it was never a real strategy. Helm’s Deep had multiple perimeters and controls that held back a DDoS of orcs. ↩
Microsoft can afford this posture because of the layered infrastructure behind it—the EDR, the compliant device gating, the identity controls. The advice to “facilitate rather than block” assumes you’ve already built the foundational controls listed earlier in this section. Notably they are hardcore about PAWs—Privileged Access Workstations—completely separate machines that are locked down and used only for accessing commercial production environments. They also enforce JIT access with secondary approval for prod elevation and more alongside that. If you haven’t, start there—but start now, because those controls are what enable velocity. ↩
“JIT for NHIs” sounds rigorous until you realize it just moves the credential—it’s turtles all the way down. Whether you hand a robot a gun or tell it where the safe is and give it the combination, the robot still has access to a gun. ↩
But you have backups, right? ↩
The subtler risk is worth naming: an agent could write code that looks correct, passes review, but contains a flaw introduced by a poisoned context—and that’s genuinely harder to catch than a malicious VBA macro. This is why the “diff-based code review” control listed earlier matters more, not less, in a world of agent-generated code. ↩
By the way, it appears now that Seatbelt is available by default and I think it may make sense to use /sandbox the more you “take your hand off the wheel” with these tools. ↩
And no, using MCP inappropriately as a data pipeline is not a valid use case to build your own MCP server for a third-party service. You’ll hit rate limits anyway, you silly goose. Be an adult and do some real data engineering. ↩
I imagine this will get better over time, but it was a vivid reminder that agents optimize for “get it working” rather than “get it right.” Those are not the same thing. ↩
Corporate Personhood has a strange history in the United States (and elsewhere), culminating in one of the most terrible recent Supreme Court decisions on this subject in the Citizens United case in 2010. So, as we’re worried about agents becoming human-like and the implications of that, understand that corporations themselves have been treated as people for a very long time… ↩
Joe just better not get upset that the agent forgot to put mustard on his burger. ↩
Be nice to the AI. It will have receipts. I’m not joking. ↩
In fact, I would argue there’s an older piece of advice here, independent of agents: Don’t roll your own cryptography. ↩
If you have set up a global geo-distributed secret service at your company, I would genuinely love to hear about it. ↩

One Day, Nine Phases, 93% Less CSS

Ryan Spletzer — Sun, 08 Feb 2026 00:00:00 +0000

For years, this blog hosted on GitHub Pages (with Cloudflare in front) ran Bootstrap 3.2.0—a CSS framework released in 2014—despite using roughly only six of its classes.

I have previously written in the very first blog post here about how I first got this blog up and running.

Then I took Rob Eisenberg’s Web Component Engineering course over the most recent holiday break (and have been studying and getting back into frontend in general), and it rewired how I think about this.

I always had a feeling that modern CSS had made frameworks like Bootstrap largely unnecessary for a site this simple, but Rob’s course confirmed this for me.

The Philosophy

The course covers everything from Shadow DOM and custom elements to modern CSS, accessibility, and Playwright testing—all built on a single premise: the web platform (minus all the Bootstrap/React cruft) is usually enough. (And anything we add on top should be selectively chosen.)

CSS Grid, Flexbox, custom properties, and a dozen other capabilities didn’t exist back in the day when Bootstrap used to be the answer to everything. Why ship a CSS framework when the browser already knows how to do it natively?

And especially when AI can easily help you make it happen?

The Safety Net

Before touching a single line of CSS, I had already invested in a Playwright visual regression test suite (again, assisted by AI): 27 tests across three viewports (desktop, tablet, mobile), with pixel-comparison screenshots that catch any unintended visual change.

Without automated visual tests, a refactor like this is a leap of faith. But with those tests, it’s a controlled experiment. Every change gets verified. Every phase ends with green tests or it doesn’t end.

Nine Phases, One Day

Working with Claude Code, I broke the work into nine phases, each independently testable:

Dead code removal — Deleted unused includes and dead CSS
Bug fixes — Fixed a missing <!DOCTYPE html>, broken OG tags, stray HTML
Semantic HTML — Replaced <div> soup with <header>, <nav>, <main>, <article>, <aside>, <footer>
Meta tags and structured data — Added JSON-LD schemas, fixed Twitter cards, added descriptions
CSS architecture — Built a replacement stylesheet from scratch using CSS Grid, Flexbox, and custom properties
Bootstrap removal — The big swap: deleted Bootstrap, removed all its classes from HTML
CSS cleanup — Removed vendor prefixes, modernized layout patterns, converted to rem/em units
Accessibility testing — Added axe-core WCAG 2.0 A/AA checks to every test
Final verification — Full suite pass, metrics validation, cross-browser spot checks

The site remained functional after every single phase. No “big bang” rewrite.

The AI in the Room

I brought the vision, the (learned) philosophy, and the years of experience knowing what “good” looks like. Claude Code brought the ability to read every file in the codebase, understand the relationships between templates and stylesheets, write precise CSS replacements, fix accessibility violations, and iterate through test failures—all at a pace that would have taken me days to match on my own.

The entire nine-phase refactor was completed in a single day.¹

I set the constraints: zero JavaScript (besides Google Analytics), zero framework dependencies, WCAG 2.0 AA compliance, identical visual appearance. Claude Code operated within those constraints. And when it proposed a solution I didn’t agree with—like adding a JavaScript snippet to make horizontal scrollable code blocks keyboard-accessible—I pushed back, and we found a CSS-only alternative.²

I couldn’t have done this in a day without AI assistance. And the AI couldn’t have done it at all without knowing what to aim for.

The Numbers

Metric	Before	After	Change
CSS (uncompressed)	143 KB	10.7 KB	-92.5%
CSS (gzipped)	25.1 KB	2.7 KB	-89.2%
Framework dependencies	Bootstrap 3.2.0	None	Eliminated
JavaScript	None	None	Same
Semantic HTML landmarks	None	Full	New
WCAG automated checks	0	27 tests	New
JSON-LD schemas	0	4	New

93% less CSS, and the site looks exactly the same. Except now it’s accessible, semantic, and has better structured metadata.

Under the Hood

For the technically curious, the new CSS architecture is six small SCSS files:

variables.scss — CSS custom properties for colors, spacing, typography
typography.scss — Font stacks, heading scales, prose styling
layout.scss — CSS Grid for the content/sidebar layout
components.scss — Navigation, cards, sidebar panels, footer
code.scss — Syntax highlighting with WCAG-compliant Solarized colors
utilities.scss — A handful of helper classes

I wanted to just use pure CSS and some type of bundler, but that would have just traded one tool for another—I already have to use Jekyll, and that already comes with a Sass/SCSS bundler. (Else it would have just shifted that to some other node-based bundling approach.)

The HTML moved from Bootstrap’s col-md-9 / col-md-3 grid classes to semantic elements with meaningful names. A screen reader now knows what’s the navigation, the main content, and the sidebar—because the HTML tells it.

Accessibility Wasn’t an Afterthought

I have noticed a trend lately where Accessibility by various AI tools is not treated as a first-class consideration, and from an ethical point of view I strongly resonated with Rob Eisenberg’s focus on accessibility while watching his course.

The Web is for everyone, and must accommodate for folks with various impairments.

And now with AI assistance, there really is no excuse not to do so.

I’m not going to pretend that this initial refactor is “perfect” from an accessibility standpoint (or even the best from an underlying HTML/CSS cleanliness standpoint)—it’s an iteration that I’ll be building on to ensure my site is as accessible as it can be.

One of the most rewarding phases was adding automated accessibility testing with axe-core. Every one of the 27 visual regression tests now also runs a WCAG 2.0 A/AA audit. If a future change introduces a color contrast violation or a missing label, the tests catch it before it ships.

The initial audit uncovered two classes of violations:

Color contrast. Seven of nine Solarized syntax highlighting colors didn’t meet the 4.5:1 ratio against the dark code block background. Each was lightened to the minimum value that passes—close enough to the original palette that you’d never notice by eye.

Link distinguishing. Links within body text matched the surrounding text color with no underline, making them invisible to anyone who can’t perceive the accent color. We added underlines to in-content links while keeping navigational elements (tags, menus) underline-free.

These are the kinds of issues that exist on millions of websites. Automated tooling makes them trivially detectable.³

Less Is More

The web spent the last decade accumulating dependencies: build tools, framework CSS, JavaScript bundles, polyfills for features that browsers implemented years ago.

For a personal blog—a site that serves static HTML and CSS—almost none of that is necessary anymore. CSS Grid replaced Bootstrap’s grid. Custom properties replaced SCSS variables for theming. <nav>, <main>, and <article> replaced the <div role="navigation"> hacks we used to write.

The web platform has caught up. And many of us need to let go of layers of scaffolding that have emerged, and which, frankly, for a lot of scenarios, are no longer necessary.

If you’ve been looking for a concrete example of AI-assisted development that isn’t vibe coding a greenfield app from a prompt, this is it: a precise, phase-by-phase modernization of an existing codebase, with automated tests as the safety net and a human at the wheel.⁴

The AI didn’t dream up the philosophy. It didn’t decide that Bootstrap should go (I did), or that accessibility mattered (I said it did), or that zero JavaScript was a worthy constraint (I felt like it was).

What it did do, was make this vision that I’ve had kicking around for months, finally—and in a timeframe that I could accommodate—a reality.

Footnotes

Spread across a few focused sessions, but the total active time was well under a day. The pace was limited more by running tests and reviewing output than by figuring out what to change. ↩
The issue: wide code blocks created horizontal scrolling, and axe-core flagged the scrollable regions as needing keyboard focus (tabindex="0"). Claude Code proposed a small JavaScript snippet to add the attribute. I said no—zero JavaScript is zero JavaScript. We landed on white-space: pre-wrap instead, which wraps long lines and eliminates the scrollable region entirely. Problem solved, principles preserved. ↩
If you have a Playwright test suite, adding axe-core takes about five minutes and an npm install. There’s really no excuse. ↩
There’s something meta about this. I wrote a post about tidying your data house a few weeks ago and then turned around and tidied my own. Practice what you preach, I guess. And yes, if you think about it: code is a form of data. And the higher quality data (code) you feed to AI in the future, in my opinion, the better off you will be. ↩

McDonald's, Burger King, and the Innovator's Dilemma

Ryan Spletzer — Sun, 01 Feb 2026 00:00:00 +0000

There’s a running joke in the fast food industry about Burger King’s real estate strategy: find out where McDonald’s is building, and set up shop next door.

Sounds lazy. Maybe parasitic. But it’s genius. McDonald’s spends millions on site analysis—traffic patterns, demographics, parking lot geometry, the whole nine yards. Why wouldn’t you draft off their homework?

I grew up down the street from one of these pairings: a McDonald’s with a Burger King literally adjacent to it. (My childhood diet is between me and my cardiologist. The ’90s were a different time.)

Lately, I’ve been thinking about this dynamic in a different context: AI developer tools.

Startups Scout Ahead, Big Companies Follow

This is the Innovator’s Dilemma, playing out in real-time.

Clayton Christensen’s thesis wasn’t that big companies are dumb or poorly managed—it’s that they’re rationally constrained. They listen to their best customers. They invest in high-margin products. They optimize for existing business models. And in doing so, they systematically ignore the small, scrappy markets where disruption begins.

Startups don’t have quarterly earnings calls or enterprise sales teams to placate. So they scout the next problem domain. They run experiments, poke at the edges of what’s possible, figure out what people actually want before anyone has a playbook.

Then the big companies show up.

Sometimes they eat the startup’s lunch. Sometimes there’s room for both. And sometimes—the interesting case—the big company can’t reproduce the magic, even with all their resources.

Beating the Lunch Rush 🍟

GitHub Copilot launched as a technical preview in June 2021—earlier than most people remember. It was genuinely pioneering, the first major AI pair programmer. General availability came a year later. For a moment, they owned the category. Market share: 100%.

Then Cursor showed up in March 2023. Four MIT grads with a VS Code fork and a bet that AI shouldn’t be bolted onto an existing editor—it should be the editor. Co-founder Sualeh Asif explained: “We needed to own our editor and could not ‘just’ be an extension, because we wanted to change the way people program.”¹

What happened next is wild. Cursor shipped chat integration at launch; GitHub Copilot Chat didn’t reach GA until December 2023—nine months later. Cursor had codebase-wide context from day one. Agent mode? Cursor shipped it in late 2024; GitHub’s version landed in February 2025. One analysis noted that Copilot’s multi-file edits were “heavily inspired by another editor called Cursor.”

Read that again: Microsoft—one of the largest, most well-resourced technology companies on Earth—was, for a time, trailing four guys with a fork.²

The pattern repeated in the terminal. Aider pioneered CLI-based agentic coding in 2023.³ Claude Code launched in February 2025 and hit $1 billion run-rate in six months. Dario Amodei says some Anthropic engineers “don’t write any code anymore”—they just let the model do it. The devs I know who ride every wave have moved to Claude Code; those less terminal-inclined may stay parked on Cursor (or wait until there’s a GUI to accompany or wrap tools like Claude Code—something I don’t recommend waiting for).

The result: GitHub Copilot’s market share dropped from near-100% to roughly 42% in a year. Cursor crossed $1 billion ARR faster than any B2B software company in history.

This is not normal. This is not how markets usually work.

Why Can’t Giants Keep Up? 🦕

So why can’t a $3 trillion company stay ahead of four MIT grads?

Microsoft’s own leadership has been asking the same question.

Satya Nadella reportedly told employees that Microsoft’s size is a “massive disadvantage.” In a December 2025 internal email, he criticized Copilot programs as not “really work[ing]” and being “not smart.”

Former GitHub CEO Thomas Dohmke admitted: “You can always move faster and be more convicted. In the beginning we kept the team intentionally small. Small teams can move fast.”

Amjad Masad, Replit’s CEO, described the copying dynamic with candor: “I used to feel really bad. I take it for granted now that it’ll happen. It’s still annoying—especially when people don’t know we innovated that. A lot of Microsoft products are designed by Replit.”

Within a week, he says, competitors copy UI innovations. A week!

What can’t be copied that fast? “Things that are trial by fire,” Masad said. “Things with a lot of pain associated with them, especially on the infrastructure side.”

Features copy in a week. Infrastructure knowledge doesn’t.

Research suggests approval processes and org complexity delay innovation timelines by 30-50%. Technical debt may represent 40% of tech estate in large enterprises. Startups don’t have legacy architecture to protect. Microsoft has to weigh every decision against Azure, Office, Windows, and existing Copilot pricing baked into the codebase.

The big-company playbook has largely been: “Add AI to the thing we already have.”

That’s not nothing. Microsoft has 20 million GitHub Copilot users and 90% of Fortune 100 companies as customers. The distribution moat is real. Whether it outweighs being months behind on features—that’s the key question, and honestly it just might work. I don’t see every colleague I know flocking to Claude Code right now⁴, and frankly it was pulling teeth to get folks to adopt the first wave of tools that came along. (Some dark matter developers live under a rock.)

So some folks may happily stay put for a while, and wait till the next wave of agentic coding innovation is taken out of the terminal and wrapped up very neatly in a productized bow for them. (I’m not waiting. There are outsized gains to be had now, and the Claude Code burger tastes way too good. Addictive, I might say; there’s extra salt and special sauce on this one, for sure.)

They’re All Burgers 🍔

The copying isn’t slowing down.

The next frontier is orchestration—factories of agents coordinating work across codebases. Bleeding-edge community approaches exist now (bring your checkbook), but the minute a big player ships something polished, everyone copies it. Anthropic already borrows community ideas and brings them into Claude Code. The first-party tasks feature is evidence: the community invented persistent memory hacks for agents, and Anthropic productized it.

Commoditization has never moved this fast. At some point, you’re just deciding: Quarter Pounder or Whopper?

Once the market matures, the differences shrink. The startups that scouted the territory proved it was real. Product vision and execution count for a lot, but if the execution is readily clone-able, then you find yourself competing to some degree on distribution and brand loyalty.

Golder and Tellis analyzed 500 companies across 50 product categories: 47% of first movers failed versus 8% of fast followers. Steve Blank put it bluntly: “The jury is in. There’s no advantage to being first. Astute fast-followers learn from first-movers by looking at the arrows in their backs.”

So does that mean Microsoft wins in the end? They have VS Code at 75.9% market share. GitHub is where all the code lives. They have enterprise procurement on their side.

I used to work at a company that was a Microsoft shop, full stop. They always chose the Microsoft option first; you only ventured outside the ecosystem with a compelling reason. Some people only eat McDonald’s.⁵

But here’s what I think a lot of folks are missing: the capability matters more than the wrapper.

Cursor can ship the slickest editor in the world (they don’t), but if Claude Code or Codex or whatever comes next is dramatically better, it doesn’t matter whose IDE you’re using. The intelligence is the product. Everything else is a delivery vehicle.

That’s why I think the real winners in this market aren’t the tool makers—they’re the model makers.⁶ Cursor and GitHub Copilot are fighting over who gets to put the burger in your hand. They don’t control the beef.⁷

The burger wars didn’t end with one chain winning. They ended with multiple billion-dollar franchises serving similar food to different customer segments. That’s probably where we’re headed.

The Sodium Hangover 🧂

So what do you actually do with this information?

If you’re a developer: stop worrying about picking the “right” tool. (But seriously, look at Claude Code, right now.) They’re all converging. Use whatever makes you productive today, switch when something better comes along, and don’t build your identity around your editor. The tool wars are a spectator sport.

If you’re building a startup in this space: you’re in a foot race where the giants are many months behind you. That’s your window. Ship fast, accumulate learnings, build the things that can’t be copied in a week (or even if they can, in a poor facsimile of what you did because you have the secret infra sauce). And know that your real competition isn’t Microsoft—it’s the next startup that hasn’t launched yet, and will find a new angle that builds on prior learnings.

If you’re at a big company: maybe the honest move is to admit you’re playing Burger King’s game. Drafting off good ideas isn’t shameful—it’s a strategy. But your size is, as Nadella put it, a “massive disadvantage” when the underlying technology is shifting this fast. You’re not going to out-innovate four people in a room who don’t have to schedule a meeting to make a decision.

Cursor CEO Michael Truell articulated the startup bet: “The goal is to replace coding with something much better. Our aim is to build a magical tool that will one day write all the world’s software.”

That’s not acquisition-speak. That’s someone who thinks they’re McDonald’s.

And yet: competition has been unambiguously good for all of us. GitHub Copilot has a free tier now. Amazon CodeWhisperer is free for individuals. Pricing pressure has driven Pro tiers down while features accelerate.⁸ Developers capture the surplus.⁹

The burger wars didn’t produce a winner. They produced an industry, and gave cardiologists lots of work for the rest of time.

Pick your burger. They’re all pretty good now.¹⁰

Footnotes

There’s irony in that statement. Being a VS Code fork, Cursor will always be tracking upstream and stuck with an alternate extension marketplace. They also don’t own any models or have advantageous positions around them like the cloud providers do. That puts them in a precarious spot, IMO. But I digress! ↩
Cursor’s secret sauce seemed to revolve around Merkle Trees as an efficient way to index your codebase, which was cool, but ultimately an approach that can be copied, and we’ve quickly trotted past with Claude Code; though Claude Code doesn’t do it directly built-in, it does have an extensive community and you can get code search going quickly with a plugin. I’m certain Anthropic will do something native in the future for this, as they have all the means to do so. But also don’t discount the power of grep in a terminal setting—it’s powerful, and thus far I haven’t necessarily felt the need to have an index per se. ↩
I’m ashamed to say I had never heard of Aider before researching this post. ↩
Even though they should, since despite the fact that better things may come from well-known players, they’re losing valuable time that could have helped them learn how to wrangle agents in higher-order potentially GUI-ful tools in the future. ↩
I’ve also seen the opposite: Amazon fanbois who have an unending, irrational hatred of Microsoft. Can’t we all just agree to hate Oracle? ↩
And perhaps after that, cloud providers that have an advantageous position of hosting models for them and deals to use those models in their own products; Google is in a very interesting position of having both of those things, and while I don’t think they have something rivaling Claude Code the tool, or Opus 4.5 the model, yet, you know how this cat and mouse game goes. We’ll see! ↩
And they need to ensure there is enough beef to begin with; anecdotally everyone has a CLI now like Claude Code, but some have pointed out to me how other CLI’s don’t have “the same taste” in terms of it providing as quality of output based on the techniques that are used. ↩
All signs are pointing to you being able to use more features of ChatGPT free soon, supported by ads—one of the final stops of commoditization in various markets, having services driven to “Free,” means inevitably you become the product. So don’t be surprised if Mickey D’s shows up inline in your ChatGPT response touting their newest scientific advances in the Egg McMuffin. (And now this blog post has truly come full circle.) ↩
Go read Wardley Maps if you want to understand the forces behind why commoditization comes for everything eventually. ↩
I prefer the Bacon King, and my current Bacon King is Claude Code. But I also like some modifiers. No ketchup, light mayo, add light mustard, add pickles. (Fun fact: I have a former manager who cannot tolerate anything in the cucumber realm—hello, Augusto—in almost violent ways where he’d flip over the restaurant table and walk out if you serve him this particular strain of gourd. We’ve had to restrain him several times.) I’m currently working on modifying Claude Code to my liking, and figuring out if I want regular fries or chicken fries (flavors of orchestration) and my choice of dipping sauce, probably ranch (my personal taste injected into these tools). I wish Burger King had caffeine free Diet Dr Pepper, then everything would be perfect. Where did Dr Pepper get his PhD from? Ok this ADHD footnote is getting weird. Bye! ↩

Tidying Your Home for Your AI Guests

Ryan Spletzer — Thu, 15 Jan 2026 00:00:00 +0000

I want you to imagine the place that you live in.

Are your papers in a tidy pile, with the ones you want to keep cleanly filed in some kind of filing cabinet?

Are your books scattered throughout the house on disparate shelves?

Are your clothes organized with a certain strategy (by type, by season)… or is it a free-for-all? (I’m going to guess at minimum you have a sock drawer.)

At our last place, we had the hand drill sitting in the coat closet, with other tools scattered in different places throughout the house… so trust me, we are not saints in this regard.

Now imagine if you invited someone into your house and asked them to categorize your belongings, or even just figure out how to live and take care of things in your place; maybe you’re starting an Airbnb!

If the frying pan is in the upstairs bedroom closet, then people will have a hard time finding it…

This is not an advocacy post for pursuing a Marie Kondo level of tidiness at your house (although shout out to my awesome mentor whose books finally taught me as a full-grown human how to fold clothes properly).

This drawn-out analogy gets us to this point:

The way you work in an enterprise is likely much like an untidy house, and trying to bring AI into the mix to provide value atop that untidiness is going to have challenges.

“All your base are belong to us—OMG!”

Living With the Clutter

I’ve seen many scenarios over the last few years where there is a desire to implement AI to solve some type of problem, but an unwillingness to address any of the data untidiness, or further, the systems or ways of working that grew up around it.

A quick note on “tidy data”: there’s a definition for tidy data that revolves around rows and columns, which isn’t necessarily what I’m referring to here (although it is a great principle to abide by for structured data).

What I mean is how datasets and documents and artifacts are organized among systems, and within systems, and especially the boundaries we create around organization and access.

Maybe you have duplicative tools that different folks are using to store similar kinds of data.

Maybe you are all within the same tool, but have organized your data along boundaries in a way that is not conducive to consumption, either by AI, and honestly, sometimes by end users, too.

Some business requirements and processes around how data is created and used are hard and fast and unmovable.

But many are not, and could be re-oriented to be not only more friendly for AI, but also more friendly for the people operating within these environments.

This is where things get expensive in a very predictable way…

I’ve encountered several concrete examples where there was a desire to set up an AI RAG approach as a “band-aid” to accommodate the scattered nature of unstructured data across various systems (and within those systems) when, in actuality, a “house tidying” effort would have more readily allowed for using AI tools—some of which are off-the-shelf and readily available—to accommodate the desired outcomes.

The intention is not to dunk on RAG (retrieval augmented generation), but it becomes telling when the primary purpose for rolling up your sleeves and taking the time to populate a vector database is:

“We don’t want to change anything about how we work… just make the AI deal with it.”

In one case I saw, there was a desire to provide RAG for data that people didn’t have access to in the source systems… but could allow for folks request access “on demand” when linked to the source through an LLM answer backed by retrieved data.

The interesting thing about this scenario was: the data was not sensitive; access could have been easily granted, or the data could have been reorganized so it lived in one place, with boundaries aligned to the use case.

But the ask was catered around a desire to not change the way that people were currently working.

If we took a moment to reorganize the data so people did have access to it, not only would it have been an improved way of working that benefited everyone, but AI—and particularly in this case, existing off-the-shelf AI capabilities—could have accommodated the approach with far less bespoke infrastructure.

A Bigger House Isn’t a Cleaner One

One might argue at this point: is this not where a data platform comes in to save the day?

Sometimes, yes.

But the issue with that thought process is that it often just shifts the problem: You still have to decide what to ingest, concern yourself with security trimming from the source, and you may still have to tidy and organize the data when things hit the platform.

Also, if the current ways of working lend themselves to new sources of data springing up all the time, you’re going to be constantly doing repeat trips to the folks responsible for ingestion… or you may decide to ingest every new source that appears blindly and figure it out later, which can get costly and unwieldy and unapproachable very quickly.

(That is, unless you have ascended to the most elite levels of Data Mesh implementation, at a maturity level where a business user can simply click a button to onboard a new data source to your data platform for downstream consumption; if this is you, you are in an enviable position in the corporate landscape, and I’d like to have a chat with you.)

This is analogous to adding on a new room or wing to your home, and having to decide what from the various piles should go there…

You aren’t really avoiding the mess.

You’re sifting through it.

The Life-Changing Magic of Tidying Up

Can you implement AI with untidy data spread across disparate systems?

Yes, you can, but it winds up being vastly more expensive to implement.

Worse, it can lock you into a way of working that you didn’t actually want to cement for the next several years.

Don’t let AI become the reason you never address any of your problematic processes or data organization.

In fact, AI is the perfect opportunity to take a long, hard look at your systems and tools landscape and decide if this is how you want your AI guests to see your house.

Also: you’re going to have multiple AI house guests over time… the ones that are here today will be totally different in the future.

The costs of not dealing with untidy data and processes now are only going to be repeatedly borne out and compound over time when you decide you want to switch models, tools, vendors, or strategies later.

Tidy the house.

Then, invite the robots in.

Is It Safe to Write a Blog Post That Is Not About AI?

Ryan Spletzer — Sat, 13 Dec 2025 00:00:00 +0000

Warning: Occasional pithy humor and light-hearted sarcasm ahead.

I have a confession to make…

Lately, I’ve been thinking about writing a blog post that is not about AI.

Is it safe to do this?

Or will it pull me further away from the Light?

The Debate About the Possible

“When a distinguished but elderly scientist states that something is possible, he is almost certainly right.

When he states that something is impossible, he is very probably wrong.”

- Arthur C. Clarke’s First Law

I often let colleagues know that I don’t have a great photographic memory for verbal conversations (and therefore tools like Teams meetings with recordings, transcripts, AI recaps and the ability to ask questions retroactively about the meeting are an absolute game-changer and life-saver for me).

My lack of great conversational photographic memory is definitely a “me” thing, because I have other colleagues who have an almost spooky ability to remember every single aspect of what was said in a conversation—in many ways, those types of folks are their own personal Copilot (whereas for folks like me, I very much need to lean on note-taking and tools and systems to help me stay organized).

I am telling you this because the conversations I do tend to remember the most are ones tied to emotions that lodge themselves into my amygdala.

And because of that, I have a vivid recollection of what I was thinking and how I felt in 2023.

When ChatGPT became popular, the public discourse was starting to rage about what LLM-based AI could and couldn’t do.

Further, people began delving into what AI would be able to do in the future and what would remain in the realm of “Sci-Fi” (at least, in our lifetimes).

2023, frankly, was a bit of a mentally scary time for critical thinkers, because any hint of skepticism about what AI could or couldn’t do was often met with the retort:

“Well, it’s not possible yet.”

This dead end line of discussion, I felt, wasn’t helpful, and further the pushed notion that AI would solve ‘X’—where ‘X’ was, in many cases, an already solved problem with existing technology—often dismissing all the years of innovation and approaches that have been around solving problems for people for many, many years; the humble IF/ELSE deterministic logic has propelled us as a society forward and shouldn’t be tossed out with the proverbial bath water.

What Has Been Possible for a Long Time

“The only way of discovering the limits of the possible is to venture a little way past them into the impossible.”

- Arthur C. Clarke’s Second Law

Machine learning is not new, and has existed in some shape or form for many decades, with papers going back to the 1940s.

Further, computer science and software engineering are not new, and have been deployed for many years in many facets of our existence to help solve real problems and (hopefully, but not always) make our lives better and improve the human condition.

But it always begets the question of what else is possible given the technology we have now, and will have in the future.

In 2025, I’ve seen enough of AI the past couple of years to form some opinions.

And so have others:

“I see no line of sight into AI completely replacing programmers.”

- Mark Russinovich

It is ironic that the relatively mild and reserved “hot takes” about AI technology have become the “provocative” or “controversial” ones.

I would rather be confident in my (rather mild and grounded) opinions and be wrong, and change my mind in the future, than to forever remain waffle-y about what is possible (or not possible).

I also believe, like Mark, that hope is not a strategy, and I am not going to make decisions based on the pure dream that “One day, AI will handle all of this.”

In addition to that, we already have so much existing technology today that has made what used to be considered “the impossible,” possible.

However most of that progress happens by venturing a little bit beyond today’s limits; in stark contrast, we have folks engaging in wild speculation about what could happen way beyond today’s limits, not just about what is possible, but also on what timeframes, and in what ways.

An Alien From Outer Space 👽

“Any sufficiently advanced technology is indistinguishable from magic.”

- Arthur C. Clarke’s Third Law

This debate about the possible versus the impossible happens in many corners of the internet, but typically many technical folks who delve into aspects of AI tend to engage in conversations the most intensely.

Which begs the question: Where does this leave the rest of humanity that may not be technical, and for whom AI is a totally foreign entity?

Many are now trying to (or perhaps, due to fear, in many cases, consciously or unconsciously, trying not to) wrap their heads around this brand new “thing.”

Folks often liken the arrival of modern LLM-based AI to the creation of something analogous to HAL 9000 from the movie 2001: A Space Odyssey.

However, I often wonder if the arrival of AI is less like HAL, and more like the Monolith that appears in multiple parts of that movie.

In many ways it feels like an alien technology showed up in our neck of the woods in this Milky Way galaxy, and everyone is trying to wrap their heads around it.

Another unfortunate byproduct of this is, because AI may feel like “magic” to many people, they pin all of their hopes and dreams onto the technology, and hallucinate things that AI can and can’t do.

The problem is that these human hallucinations aren’t just incorrect—they’re expensive. They turn into roadmaps, budgets, and organizational decisions.

And that’s when the “magic” stops being fun, and often turns into a CFO discussion of “What ROI and/or value are we getting out of this?”

And it’s not because the technology doesn’t have potential, but rather because it was deployed in a “wishful thinking” way that dead-ended either due to very real data or technical or even economical constraints, or perhaps simply due to the fact that it didn’t solve real problems for real people, or a myriad of other potential issues. (This is yet another entire rabbit hole suitable for another blog post.)

Maybe the alien life form not only gave us something perceived as “magic,” but perhaps by extension we contracted a disease from it in the form of “AI Rabies,” where we are metaphorically foaming at the mouth about what is possible and losing our sense of rationality and once again, hallucinate about the possible, or barring that just lose our common sense about what can be helpful to meet real user needs.

Deployed use case issues aside, this is also how you get to one of the weirdest parts of this moment: not the technology itself, but the mythology we’re building around it.

Which is why I flinch a little bit every time I see a certain phrase making the rounds…

Beyond the Infinite ♾️

“This is the worst AI is ever going to be.”

- A trite saying floating around the internet

Dissecting that annoying line could fill up a whole additional blog post. (Is it just the models getting better? And purely through more data? Or is it new type of model techniques? Or is it techniques we use around the models that get better? What do you mean?!)

This utterly useless phrase basically amounts to: “Things improve over time.”

Wow.

Beyond the Infinite(ly) stupid.

Everything tends to improve over time, not just AI…

The humble washing machine has improved over time.

I could cite a ton of additional mundane examples here, but you get the idea…

Bringing it back to an example that is less mundane: Models will improve over time, and model layers and weights are one thing, but few appreciate the immense amount of work that goes into model serving technologies like Ray Serve and other libraries that make models like LLMs actually possible to use, and improvements will be realized in the surrounding technologies that are all absolutely essential to making AI possible to use, but may not get their full due, nor the meaningful amount of mind share to discuss openly.

But the most insidious thing about this idiotic phrase is the simultaneous sense of wonder and awe and existential dread it instills in people while we are all collectively barreling through the psychological daily onslaught of new AI advancements, and it is completely uncalled for and unnecessary…

If it is truly the worst it is ever going to be, I still at this time find AI immensely useful.

But to find AI useful does not automatically negate the utility of everything else in the technology world that has come before it, and is still alive and well and being used today.

My coffee maker at home is useful.

This blog post that I am writing right now and host on GitHub Pages with Cloudflare in front of it is (hopefully) useful, and fueled by 100% organic pure human thought and creativity—the only AI help for this post came from creating alt text for the images, to improve accessibility (which is a great use case for AI).

My dev machine setup scripts are useful in saving days setting up a machine by hand, both for me and for others I know that are using them. (AI helped me write the latest versions of those, but AI didn’t instill the philosophy of design nor the essence of simplicity I wanted to achieve in the latest iterations of them, nor did it provide the years of learning and the acquired taste of what “good” looks like to hone my approach, nor does it perform the actual setup.)

And in consideration of the time span of all these decades of useful technologies that we’ve accumulated, and will continue to accumulate independently of AI, we need to be able to have open ways to talk about them, and not have those discussions be dismissed out of hand because they are somehow “boring” and not aligned with the prevailing AI narrative.

For example, to get MCP to work properly, you have to build up an understanding of a web of interrelated IETF specifications around OAuth and adjacent technologies, which to some may feel “boring,” but for us engineers, is essential to produce something that is not a walking security hazard with no auth (or dubious auth) that vibe coded its way out of its containment zone of Lovable or Replit.

In fact, most of the time I’ve spent at work with our teams delivering AI for the enterprise involves using technologies and techniques around data engineering and data science and automation and full-stack software engineering and cloud infrastructure (and more) that have nothing to do with AI whatsoever.

Thankfully, I don’t think I will be smote by a bolt of lightning if I write a future blog post that has nothing to do with AI.

(And I won’t be smote for this one because by my count there are 38 mentions of the word “AI” in this blog post, including the one in this sentence—but please count for yourself, because I am a human that can make mistakes!)

AI didn’t create our current set of technology—rather, our existing technology helped us create AI.

The introduction of AI is additive, not subtractive, to the existing technology we have.

And, coupled with existing technology, AI is more than likely going to help us do amazing things in the future.

With this in mind, maybe, just maybe, the mental model of how we got here, and where we’re going, needs to be thought about a little bit differently, and should allow more space for more discussion about all types of technology, not just AI.

And perhaps we can try to take some things a little less seriously and remember to have a little bit of fun along the way, too—so in the spirit of fun, let’s flip this script:

Visualizing the OAuth & OpenID Connect Spec Graph

Ryan Spletzer — Sun, 09 Nov 2025 00:00:00 +0000

I have a solid grasp of OAuth 2.0 and OpenID Connect (OIDC) from past projects, notably from going heads-down and reading the core specs for those protocols to implement a federated identity provider with open-source frameworks. (It was a doozy.)

But then along came the Model Context Protocol (MCP)—an emerging standard for AI agents that leans on OAuth 2.1 and various OAuth extensions.

Suddenly I was up to my eyeballs in specs again: dynamic client registration, PKCE, you name it. Each spec I opened would reference two more that I “should” go read.

Before long, I had a cascade of browser tabs open and was struggling to keep track of how all these documents fit together.

If you’ve ever tried to implement OIDC/OAuth, you probably know the feeling—a specification sprawl where every RFC and spec points to another, and you’re left juggling a dozen tabs in search of the full picture.

In my case, the trigger was MCP. It adopts OAuth 2.1 for authorization, which meant revisiting the OAuth core spec (RFC 6749) and many related ones. Reading the MCP spec led me to open the OAuth Dynamic Client Registration spec (RFC 7591), which led me to the OAuth 2.0 Authorization Server Metadata spec (RFC 8414), which led me back to OIDC Discovery… you get the picture.

Even experienced developers can find this overwhelming—it’s easy to lose track of which specs are fundamental, which are optional extensions, and where to even begin.

Trying to navigate the OAuth/OIDC ecosystem through official specs alone often results in “browser tab explosion.” For example, you start with the OpenID Connect Core spec, which tells you it’s built on OAuth 2.0 (so off you go to read RFC 6749). OAuth 2.0 in turn mentions bearer tokens (RFC 6750), so open that too. OIDC also relies on JSON Web Tokens (JWT, RFC 7519), which themselves rely on JOSE specs like JWS, JWE, and JWK—there go three more tabs.

Pretty soon you’ve got 10+ spec documents open side by side, and you’re doing a lot of mental context-switching to piece together how they all relate.

This spec overload isn’t just an academic problem; it hits practical questions:

Understanding dependencies: Which specs build on which others? (e.g. does implementing feature X require reading another spec?)
Finding the right document: If you need to implement Proof Key for Code Exchange (PKCE), do you read the OAuth 2.0 core spec, an extension RFC, or something in OIDC?
Not missing anything important: Are there security best-practice specs (like OAuth 2.0 Security BCP) or extension drafts that you should be aware of?
Teaching or documentation: How do you explain this tangle of specs to someone else without their eyes glazing over?

In short, the OAuth/OIDC standard family is rich but complex, and it’s easy to get lost in the cross-references. I personally hit a point where I knew there had to be a better way to see “the big picture” than control-tabbing between RFCs.

The Solution: A Visual Map of the Specs

To tackle this, I decided to map out the spec ecosystem visually.

Instead of manually drawing boxes and arrows (and inevitably forgetting some), I leveraged AI coding assistance to help build a Mermaid diagram of the specifications and their references to each other.

Mermaid lets you write text-based diagrams, and GitHub can render them natively. I fed my assistant (think ChatGPT/Copilot) an initial list of the OAuth, OIDC, and JWT specs, and asked it to help generate an initial graph definition, then continued to add more ancillary specs from there.

The result is an “OpenID Connect and OAuth Specification Graph” that now lives in a GitHub repo.

This visual graph shows most of the major specifications in the OAuth/OIDC universe and draws arrows to represent their reference relationships. I grouped the nodes by category to make it clearer:

OAuth 2.0 Core & Extensions – the base framework (RFC 6749) and its many RFC extensions (PKCE, token revocation, device flow, etc.).
JOSE/JWT specs – the JSON Web Signature/Encryption/Key standards and JWT itself, which provide the cryptographic backbone for tokens.
OpenID Connect – the identity layer specs (OIDC Core, Discovery, Dynamic Client Registration) built on top of OAuth 2.0.
UMA (User-Managed Access) – an OAuth-based protocol for user-controlled resource sharing.
Miscellaneous/Modern – newer drafts and best practices (OAuth 2.1, Security BCP, DPoP, etc.) that integrate or update the above.

Crucially, each node in the graph is clickable, linking directly to the official spec document or RFC text.

By visualizing the web of specs, this fun weekend project aims to turn that intimidating tangle into a navigable map.

For me, this has already been a huge help in regaining context quickly when diving back into OAuth land to read up on the dynamic client registration and other specs referenced by MCP, after having read the core OIDC and OAuth and JWT specs many years ago…

Building the Spec Graph (with a Little AI Help)

One fun aspect of this project was using AI pair-programming to build the graph.

Hand-coding a Mermaid diagram with ~50 nodes and dozens of arrows would be tedious (and error-prone) to do solo.

Instead, I iteratively prompted a coding assistant to help generate the Mermaid markup.

Using AI in this way felt like having a knowledgeable co-author: it could recall obscure RFC numbers and the relationships mentioned in their text, which saved me a ton of manual cross-checking.

This was a great reminder that LLMs didn’t make understanding specs obsolete—but they did make assembling reference materials faster. Instead of manually collating references from dozens of documents, I could focus on validating and fine-tuning the output.

The AI assistance turned a daunting documentation task into something almost fun.

Mapping the OAuth/OIDC Ecosystem: Key Relationships

So what does our spec graph reveal?

At a high level, it confirms the layered architecture of modern auth standards. Here are some of the key relationships visualized:

OAuth 2.0 is the foundation – Nearly every other spec extends or references the core OAuth 2.0 framework (RFC 6749). If you’re dealing with tokens or authorization flows, OAuth2 is the base context.
JOSE is the cryptographic foundation – JSON Web Signature, Encryption, Keys, etc., and JWT all provide the security underpinnings. Many higher-level specs depend on these for signing/encrypting tokens.
OpenID Connect builds on OAuth + JWT – OIDC uses OAuth 2.0 for its authentication flows (delegating login via authorization code, etc.) and uses JWT as the format for ID Tokens (the bit that carries user identity). In other words, OIDC is an identity layer over OAuth.
UMA extends OAuth – The User-Managed Access 2.0 specs build on OAuth 2.0 (and use bearer tokens) to enable a resource owner consent workflow that goes beyond simple scopes, allowing users to delegate fine-grained access to their data.
Newer extensions mix and match – Recent drafts and extensions often combine pieces of the above. For example, JWT access tokens (OAuth RFC 9068) marry OAuth with JWT/JOSE, and the upcoming OAuth 2.1 consolidates core OAuth 2.0 with required security best practices.

These relationships help illustrate why certain specs exist. Some fill gaps in the core OAuth 2.0 spec (like additional security, token management, or special flows for devices), while others build higher-level functionality on top of OAuth (like federated identity in OIDC or user-mediated sharing in UMA).

With one glance at the graph, you can glean which specs are core vs. peripheral, and how they logically group together. This is super helpful when planning what to implement or what to study next. Instead of treating the specs like an amorphous laundry list, you start to see an architecture emerge: a few foundational layers, and many optional modules on top.

Beyond Tabs: Gaining Insight and Context

Ultimately, this spec graph isn’t just about avoiding browser tab overload—it’s about building a mental model of the OAuth/OIDC landscape.

Instead of treating each RFC or spec as an isolated piece of the puzzle, you start to see how the whole puzzle fits together.

This has several benefits:

Faster learning: If you’re new to these standards, a visual map helps you decide where to begin (perhaps start with OIDC to see a full use-case, then drill down into OAuth 2.0, then JWT… rather than reading specs in a random order).
Implementing with confidence: When working on a feature (say, adding device login to your app), you can consult the map to ensure you’ve pulled in all the relevant specs (Device Authorization Grant, PKCE, maybe OAuth Security Best Practices) and understand their dependencies.
Better communication: As a developer or architect, you can use this diagram to explain the ecosystem to colleagues. It’s a lot easier to justify “we need to support spec X” when you can show how X connects to the standards you already use.
Keeping up to date: The OAuth/OIDC world is still evolving (OAuth 2.1, new best practices, etc.). A living graph can be updated, so you can visually track new additions or changes in the framework over time.

I open-sourced the OIDC/OAuth Spec Graph with the hope that it will be a useful reference for others, not just myself. If you think I missed an important spec or got a relationship wrong, I’m all ears—feel free to open an issue or PR on the repo.

What started as a personal exercise to manage my own spec overload has turned into a shareable artifact.

I’ve found that understanding OAuth, OIDC, and their related specs becomes much easier when you can see the forest rather than just the trees.

After all, the goal is to make this stuff accessible—it shouldn’t require a PhD in browser tab management to learn web authentication standards.

Sometimes the best way to untangle complexity is to visualize it.

Pinocchio is Not a Real Boy

Ryan Spletzer — Sat, 06 Sep 2025 00:00:00 +0000

LLMs didn’t make code literacy optional—in fact, quite the opposite: they raised the bar for what you need to learn so that you can effectively steer these AI assisted tools.

If anything, you’re going to have to study programming languages, frameworks, and systems design more going forward, not less.

Because when a model spits out a slick-looking solution, you still need to actually understand it.

And if you’re going to ship that solution, you need to be able to reason about it when something breaks at 2 A.M.

Or worse, when it gets hacked.

And when that happens, you’ll have no one to point the finger at except yourself.

“This shouldn’t be possible”… and yet, it is

I’m not a frontend specialist.¹

I know HTML, CSS and enough raw JavaScript with accompanying design patterns to be dangerous, and I know enough about the fundamentals of the web and about tracing down errors and bugs with browser web developer tools to even identify and fix a problem; but most of my web work is from 12+ years ago—back when SPA frameworks were still figuring themselves out.²

And yet, despite my limited hands-on expertise in frontend development, I recently used LLM-assisted development to contribute fixes around unnecessary auth redirects to the identity provider from msal.js in a team’s React/TypeScript frontend project, and got them past the issue.³

That ability to contribute to something where I don’t even have strong enough expertise nor muscle memory to do it by hand—with or without a lot of googling—is the magic of AI-assisted code generation.

And it is also the trap.

This generated code came (largely) from an LLM-enabled software process, that was carrying out my directives.

It didn’t come from an intern or a junior developer or a non-technical person, but it could have easily also come from anyone with any background, provided they knew the right questions to ask…

Despite knowing those right questions to ask, and despite my very careful crafting of the prompt to describe precisely what I wanted, and despite giving the tool a “short leash” by doing scoped edits in the relevant files, I still found myself spending an undue amount of time reviewing the diff of the output that came out of my prompt for this issue.

I realized through this process that suddenly, I was my own worst enemy.

I had to study the output ~10x more carefully precisely because it felt “too good.”

It turns out, the assistant made a couple of mistakes—of course it did—and only by reading the code, asking the right questions, and generally just having a gut feeling that “something is not right here” (“Why did it do that?”), did I catch the issues involved.

The lesson here is: AI coding assistance accelerates you very quickly into terrain you don’t fully know or understand.

And because of that, your only safety net is your willingness to continuously learn fast and verify everything.

You vibe it, you run it

There’s a line I like from a recent blog post from Uptime Labs: “You vibe it, you run it.”

I’m not here to out-write nor out-research that original essay; I’m here to double down on the core points with lived experience.

If you “vibe code” something you don’t understand and then hand it off, no one truly understands it.

At best, the next team inherits a black box with nice comments.

At worst, they inherit a fire.

Yes, you can ask an LLM to explain a codebase to you.

You can be diligent with Cursor or GitHub Copilot rules, ensuring that it always generates docstrings and Mermaid diagrams (token consumption galore).

Some people will do that.

Most inexperienced vibe coders won’t—at least, not yet.

The folks who will do well and enjoy continued success in this era aren’t the ones who ship the first demo (which will become, on some level, commoditized to the point where anyone can do that); they’re the ones with endless curiosity who keep learning between prompts and after the demo.

Because of this, I would really discourage using Max / YOLO Mode for everything and not taking the time to understand what has been generated afterwards.

I’ve dabbled in digital photography in the past, and I’ll never forget what my friend’s dad (a professional photographer) said to me once: “You know, the thing is, the more shots you take, the more you have to go through and review later.”

I learned that lesson early on when I was shooting photos with my DSLR, and it’s even more of a prescient lesson now as I’m a few years older and generally have less time to afford to give to endlessly reviewing and editing photos; whether it’s a nice mirrorless camera or my phone’s camera, I’m much more deliberate about what I snap a photo of now.

In that vein, I recommend being much more deliberate about the “shots” you’re taking with these coding tools.

There are times when you want to generate something quickly from scratch as a starting point, and letting the agent take over the whole project may be warranted in that initial prompt.⁴

Maybe the thing you’re generating is just for you and is simply a tool you’re going to run locally and may even throw away later—fine.

When things get “real” though, it’s a different story.

You have to understand that the universe has enough entropy as it is, and you’re not helping the situation by “pooping out” drivel and nonsense.⁵

When you’re trying to bring something to life that may be used by others and perhaps eventually put into a production setting, I recommend orchestrating the LLM with a sliding scale of autonomy: ask relevant questions to the tool first to gain an understanding, then based on that understanding try some approaches that tightly scope the autonomy of the agent as possible for the problem at hand, commit often so you have coherent states to roll back to (and you can leverage the tools here by crafting more detailed commit messages, too), take some suggestions from the tools; but sometimes, you need to let the tool sit there quietly while you read.

In any case, you own the responsibility of comprehension afterwards.

The toy that went to production

Every day I see vibe-coded projects deployed to the open internet with:

no authentication
no authorization
unvalidated inputs
secrets in plain text
no TLS termination at the edge
no proper LB/WAF/DDOS protections
zero observability (no logs, no metrics, no traces)
no rollback plan

In toy cases with silly frontends, maybe this doesn’t matter.

The problem is people are treating toys like production.

Pinocchio is not a real boy, and his nose gets longer every time he promises you that his created puppet show is “good enough for prod.”

Your creation will gather dust on a shelf—or worse, you and Pinocchio will get swallowed by a whale, in the form of your neighborhood friendly script kiddie that’s taken over your website just for the lulz.

Study more, not less

I’m going to keep studying languages, frameworks, auth, networking, zero trust, systems design patterns—because the work that actually matters isn’t a weekend toy.

It’s lived in the messy world of legacy constraints, enterprise identity, real users, and uptime.

Knowing what you want counts for a lot.

Knowing what it takes to make it real counts for everything.

Quality is recognizable—even to your contractor

You’ve likely had a contractor open a wall at your home residence at some point and say, “what the f— happened here?”

That’s the sound of shortcuts and cut corners hitting their expiration date.

You also know the feeling of opening a new iPhone and everything just… works.

There’s a gap between “assembled parts” and “integrated product.”

This is analogous to asking your buddy to build you a tower PC when you have zero knowledge of how to build one yourself: it sounds like a great idea and looks wonderful, until a random part dies, then suddenly you’re dependent on that one friend—or a repair shop—because you don’t know how to replace the part yourself.

A family member of mine is a commercial pilot with decades of experience, and in recent years he has taken up the hobby of flying small drones, and while he knows it is on some level a fun toy, on another level he takes the FAA rules around drones very seriously.

Further, he will likely never lose his drone in a lake like you’ve seen in those many funny YouTube videos.

He enjoys flying drones, but at the same time he does not trust them like a real aircraft.

He understands capability versus consequence.

We should, too.

The enterprise “intern effect”

Even before the AI craze and hype hit, at multiple companies, I’ve long seen non-software engineering business teams hire a summer intern to build something the org won’t prioritize—and on the surface, this is often a perfectly reasonable move to explore.

And the intern might do a great job for their level.

But because the team around them has no software engineering experience, they don’t understand the company’s ecosystem, identity model, deployment standards, data policies, etc.

When the internship ends, so does ownership of the project.

And the team is left holding the bag, desperately asking other teams if they can take what the intern built (which due to lack of guidance is nowhere near being production-ready) and “host” it somewhere besides the local laptop it was built on…

That pattern is now widespread—a director of marketing with no programming experience can now vibe code a solution into reality (and even commit it to git—I’ve seen it happen!) and the “handoff” from the coding tool to person with the idea is immediate.⁶

It’s like everyone now has their own personal intern, who can do an entire intern project in a day.

But it turns out, the intern is a model, and much like a real-life software engineering intern embedded in a team of non-software engineers, there probably wasn’t the right guidance available during those vibe coding sessions…

If you don’t build the muscle of understanding, you get all of the demo and none of the durability.

Yes, you still have to study

Notice the wave of new O’Reilly-style books and courses popping up: “Build X with Y,” “Web Apps with Bolt,” “Secure LLM Apps on Z.”

If you thought “AI means I don’t have to study,” think again.

The material is evolving because the work is evolving, and the work still rewards people who can read code, reason about systems, and make good trade-offs.

We can all experiment.

And we should.

But experiments don’t relieve us from the burden of understanding.⁷

Pinocchio can sing, dance, and put on a great demo.

Just remember: he’s not a real boy.

He’s not even a puppet with sentience—just a wooden marionette where you are pulling the strings and telling him the songs to perform.

And his nose grows longer with every lie he tells you about the production-worthiness of the things you are vibing into existence.

If the songs he sings are poorly written and out of tune, no one will be willing to pay for the puppet show (or give you the time of day to put the show on in the first place), and you will have no one to blame but yourself.

In this way, vibe coding (and to some degree, using AI in general) is very much like holding a mirror up to yourself.

Yes, it will allow you to do things you never thought possible.

But it will also reflect back the gaps in your own understanding, especially if you lack the self-awareness to recognize your own weaknesses that prevent you from fostering curiosity and asking the right questions.

Just because you can will something into existence, does not mean that you are absolved of personal responsibility over what you created.

If you want to build something you can trust, you still have to do the work—study harder, continuously learn, stay curious, give the AI a shorter leash (when possible), and own what you vibe.

Footnotes

But I’m working on it. ↩
Sidebar, for the continous learners out there, Scalable and Modular Architecture for CSS will change your life. It certainly might not be the most modern tome on CSS best practices, but it certainly changed my entire perspective on what maintainable CSS could look like when I read it a number of years ago. ↩
Really, a large part of contribution to that frontend fix was in large part my deep, years-long expertise in OpenID Connect/OAuth 2.0/JWT tokens, etc., but I digress. ↩
I find spec-driven development fascinating in this regard, because the act of forcing yourself to write out a spec goes a long way towards helping to craft your understanding of the problem, and the code that follows—but this still doesn’t absolve you of understanding the generated output. ↩
I didn’t invent the “pooping out code” phrase—many years ago, far before the dawn of AI coding assistance, I had a colleague who enjoyed scaffolding tools a lot and wanted the tools to just “poop out the code for me.” ↩
I’ve seen these patterns play out before, too, with low-code/no-code tools and the notion of “citizen developers”—even if something is created that is well-done, at a certain point I have often observed that the citizen developer doesn’t want to maintain the thing they have built indefinitely, and really wants to hand it off to some other group to run and maintain. ↩
What’s so funny ‘bout peace, love, and understanding? ↩

Ask vs Act: Applying CQRS Principles to AI Agents

Ryan Spletzer — Sun, 17 Aug 2025 00:00:00 +0000

Asking an AI agent a question is a whole different ballgame from letting it take action.

Imagine chatting with an AI assistant that not only answers your questions but can also perform tasks on your behalf. For example, you ask, “What’s our PTO policy?” and get a helpful answer—and then you say, “Please submit my PTO request for next week,” and the agent actually goes off to do it. These two requests might seem similar (both are things you ask the AI), but under the hood they are very different operations. In software architecture, we have a name for recognizing and handling this difference: CQRS, which stands for Command Query Responsibility Segregation. CQRS essentially says we should split the world into queries (reads) and commands (writes) and handle each with its own pathway.¹ In this post, we’ll explore how CQRS concepts apply to building AI agents—and why retrieving information (think of Retrieval-Augmented Generation, or RAG) versus taking actions (executing some operation) need to be treated differently at every step.

What is CQRS?

CQRS is an architectural pattern that separates the responsibility of reading data from writing data. In traditional systems, you often have one architectural stack or service handling both reads and writes, which can lead to all sorts of issues—performance bottlenecks, scaling difficulties, and conflicts between read and write workloads. CQRS addresses this by using different architectural models (or even different services and data stores) for writes versus reads. In other words, the part of your system that processes commands (things that change state) is kept separate from the part that handles queries (things that only retrieve data). This allows each side to be optimized and scaled independently. For example, the write side (commands) can focus on transactional integrity and business logic, ensuring every change follows the rules and is consistent, while the read side (queries) can be optimized for fast lookups and flexible querying, often using techniques like caching or denormalized view models. The result is a more robust system: each half does one job and does it well, without stepping on the other’s toes.

Why does this matter for AI? Modern AI agents are, at their core, software systems that both answer questions and perform actions. If we draw a parallel, an AI agent answering your question (“What’s the PTO policy?”) is akin to a query—it’s reading from some knowledge source and returning information. But when that agent performs an action (“Submit PTO request”), it’s like a command—it’s trying to change the state of some system (your HR system in this case). The CQRS principle tells us it’s wise to handle those two types of operations separately. Just as in a well-architected app we wouldn’t use the exact same process to both fetch data and update a database, in an AI agent we shouldn’t treat information retrieval the same as we treat action execution. We’ll delve into exactly how to separate these in an AI context, but first let’s break down the two sides: the read path (our AI agent’s knowledge retrieval, often using RAG) and the write path (the agent’s action-taking via tools or APIs).

The Read Path: Retrieval-Augmented Generation (RAG)

When an AI agent answers questions or gathers context, it’s operating on the read side of the world—no different from a user running a search or query. A popular approach for AI agents to answer user questions is Retrieval-Augmented Generation (RAG). RAG is essentially a two-step process: first retrieve relevant content from your knowledge sources, then have the AI model generate an answer using that content. In practice, this means the agent might take your question, use it to look up documents or data (for example, searching a vector database of internal docs, or hitting a knowledge base API, perhaps via an MCP proxy endpoint), and then compose a response based on what it found. Crucially, this read process does not alter any system state—it’s pulling information, not changing information.

Because reads are read-only, we have a lot of flexibility in where the data comes from. An AI agent’s retrieval step can tap into all sorts of sources:

Operational data sources: Sometimes the freshest or most authoritative info lives in operational systems (e.g. an airline reservation system for up-to-the-minute flight status). In fact, many real-world RAG setups use a hybrid approach that draws on structured operational data (like databases, CRMs, or transactional systems) to fetch precise facts (customer records, inventory levels, latest transactions) in addition to unstructured docs.²
Analytical or external knowledge bases: Other times the agent might query analytical data products or data warehouses—for example, to get aggregated metrics or historical trends—or use a vector search index populated with company documents for semantic search. The key is that for reads, it’s fair game to pull from either operational or analytical stores. (But there are considerations as to when to go to one versus the other.) The agent could query a Snowflake dataset for last quarter’s sales number and also query a vector DB for the latest policy document text. As long as the data is accessible (and not too stale to be irrelevant), the agent can use it.

One thing to keep in mind is that the quality of these read-side sources directly affects the agent’s output. If the underlying knowledge is outdated or disorganized, the AI’s answer will suffer—garbage in, garbage out. AI will confidently return an outdated policy as an answer because an old document hadn’t been marked obsolete in the knowledge index. So, even though reads don’t modify anything, they do require us to think about data curation and freshness (much like any search system). But from a system design perspective, reads are the “easy” part: they’re fast, parallelizable, and forgiving. You can cache results, pre-compute indexes (which in a way is analogous to embeddings for vector search), and generally throw engineering tricks at making reads as efficient as possible. If a read query hits a slightly stale replica, usually it’s not the end of the world (but this depends on the scenario)—you might just get a slightly outdated answer, which can be corrected next time or with a refresh. This tolerance is one reason CQRS often allows the read model to be eventually consistent with the write model. The takeaway is that an AI agent’s RAG-based retrieval is its query model—and we can scale and optimize that independently from any action-taking logic.

The Write Path: Taking Actions (Commands with Consequences)

Now let’s talk about the more consequential side of the house: when an AI agent takes an action that changes something in the world. This is the write path, analogous to the command side in CQRS. In an AI agent context, a “write” usually means invoking some tool or API call that performs an operation—booking a meeting, creating a support ticket, updating a database record, sending an email, you name it. Unlike a read, which might draw on multiple sources, a write typically must go to a specific operational system—the system of record that actually accepts the change. If our agent is to submit a PTO request, that needs to happen in the HR system (the authoritative operational source for PTO data), not in, say, a reporting data warehouse. Writes have to hit the source of truth so that the action is real and persistent.

(As an aside: lately I’ve also been referring to the source of truth using an alternative term that may be familiar to those who know DNS: the source of authority. DNS records have a source of authority on the internet—often at a domain registrar—and DNS records are propagated out to other DNS servers on the internet. All the requests for DNS queries to any DNS server return records that are “true,” but only the origin DNS servers are the “source of authority,” represented by SOA records in the DNS system. When a change is made in a DNS record at the source of authority, there is a brief period of time where that change needs to propagate out to other servers, and eventually it becomes consistent throughout the internet. Relating this back to data: I find there is often confusion around what data is “true.” Data may very well be replicated to multiple places in a given ecosystem, very typically to analytical data platforms—and the data at all those locations all is typically “true”!—however, there’s only one place where the data is “authoritative.” Inside the words “authoritative” and “authority” is the word “author,” so another way to think of this is: the source of authority for data is typically where the data is authored, and that typically makes it the “authoritative” source of the data. Following this, much like DNS records, when an authoring change is made to data, it takes a little bit of time for it to eventually replicate and become consistent in a given ecosystem, typically through combinations of periodic ELT jobs or event-driven syncs.)

Because actions carried by agents actually change the state of our systems, they come with a lot more responsibility and required care:

Business Logic and Validation: Operational systems don’t (and shouldn’t) allow just any change willy-nilly. There are business rules and validations in place. For example, submitting a PTO request might require that the dates don’t clash with a holiday, or that the user has enough PTO balance, etc. An AI agent needs to respect those rules. If you simply feed the LLM a text instruction “ensure the user has enough PTO days” and hope it does the right thing, you’re taking a big risk. As industry experts have noted, relying on an LLM alone to enforce complex business policies is risky—the model might misunderstand the rule or be manipulated by a crafty prompt. Therefore, the write path often involves explicit checks or calls to validation services. In a sense, the write model of our AI agent includes this deterministic guardrail logic to make sure the action is valid (just as a traditional write model in CQRS would encapsulate all the business rules for a transaction).
Confirmation and Safety: Another key difference with writes is the potential impact of getting it wrong. A bad read (e.g., mis-answering a question) might confuse or annoy a user, but a bad write could create real damage—think of an agent accidentally ordering 1000 units of something or deleting the wrong user account. 😬 For this reason, it’s wise to confirm actions with a human before executing them, especially in a chat or interactive setting. Many agent frameworks build this in. For instance, Amazon’s agent toolkit suggests requiring the user’s explicit confirmation before an agent executes certain functions, precisely to safeguard against malicious or unintended commands.³ In our PTO example, the agent should probably respond with “I can submit a PTO request for you from Sept 25 to Sept 29, 2025. Shall I go ahead and do that?”—only on a “Yes” or a confirmation through a button click should it actually call the PTO API. This aligns with a broader principle: no autonomous writes without user oversight (at least not until we’re very confident in the agent’s judgment). In non-chat scenarios, this might be implemented via a human-in-the-loop queue or approval workflow for agent-initiated actions.
Transactional Integrity: When the agent does perform the action, it needs to handle success or failure robustly. The operational system might reject the request (e.g., if the user didn’t have enough PTO days, the HR API will return an error). The agent’s write path has to be ready to catch errors, possibly roll back or try a different approach, and inform the user gracefully. This is analogous to how a well-designed command handler in a system ensures that either the transaction fully succeeds or it cleanly fails and reports back. In an agent, you don’t want it to just silently fail or, worse, assume success when the operation actually didn’t go through.

All these factors mean the write path is usually more complex and needs tighter control than the read path. In CQRS terms, the command side is optimized for correctness, consistency, and safeguards, not raw speed. We often accept a bit more latency or friction for writes (like an extra confirmation step or a validation call), because the priority is doing the right thing and maintaining system integrity. Indeed, one of the benefits of classic CQRS is that your write model can be built with all the heavy checks and balances needed, while your read model remains snappy for queries. We see the same in AI agents: the retrieval (reading) can be free-flowing and creative, but when it’s time to execute an action, the agent should switch into a more controlled, rule-abiding mode.

Designing AI Agent Architecture with Separate Read/Write Pipelines

Embracing CQRS in an AI agent doesn’t necessarily mean you literally have two completely separate codebases, but it does mean architecting the agent’s “brain” to have a clear split between reading from knowledge versus acting on the world. Practically, this often translates to using different modules or components in your agent system. For example:

You might have a retriever module (or chain) that handles all RAG-related tasks: embedding the query, searching the vector database or other data sources, and returning supporting facts.
Then a separate action module (or planner/executor) that decides when and how to invoke tools or APIs to make changes, and handles the responses from those actions.

Some emerging frameworks make it easier to build this kind of separation. One example is LangGraph⁴, an open-source library built on the core of LangChain⁵ that lets you define agent workflows as graphs. With LangGraph, you can explicitly model the agent’s decision process as nodes and edges—for instance, one branch of the graph could be a “Retrieve knowledge” step and another branch a “Perform action” step, each with its own logic. Crucially, LangGraph was designed with human-agent collaboration in mind: it enables agents to write drafts for review and await approval before acting, and makes it easy to insert human-in-the-loop checks for any action. In other words, it provides a structured way to say “this part of the agent’s workflow is a query (no approval needed), and this part is a command (pause and get approval)”. Even if you don’t use such a framework, the concept is clear—treat the action-taking part as a first-class concern, with its own state and control flow, separate from the Q&A part.

Another consideration is data flow between these components. Often the result of the read path will inform the write path. For example, if the user asks, “Order more laptops for the team,” the agent might first do a read step: query an inventory or procurement knowledge base to see what the current stock and policy is. Based on that, it forms a plan and then triggers the write step: calling the procurement system’s API to place the order. By separating these concerns, you can even choose to run them on different infrastructure—the retrieval might run against a vector search service or analytical database, whereas the action might run via a secure connector to the operational system. (It should go without saying that reading from an out-of-date analytical database may be a bad idea if you need that data to be accurate up-to-the-minute for the subsequent write operation; therefore in agents with action capabilities, it may be better to read from the operational system first before performing the action, but other standalone read prompts may be just fine reading from an analytic system. Use your best judgment here, and always do a “sanity check” read of the operational data store before doing a write!) In a way, this mirrors the idea of having separate read and write databases in advanced CQRS implementations⁶ (for instance, a denormalized read store vs. a normalized transactional write store). In AI agents, you might have a vector DB + cache for reads and a set of authenticated API clients for writes.

Key Differences Between Reading and Writing (RAG vs Actions) in Agents

To summarize the lessons of applying CQRS to AI agents, let’s highlight the key differences and best practices when handling “reads” versus “writes” in an agent:

Data Sources and Freshness: Reads (RAG) can draw from a wide array of sources—from real-time operational databases to analytical warehouses or indexed documents—to gather context. Writes (actions) must target the authoritative operational system for the task at hand. If an agent needs to record or change data, it has to go through the system-of-record (be it an HR system, CRM, etc.), not a read-only cache or replica (obviously). This ensures the action is reflected in the source of truth. (In short: read from anywhere that makes sense, but write only to where it truly counts.)
Performance vs. Integrity: Reads are optimized for performance—low latency, high throughput, and often eventual consistency is acceptable. You might use denormalized data or briefly stale indices to answer quickly. Writes prioritize integrity and correctness over raw speed. It’s better for a write action to be a bit slower or go through checks than to be fast and wrong. The write path should include all necessary validation and follow the business process exactly (e.g., a “Book hotel room” command goes through all the steps a proper booking requires).
Business Logic Enforcement: Reads typically involve minimal business logic—mostly filtering or formatting data—since they don’t change anything. Writes carry the full weight of business rules. The agent’s action module needs to enforce policies and rules either by invoking back-end validations or through explicit logic. As noted, don’t trust the LLM to enforce all rules implicitly. Instead, encode critical rules in code or use external validators to ensure compliance (for example, require certain fields or approvals for specific actions). Some companies are even developing “policy engines” for AI agents to reliably translate high-level rules (“User must have VP approval to spend over $10k”) into checks before an action executes.⁷
Error Handling and Feedback: On the read side, if something goes wrong (say a data source is unreachable), the agent can often fail gracefully—maybe answer “I don’t have that info right now,” or just produce an answer with what it has. On the write side, failures need careful handling. The agent should catch errors from the tool/API (e.g., insufficient permissions, validation failed, network timeout) and decide on next steps: maybe ask the user for a different input, try an alternate method, or at least report the failure. In essence, write operations in an agent should be treated like transactions—with commit/rollback semantics or compensating actions if possible (this is analogous to how a robust CQRS command side might use techniques like sagas⁸ to handle failures, especially for very complex actions that could span multiple systems). One of your agent’s actions will fail at some point, whether it’s due to an upstream outage or a transient failure, and you want to ensure you leave your systems in a consistent state, and also be able to show the best possible error message you can to the user, and further see if there is a graceful way to provide them an alternative method for carrying out their task. For example: if the agentic action fails, maybe you can provide the means to retry if it was a transient error, or maybe you can redirect the user to a viable non-agentic alternative, or send them to a relevant support channel, connect them with a human, or many other potential alternatives.
User Confirmation and Trust: Reads can usually be executed automatically without user intervention—you don’t need to ask permission to look something up (aside from access control concerns). Writes should earn trust. It’s a good practice to involve the user when an agent is about to do something non-trivial or irreversible. As we discussed, a simple confirmation step (“Are you sure?”) goes a long way. In more complex workflows, a human reviewer might need to approve a batch of actions the agent wants to perform. The goal is to keep a human in the loop for oversight, until we’re as confident in the AI as we are in autopilot flying a plane. 😉 Even advanced frameworks encourage this—for example, LangGraph allows inserting moderation or approval nodes so that certain agent actions pause and wait for a human to OK them. This not only prevents mistakes or mischief but also improves user confidence that the AI isn’t running wild.

By acknowledging these differences, we essentially set up our AI agent with two mental models: one for retrieving knowledge and one for executing decisions. This duality is exactly what CQRS preaches, applied to the realm of AI-driven systems.

Conclusion: Embracing the Separation of Reads and Writes in AI Agents

The main takeaway here is simple but profound: asking and acting are fundamentally distinct operations, and we should design AI agents with that distinction front and center. Borrowing the CQRS mindset helps because it reminds us to use the right tools and safeguards for each job. An AI agent leveraging RAG is fantastic at pulling in information (sometimes from all corners of your enterprise data), but that same agent needs a very different approach when it comes time to do something with that information. In practice, this means building agents that have clear “query modes” and “command modes,” with appropriate pipelines for each. Reads can be liberal—grab whatever data might help, combine it, summarize it—while writes must be conservative—follow the rules, double-check, get approval, log everything.

As AI agents become more powerful and autonomous, this separation will only grow in importance. We’ve all seen the hype around agents that can plan and execute tasks; it’s exciting, but without a grounded architecture it can also be a recipe for chaos. By treating reads and writes differently (like different species in the AI ecosystem), we ensure that our agents can be both useful and trustworthy. They’ll give great answers and perform reliable actions, without confusing the two. In essence, we’re teaching our AI agents a lesson that seasoned software architects have known for years: there’s a time to observe, and a time to act—and handling each properly makes all the difference. Adopting a CQRS-like approach for AI won’t stifle their abilities; on the contrary, it will let us scale up their knowledge powers and automation powers side by side, safely and effectively.⁹ So the next time you design an AI agent, remember to ask yourself: Which parts of this are queries, and which are commands? By designing with that question answered, you’ll be well on your way to an agent that can both speak intelligently and act responsibly.

(For more information on the patterns in this space, I encourage the reader to dig into the following software engineering principles, in this order: SOLID, Domain Driven Design, CQRS and Event Sourcing. These principles build on each other and will help you to find your way through building these complex agentic systems. Some of the best books on this subject that bring this all together are oldies but goodies, and I have them listed on my linkfarm page: Adaptive Code: Agile coding with design patterns and SOLID principles (2nd Edition) (Developer Best Practices) and Microsoft .NET - Architecting Applications for the Enterprise (2nd Edition) (Developer Reference). Whether or not you’ve worked in the Microsoft ecosystem and with the languages described within those texts, the patterns are relevant to many languages and frameworks and software stacks.)

References

MCP is a USB Port, Not a Hard Drive

Ryan Spletzer — Sun, 03 Aug 2025 00:00:00 +0000

Or:

Why your “just grab every message in this Slack channel since 2017” request is going to hurt.

A funny thing happens any time a new standard or protocol lands in the AI world: people assume it magically solves everything upstream and downstream of it. Anthropic’s Model Context Protocol (MCP) is the latest occurrence of this. I keep hearing versions of:

“Sweet, so with Slack MCP I can just ask Claude to give me all links anyone ever posted in the #eng channel, right?”

Sure. This is like saying you “just” drink from a firehose with a coffee straw.

This post is my attempt to de-hype MCP—not because it’s bad (it’s actually great!), but because it’s being asked to do jobs it was never designed for, and we are collectively “hallucinating” that it can give us lots of things “for free” without a lot of elbow grease behind-the-scenes. Think of this as a sanity guide for folks building AI integrations, and a reminder that protocols don’t magically turn rate‑limited APIs into data lakes. It’s all about the difference between operational vs. analytical use cases…

What MCP Actually Is

MCP is a protocol: a JSON-RPC 2.0 contract over stdio or WebSocket that lets an LLM client discover and call “tools” exposed by separate processes (“servers”). That’s it. It’s a clean, low-level “USB port” for your LLM to understand and direct your agent software process to talk to something else—files, databases, ticketing systems, whatever.

What MCP gives you:

A standard handshake to list tools, resources, and prompts
A way to call those tools and stream back results, logs, and progress
A boundary between the model client and the side-effectful world (permissions, isolation, auditing)

What MCP does not give you:

Indexing, caching, or search
Pagination or resumable jobs semantics
Faster upstream APIs
Infinite context windows (or cheap ones)

It’s closer to fetch() than it is to “Snowflake + Airflow + dbt”.

Where the Pain Actually Lives

When someone asks for scraping “all the links since the beginning of time,” MCP is just the messenger. The real bottlenecks are:

Source API limits Slack, Jira, Confluence, you name it—these APIs paginate, throttle, and often cap historical access. MCP can’t negotiate you a higher rate limit with downstream services.
Network & serialization overhead Every page of 200 messages gets marshaled into JSON, shipped over the wire, unmarshaled, and then—if you’re unlucky—stuffed into the model’s context. That’s a lot of busywork for everyone involved.
Token budgets “Just dump it into the model” is how you burn through a 200k context window in one shot. You’ll end up chunking and summarizing anyway. Better to design for that from the start.
Lack of long-running job semantics MCP doesn’t define a job queue, checkpoints, or retry policies. If you need to crawl 8 years of chat logs, do it like a grown-up: in a background job with state, not an interactive REPL loop with a model.

A Rubric: Good vs. Bad Fits for MCP

Great Uses

Small, deterministic calls: “Create a Jira ticket with these fields.”
Scoped reads: “Grab and summarize the last 50 messages from the #design channel.”
Controlled writes: “Rotate this credential,” “Run this Terraform plan (and show me the diff).”
Thin indexing facades: “Search the issue index for ‘SPIR-V error’ (the heavy lifting was precomputed elsewhere).”

Terrible Uses

Bulk export / ETL: “Give me every Confluence page created since 2015 and summarize.”
Analytics scans: “Compute the average time-to-close for 400k tickets.”
Multi-GB artifacts: “Send the full build logs for the last 10,000 pipelines.”
Anything that needs orchestration: map/reduce, long-running jobs, backoff retries, partial failure handling.

If a human engineer would script it to run overnight with a cron job and some careful retry logic, don’t expect MCP + a chat loop to brute-force it interactively.

Patterns That Actually Work

When you do need big-ish data or repeated access, pattern your MCP servers like this:

1. Precompute and Index Elsewhere

Nightly ETL Slack → S3 → Snowflake (or a vector store). Then expose a search tool over that index through MCP. The model sees a fast query surface—not the raw firehose.

2. Server-side Summaries & Pagination Links

Have your tool return: { summary: "...", next_cursor: "abc123" } instead of 10,000 raw rows. Let the model decide if it needs page 2.

3. Background Jobs + Job IDs

Expose a start_export tool that kicks off a real background job (queue, cron, whatever) and returns a job ID. Provide get_status(job_id) and get_results(job_id) tools. MCP stays snappy; heavy lifting lives elsewhere.

4. Advertise Limits Up Front

During capability discovery, include metadata: max_rows, max_bytes, max_duration. If the model knows it only gets 1,000 rows at a time, it can plan a more efficient strategy.

5. Shard Large Tasks

Instead of “export all messages”, expose tools like get_channel_links(channel_id, start_ts, end_ts, limit). The model can iterate by month or by cursor token, not by “hope and pray.”

Snappy Lines for Slide Decks & Exec Briefings

Feel free to steal these:

“MCP doesn’t turn rate‑limited APIs into data lakes.”
“It’s a function-call adapter, not a data pipeline.”
“Your throughput is min(slowest upstream API, model token budget).”
“If you wouldn’t do it in a single curl call, don’t try to do it in a single MCP call.”

A Tiny Visual

User → LLM Client ──(MCP)──> Tool Server ──> Real System (Slack/Jira/DB)
               ^              ^
           Protocol        All the speed, limits,
                            and pain live here

So… How Do I Pitch MCP Without the Hype?

Try this analogy:

“MCP is a USB port. It standardizes the plug. It doesn’t make the disk spin faster, increase its capacity, or magically reorganize your files.”

Then follow with a practical recipe:

Use MCP to wire the model into well-scoped actions.
Do heavy data work outside the loop. Precompute, cache, index.
Expose smart, limited endpoints that a model can compose.
Instrument and log everything. (You’ll need to debug the LLM’s tool usage.)

Finish with a reality check: “Yes, we can hook Claude up to Slack. No, it will not be your compliance archive or analytics warehouse. Different problems, different tools.”

Closing Thought

Protocols are boring on purpose. That’s their value. MCP gives us a boring, predictable way for models to poke the world. Treat it like a stable connector—and build the real data plumbing where it belongs. That’s how you get something reliable and fast, without selling fantasy bandwidth to your stakeholders.

Your Information Diet in the Age of AI

Ryan Spletzer — Mon, 21 Jul 2025 00:00:00 +0000

I was on a call with a colleague, walking through some Wardley Maps I’d drawn for various AI tools. Partway through my overview, he stopped me and asked a question I’ve heard a couple of times before: “How do you find out about these things?” I paused, realizing that the answer has less to do with any magic trick and more to do with habits—essentially, what I choose to read, watch, and listen to on a daily basis. In other words, it comes down to my information diet.

We live in an age where AI can answer questions, recommend content, and even generate entire articles for us. You might lean on these AI tools for quick answers, but at the end of the day you are still the one reading, watching, and listening to information sources. The critical question is: are those sources high quality? Or are you unwittingly on the receiving end of the Gish gallop? If you’re not familiar with that term, the Gish gallop is a rhetorical technique where someone overwhelms you with a barrage of arguments (often of dubious quality) with no regard for accuracy. In an era of infinite content—much of it auto-generated or spammy—it’s dangerously easy to get galloped by misinformation or superficial noise.

The Importance of a Healthy Information Diet

Just like your body needs a balanced diet of nutritious food, your mind needs a balanced intake of information. The core message I want to share is that for your day-to-day work and life, it’s vital to cultivate a healthy information diet—especially if you work in tech or any fast-moving field like AI. Even if you don’t work directly on AI, there’s a good chance you’re using it indirectly. If you’re not consciously mixing the right information into your brain and applying it to your day-to-day decisions, you risk stagnating in your approach. In the tech world, things change quickly and the moment you stop learning is the moment you start losing your edge.

Think of it this way: you are what you eat, and by the same token “you are how you search.” Feeding solely on easy, processed info—the equivalent of junk food—can leave your knowledge malnourished. Sure, an AI-generated summary or the top Google result might be a quick fix for a simple question, or a Yahoo News headline might leave you feeling like you’ve gotten a real picture of some type of new advancement in some space, but be cautious about making that your main sustenance. One information scientist aptly compared AI’s instant answers to fast food: convenient and tasty, but not the healthiest choice in the long run. But this goes beyond AI and into your everyday information intake habits: If you only consume bite-sized, context-free tidbits (think of those sensational one-line news alerts or random social media takes), you’re likely missing the vitamins and fiber of deeper understanding. A healthy information diet means balancing those quick bites with more substantive meals—the reports, articles, books, and conversations that challenge and grow your thinking, and add to your repertoire of skills.

Quality Over Quantity: Beware the Flood of Junk Info

Having a healthy info diet is as much about avoiding bad info as it is about finding good info. The term “Gish gallop” I mentioned is a warning sign: when you notice you’re being inundated with a firehose of arguments or facts of questionable quality, step back and assess. This can happen when scrolling through algorithm-driven feeds that prioritize engagement over accuracy—you get spammed with clickbait, outrage, and half-truths all mixed together. It can even happen when using AI tools if you treat their output as gospel. AI might confidently present information that is incorrect—but I would argue there’s a worse scenario than blatantly incorrect info (which can be fact-checked), and this is when you’re getting answers and information that is woefully oversimplified. (Remember, these models predict plausible answers; they don’t guarantee truthful ones, or further ones with all the necessary context for you to understand a topic, unless you ask the right questions—more on that later…). If you rely on these without critique, you might end up with a head full of fluff or outright falsehoods, which is the intellectual equivalent of subsisting on cotton candy.

So how do you ensure quality? One way is to slow down and cross-check information instead of accepting the first answer. It might take a bit more work (like reading a full article or checking multiple sources), but doing so gives you the ability to compare evidence and form your own judgment. Think of consuming information like eating: grabbing fast food on the run (a quick AI answer or a single Twitter thread) is fine once in a while, but you don’t want every meal to be from the drive-through. AI-generated overviews are like drive-through burgers—quick and hot, but not very nutritious. Taking the time to read in-depth sources or multiple perspectives is more like cooking a balanced meal at home: it’s slower and requires effort, but you know exactly what you’re getting. This extra effort “gives you back the ability to examine multiple sources… and leaves open the possibilities for learning, discovery and serendipity” that a one-and-done info snippet would never provide.

In short, be vigilant about info quality. If something smells fishy or too hypey, question it. Develop a bit of a BS filter. Over time, you’ll get better at distinguishing a well-researched piece from a regurgitated press release, or a genuine expert opinion from a performative rant. (Oh hey, maybe this blog post is a performative rant—check and question your sources!) High-quality inputs lead to high-quality outputs in your own work and decisions—garbage in, garbage out, as the saying goes…

Curate What You Consume

Since there’s an infinite amount of content out there, a key skill is curation—deliberately choosing what to read/watch/listen to. Don’t passively scroll whatever the algorithm shoves at you. Take control of the menu. For me, a big part of curation is on social platforms. (I miss old school Twitter for this.) Over the years, I’ve sought out credible voices in the tech industry and followed them to essentially build a custom “feed” of quality insights. Folks like Camille Fournier, Will Larson, Michael Lopp, Kelsey Hightower, Nicole Forsgren, Bryan Liles, Gergely Orosz, Laura Tacho, Abi Noda, Sam Schillace (to name just a few) consistently share valuable perspectives on engineering leadership and emerging technology. By following leaders of this caliber, I ensure that my timeline isn’t just the outrage-of-the-day or viral memes, but rather snippets of wisdom, links to great articles, and heads-up on new tools worth looking at. In other words, I let respected experts and practitioners populate my brain’s newsfeed, not random influencers or click-chasers.

Social media can be a double-edged sword. It’s where a lot of breaking info and niche discussions happen first, but it’s also a source of endless junk. The trick is aggressively filtering who you follow. (Sidebar: this is a great trick to use with Slack/Teams channels and email in your inbox as well—aggressively mute/leave channels to reduce noise, organize them with something like the PARA method, and follow some tips on how to get control of your inbox—I learned all these things from people I follow…) If someone consistently adds value (teaches you something, makes you think, points you to good resources), keep them. If they mostly spout hot takes or amplify noise, mute or unfollow. The information you consume is just as important as the food you eat—doomscrolling Twitter aimlessly is the mental equivalent of chowing down on candy bars all day. I try to limit the empty calories and focus on accounts and publications that are more like a balanced meal.

Curating your information diet goes beyond just who you follow on Twitter. It also means picking good long-form sources to balance out the short-form stream. For example, I subscribe to a few weekly newsletters (like Gergely Orosz’s Pragmatic Engineer) and industry blogs that summarize what’s new in tech. These act like a hearty weekly dinner—more substantial and organized than the constant snacking of social media. I also make time for books and deep-dive articles when I can, because often the best insights come from longer, more structured content that an author spent months or years refining. (Only the best ideas tend to survive the gauntlet of writing a full book or peer-reviewed paper, so those can be high nutritional value, so to speak.) By reading books or listening to hour-long podcast interviews, you force yourself to engage with ideas at a deeper level than a tweetstorm or headline can offer. It’s like the difference between a full-course meal and a quick bag of chips.

Finally, remember that your peers and community are part of your information diet too. Find the people in your network who are always tinkering, always learning, always sharing cool stuff they discovered. Every office or friend group has a few of these “information chefs” who love to cook up new ideas. Take advantage of that—ask them what they’re excited about lately. If you’re active in any Slack/Discord groups, forums, or local meetups, pay attention to what those trusted folks are buzzing about. Often I learn about the next big library, or a handy AI tool, from an engineer in my team or a friend in the industry who says “Hey, have you seen this yet?” Those conversations can be golden. They’re like getting a recommendation for a great new restaurant from a friend: personalized and usually reliable.

Keep Your Mind Open to New Ideas

If you only consume the same type of information over and over, you’re going to get the same results. A healthy diet has variety—some new flavors and cuisines in addition to your comfort food. Likewise, a healthy information diet means embracing things that are unfamiliar or even initially uncomfortable. Human nature being what it is, most new ideas and technologies get some pushback at first. Maybe it’s skepticism (“That’ll never work here”), maybe it’s discomfort (“This is too different from what I know”), or plain inertia (“I don’t have time to learn that”). But if you dismiss every new development out of hand, you’ll wake up one day stuck with a very outdated worldview.

I encourage you to push beyond the initial pushback and keep going a little further than feels natural. Often, once you get past that first hill of resistance, you start seeing the value on the other side. We’ve all seen colleagues who refused to learn the new tool or ignored the new trend; a year or two later, they find themselves playing catch-up. Don’t let that be you. The tech landscape moves fast. “If you’re not moving forward, you’re falling behind,” as one engineer wrote about avoiding stagnation. We see this over and over: cloud computing, DevOps, containerization, serverless, machine learning—pick any tech trend, and early on there were plenty of naysayers saying it’s all hype or not worth the effort. But those who took the time to explore new approaches often reaped huge benefits, while the skeptics were left scrambling to adapt once the change became inevitable. A very current example is AI-assisted development. When GPT-based tools first emerged, a lot of folks (understandably) pushed back: “Can we trust the code it writes?”, “Won’t this take our jobs?”, etc. Yet here we are a short time later, and such tools have gone mainstream. In fact, three-quarters of workers are now using AI in their jobs, one way or another (if this Microsoft study referenced in that prior link is to be believed—again, check and question your sources!). The initial fears and objections are giving way to “Okay, how do we actually leverage this properly?” Those who kept an open mind early on are now ahead of the curve in using AI to be more productive. The point is, don’t reject a new idea just because it’s new. Sample it. Even if it ends up not to your taste, at least you made an informed decision. And if it does turn out to be the next big thing, you’ll be glad you got a head start.

Keeping an open mind also means sometimes venturing outside your usual bubble of information. If you’re a software engineer, it can be useful to occasionally read something from, say, a design blog or a product management podcast, just to see how adjacent fields are thinking. If you’re deep into AI, it might pay to read what skeptics or ethicists are saying about it, not just the cheerleading press releases. A diverse information diet ensures you’re not lopsided in your understanding. It inoculates you against groupthink. Sure, most of the time I read my preferred tech sources, but every now and then I deliberately read an opposing viewpoint or an analysis from a different domain. Despite working in AI, I recently read The AI Con: How to Fight Big Tech’s Hype and Create the Future We Want. Did I agree with 100% of that book? No. But much of it did resonate with me, and it has helped me to hone my BS filter when I see some new, over-hyped headline about something that’s going to “change everything.” Forcing yourself to read things that run counter to your world view or challenge the prevailing mainstream media hype is a bit like eating your vegetables—not always the most immediately fun part, but it makes you stronger in the long run. (And steering away from AI hype for a second, if you want some critical thinking and de-hyping of quantum computing, especially as it relates to un-founded fears about it “breaking cryptography,” take a look at this paper which is dense and mathy, but basically dissects how factorisation exercises in quantum computers have been a sleight-of-hand magic trick, or watch 30-minutes of this YouTube video at 14:49 from Security Now which explains these quantum factorisation sleight-of-hand tricks in a relatable way.)

Beyond the Headlines: Choose Trusted Voices

Another aspect of a good information diet is knowing who (and what) to trust, especially when it comes to news and analysis. We’ve all seen how a story can get distorted as it’s rehashed across the internet. A company might release a nuanced research paper, and by the time it makes it to a Yahoo News headline, it’s been dumbed down to “Scientists Create Monster AI That Will Take Your Job!” Getting your tech news solely from generic mainstream outlets is like trying to sustain yourself on popcorn—it’s mostly air, with a bit of salt and butter. It’s easy to fall for sensational headlines and bite-size news that lack context, because they’re designed to grab attention, not to inform deeply. So when possible, go beyond the headline. If something interests you, click through and read the details (or find the original source). Often the reality is more complex—and more interesting—than the blurb.

Let’s use a non-tech analogy. Say a new Ghostbusters movie comes out. Who are you going to trust with a review? Perhaps one of the following:

The Today Show: a quick segment on morning TV with a cheerful host reading talking points.
Rotten Tomatoes: an aggregation of many critics’ and viewers’ opinions, averaged into a score.
A like-minded friend who already saw it: someone whose taste in movies usually matches yours.

For me, the choice is obvious: I’d call up the friend (or read their texts about it). Why? Because I know how they think and I trust their perspective. The Today Show might give a superficial take aimed at a broad audience (“It’s a fun summer romp for the family!”)—that’s like a headline with no depth. Rotten Tomatoes is more informative, but it’s an impersonal crowd score. My friend, on the other hand, will tell me candidly, “Yeah, it’s a decent throwback but the plot was weak, I know you hate those cheesy jokes so you’ll probably find it mediocre.” That’s actionable intel for me.

Translate this back to technology and work. When you’re evaluating a new programming framework, a new SaaS product, or any new “shiny thing” in tech, consider the sources of info in a similar vein:

Official marketing or mainstream media (equivalent to the Today Show): quick soundbites, maybe biased towards positivity (or in the case of AI doomers, stoking your fears and distracting you from real-world current day problems), not deeply technical. Useful for a high-level gist, but not entirely trustworthy for making a decision.
Aggregated opinions (equivalent to Rotten Tomatoes): for example, a Stack Overflow thread, a collection of reviews, a Gartner Magic Quadrant. These give you a broader sense of consensus or common pros/cons. Better, but can lack context of your specific needs.
Trusted peers or experts who’ve tried it (equivalent to your friend): a blog post by an engineer who implemented the tool in a project, a conversation with someone at a meetup who has hands-on experience, or an in-depth YouTube review by a respected techie. These sources are often the most valuable because they can tell you the nuances—the “gotchas,” the unexpected benefits, how it compares to similar tools, etc., all from a point of view you recognize as honest.

Whenever possible, put more weight on the voices that have proven trustworthy and aligned with your context. In my case, if a Camille Fournier or Kelsey Hightower shares an opinion on a new infrastructure tool, I’m inclined to listen closely. If randomsocialmediauser123 hypes the same tool with zero track record, I take it with a big grain of salt. This isn’t to say famous or well-known folks are always right (they aren’t), but over time you learn which people or outlets tend to have signal and which are just noise.

And if you don’t have a go-to friend or expert on a topic? Consider becoming that person yourself. Dive in and experiment so you can share first-hand knowledge. It’s the principle of “trust but verify” in action: take in others’ opinions, but validate through your own exploration when you can. Not only will you get a better understanding, you’ll also be contributing back to the information ecosystem with your findings—maybe writing your own blog post or giving your colleagues the real scoop around the proverbial water cooler.

Final Thoughts: You Are What You Read (So Choose Wisely)

Circling back to that question that started it all—“How do you find out about these things?”—the answer boils down to cultivating and maintaining a healthy information diet. In the age of AI, this is both more challenging and more important than ever. More challenging, because there’s an overwhelming buffet of content being served up, not all of it healthy or even true. More important, because the pace of change is blistering; falling behind can have real consequences for your career and understanding of the world.

The good news: you are in control of what you consume. Be intentional. Feed your mind with high-quality information from a variety of sources. Balance the quick bites of AI-generated answers or social media chatter with the slower digestion of books, articles, and thoughtful conversations. Curate who you listen to, so that you’re hearing from people who inspire and educate you, not just echo chambers that amplify hype. Experiment and taste new ideas, even if they seem strange at first—you might discover something game-changing, or at least you’ll expand your palate. And above all, stay curious. A curious mind naturally seeks out better information and resists the junk food of shallow content.

If you do all that, you’ll have an information diet that keeps you sharp, informed, and ready for whatever comes next. And when a colleague or friend marvels at how you’re always ahead of the curve, you can smile and say, “I just watch what I consume.” In the end, the quality of what you put into your brain determines what you get out of it—so choose wisely. Bon appétit!

The Many Contexts of Model Context Protocol

Ryan Spletzer — Fri, 30 May 2025 00:00:00 +0000

Or: why “where” your MCP server runs matters just as much as “what” it does.

TL;DR Running an MCP Server on localhost is a night-and-day difference compared to running one remotely.

Local == single user, implicit trust, almost zero auth to the server.

Remote == multi-user, explicit trust, real authentication to the server, security trimming, token exchange, and all the headaches (and rewards) that come with it.

1. Introduction – Context in Two Senses

When people hear Model Context Protocol (MCP), they usually think about the data context an LLM needs—documents, code, tickets, whatever.

Less obvious, but just as important, is the deployment context in which the MCP server itself lives.

Local sandbox: One dev, one laptop, one loopback interface.
Remote service: Many users, many clients, many downstream systems, and a permanent address on the public internet (or barring that, at least on your private network).

Those two worlds demand radically different security and architecture choices.

MCP has only existed as a “thing” since late 2024, and has only really gotten popular (and only gotten serious with auth specs) as of the last couple of months, and the mere ~6 or so months of its existence is part of the reason why the state of affairs with respect to running these in advanced remote scenarios is relatively immature. (Yes, I said it—for those on the agentic hype trains, don’t @ me.)

2. Local MCP – The Joy of Single-User Simplicity

I am going to pick on the GitHub MCP Server as an example, because despite it being relatively popular—with GitHub being one of the first things developers would likely want to connect to with agentic tools—it is imbued with all the naïveté I’ve come to expect in the MCP servers that I’ve looked at thus far, making it effectively a non-starter for sophisticated enterprises who would wish to run it remotely (unless you applied some serious elbow grease—more on that later).

Running the GitHub MCP Server locally could look like the following:

# Provide your GitHub (preferably Fine-Grained) Personal Access Token
read -rsp "GitHub PAT: " GITHUB_PERSONAL_ACCESS_TOKEN

# Run the GitHub MCP Server
docker run --rm -i \
  -e GITHUB_PERSONAL_ACCESS_TOKEN=$GITHUB_PERSONAL_ACCESS_TOKEN \
  ghcr.io/github/github-mcp-server:latest

You’ll notice that last message: GitHub MCP Server running on stdio.

stdio stands for “Standard Input/Output” and represents an inter-process communication approach to providing an MCP Server to a local MCP client, like Visual Studio Code.

stdio only is feasible for local scenarios, so that leads us to the next logical question: how would we make the GitHub MCP server available over HTTP / SSE (Server-Sent Events) transport, or the even more recent and modern Streamable HTTP transport (which literally was conjured up alongside the MCP specs in the last few months)?

The answer to this question illustrates a key point: The GitHub MCP Server doesn’t support HTTP + SSE, and thus doesn’t actually support being run in a remote setting at all. (Yet.)

Many of the MCP servers that have been created over the last few months didn’t take into account remote scenarios, as they were coded and designed purely with these local clients in mind.

So let’s turn our attention to an MCP Server that does have HTTP+SSE transport, the Semgrep MCP Server:

# There are multiple ways to run this, but to stay consistent we'll run it with docker:
docker run -i --rm ghcr.io/semgrep/mcp -t sse

The examples above just show how to run the servers, but in reality, when consuming via something like Visual Studio Code, you would wire this up in something like .vscode/mcp.json so your IDE is aware of how to start it up when you open a project.

2.1. Why No Auth Feels Okay

You may see this as a developer and think “I don’t get it, this seems totally fine so far”—and you may actually be correct. See, running these MCP servers locally has certain aspects to it:

Physical access is implicit authentication.

If you can hit localhost, you’re already inside the blast radius (your laptop).
One tenant = one permission set.

There’s no question who the user is in this scenario, it’s all just you.

The server can safely assume “allow everything.”

To be honest, this is what developers like: developer ergonomics reign supreme when there is zero OAuth dance, no JWT debugging, no credential managers to deal with. It’s the fastest path from idea to working prototype. But it is these same ease of use tendencies for working on our local machine that can make us blind to the very real issues of trying to make these types of MCP servers work in a remote setting. For example, lack of HTTP transport aside, if you attempted to move the GitHub MCP Server into a remote setting, whose personal access token are you going to use? And if it’s a privileged personal access token, how do you know the user behind the calling client isn’t accessing repos or other data that they don’t have access to natively?

2.2. The Boundaries of “Safe Enough”

The moment you expose that port beyond loopback—say, you port-forward via ngrok so a coworker can demo your MCP server—all bets are off. Anyone who has the URL owns your MCP, and by that, any permissions you granted to that GitHub personal access token. That’s fine for a five-minute demo, catastrophic for anything longer.

If all MCP was is a way to wrap external resources (be they your local file system, or remote database or REST API, etc.) with your own local MCP servers to provide it to your own local agent tools, then there would be pretty much zero issues here, and no need for this blog post—it would just be developers doing their thing on their local dev machines. But the issue with MCP right now is it is trying to be more than that, and this is where pretty much any off-the-shelf MCP server is going to struggle in a remote setting.

In a very real way, many available MCP servers today are at about the same level as a “Hello World” local web app or API running on localhost with no authentication (and without auth, further you really have no concept of multiple users or multi-tenancy)—if you’ve created something like that, I would hope that you would understand intuitively that it in no way could just simply go to production as-is.

3. Remote MCP – Welcome to Multi-User Reality

Move the same MCP server to https://mcp.acme.dev and everything changes.

3.1. Authorization is Now Mandatory

Every request must identify who is calling. The mainstream way is OAuth / OpenID Connect (OIDC), and the MCP specs themselves call this out as “optional”, but something that SHOULD be implemented for HTTP transport scenarios—in reality, authorization was an afterthought added to the spec after many individuals and companies pointed out the issues of not defining a way to handle authorization, since not only would it likely lead to naïve security hazards, but would hurt the interoperability and discoverability aspects of the ecosystem.

A call to an MCP server with authorization would look something like the following with a JWT access_token passed in the Authorization HTTP header with the Bearer scheme:

POST /query
Authorization: Bearer eyJhbGciOiJSUzI1N...
...

This:

Validates the JWT access_token’s signature against your identity provider’s (IdP’s) jwks_uri which provides the public keys and is discoverable via the OpenID Connect metadata endpoint.
Rejects expired JWT access_tokens (concretely, ensuring the current date time is not past the exp claim in the token, represented Unix epoch time).
Checks that the JWT access_tokens aud claim is in fact the string you expect that represents your MCP resource, not an arbitrary API. (The aud is often represented as a GUID, or some type of unique identifier for your target resource.)
In cases where there is a human user involved, it should utilize something like the user’s sub (subject) claim from the JWT access_token, to understand who the user is for filtering for downstream calls. (Which, no server I’ve ever seen so far does this—the validation typically stops with only the checks above, if you’re lucky that they’ve done all of them right and not missed something. Beyond this authorization check, they typically don’t implement further fine-grained authorization logic for security trimming, because frankly that can get enterprise-specific and is difficult to assume or make configurable for a wide variety of scenarios.)

A reverse proxy (→ Envoy, Traefik) can handle this so your app code can assume a verified user context—but again, you may need that user context in your actual MCP server code to make further authorization determinations. Most off-the-shelf MCP servers do not—and often cannot—make those authorization decisions for you.

Note that we just say “Authorization” here and not “Authentication”—that is because technically the authentication portion happens between the client and the identity provider (IdP) you’re talking to, like Okta or Microsoft Entra ID or a consumer third party identity provider like Google or Apple or Microsoft.

3.2. Authorization & Security Trimming

Authentication (AuthN) only says “Alice is Alice.” Authorization (AuthZ) answers “Which docs/repos/items/objects can Alice actually read?”

An enterprise MCP ecosystem could often fan out to services like GitHub, Jira, Confluence, Artifactory, Microsoft Graph, or a vector DB keyed by user or security group ID’s. That means:

Row-level filters: SELECT … WHERE user_id = <aliceUserId> OR group_id = <securityGroupIdAliceIsIn>
Search ACLs: e.g. Elasticsearch or OpenSearch query-time filters
Downstream: on-behalf-of (OBO) flows; exchange Alice’s inbound JWT access_token scoped for the MCP server resource with your IdP for another JWT access_token to access an API that accepts user-scoped tokens from your IdP for purposes of security trimming.

Fail to trim, and your nice AI agent becomes a data-leak vending machine.

3.3. Multi-Tenant Isolation

Thus far we have approached local and remote scenarios only from the standpoint of a developer who is using local tools to connect to local and remote MCP servers.

But MCP servers are not destined just for developer tools—they can and will be used in scenarios like enterprise chatbot assistants, where a lot of the same remote lessons apply with regards to user identity and security trimming, etc.

But what about when you’re running a multi-tenant SaaS product that has agentic capabilities that wish to consume MCP Servers?

You are now storing embeddings, indexes, caches, and data for many customers. Everything needs a partition key. Consider:

$TENANT_ID:$DOC_ID keys in Redis.
Distinct Milvus/Mongo/Postgres collections/tables/databases per tenant.
S3 prefix per tenant (s3://mcp-prod/tenants/$id/).
Database filters based on tenant ID.

When it comes to running products, you are now firmly out of the realm of off-the-shelf MCP servers and now in a place where you are creating them from scratch—sure, you can derive some “inspo” from some of the MCP servers out there, but what you’re creating inherently now has to be catered to your products and SaaS ecosystem.

These considerations even extend to scenarios beyond your core products and into tools your customers may be using, like Microsoft 365 Copilot, where they may want to connect their own enterprise Copilot chat to MCP Servers that wrap various APIs of your products, and you have to provide facilities for users at your customer’s enterprise to authenticate to provide proper security trimming, in much the same way that you authenticate to any one of a number of Slack apps as a user to connect to various tools.

4. Consumer Patterns Drive Security Posture

This table gives a brief breakdown of some of the remote authentication patterns for various scenarios.

Pattern	Who calls MCP?	Typical Auth Flow
IDE plugin (Visual Studio Code, JetBrains)	The developer’s desktop	Often local MCP → no auth. If remote: OAuth device flow (perhaps with PKCE); store refresh token in keychain.
Web/Mobile app feature (“Summarize this ticket”)	Product backend	Backend → MCP: service credential (client cred). User context propagated as JWT or user ID header.
Pure unattended (nightly batch, scheduled agents)	Job runner / microservice	Service-to-service token only. User context often absent or deferred via signed job token.

The security envelope expands as soon as humans other than the owner of the target resources are involved.

When it comes to local and remote MCP servers in an enterprise setting, the reality is you will wind up forking or wrapping off-the-shelf MCP servers to accommodate your authentication/authorization needs.

But when it comes to building SaaS products, it gets a little bit more involved, and warrants some more ideation and discussion.

Not much of what is described in this blog post is unique to MCP—you could easily replace “MCP Server” with “Rest API” and most of these considerations still apply—it’s simply that we are re-learning these lessons in the realm of MCP as the spec and implementations and thought leadership evolves around it.

5. Propagating Identity – Dual Tokens & OBO

It is worth digging into a hypothetical example of a real world SaaS product to illustrate the inherent unavoidable complexity involved in properly propagating user context through a call chain from a frontend through a backend and through a set of collaborating services, which in this case includes an MCP server.

Imagine your MCP server receives a call from your SaaS product backend. In an ideal world (which we are so often very far away from), that backend authorizes itself in with its own token issued through client credentials grant flow, and forwards the end-user’s JWT access_token, so the MCP server can do user-level validation and security trimming.

Key points:

Two tokens travel together (Authorization: Bearer <serviceJWT> and x-user-token: Bearer <userJWT>).
MCP validates both.
If CalendarAPI trusts your IdP, MCP can perform an On-Behalf-Of exchange to get a new access token valid for CalendarAPI.

As always, treat user tokens as PII; never log them.

6. Long-Running & Async Tasks

Finally we arrive at the very complex scenario of long-running (perhaps hours or days long) tasks that may be initiated by humans. I am going out on a limb when describing the approaches that can be taken here, because they are not very standardized, but I believe I have some ideas that can assist in making call chains like this more cryptographically secure.

Example: A user kicks off, or configures, an AI-driven workflow that runs for multiple hours and needs to utilize the user’s identity to access various services. This workflow may even run in some type of scheduled way unattended from there on out. The original JWT access_token representing the user expires after an hour, as is typical and proper with most IdP’s. (Note: this is not unlike scenarios you can find in low-code tools like Power Platform.) We have some options:

6.1. Refresh Tokens

The agent process (acting as an MCP client in this case) stores an (ideally encrypted at rest) refresh_token representing the user.
The agent process swaps the refresh_token for fresh JWT access_tokens with the IdP, scoped to desired target resources, as needed.

This is likely the most “proper” approach to this today, and has precedence in various services out there. But there is a trade-off: storing a person’s refresh_token is essentially storing a session for them, and is inherently high-privilege; thus these refresh_tokens should (and I would say, must) be protected at rest. They can also be revoked on user off-boarding, which is usually the responsibility of the IdP, but your implementation should gracefully handle if/when a refresh_token is revoked.

6.2. Signed Job Token

At task creation, the agent itself issues a JWT, perhaps by using the original user’s JWT or its signature to sign the new JWT with a timestamp claim inside of it, with storage mechanisms for both user + newly issued JWT for later verification (think along the lines of how we use code signing certs to sign source code—the code signing cert may be long expired, but as long as it was valid at signing time, it provides those integrity assurances):

{"sub":"alice", "scope":"export:1234", "exp":now+3h, ...}
Agent presents the original user’s now-expired JWT access_token plus the newly minted JWT signed by the original JWT access_token in HTTP headers to MCP endpoints dedicated to the job, and MCP knows to validate the original JWT signature, aud, sub, but not to pay attention to its exp (expiration), but rather utilize the iat (issued at) claim to understand if the child JWT with the timestamp was signed during a period when the original JWT was valid.

In this approach, there is no long-lived refresh_token required, and the scope is tightly bounded—however, it also requires any endpoints that might receive such a JWT to implement very sophisticated logic of validating a child JWT token based on its signature from another JWT, which many off-the-shelf JWT validation libraries will struggle to do (which means you may be rolling up your sleeves to implement some of this).

6.3. Service account with embedded user claims

Worker runs with service auth only.
It stores userID + allowed resources in job metadata.
Every downstream query constrains rows by that userID.

The tolerance level for using basic strings to represent user context throughout short-lived and long-lived call chains depends on many factors. Your ecosystem may or may not be tolerant to simple user id strings that could potentially be spoofed if someone had an anchor point to call a backend service in your ecosystem. Further, there may be third party API’s you wish to hit as the user which you do not control, in which case you would inherently need some type of super principal that has access to everything in that third party and extra logic to filter based on the user.

Some type of cryptographic material representing the user (a refresh_token potentially that could be exchanged for fresh JWT access_tokens for a given user when needed) will always be stronger than just representing the user in an HTTP header like x-user-id: <userId>.

Pick the approach that aligns with your org’s security model and compliance burdens.

7. State of Today’s MCP Servers – A Gentle Rant

Most open-source MCP implementations assume a trusted localhost and a single user. Spin one up for a hackathon and it’s magic; deploy it in prod and you’ll suddenly need:

Plug-and-play OAuth/OIDC support (ideally configurable with an IdP).
Multi-tenant storage abstractions out of the box for product scenarios.
Hooks for per-request security trimming.
Token exchange helpers (OBO, OAuth 2 Token Exchange draft).
Auditing and rate limits.

Today you’ll find yourself bolting these on—or writing a custom gateway—because upstream doesn’t ship them. That’s not a knock on the maintainers; it’s a sign the ecosystem grew up in a local-first culture. But the demand for hosted MCP is exploding, so the tooling, and more importantly the patterns and practices, need to catch up.

8. Conclusion

Context in MCP isn’t just the corpus you feed the model—it’s the environment your server inhabits. Keeping everything on your laptop? Enjoy the blissful simplicity. The second you move to a team, a customer, or the public cloud, bring your security A-game:

Authenticate every request.
Authorize every slice of data.
Propagate identity with care.
Design for token lifecycles, not solely one-shot calls.

If you’re building an MCP server, plan a two-mode strategy:

Dev mode: Runs wide open on localhost for fast iteration.
Prod mode: Pluggable OAuth, per-tenant isolation, audit logs, and the rest.

Your future self—and your security team—will thank you.

Ultimately, context is everything—not just in prompts, but in the infrastructure that serves them. Let’s make our MCP tools as context-aware as the models they power.

Bonus: A2A and ACP Protocols

Hot on the heels of MCP are the new A2A and ACP protocols which can be used to complement MCP, and while I won’t delve into the details of those in this blog post, I’ll just note that both require similar authentication and authorization considerations to be enterprise-grade and production-worthy, and I may delve more into specifics around those in a future blog post.

I wrote this post mostly to address a lot of FUD (Fear, Uncertainty, and Doubt) around how MCP servers operate today and their AuthN/AuthZ strategies (or lack thereof), and should the need arise I can take a similar stab at addressing A2A and ACP.

The world of AI is moving fast, and on the one hand it is great that we are getting open standards and specs so quickly, but on the other hand it will take a little time for them to become fully “fleshed out” with enterprise-grade patterns and practices.

Ryan Spletzer

How to Disable OTP on Your YubiKey

Quick Reference

Via YubiKey Manager (GUI)

Via ykman (CLI)

Disabling OTP vs. Removing the Credential

What This Doesn’t Affect

Re-Enabling OTP

What OTP Actually Is (And Why You Probably Don’t Need It)

Yubico OTP

Static Password

OATH HOTP

OATH TOTP (A Common Source of Confusion)

Why FIDO2/WebAuthn Replaced All of This

Conclusion

Footnotes

A No-Nonsense Guide to GPG Commit Signing with a YubiKey

Why Bother?

Overview / TL;DR

Planning Your Key Identity

Prerequisites

Step 1: Generate Your GPG Key Pair

macOS – Key Generation

Ubuntu – Key Generation

Windows – Key Generation

Step 2: Move the Signing Subkey to Your YubiKey

Change Your YubiKey PINs

Changing PINs with ykman (Recommended)

Changing PINs with gpg --card-edit (Alternative)

Ghostty Quirk: “Screen or Window Too Small”

GUI Pinentry Quirk: “Sorry, No Terminal at All Requested”

Move the Subkey onto the Card

Loading the Same Key onto a Second YubiKey

Removing the Master Key from Your Machine

Step 3: Configure Git for GPG Signing

macOS – Git Config

Ubuntu – Git Config

Windows – Git Config

Step 4: Upload Your Public Key to GitHub

Making Your Public Key Easy to Find Later

Step 5: Test It

Setting Up on a New Machine

macOS – New Machine

Ubuntu – New Machine

Windows – New Machine

Re-Keying: Updating Your Email Addresses

Adding a New UID

Revoking an Old UID (Optional)

Re-Uploading Your Public Key

Cleaning Up

Updating Git Config

Revoking a Compromised Subkey

Revoke the Subkey

Publish the Revocation

Generate a New Signing Subkey

Load the New Subkey onto Your YubiKey(s)

Update Git Config and GitHub

Clean Up

Starting Over: Complete Re-Key (Fresh Start or Master Key Compromise)

Revoke the Old Key (if You Still Have Access)

Generate New Keys and Set Up from Scratch

Updating Other Machines

Extending Key Expiration

Extending Your Master Key’s Expiration

Extending a Subkey’s Expiration

After Extending

Troubleshooting

“gpg: signing failed: No secret key”

“gpg: signing failed: Inappropriate ioctl for device”

PIN Entry Dialog Doesn’t Appear on macOS

GPG Keeps Prompting for PIN Even With the YubiKey Inserted

GPG Can’t Find Keys After Upgrading to GnuPG 2.4+

“Please Insert Card with Serial Number…”

YubiKey Not Detected After Removing and Reinserting

scdaemon and pcscd Conflict (Linux)

Wrong gpg Binary in Git

Accidental OTP Output When Touching YubiKey

“Unusable Secret Key” (Expired Subkey)

Bonus: Using Your YubiKey for SSH Authentication

Setup

Changing PINs with `ykman` (Recommended)

Changing PINs with `gpg --card-edit` (Alternative)

`scdaemon` and `pcscd` Conflict (Linux)

Wrong `gpg` Binary in Git