YubiKey SSH Authentication & Commit Signing

Caution

Portion of document and debugging was done with help of ClaudeAI, especially Sonet 4.6 Extended.

1. Problem Statement

The objective was to establish a complete, hardware-backed SSH and Git commit signing workflow using a YubiKey 5 Nano with FIDO2 resident keys across two platforms:

macOS (primary workstation):

• SSH authentication to GitHub using a resident key stored on the YubiKey
• A tactile user-presence prompt (touch confirmation) when the key is exercised
• Hardware-backed Git commit signing using the same resident key

Linux (remote development box, accessed via SSH from Mac):

• SSH authentication to GitHub from the Linux box, proxied through the Mac’s YubiKey
• Git commit signing from Linux, with the signing operation physically performed by the YubiKey plugged into the Mac

The surface area of this problem spans four distinct subsystems: FIDO2/libfido2, OpenSSH agent protocol, macOS launchd service management, and Git’s SSH signing pipeline. Failures in any one layer propagate silently to the others, making diagnosis non-trivial.

---

2. Initial State Audit

Before any changes were made, a systematic audit was performed to establish ground truth. This is the most critical phase, attempting fixes without understanding the current state is the primary cause of thrash in infrastructure debugging.

2.1 YubiKey State

Device type: YubiKey 5 Nano
Firmware: 5.4.3
Enabled: OTP, FIDO U2F, FIDO2, OATH, PIV, OpenPGP

Running ykman fido2 credentials list revealed two resident credentials:

Credential ID	RP ID	Notes
7f85e829...	ssh:github-yk-s5-nano-a1	Well-formed RP ID
e2e7ddb3...	ssh:	Bare RP ID — problematic

The second credential (e2e7ddb3) was being used as the Git signing key (id_ed25519_sk_git.pub). A bare ssh: RP ID is what OpenSSH generates when no -O application= flag is passed during key creation. It functions locally but is brittle in forwarded-agent contexts because the FIDO2 assertion request must use the exact RP ID stored in the credential. This was flagged early as a root cause candidate for potential forwarding failures.

2.2 SSH Agent State

Two agents were discovered running simultaneously:

PID 97848  /opt/homebrew/bin/ssh-agent -s    ← owns SSH_AUTH_SOCK
PID 97790  /usr/bin/ssh-agent -l             ← launchd-managed, orphaned

The -l flag on the system agent is macOS’s launchd socket activation mode. The homebrew agent was the active one, evidenced by $SSH_AUTH_SOCK pointing to its socket. Critically, the socket path was non-standard:

/Users/grim_reaper/.ssh/agent/s.ew7K20NnUU.agent.0Ms6aR4TwF

This is neither the homebrew default (/tmp/ssh-*/agent.*) nor the system launchd socket. The random components suggested it was generated by eval $(ssh-agent) at some point and propagated by tmux. This was confirmed by finding eval "$(ssh-agent)" in ~/.config/tmux/scripts/update-env.sh.

Implication: The agent had no stable lifecycle management. Its survival was accidental, it had been running since Fri 02AM only because nothing had killed it. A reboot or fresh login outside tmux would leave $SSH_AUTH_SOCK pointing at a dead socket.

2.3 Git Config State

[gpg "ssh"]
    program = "/opt/homebrew/bin/ssh-keygen"

gpg.ssh.program is not a recognized git-config key. Git’s SSH signing pipeline hardcodes its ssh-keygen invocation, this key is silently ignored on every signing operation. This was a vestigial config entry with no effect.

gpg.ssh.allowedSignersFile was absent entirely, meaning git verify-commit and git log --show-signature would fail regardless of whether signing itself worked.

2.4 Agent Forwarding State

ssh -A linux-box was already in use. ssh-add -L on the Linux box confirmed both SK keys were visible in the forwarded agent. $SSH_AUTH_SOCK on Linux pointed to /tmp/ssh-wcBOjoXY7N/agent.8676 — a path that changes on every new SSH connection, causing tmux panes opened after a reconnect to hold a stale socket reference.

---

3. Problem Decomposition

With the audit complete, the problem was decomposed into discrete, independently solvable components:

#	Component	Platform	Severity
1	No touch prompt on SK key operations	Mac	High
2	Agent lifecycle has no stable foundation	Mac	High
3	Orphaned system agent	Mac	Low
4	Dead gpg.ssh.program config	Mac	Medium
5	Missing gpg.ssh.allowedSignersFile	Both	Medium
6	Bare RP ID on signing key	Both	Low/Informational
7	SSH_AUTH_SOCK changes per SSH session on Linux	Linux	High

---

4. Root Cause Analysis

The RCA!

4.1 No Touch Prompt (Problem 1)

Hypothesis: The agent has no mechanism to surface a UI prompt when a FIDO2 operation requires user presence.

Investigation: When ssh-agent receives a signing request for an SK key, it invokes ssh-sk-helper (the FIDO2 helper binary). ssh-sk-helper calls into libfido2, which triggers the physical key to request user presence (the flashing LED). The key was indeed flashing, the hardware path was working. The gap was entirely in the notification layer between the agent and the user.

OpenSSH’s read_passphrase() function, used for both PIN collection and presence prompts has a fallback path for processes with no controlling TTY: it will exec the binary pointed to by $SSH_ASKPASS if set, and $SSH_ASKPASS_REQUIRE=force overrides the TTY check. Without these, a daemon-mode agent simply gets nothing from read_passphrase(), the operation either succeeds silently (touch-only keys) or fails (PIN-required keys).

Resolution path: Set SSH_ASKPASS to a custom script that presents a macOS-native dialog. Set SSH_ASKPASS_REQUIRE=force to bypass the TTY check. These must be set in the agent’s environment, not the shell’s, because read_passphrase() is called by the agent process, not the SSH client process.

4.2 Agent Lifecycle (Problem 2)

Hypothesis: The tmux-managed agent is fragile and will not survive across reboots or sessions.

Confirmed by: The custom socket path, the tmux script that spawns a new agent, and the absence of any launchd plist managing the homebrew agent.

Resolution path: A launchd plist with KeepAlive=true and a stable, hardcoded socket path (~/.ssh/agent.sock). The -D flag keeps the agent in the foreground (required by launchd, a forking process that exits confuses launchd into restart loops).

4.3 Linux SSH_AUTH_SOCK Instability (Problem 7)

Hypothesis: The forwarded agent socket path changes on every SSH connection, breaking tmux panes that hold the old value.

Confirmed by: Observing /tmp/ssh-wcBOjoXY7N/agent.8676 on one connection vs a different path on reconnect.

Resolution path: ~/.ssh/rc, a file sshd executes after authentication, before starting the user’s shell. It receives $SSH_AUTH_SOCK set to the current forwarded socket. A symlink at a stable path (~/.ssh/agent.sock) updated on every login is the canonical solution. The shell then exports SSH_AUTH_SOCK="$HOME/.ssh/agent.sock" permanently.

---

5. Implementation & Issues Encountered

5.1 launchd Plist (Agent Stable Lifecycle)

What was done: Created com.user.homebrew-ssh-agent.plist with:

• -D flag (foreground mode, required by launchd)
• -a ~/.ssh/agent.sock (stable, predictable socket path)
• SSH_SK_HELPER set to a wrapper script in EnvironmentVariables

Issue encountered — SSH_SK_HELPER path resolution:
The wrapper script was initially set via $HOME expansion. launchd does not expand $HOME in plist EnvironmentVariables. It must be an absolute path. Fixed by hardcoding /Users/grim_reaper/.

5.2 sk-helper-wrapper (Touch Notification)

What was done: A wrapper script set as SSH_SK_HELPER that fires an osascript notification then execs the real sk-helper.

Issue encountered — AppleScript line continuation:
The initial wrapper used \ for line continuation in the AppleScript string:

osascript -e '
  display notification "Touch your YubiKey now" \
    with title "SSH: Key Operation" \

AppleScript’s line continuation character is ¬ (option-L on macOS), not \. The \ is an unknown token to the AppleScript parser. Error from the agent log:

49:50: syntax error: Expected end of line, etc. but found unknown token. (-2741)

Fix: Collapse to a single-line -e argument. No line continuation needed.

5.3 PIN Collection (verify-required Keys)

What was done: Key was created with -O verify-required, setting the FIDO2 UV (user verification) bit, requiring PIN at every assertion.

Issue encountered — PIN dialog never appearing:

Initial hypothesis: SSH_ASKPASS and SSH_ASKPASS_REQUIRE set in the wrapper script would be inherited by the sk-helper.

Disproven by the debug log:

SSH_ASKPASS=
SSH_ASKPASS_REQUIRE=

The wrapper’s debug block ran before the exports, showing the vars were empty at entry, confirming they were not being passed in from outside. More critically: the agent collects the PIN via read_passphrase() before invoking the sk-helper. The sk-helper never sees SSH_ASKPASS, the agent process does. Setting these vars in the wrapper is after the fact.

Correct fix: SSH_ASKPASS and SSH_ASKPASS_REQUIRE must be in the launchd plist’s EnvironmentVariables block, so the agent process itself inherits them. This was the definitive resolution.

5.4 sk-askpass Script (PIN Dialog)

What was done: A bash script that uses osascript to present a native macOS PIN dialog.

Issue encountered — AppleScript double-quote injection:

The real prompt string from OpenSSH 10.2 was:

Enter PIN and confirm user presence for ED25519-SK key SHA256:qwho5NkwOCFf0b/yIil2l/vXowxVw8hCF3eqN+uevaw:

When interpolated into an AppleScript string literal without escaping, the " characters in any variant of the prompt (particularly around key identifiers in some OpenSSH builds) would produce """ a syntax error. The fix was to escape double quotes before interpolation:

safe_prompt="${prompt//\"/\\\"}"

Issue encountered — Two-call sequence (OpenSSH version delta):

Initial implementation branched on prompt content, expecting two separate read_passphrase() calls:

1. "Confirm user presence for key..." → presence check
2. "Enter PIN for..." → PIN entry

This was based on older OpenSSH behavior. OpenSSH 10.2 (the homebrew version) combined these into a single call:

Enter PIN and confirm user presence for ED25519-SK key SHA256:...

This was discovered by instrumenting the askpass script with a debug log:

echo "$(date) PROMPT=[$1]" >> /tmp/sk-askpass-debug.log

However, a second call was still observed. Investigation revealed this is a second separate invocation that occurs after PIN collection. The two-call sequence is:

1. "Enter PIN and confirm user presence for ED25519-SK key SHA256:..." → combined, returns PIN
2. "Confirm user presence for key ED25519-SK SHA256:..." → presence-only, must return empty string and exit 0

The correct implementation: match on ^confirm user presence (case-insensitive) for the second call and exit 0 silently. The sk-helper-wrapper’s touch notification fires after both askpass calls return, providing the visual cue at exactly the right moment.

5.5 Resident Key RP ID (Bare ssh:)

What was found: The signing key was created without -O application=, producing RP ID ssh: (bare).

Risk assessment: A bare ssh: RP ID functions because OpenSSH defaults to ssh: when no application string is specified in the assertion request. The credential’s stored RP ID matches the assertion’s RP ID, so it works. However, it’s non-portable, non-auditable, and a security smell.

Resolution: Delete the old credential, recreate with explicit RP ID:

ssh-keygen -t ed25519-sk \
  -O resident \
  -O application=ssh:github \
  -O verify-required \
  -C "github-yk-s5-nano" \
  -f ~/.ssh/id_ed25519_sk_git

This also served as the end-to-end integration test — if the new key worked with the full PIN + touch flow, all components were correctly wired.

---

6. Architecture of the Final Solution

6.1 Mac

┌─────────────────────────────────────────────────────────────────┐
│ launchd (com.user.homebrew-ssh-agent)                           │
│                                                                 │
│  /opt/homebrew/bin/ssh-agent -D -a ~/.ssh/agent.sock            │
│                                                                 │
│  EnvironmentVariables:                                          │
│    SSH_SK_HELPER  → ~/.ssh/sk-helper-wrapper                    │
│    SSH_ASKPASS    → ~/.ssh/sk-askpass                           │
│    SSH_ASKPASS_REQUIRE → force                                  │
└───────────────────────┬─────────────────────────────────────────┘
                        │
                        │ on FIDO2 signing request
                        ▼
┌─────────────────────────────────────────────────────────────────┐
│ ssh-agent process                                               │
│                                                                 │
│  read_passphrase("Enter PIN and confirm user presence...")      │
│    → execs SSH_ASKPASS (sk-askpass)                             │
│    → macOS PIN dialog shown, user enters PIN                    │
│    → PIN returned via stdout                                    │
│                                                                 │
│  read_passphrase("Confirm user presence...")                    │
│    → execs SSH_ASKPASS (sk-askpass)                             │
│    → silent exit 0, empty string returned                       │
│                                                                 │
│  invokes SSH_SK_HELPER (sk-helper-wrapper)                      │
│    → osascript notification: "Touch your YubiKey"               │
│    → execs real ssh-sk-helper                                   │
└───────────────────────┬─────────────────────────────────────────┘
                        │
                        ▼
┌─────────────────────────────────────────────────────────────────┐
│ /opt/homebrew/opt/openssh/libexec/ssh-sk-helper                 │
│                                                                 │
│  libfido2 → YubiKey 5 Nano (USB)                                │
│    → UV: PIN already collected and passed                       │
│    → UP: waiting for physical touch                             │
│    ← FIDO2 assertion returned                                   │
└─────────────────────────────────────────────────────────────────┘

6.2 Linux (Agent Forwarding Chain)

Linux shell                    Mac ssh-agent
     │                               │
     │  git commit -S                │
     │    → ssh-keygen -Y sign       │
     │    → agent socket request     │
     │         ($SSH_AUTH_SOCK       │
     │          = ~/.ssh/agent.sock  │
     │          → symlink            │
     │          → /tmp/ssh-.../      │
     │          → forwarded socket)  │
     │                               │
     └──────── sign request ────────►│
                                     │  PIN dialog on Mac
                                     │  Touch notification on Mac
                                     │  YubiKey touched on Mac
                                     │
     ◄──────── signature ────────────┘
     │
     │  commit object written

6.3 Linux: Stable SSH_AUTH_SOCK via ~/.ssh/rc

# ~/.ssh/rc — executed by sshd after auth, before shell
if [ -S "$SSH_AUTH_SOCK" ]; then
    ln -sf "$SSH_AUTH_SOCK" "$HOME/.ssh/agent.sock"
fi

$SSH_AUTH_SOCK in this context is set by sshd to the forwarded agent socket for this session. The symlink at ~/.ssh/agent.sock is a stable pointer that tmux panes always use, regardless of when they were opened relative to the SSH connection. Shell config exports SSH_AUTH_SOCK="$HOME/.ssh/agent.sock" permanently.

---

7. Signal vs. Noise: Diagnostic Methodology

7.1 Read the agent log before guessing

/tmp/com.user.homebrew-ssh-agent.err was the single most valuable diagnostic source. Every agent-side failure was logged there. The key lesson: agent refused operation from the SSH client side is opaque — the agent log translates it to specific errors:

Agent log message	Root cause
syntax error: unknown token	AppleScript \ continuation in osascript call
incorrect passphrase supplied to decrypt private key	sk-helper exited non-zero due to PIN collection failure
execution error: User canceled (-128)	User dismissed the osascript dialog

7.2 Instrument before modifying

When the PIN dialog wasn’t appearing despite SSH_ASKPASS being set in the wrapper, the debug approach was to add a log block to the wrapper itself:

{
  echo "=== $(date) ==="
  echo "SSH_ASKPASS=$SSH_ASKPASS"
  echo "SSH_ASKPASS_REQUIRE=$SSH_ASKPASS_REQUIRE"
} >> /tmp/sk-helper-debug.log

This revealed SSH_ASKPASS= (empty), definitively proving the vars were not in the agent’s environment — the wrapper was being invoked but the agent that spawned it had no SSH_ASKPASS set. This collapsed a three-hypothesis space (wrong path, wrong timing, wrong process) into a single confirmed cause.

7.3 Capture the real prompt before writing the handler

Rather than assuming the prompt string format, the askpass script was instrumented to log the actual argument:

echo "$(date) PROMPT=[$1]" >> /tmp/sk-askpass-debug.log
exit 1

This revealed the OpenSSH 10.2 combined prompt format, which differed from what older documentation described. Writing the handler against the real data rather than assumed data prevented a class of string-matching bugs.

7.4 Test components in isolation before integrating

Every script was tested standalone before being integrated into the agent flow:

~/.ssh/sk-askpass "Enter PIN for YubiKey:"      # verify PIN dialog
~/.ssh/sk-askpass "Confirm user presence..."     # verify silent exit
~/.ssh/sk-helper-wrapper                         # verify notification fires

This established a known-good baseline for each component, making integration failures attributable to the wiring rather than the components.

---

8. Lessons & Design Principles

1. Daemon environment ≠ shell environment. A launchd-managed process inherits only what its plist specifies. Environment variables set in your shell, your .zshrc, or even in a child process of the daemon are invisible to the daemon itself. Any configuration that a daemon needs must be declared in its EnvironmentVariables plist block. This is a frequent source of “works in terminal, fails in daemon” failures.

2. Agent forwarding is a proxy, not a copy. The forwarded agent socket on Linux does not run a local agent — it’s a Unix socket whose other end is the Mac’s ssh-agent. Every signing request travels over the SSH connection, is serviced by the Mac agent, and the result is returned. This means all UX (PIN dialogs, touch notifications) happens on the Mac, not Linux. Any attempt to configure askpass on Linux for forwarded SK key operations is misguided.

3. RP ID matters for portability. FIDO2 credentials are bound to their RP ID at creation time. OpenSSH defaults to ssh: when no -O application= is given. While this works, it produces credentials that are indistinguishable from each other in audit and creates assertion requests with no meaningful relying-party identifier. Always specify -O application=ssh:<something> when creating resident keys.

4. verify-required has real UX cost — plan for it. The -O verify-required flag sets the FIDO2 UV bit. Every assertion requires PIN + touch, not just touch. This is the correct security posture for signing keys, but it requires the full SSH_ASKPASS infrastructure to be working before it’s usable in a daemon context. Touch-only keys work without SSH_ASKPASS because read_passphrase() is called once and returns empty string, which the key accepts. PIN-required keys fail silently without it.

5. Stable socket paths are infrastructure, not convenience. The tmux-managed, randomly-named agent socket was a ticking clock — it survived by accident. Stable socket paths (~/.ssh/agent.sock) managed by a proper service supervisor (launchd on Mac, a symlink-on-login mechanism on Linux) are a first-class architectural requirement for any workflow that spans terminal sessions, tmux attach/detach cycles, and SSH reconnects.

---

9. Final File Inventory

File	Platform	Purpose
~/Library/LaunchAgents/com.user.homebrew-ssh-agent.plist	Mac	Launchd service definition for homebrew ssh-agent
~/.ssh/sk-helper-wrapper	Mac	SSH_SK_HELPER override — fires touch notification, execs real sk-helper
~/.ssh/sk-askpass	Mac	SSH_ASKPASS handler — PIN dialog via osascript, silent exit for presence prompt
~/.ssh/agent.sock	Both	Stable socket symlink (Mac: real socket; Linux: symlink to forwarded socket)
~/.ssh/rc	Linux	sshd post-auth hook — updates ~/.ssh/agent.sock symlink to current forwarded socket
~/.ssh/id_ed25519_sk_git.pub	Both	FIDO2 resident key public stub, RP ID ssh:github
~/.ssh/allowed_signers	Both	Git SSH signature verification allowlist
~/.config/github/git_config_global	Both	Git config — GIT_CONFIG_GLOBAL target, SSH signing configured

---

10. Verification Checklist

A complete end-to-end verification must exercise each link in the chain independently:

# 1. Agent is alive and has the key
ssh-add -L | grep "ssh:github"

# 2. GitHub auth works (touch notification fires, YubiKey touch required)
ssh -T git@github.com

# 3. Commit signing works (PIN dialog, touch notification, signature verifiable)
cd /tmp && git init test && cd test
git commit --allow-empty -m "test signing"
git log --show-signature -1
# Expected: "Good "git" signature for <email> with ED25519-SK key SHA256:..."

# 4. (Linux) Forwarded agent visible
ssh -A linux-box "ssh-add -L | grep github"

# 5. (Linux) GitHub auth via forwarded agent (prompts on Mac)
ssh -A linux-box "ssh -T git@github.com"

# 6. (Linux) Commit signing via forwarded agent (prompts on Mac)
ssh -A linux-box "cd /tmp && rm -rf test && git init test && cd test && \
  git -c user.email='...' -c user.signingkey=~/.ssh/id_ed25519_sk_git.pub \
  commit --allow-empty -m 'test linux signing' && \
  git log --show-signature -1"

All six checks passing constitutes a verified, production-ready setup.