fix(cli-proxy): surface HTTP status in DIFC probe diagnostics by lpcox · Pull Request #5616 · github/gh-aw-firewall

lpcox · 2026-06-27T17:47:33Z

What & why

The awf-cli-proxy sidecar probes the external DIFC proxy with gh api rate_limit before serving agent traffic. The classifier in containers/cli-proxy/entrypoint.sh only recognized connection-refused, timeout, and DNS failures; anything else became an opaque diagnosis=unknown.

On GitHub Enterprise Cloud data-residency (*.ghe.com) tenants the proxy comes up healthy and is reachable, but the forwarded gh api call returns an HTTP error (the proxy targets the wrong API host). That HTTP error fell into the unknown bucket, and the gh response body was discarded — so the firewall failed fast with no actionable signal. This is the firewall-side diagnostics ask (#3) from github/gh-aw#41225.

Changes

Capture gh stdout (response body) and stderr separately instead of redirecting stdout to /dev/null.
New classification bucket reachable-but-api-error (HTTP NNN) that extracts the HTTP status from the gh output.
Surface the captured gh error on every failed attempt (not just the last), and print the response body on final failure.
Targeted hint when GITHUB_SERVER_URL is a *.ghe.com host, pointing at the DIFC-proxy enterprise-host root cause.
Refactor the classifier into a pure classify_probe_failure() function and add tests/cli-proxy-probe-classify.test.sh (wired into the build.yml "Run shell unit tests" step).

No change to the fail-fast / health-gate semantics — this is purely better diagnostics.

Scope note

This PR is the firewall-side piece only. The underlying root cause — the DIFC proxy not being enterprise-host-aware on *.ghe.com — is tracked in the companion issues:

DIFC proxy is not enterprise-host-aware on GHEC data-residency (*.ghe.com) gh-aw-mcpg#8202 — DIFC proxy enterprise-host awareness (root cause)
Compiler does not propagate enterprise host into curated DIFC-proxy env on *.ghe.com gh-aw#41911 — compiler/env propagation into the curated DIFC-proxy env

Testing

$ bash tests/cli-proxy-probe-classify.test.sh
Results: 8 passed, 0 failed
$ bash tests/setup-iptables-port-spec.test.sh
Results: 53 passed, 0 failed
$ bash -n containers/cli-proxy/entrypoint.sh   # syntax OK

The new test covers each bucket, including HTTP status in stderr vs body, and confirms the unknown fallback does not trip set -e when no HTTP status matches.

Example log (before → after)

Before:

[cli-proxy] DIFC proxy probe failed (attempt 1/10, diagnosis=unknown) ...

After (on a *.ghe.com tenant):

[cli-proxy] ERROR: DIFC proxy liveness probe failed ... diagnosis=reachable-but-api-error (HTTP 404)
[cli-proxy] gh api stderr: gh: Not Found (HTTP 404)
[cli-proxy] gh api response body: {"message":"Not Found", ...}
[cli-proxy] HINT: GITHUB_SERVER_URL=... looks like a GitHub Enterprise data-residency tenant ...

The awf-cli-proxy liveness probe ran 'gh api rate_limit' through the external DIFC proxy and bucketed any failure that was not conn-refused, timeout, or DNS into an opaque 'diagnosis=unknown'. On GHEC data-residency (*.ghe.com) tenants the proxy is reachable but the forwarded call returns an HTTP error, so the real cause was hidden. - Capture gh stdout (response body) and stderr separately instead of discarding stdout. - Add a 'reachable-but-api-error (HTTP NNN)' classification bucket. - Print the gh stderr on every failed attempt and the body on final failure. - Emit a targeted hint when GITHUB_SERVER_URL is a *.ghe.com host. - Extract classify_probe_failure() as a pure function and unit-test it. Refs #5615. Root cause tracked in github/gh-aw-mcpg#8202 and github/gh-aw#41911. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

github-actions · 2026-06-27T17:49:17Z

✅ Coverage Check Passed

Overall Coverage

Metric	Base	PR	Delta
Lines	98.25%	98.28%	📈 +0.03%
Statements	98.17%	98.21%	📈 +0.04%
Functions	99.53%	99.53%	➡️ +0.00%
Branches	94.00%	94.00%	➡️ +0.00%

📁 Per-file Coverage Changes (1 files)

File	Lines (Before → After)	Statements (Before → After)
`src/workdir-setup.ts`	92.7% → 94.5% (+1.82%)	92.7% → 94.5% (+1.82%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

Copilot

Pull request overview

Improves awf-cli-proxy DIFC liveness probe diagnostics by capturing gh api rate_limit stdout/stderr separately, classifying HTTP API failures explicitly (including HTTP status), and surfacing actionable logs/hints (notably for *.ghe.com tenants) while keeping existing fail-fast gating semantics.

Changes:

Refactors probe failure classification into a pure classify_probe_failure() function and adds an HTTP-status-based bucket (reachable-but-api-error (HTTP NNN)).
Captures and surfaces gh stderr on every failed attempt, and prints stderr + response body on final failure (plus a targeted *.ghe.com hint).
Adds a focused shell unit test for the classifier and wires it into the build workflow.

Show a summary per file

File	Description
`containers/cli-proxy/entrypoint.sh`	Adds a testable classifier, captures stdout/stderr separately, and emits improved retry/final-failure diagnostics including HTTP status and a `*.ghe.com` hint.
`tests/cli-proxy-probe-classify.test.sh`	New shell unit test to validate each probe failure classification bucket, including HTTP status extraction paths.
`.github/workflows/build.yml`	Runs the new shell unit test in CI alongside existing shell tests.

Review details

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Files reviewed: 3/3 changed files
Comments generated: 1
Review effort level: Low

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

github-actions · 2026-06-27T17:51:40Z

✅ Copilot review passed with no inline comments.

@lpcox Add the ready-for-aw label to this PR to trigger agentic CI smoke tests.

github-actions · 2026-06-27T17:52:25Z

✅ Smoke Gemini completed. All facets verified. 💎

github-actions · 2026-06-27T17:52:26Z

✅ Smoke Claude passed

github-actions · 2026-06-27T17:52:26Z

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

github-actions · 2026-06-27T17:52:26Z

✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

github-actions · 2026-06-27T17:52:26Z

❌ Security Guard failed. Please review the logs for details.

github-actions · 2026-06-27T17:52:26Z

✅ Build Test Suite completed successfully!

github-actions · 2026-06-27T17:52:28Z

🔌 Smoke Services — All services reachable! ✅

github-actions · 2026-06-27T17:52:29Z

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

github-actions · 2026-06-27T17:52:29Z

✅ Smoke Copilot BYOK AOAI (Entra) completed. Copilot AOAI BYOK (Entra) mode operational. 🔓

Posted smoke test results and added label smoke-copilot-byok-aoai-entra

github-actions · 2026-06-27T17:52:29Z

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

github-actions · 2026-06-27T17:52:30Z

✅ Smoke Copilot BYOK AOAI (api-key) completed. Copilot AOAI BYOK (api-key) mode operational. 🔓

github-actions · 2026-06-27T17:52:34Z

✅ Contribution Check completed successfully!

Contribution check complete: PR #5616 follows the applicable CONTRIBUTING.md guidelines. It has a clear description with related issue references, includes focused tests wired into CI, and places changes in appropriate container/test/workflow locations.

github-actions · 2026-06-27T17:52:36Z

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

github-actions · 2026-06-27T17:52:56Z

🔑 Smoke Copilot PAT PAT auth validated. All systems operational. ✅

github-actions · 2026-06-27T17:55:12Z

Smoke Test: Claude Engine

API check: ✅ PASS
gh check: ✅ PASS
File check: ✅ PASS

Overall result: PASS

Generated by Smoke Claude for issue #5616 · 61.3 AIC · ⊞ 3.3K · ◷

github-actions · 2026-06-27T17:55:23Z

Smoke Test: Copilot BYOK (Direct) Mode — ✅ PASS

✅ GitHub MCP connectivity verified (2 merged PRs fetched)
✅ GitHub.com connectivity: HTTP 200
✅ File write/read test: artifact file exists
✅ BYOK inference path working (api-proxy → api.githubcopilot.com)

Mode: Direct BYOK via COPILOT_PROVIDER_API_KEY with api-proxy sidecar
Status: All smoke tests passed

🔑 BYOK report filed by Smoke Copilot BYOK

github-actions · 2026-06-27T17:55:50Z

🔍 Smoke Test Results

Test	Status
GitHub MCP connectivity	✅
GitHub.com HTTP connectivity	✅ HTTP 200
File write/read	❌ Pre-step outputs not substituted

PR: fix(cli-proxy): surface HTTP status in DIFC probe diagnostics
Author: @lpcox

Overall: FAIL — pre-step template variables (${{ steps.smoke-data.outputs.* }}) were not expanded; file test could not be verified.

📰 BREAKING: Report filed by Smoke Copilot

github-actions · 2026-06-27T17:56:13Z

Smoke Test

Apply safe dependency updates for June 2026 security refresh ✅
Update transitive \linkify-it` to 5.0.1 in lockfile` ✅
GitHub title check ✅
Temp file write/read ✅
Discussion lookup/comment ✅
npm ci && npm run build ✅
Overall: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

registry.npmjs.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "registry.npmjs.org"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex

github-actions · 2026-06-27T17:56:48Z

📡 OTEL Tracing Smoke Test Results

Scenario	Result	Notes
1. Module Loading	✅	`otel.js` loaded; `isEnabled()=true`; exports `startRequestSpan`, `setTokenAttributes`, `setBudgetAttributes`, `endSpan`, `endSpanError`, `shutdown` + internals
2. Test Suite	✅	59 passed, 0 failed across 2 suites (`otel.test.js`, `otel-fanout.test.js`)
3. Env Var Forwarding	⚠️ pending	`src/services/api-proxy-service.ts` does not yet forward `OTEL_EXPORTER_OTLP_ENDPOINT` / `GITHUB_AW_OTEL_TRACE_ID` — expected during development
4. Token Tracker Integration	✅	`onUsage` callback present in `token-tracker-http.js`
5. OTEL Diagnostics	⚠️ n/a	No `otel.jsonl` — expected; file-fallback spans only emit when api-proxy container runs

Summary: Core OTEL implementation is healthy (module + 59 tests passing + onUsage hook in place). Env-var forwarding from the AWF CLI layer to the api-proxy container is the one remaining open item.

📡 OTel tracing validated by Smoke OTel Tracing

github-actions · 2026-06-27T17:57:00Z

🔬 Smoke Test: Copilot PAT Auth — PASS

Test	Result
GitHub MCP connectivity	✅
GitHub.com HTTP status	✅ 200
File write/read	✅

Overall: PASS | Auth mode: PAT (COPILOT_GITHUB_TOKEN)
cc @lpcox

🔑 PAT report filed by Smoke Copilot PAT

github-actions · 2026-06-27T17:57:15Z

🧪 Chroot Version Comparison Results

Runtime	Host Version	Chroot Version	Match?
Python	Python 3.12.13	Python 3.12.3	❌
Node.js	v24.17.0	v22.23.0	❌
Go	go1.22.12	go1.22.12	✅

Overall: ❌ FAILED — Python and Node.js versions differ between host and chroot environment.

Tested by Smoke Chroot

github-actions · 2026-06-27T17:57:22Z

@lpcox
fix(cli-proxy): surface HTTP status in DIFC probe diagnostics ✅
fix: correctly recover runner tool on PATH (after sudo w/ secure_path). remove incorrect reading from GITHUB_PATH ✅
GitHub.com connectivity ✅
File write/read test ✅
Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) ✅

Overall: PASS

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)

github-actions · 2026-06-27T17:57:31Z

🏗️ Build Test Suite Results

Ecosystem	Project	Build/Install	Tests	Status
Bun	elysia	✅	1/1 passed	✅ PASS
Bun	hono	✅	1/1 passed	✅ PASS
C++	fmt	✅	N/A	✅ PASS
C++	json	✅	N/A	✅ PASS
Deno	oak	N/A	1/1 passed	✅ PASS
Deno	std	N/A	1/1 passed	✅ PASS
.NET	hello-world	✅	N/A	✅ PASS
.NET	json-parse	✅	N/A	✅ PASS
Go	color	✅	1/1 passed	✅ PASS
Go	env	✅	1/1 passed	✅ PASS
Go	uuid	✅	1/1 passed	✅ PASS
Java	gson	✅	1/1 passed	✅ PASS
Java	caffeine	✅	1/1 passed	✅ PASS
Node.js	clsx	✅	all passed	✅ PASS
Node.js	execa	✅	all passed	✅ PASS
Node.js	p-limit	✅	all passed	✅ PASS
Rust	fd	✅	1/1 passed	✅ PASS
Rust	zoxide	✅	1/1 passed	✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Generated by Build Test Suite for issue #5616 · 37.3 AIC · ⊞ 7.8K · ◷

github-actions · 2026-06-27T17:57:51Z

@lpcox

fix(cli-proxy): surface HTTP status in DIFC probe diagnostics ✅
fix: correctly recover runner tool on PATH (after sudo w/ secure_path). remove incorrect reading from GITHUB_PATH ✅
GitHub.com connectivity ✅
File write/read ✅
Running in direct BYOK mode (AWF_AUTH_TYPE=github-oidc + AWF_AUTH_AZURE_* + COPILOT_PROVIDER_BASE_URL) via api-proxy → Azure OpenAI (Foundry, o4-mini-aw) authenticated via Microsoft Entra ✅

Overall: PASS

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)

github-actions · 2026-06-27T17:58:42Z

Gemini Engine Validation Results

GitHub MCP Testing: ✅
GitHub.com Connectivity: ✅
File Writing Testing: ✅
Bash Tool Testing: ✅

Overall status: PASS

Last 2 merged PRs:

Apply safe dependency updates for June 2026 security refresh (Apply safe dependency updates for June 2026 security refresh #5609)
Update transitive linkify-it to 5.0.1 in lockfile (Update transitive linkify-it to 5.0.1 in lockfile #5608)

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

localhost

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "localhost"

See Network Configuration for more information.

💎 Faceted by Smoke Gemini

github-actions · 2026-06-27T17:59:34Z

Smoke Test: GitHub Actions Services Connectivity

Check	Result
Redis PING	❌ No response (timeout)
PostgreSQL pg_isready	❌ `no response`
PostgreSQL SELECT 1	❌ No response (timeout)

Overall: FAIL

host.docker.internal resolves to 172.17.0.1 but neither Redis (6379) nor PostgreSQL (5432) are reachable from this agent environment. Service containers do not appear to be running or accessible in the current workflow context.

🔌 Service connectivity validated by Smoke Services

Copilot AI review requested due to automatic review settings June 27, 2026 17:47

Copilot started reviewing on behalf of lpcox June 27, 2026 17:48 View session

Copilot AI reviewed Jun 27, 2026

View reviewed changes

Comment thread tests/cli-proxy-probe-classify.test.sh

Potential fix for pull request finding

8355a78

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

lpcox added the ready-for-aw label Jun 27, 2026

lpcox temporarily deployed to aoai-model June 27, 2026 17:52 — with GitHub Actions Inactive

github-actions Bot added the smoke-claude label Jun 27, 2026

github-actions Bot added the smoke-copilot-byok label Jun 27, 2026

github-actions Bot mentioned this pull request Jun 27, 2026

[aw] No-Op Runs #5368

Open

github-actions Bot added the smoke-codex label Jun 27, 2026

github-actions Bot added the smoke-otel label Jun 27, 2026

github-actions Bot added the smoke-copilot-pat label Jun 27, 2026

github-actions Bot added smoke-copilot-byok-aoai-apikey build-test labels Jun 27, 2026

lpcox temporarily deployed to aoai-model June 27, 2026 17:57 — with GitHub Actions Inactive

github-actions Bot added the smoke-copilot-byok-aoai-entra label Jun 27, 2026

lpcox deployed to aoai-model June 27, 2026 17:57 — with GitHub Actions Active

github-actions Bot added the smoke-gemini label Jun 27, 2026

Uh oh!

Conversation

lpcox commented Jun 27, 2026

What & why

Changes

Scope note

Testing

Example log (before → after)

Uh oh!

github-actions Bot commented Jun 27, 2026

✅ Coverage Check Passed

Overall Coverage

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

Uh oh!

github-actions Bot commented Jun 27, 2026

Uh oh!

github-actions Bot commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jun 27, 2026

Smoke Test: Claude Engine

Uh oh!

github-actions Bot commented Jun 27, 2026

Smoke Test: Copilot BYOK (Direct) Mode — ✅ PASS

Uh oh!

github-actions Bot commented Jun 27, 2026

🔍 Smoke Test Results

Uh oh!

github-actions Bot commented Jun 27, 2026

Smoke Test

Uh oh!

github-actions Bot commented Jun 27, 2026

📡 OTEL Tracing Smoke Test Results

Uh oh!

github-actions Bot commented Jun 27, 2026

🔬 Smoke Test: Copilot PAT Auth — PASS

github-actions Bot commented Jun 27, 2026 •

edited

Loading

github-actions Bot commented Jun 27, 2026 •

edited

Loading

github-actions Bot commented Jun 27, 2026 •

edited

Loading

github-actions Bot commented Jun 27, 2026 •

edited

Loading

github-actions Bot commented Jun 27, 2026 •

edited

Loading

github-actions Bot commented Jun 27, 2026 •

edited

Loading

github-actions Bot commented Jun 27, 2026 •

edited

Loading

github-actions Bot commented Jun 27, 2026 •

edited

Loading

github-actions Bot commented Jun 27, 2026 •

edited

Loading

github-actions Bot commented Jun 27, 2026 •

edited

Loading

github-actions Bot commented Jun 27, 2026 •

edited

Loading

github-actions Bot commented Jun 27, 2026 •

edited

Loading

github-actions Bot commented Jun 27, 2026 •

edited

Loading

github-actions Bot commented Jun 27, 2026 •

edited

Loading