Implement copilot quality alt by kzhou314 · Pull Request #38 · github/accessibility-scanner-alt-text-plugin

kzhou314 · 2026-06-18T23:19:00Z

Closes github/accessibility#10635

Adds an opt-in, model-backed alt-text-quality rule that judges whether an image's alt text actually describes the image — a gap the deterministic rules can't reach.

Design & rationale: ADR

Copilot

⚠️ Not ready to approve

There are correctness and reliability issues in new code paths (notably link-context wording in prompts and loadImageAsDataUrl handling of data: URLs / timeouts) that should be addressed before merging.

Pull request overview

Adds a new opt-in, model-backed alt-text-quality rule to evaluate whether an image’s alt text actually describes the image (using GitHub Models, optionally enriched by an Azure AI Vision pre-pass), and extends image extraction to capture additional context for higher-quality judging.

Changes:

Introduces the async, opt-in alt-text-quality rule plus a new judge layer (Copilot judge, optional Azure-augmented judge, caching).
Enhances extractImages and ImageRecord with intrinsic dimensions and page/section/prose context used by the model-backed rule.
Adds an offline probe harness + fixtures and updates docs/schema/tests to cover the new rule and async rule evaluation.

File summaries

File	Description
tsconfig.json	Includes `scripts/*/.ts` in typechecking.
tests/utils/helpers.ts	Adds `ImageRecord` defaults for new fields; adds sync-only rule evaluation helper.
tests/unit/vague-alt-text.test.ts	Updates typing to account for async-capable `Rule.evaluate`.
tests/unit/missing-alt-text.test.ts	Updates typing to account for async-capable `Rule.evaluate`.
tests/unit/judges-caching.test.ts	Adds unit tests for judge/vision caching behavior.
tests/unit/alt-text-quality.test.ts	Adds unit tests for the new `alt-text-quality` rule behavior.
tests/fixtures/alt-quality/cases.json	Adds offline probe fixture cases for expected verdicts.
tests/fixtures/alt-quality/cases-github.json	Adds GitHub-specific probe fixture cases.
tests/extract.test.ts	Adds tests for capturing `pageTitle` and `sectionHeading`.
tests/example-site.test.ts	Skips opt-in rules when asserting expected findings in the example site.
src/utils/load-image-data-url.ts	Adds helper to load images as data URLs for the rule and probe.
src/utils/fetch-with-retry.ts	Adds fetch wrapper with timeout and bounded retry for transient failures.
src/types.ts	Extends `ImageRecord`; makes `Rule.evaluate` async-capable.
src/rules/index.ts	Registers `alt-text-quality` in the append-only rules list.
src/rules/alt-text-quality.ts	Implements the async, opt-in model-backed rule orchestrator.
src/judges/types.ts	Defines judge interfaces/types shared across judge implementations.
src/judges/prompt.ts	Adds the system prompt and strict JSON schema for structured verdicts.
src/judges/index.ts	Adds `createJudge()` with mode selection and wiring (copilot vs azure-augmented + caching).
src/judges/copilot-judge.ts	Implements the GitHub Models-backed judge call.
src/judges/caching.ts	Adds content-hash caches for verdicts and Azure vision analysis.
src/judges/azure-vision-api-client.ts	Implements Azure AI Vision Image Analysis client.
src/judges/azure-augmented-judge.ts	Decorates the Copilot judge with optional Azure pre-analysis context.
src/extract.ts	Enhances extraction with nearby text, figcaption, link/button context, page title, and section heading.
scripts/probe-alt-quality.ts	Adds offline probe CLI for scoring fixture cases against the judge.
schema/config.schema.json	Adds `alt-text-quality` to config schema and updates schema description.
README.md	Documents the new opt-in rule, configuration, and required secrets.
package.json	Adds `probe` script and `tsx` dev dependency.
package-lock.json	Locks new dependencies (`tsx`, `esbuild`, etc.).
index.ts	Awaits rule evaluation to support async rules.
.gitignore	Ignores `.env` (used by the probe).
.github/workflows/scan-static-sites.yml	Adds job-level env wiring for model/vision credentials.

Copilot's findings

Files reviewed: 28/31 changed files
Comments generated: 4

Note

Your feedback helps us improve the quality of this feature.
Please use 👍 or 👎 to tell us whether this assessment is correct.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

⚠️ Not ready to approve

It introduces a model call pathway that currently forwards raw outerHTML into prompts (risking sensitive URL/query leakage) and includes a few correctness/perf/test-safety issues that should be addressed before approval.

Copilot's findings

Files reviewed: 29/32 changed files
Comments generated: 6

Note

Your feedback helps us improve the quality of this feature.
Please use 👍 or 👎 to tell us whether this assessment is correct.

Copilot

⚠️ Not ready to approve

The model-backed rule currently risks leaking sensitive URL data (e.g., link hrefs and logged image URLs with querystrings) into model context and CI logs, which should be redacted before merging.

Copilot's findings

Files reviewed: 29/32 changed files
Comments generated: 5

Note

Your feedback helps us improve the quality of this feature.
Please use 👍 or 👎 to tell us whether this assessment is correct.

+function normalizeDataUrl(dataUrl: string): string {
+  const comma = dataUrl.indexOf(',')
+  if (comma === -1) return dataUrl
+  const meta = dataUrl.slice('data:'.length, comma)
+  if (/;base64/i.test(meta)) return dataUrl
+  const payload = dataUrl.slice(comma + 1)
+  let decoded: string
+  try {
+    decoded = decodeURIComponent(payload)
+  } catch {
+    decoded = payload
+  }
+  const mime = meta.split(';')[0] || 'text/plain'
+  return `data:${mime};base64,${Buffer.from(decoded, 'utf8').toString('base64')}`
+}


taarikashenafi · 2026-06-23T21:43:54Z

+  if (image.inLink) parts.push(`The image is inside a link with href="${image.inLink.href}".`)
+  if (image.inButton) parts.push('The image is inside a button (or role="button" element).')


I'm gonna add a small redactUrl() helper and apply it here + the console.error lines in evaluate() as they all leak the same querystring stuff into CI logs. The ADR calls this out too. Opening a PR (#45) so we don't end up doing this in two places.

+      } catch (err) {
+        console.error(`[alt-text-quality] failed to load ${resolved}:`, err)
+        continue
+      }


+      } catch (err) {
+        console.error(`[alt-text-quality] judge failed for ${resolved}:`, err)
+        continue
+      }


taarikashenafi · 2026-06-25T17:57:57Z

+export async function fetchWithRetry(
+  url: string | URL,
+  init: RequestInit,
+  options: FetchRetryOptions = {},
+): Promise<Response> {


+1, this one's worth doing before merge, it's in the path of every model and image call.

taarikashenafi

btw on doing the other stretch goals in this branch, might be easier to get this one in first and then branch the next thing off main? this one's already pretty big and the rules registry seems set up for adding one rule at a time, so keeping them split might make each easier to review since Joyce hasn't' started on this yet.

taarikashenafi · 2026-06-23T21:43:54Z

+  if (image.inLink) parts.push(`The image is inside a link with href="${image.inLink.href}".`)
+  if (image.inButton) parts.push('The image is inside a button (or role="button" element).')


I'm gonna add a small redactUrl() helper and apply it here + the console.error lines in evaluate() as they all leak the same querystring stuff into CI logs. The ADR calls this out too. Opening a PR (#45) so we don't end up doing this in two places.

kzhou314 · 2026-06-23T23:17:12Z

btw on doing the other stretch goals in this branch, might be easier to get this one in first and then branch the next thing off main? this one's already pretty big and the rules registry seems set up for adding one rule at a time, so keeping them split might make each easier to review since Joyce hasn't' started on this yet.

Yeah for sure, good call.

…ion, doc/context fixes

…op unsafe test casts

GitHub Advanced Security started work on behalf of kzhou314 June 18, 2026 23:19 View session

GitHub Advanced Security finished work on behalf of kzhou314 June 18, 2026 23:21

GitHub Advanced Security started work on behalf of kzhou314 June 18, 2026 23:39 View session

GitHub Advanced Security finished work on behalf of kzhou314 June 18, 2026 23:40

GitHub Advanced Security started work on behalf of kzhou314 June 19, 2026 00:01 View session

GitHub Advanced Security finished work on behalf of kzhou314 June 19, 2026 00:03

GitHub Advanced Security started work on behalf of kzhou314 June 19, 2026 00:29 View session

GitHub Advanced Security finished work on behalf of kzhou314 June 19, 2026 00:31

kzhou314 marked this pull request as ready for review June 19, 2026 00:38

kzhou314 requested a review from a team as a code owner June 19, 2026 00:38

kzhou314 requested review from accessibility-bot and Copilot June 19, 2026 00:38

Copilot started reviewing on behalf of kzhou314 June 19, 2026 00:38 View session

Copilot AI reviewed Jun 19, 2026

View reviewed changes

Comment thread src/types.ts

Comment thread src/rules/alt-text-quality.ts Outdated

Comment thread src/utils/load-image-data-url.ts

Comment thread src/utils/load-image-data-url.ts

GitHub Advanced Security started work on behalf of kzhou314 June 19, 2026 00:55 View session

kzhou314 requested a review from Copilot June 19, 2026 00:56

Copilot started reviewing on behalf of kzhou314 June 19, 2026 00:56 View session

GitHub Advanced Security finished work on behalf of kzhou314 June 19, 2026 00:57

Copilot AI reviewed Jun 19, 2026

View reviewed changes

GitHub Advanced Security started work on behalf of kzhou314 June 19, 2026 01:25 View session

GitHub Advanced Security finished work on behalf of kzhou314 June 19, 2026 01:27

Copilot AI review requested due to automatic review settings June 19, 2026 18:37

GitHub Advanced Security started work on behalf of kzhou314 June 19, 2026 18:38 View session

Copilot started reviewing on behalf of kzhou314 June 19, 2026 18:38 View session

GitHub Advanced Security finished work on behalf of kzhou314 June 19, 2026 18:40

Copilot AI reviewed Jun 19, 2026

View reviewed changes

taarikashenafi self-requested a review June 23, 2026 21:30

taarikashenafi reviewed Jun 23, 2026

View reviewed changes

taarikashenafi mentioned this pull request Jun 24, 2026

Redact query/fragment from URLs sent to model and logs #45

Open

kzhou314 force-pushed the main branch from 5cef4a8 to 80ac88f Compare June 24, 2026 21:39

kzhou314 added 21 commits June 24, 2026 15:08

Add deps and config for alt-text-quality rule

4ad74ec

Add alt-text judge provider seam (Copilot + Azure-augmented)

7681fd9

Extend image extraction with structured context fields

27e00a0

Add alt-text-quality rule (default disabled)

3c7124c

Update tests and add alt-quality fixture

1959031

Add offline probe harness

b863e95

Add GitHub-representative alt-text-quality fixture

d7e1296

Refine Azure preamble and clarify NotImplemented vision client

ad639e0

Skip Azure pre-pass for images 50px or smaller

947e05c

Pass intrinsic image dimensions through probe

7893ded

Trim verbose comments in Azure vision client

436f64d

Auto-select azure-augmented mode when Azure creds present

c9c5ee3

Document alt-text-quality rule in README

6c6663c

Add timeout and retry to model and vision requests

c7d7d44

Add alt-text-quality to config schema

9c8cb38

Add unit tests for alt-text-quality rule

b3e26c6

Pass page title and section heading to the judge

6560deb

Add judgment and vision-extraction caches to the judge layer

794ea25

Address Copilot review: fetch timeout for images, data URL normalizat…

dcaa504

…ion, doc/context fixes

Address Copilot review: sanitize image HTML, widen data URL regex, dr…

fc806c1

…op unsafe test casts

Trim verbose comments; simplify image extraction

5ae4bc1

kzhou314 force-pushed the implement-copilot-quality-alt branch from 9b5b167 to 5ae4bc1 Compare June 24, 2026 22:11

GitHub Advanced Security started work on behalf of kzhou314 June 24, 2026 22:11 View session

GitHub Advanced Security finished work on behalf of kzhou314 June 24, 2026 22:13

		if (image.inLink) parts.push(`The image is inside a link with href="${image.inLink.href}".`)
		if (image.inButton) parts.push('The image is inside a button (or role="button" element).')

Uh oh!

Conversation

kzhou314 commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

⚠️ Not ready to approve

Copilot's findings

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

⚠️ Not ready to approve

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

⚠️ Not ready to approve

Uh oh!

taarikashenafi Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

taarikashenafi Jun 25, 2026

Choose a reason for hiding this comment

Uh oh!

taarikashenafi left a comment

Choose a reason for hiding this comment

Uh oh!

taarikashenafi Jun 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kzhou314 commented Jun 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kzhou314 commented Jun 18, 2026 •

edited

Loading

taarikashenafi Jun 23, 2026 •

edited

Loading

taarikashenafi Jun 23, 2026 •

edited

Loading