Test harness CLI v1.0.32 incompatible with permission kind values and session.auth from commit dd2dcbc

[brunoborges]

Summary

Commit dd2dcbc (Per-session GitHub authentication for all SDK languages, plus update runtime and permissions to match (#1124)) introduced two protocol changes:

1. New permission kind values: approve-once, reject, user-not-available, no-result (replacing approved, denied-by-rules, denied-no-approval-rule-and-could-not-request-from-user, denied-interactively-by-user)
2. New RPC method: session.auth.getStatus for per-session authentication

However, the test harness (test/harness/package.json) still pins "@github/copilot": "^1.0.32", and CLI v1.0.32 does not support either of these changes:

• CLI v1.0.32 only accepts the old permission kind strings. Sending the new values causes Unhandled permission result kind: [object Object], which makes every tool execution fail with permission errors.
• CLI v1.0.32 does not implement session.auth.getStatus, returning JsonRpcException: Unhandled method session.auth.getStatus.

Impact

Any SDK that faithfully ports commit dd2dcbc will have broken E2E tests when run against the test harness. We observed this in the Java SDK (github/copilot-sdk-java) where 14 E2E tests fail (11 tool/permission failures + 3 auth errors).

Failing test categories

Permission/Tool failures (11): The CLI rejects the new permission kind values, so all tool executions fail. The model responds with apology messages instead of executing tools. Examples:

• ToolsTest.testInvokesBuiltInTools — expects tool output, gets permission error
• PermissionsTest.testPermissionHandlerForWriteOperations — permission handler never invoked
• CopilotSessionTest.testShouldCreateSessionWithCustomTool — custom tool never executes

Per-session auth errors (3): All PerSessionAuthTest methods throw Unhandled method session.auth.getStatus. Additionally, no test snapshots exist for auth tests under test/snapshots/.

Expected fix

Either:

1. Bump the CLI dependency in test/harness/package.json to a version that supports the new permission protocol and session.auth.getStatus
2. Or add auth test snapshots and backport the new permission kind values to the CLI

Reproduction

# In any SDK that has ported commit dd2dcbc:
# Run E2E tests against the test harness with @github/copilot@1.0.32
# Observe 14 test failures

References

• Reference impl commit: dd2dcbc
• CLI version in harness: @github/copilot@1.0.32
• Affected downstream: github/copilot-sdk-java (PR Add GitHub Actions workflow for .NET Core Desktop #122)

[brunoborges]

This is also affecting real users of the Java SDK in production — see github/copilot-sdk-java#126. Users are having to manually send the old "approved" string as a workaround.

[github-actions]

Investigation findings

Label applied: bug — this is a genuine regression in the test infrastructure.

What I investigated

• test/harness/package.json — confirms @github/copilot is pinned to ^1.0.32; the lock file also resolves to exactly 1.0.32
• nodejs/src/generated/rpc.ts — shows the new permission result kinds: approve-once, reject, user-not-available (no approved/denied-* variants)
• nodejs/src/types.ts — PermissionRequestResult uses the new kinds; approveAll returns { kind: "approve-once" }
• nodejs/src/session.ts — emits user-not-available when no handler is registered

What I found

Commit dd2dcbc (Per-session GitHub authentication for all SDK languages, plus update runtime and permissions to match #1124) made two breaking protocol changes that require a newer CLI build:

1. New permission result kinds: approve-once / reject / user-not-available replace the old approved / denied-* strings. CLI v1.0.32 does not recognise the new values, responding with Unhandled permission result kind: [object Object], which causes every tool invocation to fail with a permission error.
2. New RPC method session.auth.getStatus: not implemented in CLI v1.0.32, returns JsonRpcException: Unhandled method session.auth.getStatus.

The test harness was not bumped to a compatible CLI version when these protocol changes landed, so any SDK that faithfully ports the commit (e.g. the Java SDK) will hit 14+ E2E failures immediately.

Fix needed

The test/harness/package.json dependency "@github/copilot": "^1.0.32" needs to be bumped to the minimum CLI version that supports both the new permission protocol and session.auth.getStatus. Auth test snapshots under test/snapshots/ may also need to be added so that the replay harness can cover PerSessionAuth scenarios.

Generated by Bug Handler for issue #1146 · ● 203.6K · ◷

[brunoborges]

Update: Tested with CLI v1.0.36 (latest available on npm) — same failures. The new permission kind values (approve-once, reject, user-not-available, no-result) and session.auth.getStatus are not supported in any released CLI version up to and including v1.0.36.

The CLI's permission handler still only accepts the old values: approved, denied-by-rules, denied-no-approval-rule-and-could-not-request-from-user, denied-interactively-by-user, denied-by-content-exclusion-policy, denied-by-permission-request-hook.

This means the protocol changes in commit dd2dcbc cannot be used by any downstream SDK until a new CLI version ships with support for them.

[edburns]

See PR #1150

[SteveSandersonMS]

Update: Tested with CLI v1.0.36 (latest available on npm) — same failures. The new permission kind values (approve-once, reject, user-not-available, no-result) and session.auth.getStatus are not supported in any released CLI version up to and including v1.0.36.

I'm not sure what's causing the behavior you're observing, but 1.0.36 is the version where approve-once etc. is added. The E2E tests in CI are using this runtime version and passing (e.g., https://github.com/github/copilot-sdk/actions/runs/24973224129/job/73120080224).

Any SDK that faithfully ports commit dd2dcbc will have broken E2E tests when run against the test harness

I don't think that's the case. The test harness only uses that package to extract a TypeScript type. It's not actually invoking the binary bundled there at runtime in the tests.