Test the effective policy of every coding-agent surface

GitHub says enterprise managed settings now extend to the GitHub Copilot app and Copilot cloud agent. Enterprise owners can restrict plugins and marketplaces centrally, and managed values override local settings. The same announcement also identifies a surface boundary: approval-bypass controls apply only to interactive clients. For the cloud agent, plugin and marketplace controls take effect on its next task assignment. Read the GitHub announcement.

That is a useful implementation case for a broader operating problem. A shared configuration can describe intended policy without demonstrating identical enforcement across desktop, CLI, IDE, and cloud execution surfaces. Our recommendation is to test the effective policy of each client and retain the result as release evidence.

Build the matrix around observable behavior

Start with one row per combination of surface and control. A compact worksheet can use these fields:

Field	What to record
Surface	Desktop app, CLI, IDE integration, cloud agent, or another in-scope client
Policy source	Enterprise setting, organization setting, repository configuration, or local value
Control	Plugin restriction, marketplace restriction, approval behavior, credential scope, or update behavior
Supported here?	Documented support, documented exclusion, or unknown
Precedence	Which value wins when managed and local settings conflict
Refresh trigger	Restart, settings refresh, next task, or another documented event
Test action	The smallest action that exercises the control
Expected result	Allow, deny, request approval, or defer until refresh
Observed result	The exact UI, CLI, API, or task outcome
Evidence retained	Timestamp, client version, configuration revision, task identifier, and relevant log or screenshot
Owner	The person or team responsible for resolving a mismatch

This is a proposed audit design, not a claim that GitHub supplies this worksheet. Its purpose is to separate four states that are easy to collapse: a control exists, an administrator configured it, a particular surface consumed it, and a test observed the expected behavior.

Convert documented boundaries into acceptance tests

For GitHub’s announced change, the source excerpt supports three initial test families. First, set a managed plugin or marketplace restriction and attempt the prohibited operation from every covered surface. Second, create a conflict between a managed value and a local setting, then verify that the managed value wins. Third, change a plugin or marketplace control for the cloud agent and submit a new task, because GitHub identifies the next task assignment as its refresh point. Approval-bypass testing belongs only in the interactive-client rows because the announcement limits that control accordingly. See the documented scope and refresh behavior.

Record the denial itself, not only the configuration value. A useful result includes the attempted operation, effective client version, relevant policy revision, observed message or status, and time of execution. If a surface permits the action unexpectedly, classify that as a policy-verification failure until the documented scope, precedence, or refresh behavior explains it.

Unknown support deserves its own state. Treating unknown as denied would overstate the evidence; treating it as allowed could hide exposure. Mark it explicitly and assign an owner to obtain documentation or run a bounded test.

Connect policy checks to release gates

OWASP recommends limiting agents to the tools needed for a task, obtaining explicit approval for high-impact or irreversible actions, and retaining audit trails. It also recommends blocking releases when high-risk tool policies, approval logic, or credential scopes change without updated tests. Read the OWASP AI Agent Security Cheat Sheet.

A practical release rule derived from that guidance is to trigger matrix retesting when any of these inputs changes:

• A client, extension, or cloud-agent version
• Plugin or marketplace policy
• Approval logic
• Credential scope
• Policy precedence
• A surface-specific refresh mechanism

The release record can then link each changed input to the affected rows and their latest observations. This does not prove the absence of every bypass. It provides bounded evidence that named controls produced expected outcomes on named surfaces under a recorded configuration.

Make refresh behavior part of the control

Configuration propagation is operational behavior, not clerical detail. If one surface reads a policy immediately while another waits for a new task, a test performed before the appropriate trigger can produce a misleading result.

For every row, define the starting state, the policy mutation, the documented or hypothesized refresh trigger, and the first action after refresh. Where documentation is silent, label the trigger as a test assumption and capture what happens before and after restart, reconnect, reload, or new-task submission as applicable. Do not convert one successful refresh into a universal claim about other clients.

Keep one reusable evidence packet

The finished artifact is more useful than a screenshot of an administration page. It combines intended configuration with observed enforcement:

1. Freeze the client versions and policy revision.
2. Enumerate every in-scope surface and supported control.
3. Introduce controlled conflicts to test precedence.
4. Apply the surface-specific refresh trigger.
5. Attempt an operation expected to be denied or gated.
6. Retain the result with its timestamp and owner.
7. Repeat affected rows after relevant policy, approval, credential, or client changes.

Teams adapting this matrix to their own agent stack can use the same acceptance-test discipline when they build agent systems: define the policy source, exercise the effective behavior, and preserve enough context to reproduce the result.

The key distinction is simple. Shared settings establish administrative intent. Surface-specific tests show whether that intent became observable enforcement in the configuration that was actually exercised.