fix(install): align validation auth chain with install

[a1icja]

fix(install): align validation auth chain with install

TL;DR

apm install owner/repo/sub#BRANCH would false-fail validation when the GitHub Contents API rejected the env-var PAT but git clone (driven by the system credential helper) would have succeeded — the same package lands fine when added to apm.yml directly because that path skips validation entirely. This PR teaches validate_virtual_package_exists two new fallbacks: a directory-exists Contents API probe, and a git ls-remote chain that mirrors _clone_with_fallback's auth attempts. Validation now agrees with install for any virtual subdirectory dep that carries an explicit #ref; path-typo rejection on the default branch is preserved.

Note

The ls-remote fallback is gated on dep_ref.reference is not None so unsuffixed owner/repo/sub (no explicit #ref) still fails fast on path typos — the gate keeps that signal.

Problem (WHY)

• Live failure: apm install foo/bar/gee#sho --verbose errors with not accessible or doesn't exist while the equivalent entry in apm.yml installs cleanly. Verbose log shows the resolver found a per-org PAT (source=GITHUB_APM_PAT_FOO, type=classic) but validate_virtual_package_exists still returned False.
• validate_virtual_package_exists only walks the GitHub Contents API. _clone_with_fallback walks an authenticated PAT attempt then a plain HTTPS attempt that lets the system credential helper supply the token — two stages of auth-asymmetry today.
• [!] SSO-half-authorized PATs and EMU / fine-grained-PAT scope mismatches make this asymmetry user-visible: tokens that pass git auth but receive a 4xx on the Contents API are common in enterprise installs.

Why this matters: validation is the gate on a deterministic action, so it must agree with that action. PROSE: "Grounding outputs in deterministic tool execution transforms probabilistic generation into verifiable action." — when the validator's view of "accessible" diverges from git clone's view, the deterministic layer no longer dominates and the install pipeline never gets to run.

Approach (WHAT)

#	Fix
1	Add a directory-exists probe to validate_virtual_package_exists: when no marker file is found, request GET /repos/{owner}/{repo}/contents/{path}?ref={ref} for the directory itself. A 200 means install will find the path.
2	Add a git ls-remote --heads --tags <url> <ref> fallback gated on dep_ref.reference is not None. Walks the same auth chain as _clone_with_fallback, so any token / credential that clones is accepted by validation.
3	Thread verbose_callback from _validate_package_exists into validate_virtual_package_exists so each probe attempt is visible under --verbose ([+] path@ref / [x] path@ref (reason)) without re-running with print statements.

Implementation (HOW)

• src/apm_cli/deps/github_downloader.py — validate_virtual_package_exists rewritten with a _probe(path) helper, marker probes, and two new fallbacks. Three new private helpers: _directory_exists_at_ref, _ref_exists_via_ls_remote, _ssh_attempt_allowed. The auth chain in _ref_exists_via_ls_remote deliberately mirrors _clone_with_fallback's ordering and env builders — per-attempt env selection (self.git_env for the token attempt vs _build_noninteractive_git_env for the credential-helper attempt) is preserved so the locked-down env is only used for the token-bearing attempt.
• src/apm_cli/install/validation.py — one-line change: pass verbose_callback=verbose_log so the new per-probe diagnostics flow through --verbose.
• tests/test_github_downloader.py — adds TestRefExistsViaLsRemote (8 tests) pinning the auth chain: token short-circuits, 403-on-token falls back to credential helper, no-token skips first attempt, all-fail returns False, empty output is a miss not a hit, Artifactory short-circuits, SSH skipped by default, SSH added when ProtocolPreference.SSH is set.
• CHANGELOG.md / docs/src/content/docs/getting-started/authentication.md — one Fixed entry under [Unreleased] and a paragraph describing that validation walks the same chain as install.

Diagrams

Legend: control flow inside validate_virtual_package_exists for a virtual-subdirectory dep — markers fast-path where present, the two new fallbacks reconcile validation with _clone_with_fallback. Look first at the RefGate node — that is the new gate that preserves path-typo rejection for unsuffixed deps.

flowchart TD
    Start[validate_virtual_package_exists] --> Markers{Probe markers:<br/>apm.yml, SKILL.md,<br/>plugin.json, README.md}
    Markers -->|hit| Pass[return True]
    Markers -->|all 404| DirProbe{Contents API<br/>directory-exists<br/>at ref}
    DirProbe -->|200| Pass
    DirProbe -->|404 or non-GitHub| RefGate{dep_ref.reference<br/>is not None?}
    RefGate -->|no| Fail[return False]
    RefGate -->|yes| LsRemote{git ls-remote<br/>chain}
    LsRemote -->|attempt 1: token + git_env| Pass
    LsRemote -->|attempt 2: plain HTTPS<br/>credential helper| Pass
    LsRemote -->|attempt 3: SSH<br/>only if --ssh or<br/>--allow-protocol-fallback| Pass
    LsRemote -->|all fail| Fail

Loading

Trade-offs

• Gated ls-remote fallback on explicit #ref. Chose to require dep_ref.reference is not None; rejected unconditional fallback because path-typo rejection on the default branch is a useful fast signal. Without the gate, a typo like owner/repo/skils would pass validation just because the repo and default branch resolve.
• Directory-exists probe still uses the Contents API. Chose to keep one cheap API call before reaching for git, rejected "go straight to ls-remote" because the Contents API path stays correct for the common case (public repos, fully-authorized PATs) and avoids spawning a git subprocess.
• SSH attempt skipped by default. Chose to gate SSH on ProtocolPreference.SSH or --allow-protocol-fallback, rejected always-attempt because users without SSH configured would see noisy validation output and possibly hung passphrase prompts. BatchMode=yes plus ConnectTimeout=10 further protects users who DO opt in.
• Three private helpers on GitHubPackageDownloader, not a new class. Chose to colocate the new helpers with the existing auth state; rejected a separate Validator class because the auth resolver, git env, and protocol prefs already live on the downloader and a sibling class would duplicate that state.

Benefits

1. The reporter's failing CLI invocation (apm install foo/bar/gee#sho) now succeeds: validation defers to install for any explicit-ref virtual subdir dep.
2. Validation/install asymmetry closed for the common SSO-half-authorized-PAT case: an env-var PAT narrower than the OS keychain credential no longer breaks apm install <pkg>.
3. 8 new pytest tests pin the ls-remote auth chain (token short-circuit, 403 fallback, SSH gating, empty-output miss, Artifactory bypass) so a future refactor cannot silently regress lenience.
4. --verbose now shows every probe attempt ([+] path@ref / [x] path@ref (reason)); future user reports are debuggable without re-running with print statements.
5. No LOC-budget breach: changes land in github_downloader.py (not subject to the install-engine LOC budget) and commands/install.py is untouched.

Validation

pytest — 8 new + 24 existing validation/architecture tests:

$ .venv/bin/python -m pytest \
    tests/test_github_downloader.py::TestRefExistsViaLsRemote \
    tests/unit/test_install_command.py::TestValidationFailureReasonMessages \
    tests/unit/test_install_command.py::TestLocalPathValidationMessages \
    tests/unit/test_install_command.py::TestGenericHostSshFirstValidation \
    tests/unit/install/test_architecture_invariants.py --no-header -q
................................                                         [100%]
32 passed in 0.69s

New `TestRefExistsViaLsRemote` test list (all 8 PASSED)

test_first_attempt_with_token_succeeds_short_circuits
test_authenticated_403_falls_back_to_credential_helper
test_no_token_skips_first_attempt
test_all_attempts_fail_returns_false
test_empty_output_means_ref_not_found
test_artifactory_dep_short_circuits_without_calling_git
test_ssh_attempt_skipped_by_default
test_ssh_attempt_added_when_protocol_pref_is_ssh

Pre-existing failures observed on `main` (NOT introduced by this PR)

Reproduced on main with git stash && pytest <node-id>:

• tests/test_github_downloader.py::TestGitHubPackageDownloader::test_resolve_git_reference_commit — sandboxed clone of fake repo.
• tests/test_github_downloader.py::TestEnterpriseHostHandling::test_clone_fallback_respects_enterprise_host — same.
• tests/test_github_downloader.py::TestDownloaderCredentialFallback::test_credential_fill_used_when_no_env_token — pre-existing assertion on github_token cache.
• tests/unit/test_install_command.py::TestInstallGlobalFlag::test_global_without_packages_and_no_manifest_errors — output-string assertion that depends on terminal width.

How to test

• On a fork or org where you have a per-org env-var PAT (e.g. GITHUB_APM_PAT_<ORG>) and a separate, more-authorized credential cached via gh auth setup-git / OS keychain, run apm install foo/bar/gee#sho --verbose against a virtual subdirectory dep on a non-default branch. Expect [+] ls-remote ok via plain HTTPS w/ credential helper (or via authenticated HTTPS) in the verbose output and successful install.
• Run apm install <org>/<repo>/<wrong-sub-name> --verbose (no #ref) and confirm validation still fails fast with not accessible or doesn't exist — the gate on explicit-ref preserves path-typo rejection.
• Run .venv/bin/python -m pytest tests/test_github_downloader.py::TestRefExistsViaLsRemote -v and confirm all 8 tests pass in <1s.
• Diff vs feat/install-branch-flag: confirm this PR contains only the validation/install auth-alignment changes (no --branch flag, no new apm_cli/install/branch_flag.py, no commands/install.py edit).

Co-authored-by: Copilot 223556219+Copilot@users.noreply.github.com

[a1icja]

@microsoft-github-policy-service agree

[Copilot]

Pull request overview

Aligns apm install <owner>/<repo>/<subdir>#<ref> validation behavior with the actual install/clone authentication fallbacks, reducing false-negative validation failures when the GitHub Contents API rejects a token but git clone would succeed.

Changes:

• Extend validate_virtual_package_exists with a Contents API directory-exists probe and a gated git ls-remote auth chain fallback (mirroring _clone_with_fallback).
• Thread verbose logging through validation so probe attempts are visible under --verbose.
• Add targeted tests for the ls-remote fallback chain and document the updated validation/auth behavior.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
src/apm_cli/deps/github_downloader.py	Adds new validation probes and git ls-remote fallback helpers to reconcile validation with install auth behavior.
src/apm_cli/install/validation.py	Passes through verbose_callback so the new probes emit --verbose diagnostics.
tests/test_github_downloader.py	Introduces tests pinning the new ls-remote fallback chain behavior.
docs/src/content/docs/getting-started/authentication.md	Documents that CLI validation now follows the same auth chain as install.
CHANGELOG.md	Adds an Unreleased Fixed entry for the validation/auth alignment change.