feat(marketplace): add source parity for marketplace add

CHANGELOG.md

@@ -17,6 +17,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
   of `~/.hermes/config.yaml` (written atomically with `0o600` perms, preserving
   unrelated config keys and refusing to overwrite a malformed file). `HERMES_HOME`
   overrides the Hermes home directory. See the [Hermes integration guide](https://microsoft.github.io/apm/integrations/hermes/).
+- `apm marketplace add` now accepts git URLs with `#ref`, local file paths,
+  and hosted `marketplace.json` URLs -- so teams can consume private,
+  offline, or hosted catalogs without publishing a GitHub repo (closes
+  #676). (#1739)
 - Enterprise bootstrap mirror mode lets `install.sh`, `install.ps1`, and `apm self-update` use internal release, installer, and PyPI mirrors with fail-closed public fallback, and closes #1680. (#1733)
 
 ### Fixed

docs/src/content/docs/consumer/installing-from-marketplaces.md

@@ -65,36 +65,53 @@ reproducible across machines and CI.
 
 ## Local and self-hosted marketplaces
 
-`apm marketplace add` accepts more than GitHub-hosted repos. The same
-register / browse / install / update workflow works against:
+`apm marketplace add` can register marketplaces from git repositories,
+local files, or hosted `marketplace.json` catalogs. The same register /
+browse / install / update workflow works against:
 
 - **Local filesystem paths** -- `apm marketplace add /srv/marketplaces/agent-forge`
-  (or a relative path, or `~/code/marketplace`). Useful for
-  privacy-sensitive packages, offline workflows, and air-gapped
-  environments. Local marketplaces with relative plugin sources
-  install by copying from disk via `LocalDependencySource`.
+  (or a relative path, `~/code/marketplace`, or a direct
+  `marketplace.json` file). Useful for privacy-sensitive packages,
+  offline workflows, and air-gapped environments. Local marketplaces
+  with relative plugin sources install by copying from disk via
+  `LocalDependencySource`.
 - **`file://` URIs** -- `apm marketplace add file:///srv/marketplaces/agent-forge.git`.
   Behaves the same as a local path.
+- **Git HTTPS URLs with `#ref`** --
+  `apm marketplace add https://gitlab.com/acme/agent-forge.git#v1.0.0`.
+  APM strips the fragment into the stored ref. If you omit `#ref`
+  (or `--ref`), APM warns because the default branch can move.
+- **Hosted `marketplace.json` URLs** --
+  `apm marketplace add https://catalog.example.com/marketplace.json --name catalog`.
+  APM fetches the JSON directly over HTTPS, caches ETag/Last-Modified
+  metadata, and records the source URL plus content digest in the
+  lockfile when packages are installed from it.
 - **Generic git URLs** -- any host APM does not classify as
   GitHub or GitLab family flows through subprocess `git` and
   `GitCache`. Includes Azure DevOps (auth via `ADO_APM_PAT`),
   Gitea, Bitbucket Server, and self-hosted git servers.
 - **SSH URLs** -- `git@gitea.example.com:org/repo.git`. The host
-  is extracted, classified, and routed through the matching
-  fetcher.
+  is extracted, classified, and routed through the matching fetcher.
+
+:::note[Hosted JSON is public HTTPS]
+Hosted `marketplace.json` URLs are public HTTPS sources: APM does not
+send custom auth headers. Use git-backed marketplaces for private
+catalogs.
+:::
 
 For generic-git marketplaces, `marketplace.json` is fetched via a
 sparse-cone clone (only the manifest path is downloaded); APM does
-not forward `GITHUB_APM_PAT` or `GITLAB_APM_TOKEN` to non-GitHub /
-non-GitLab hosts. Authentication falls through to the host's
-`*_APM_PAT` (e.g. `ADO_APM_PAT`) or your local
-`git credential-manager`. See
+not forward `GITHUB_APM_PAT` or `GITLAB_APM_PAT` to non-GitHub /
+non-GitLab hosts. Authentication falls through to the host's matching
+`*_APM_PAT` variable (such as `ADO_APM_PAT`) or local git credential
+helper when one is configured. See
 [Authentication](../authentication/).
 
 **Lockfile note.** Installs from a local marketplace record a
 local-path source in `apm.lock.yaml`. Lockfiles produced this way
-are machine-specific -- do not commit them into a shared repo. See
-[lockfile reference](../../reference/lockfile-spec/).
+are machine-specific -- do not commit them into a shared repo. Remote
+`marketplace.json` installs record `source_url` and `source_digest`
+provenance. See [lockfile reference](../../reference/lockfile-spec/).
 
 ## Where next
 

docs/src/content/docs/reference/cli/marketplace.md

@@ -61,11 +61,16 @@ Register a marketplace from a source reference. Accepted forms:
 - `OWNER/REPO` -- GitHub shorthand (`acme/marketplace`).
 - `HOST/OWNER/.../REPO` -- non-GitHub host shorthand
   (`gitlab.com/team/marketplace`).
-- HTTPS URL -- any git host, including Azure DevOps, GitLab,
-  Gitea, Bitbucket Server, or a self-hosted git server.
+- HTTPS git URL -- any git host, including Azure DevOps, GitLab,
+  Gitea, Bitbucket Server, or a self-hosted git server. Add `#ref`
+  to pin the marketplace, for example
+  `https://gitlab.com/acme/marketplace.git#v1.0.0`.
+- Hosted `marketplace.json` URL --
+  `https://catalog.example.com/marketplace.json`.
 - SSH URL -- `git@host:org/repo.git` style.
 - Local filesystem path -- absolute (`/srv/marketplaces/agent-forge`),
-  relative (`./local-mkt`), or home-based (`~/code/marketplace`).
+  relative (`./local-mkt`), home-based (`~/code/marketplace`), or a
+  direct `marketplace.json` file.
 - `file://` URI -- `file:///srv/marketplaces/agent-forge.git`.
 
 ```bash
@@ -79,14 +84,18 @@ apm marketplace add gitlab.com/my-org/awesome-agents --host gitlab.com
 apm marketplace add https://dev.azure.com/contoso/eng/_git/agent-forge \
     --name agent-forge
 
-# Gitea / Bitbucket Server / self-hosted git
-apm marketplace add https://gitea.example.com/org/repo.git --name custom
+# Gitea / Bitbucket Server / self-hosted git, pinned with #ref
+apm marketplace add https://gitea.example.com/org/repo.git#v1.0.0 --name custom
+
+# Hosted marketplace.json URL
+apm marketplace add https://catalog.example.com/marketplace.json --name catalog
 
 # SSH
 apm marketplace add git@gitea.example.com:org/repo.git --name custom
 
-# Local filesystem (bare repo or working directory)
+# Local filesystem (bare repo, working directory, or marketplace.json file)
 apm marketplace add /srv/marketplaces/agent-forge.git --name agent-forge
+apm marketplace add ./vendor/marketplace.json --name vendor
 
 # file:// URI
 apm marketplace add file:///srv/marketplaces/agent-forge.git --name agent-forge
@@ -95,19 +104,22 @@ apm marketplace add file:///srv/marketplaces/agent-forge.git --name agent-forge
 | Flag | Description |
 |---|---|
 | `--name`, `-n` | Display name. Defaults to the repo name. |
-| `--ref`, `-r` | Git ref (branch, tag, or SHA). Default: `main`. |
+| `--ref`, `-r` | Git ref (branch, tag, or SHA). Default: `main`. Applies only to git-backed sources. For HTTPS git URLs, a `#ref` fragment is equivalent and is stored as the ref. |
 | `--branch`, `-b` | Deprecated alias for `--ref`. |
-| `--host` | Git host FQDN. Default: `github.com`. Ignored when `SOURCE` is a URL or local path. |
+| `--host` | Git host FQDN for `OWNER/REPO` shorthand. Default: `github.com`. Ignored when `SOURCE` already carries a host (URL, hosted `marketplace.json`, or local path); a warning is shown for hosted JSON, local paths, or a conflicting embedded host. |
 | `--verbose`, `-v` | Show detailed output. |
 
 **Trust boundary.** APM forwards its authentication tokens
 (`GITHUB_APM_PAT`, `GITLAB_APM_PAT`) only when the marketplace
-host is classified as GitHub or GitLab family. For any other host
--- generic HTTPS, SSH, Azure DevOps, self-hosted -- the
+host is classified as GitHub or GitLab family. For any other git
+host -- generic HTTPS, SSH, Azure DevOps, self-hosted -- the
 marketplace is fetched via subprocess `git` through `GitCache`,
-and authentication falls through to the host's APM PAT (e.g.
-`ADO_APM_PAT` for Azure DevOps) or your local
-`git credential-manager`. See
+and authentication falls through to the host's local git credential
+helper and matching `*_APM_PAT` variables such as `ADO_APM_PAT`.
+Hosted `marketplace.json` URLs are public HTTPS only: APM sends no auth
+headers. Use a git-backed marketplace for private catalogs. When packages
+are installed from a hosted JSON URL, the lockfile records the source URL and
+fetched content digest. See
 [`getting-started/authentication`](../../../getting-started/authentication/).
 
 **Azure DevOps.** ADO-hosted marketplaces fetch `marketplace.json`

docs/src/content/docs/reference/lockfile-spec.md

@@ -141,6 +141,8 @@ Each item in `dependencies` describes one resolved package.
 | `is_dev` | bool | no | `true` when the dep was declared under `devDependencies`. |
 | `discovered_via` | string | no | Marketplace name that surfaced this package (provenance). |
 | `marketplace_plugin_name` | string | no | Plugin name as listed in that marketplace. |
+| `source_url` | string | no | Canonical marketplace source URL when the package came from a hosted `marketplace.json` catalog. |
+| `source_digest` | string | no | `sha256:<hex>` digest of the hosted `marketplace.json` bytes used for resolution. |
 | `is_insecure` | bool | no | `true` when the source URL was `http://`. |
 | `allow_insecure` | bool | no | `true` when the manifest explicitly opted in to the insecure source. |
 | `constraint` | string | git-source semver only | The original semver range from `apm.yml` (`^1.2.0`, `~1.4`). Present when `ref:` was a range; used by drift detection so a manifest range vs. a locked tag (`v1.5.3`) is not a false positive, and by lockfile replay to pin the resolved tag deterministically across installs. |

packages/apm-guide/.apm/skills/apm-usage/commands.md

@@ -118,7 +118,7 @@ Credentials resolve via `APM_REGISTRY_TOKEN_{NAME}` env var (or `apm config set
 
 | Command | Purpose | Key flags |
 |---------|---------|-----------|
-| `apm marketplace add SOURCE` | Register a marketplace. `SOURCE` accepts `OWNER/REPO`, `HOST/OWNER/REPO`, nested `HOST/group/sub/.../REPO`, HTTPS URL (any git host -- GitHub, GitLab, ADO, Gitea, self-hosted), SSH URL (`git@host:org/repo.git`), local filesystem path, or `file://` URI. | `-n NAME`, `-r REF`, `--host HOST` |
+| `apm marketplace add SOURCE` | Register a marketplace. `SOURCE` accepts `OWNER/REPO`, `HOST/OWNER/REPO`, nested `HOST/group/sub/.../REPO`, HTTPS git URL with optional `#ref`, hosted `marketplace.json` URL, SSH URL (`git@host:org/repo.git`), local directory or file path, or `file://` URI. `--ref` applies only to git-backed sources; `--host` applies only to shorthand sources. | `-n NAME`, `-r REF`, `--host HOST` |
 | `apm marketplace list` | List registered marketplaces | -- |
 | `apm marketplace browse NAME` | Browse marketplace plugins | -- |
 | `apm marketplace update [NAME]` | Update marketplace index | -- |

src/apm_cli/commands/install.py

@@ -430,10 +430,7 @@ def warning_handler(msg):
                         if logger:
                             logger.validation_fail(package, reason)
                         continue
-                    marketplace_provenance = {
-                        "discovered_via": marketplace_name,
-                        "marketplace_plugin_name": plugin_name,
-                    }
+                    marketplace_provenance = resolution.provenance(marketplace_name, plugin_name)
                     package = canonical_str
                     marketplace_dep_ref = getattr(resolution, "dependency_reference", None)
                 except Exception as mkt_err: