Define dataflows for community-owned integrations (batch 2, individually-maintained tiles)

[philjlee]

Jira: TXP-277

What

Adds assets/dataflows.yaml to 10 integrations (11 dataflow entries: 8 metrics, 3 logs).

This is batch 2 of the integrations-extras dataflows backfill. Batch 1 is #3093 (74 Datadog-team-owned integrations). The two batches touch disjoint directories and are both based on master, so they are independent and will not conflict.

Selection criteria

An integration is in scope for this PR if all of the following hold, computed from the repo at 9c197fe:

1. It has a parseable manifest.json (the dataflows validator hard-requires one).
2. It has no existing assets/dataflows.yaml.
3. It has a hard data signal: metadata.csv with at least one data row, and/or a log pipeline under assets/logs/. No data_type was ever guessed.
4. .github/CODEOWNERS resolves <dir>/assets/dataflows.yaml (last-match-wins) to an owner set containing no @DataDog/... team.
5. It is not a RUM tile, not one of the 5 service-check-only directories, and not fluxcd/traefik (README-only, disabled in APW).

Then, within that set of 81, this PR takes the cluster with individually-maintained tiles: These paths are owned only by individual GitHub handles or personal email addresses, and the integration ships no Agent check — it is a tile-only listing.

Content

Every file follows the uniform template already used by the 14 pre-existing files in this repo:

provides:
  - id: <app_id>-<data_type>
    always_on: true
    granular: false
    data_type: <metrics|logs>
    direction: inbound

id is manifest.json's app_id plus -<data_type>. That is the mechanical rule the 17 existing entries follow 17/17 — not the directory name. For all 81 integrations in batch 2 the two happen to coincide after slugification, but app_id is the authority.

metadata.csv rows produce a metrics entry, an assets/logs/ pipeline produces a logs entry, and integrations with both get both (metrics first).

Integrations in this PR

Integration	Dataflows	Evidence	CODEOWNERS
apache-apisix	metrics	6 metric rows	@bisakhmondal bisakhmondal00@gmail.com
apollo	metrics	18 metric rows	@sachindshinde sachin@apollographql.com
artie	metrics	6 metric rows	@danafallon
contrastsecurity	logs	log pipeline	@kristianamitchellcontrastsecurity kristiana.mitchell@contrastsecurity.com
federatorai	metrics	63 metric rows	@chenyingtz @yummydsky
gigamon	logs	log pipeline	@Mrudula-Oruganti-Gigamon
loadrunner_professional	metrics+logs	9 metric rows, log pipeline	@mihneavelinOT
mongodb_atlas	metrics	204 metric rows	@Salil999 @JustinMason frank.sun@mongodb.com tonya.edmonds@mongodb.com
nomad	metrics	97 metric rows	@irabinovitch
tyk	metrics	5 metric rows	@letzya

Coverage

Assuming batch 1 (#3093) merges, coverage of assets/dataflows.yaml across the 260 integration directories in this repo moves:

Stage	Files	Coverage
Today on master	14	5.4%
After batch 1 (#3093, +74)	88	33.8%
After all of batch 2 (+81, this PR is +10)	169	65.0%

Batch 1 + batch 2 together complete the entire mechanical bucket — every integration in the repo with an unambiguous local data_type signal.

Ownership — please read

None of the directories in this PR have a @DataDog/... team CODEOWNER; they resolve to third-party organizations, vendor support aliases, or individual community maintainers.

Per #2925 ("Define dataflows for saas-integrations") and consistent repo practice, external CODEOWNERS approval is not required to merge here — master carries no branch ruleset requiring it, and Datadog employees routinely land mechanical sweeps across community-owned directories (#2810, #2829, #2878, #2957, #3000). #2925 in particular created 13 of the 14 existing dataflows.yaml files and was approved only by Datadog reviewers.

That said, this PR is intentionally a draft so that no review request has been sent to any external maintainer yet. Marking it ready will fan review requests out to the owners listed in the table above. That should be a deliberate decision, not a side effect.

Validation

These files were validated by running the real upstream validator, not by static inspection.

DataflowsValidationHandler from ddoghq/dd-source@main (domains/integrationscatalog/libs/catalogassetslib/dataflows_validation_handler.go) was executed via go test -overlay, with only the COBS-backed dataflows client and the deployment handler stubbed out. The validation code path itself is byte-for-byte upstream — the handler file was diffed against ddoghq/dd-source@main and is identical, including the current 14-value validDataTypes list (federated_sql_queries included).

Results:

• This PR's tree: 10 new files + the 14 pre-existing = 24 files, 28 dataflow IDs, 0 failures.
• The union of master + batch 1 (Define dataflows for Datadog-owned integrations (batch 1) #3093) + all of batch 2: 169 files, 187 dataflow IDs, 0 failures — which is what proves dataflow ID uniqueness holds globally, across both batches and the pre-existing files. (HandleLibrary enforces "only one app can provide a dataflow".)

The harness was negative-tested first, and correctly rejects all of: missing always_on; a data_type outside the enum; an id violating ^[a-z0-9-]+$; an id shorter than 3 characters; a file with neither provides nor uses; a .yml extension; a missing manifest.json; and the same dataflow ID provided by two apps. A known-good baseline file is accepted. So a pass here means something.

APW also posts a validate-dataflows status check on extras PRs, so there will be pre-merge feedback from the pipeline itself as well.

CI note — run / Validate redness is pre-existing

The run / Validate job is already failing on master and has been for at least the last 10 runs of that workflow. Concrete evidence at the exact SHA this branch is based on:

• https://github.com/DataDog/integrations-extras/actions/runs/30528431295 — master @ 9c197fe2c9ff8864d1fe84274298818aac67853f, job run / Validate failed at "Validate default configuration files" and "Validate configuration models", with 182 is not in sync errors spanning 39 integration directories.

Because the workflow runs with TARGET=changed on PRs, touching a directory pulls that pre-existing drift into this PR's scope even though this PR only adds a new file and changes nothing that ddev validate config/models inspects. None of the directories in this PR appear in that pre-existing failure list. This is unrelated to dataflows and is not something this PR attempts to fix.

Deferred, not in scope

Nothing was dropped from this PR for an ambiguous data_type — every integration in the batch-2 set mapped cleanly to metrics and/or logs. What remains unaddressed repo-wide after batch 1 + batch 2 (91 directories) is the research bucket, which needs schema decisions rather than scripting:

• 17 RUM SDK tiles — there is no rum value in validDataTypes, so no legal file can be written.
• 5 service-check-only directories (cfssl, cybersixgill_actionable_alerts, go_pprof_scraper, isdown, reboot_required) — service checks map to no data_type.
• ~39 outbound notification/webhook tiles — events + direction: outbound is the least-bad fit but has not been ratified by anyone.
• 5 directories with no manifest.json (aerospike_enterprise, scamalytics, warpstream, plus fluxcd/traefik which are README-only and explicitly disabled in APW) — the validator requires a manifest.

Related: #3093 (batch 1), #2925 (the original dataflows PR for saas-integrations).

🤖 Generated with Claude Code

---

Addendum: merge-gating, measured

Two things worth stating precisely, since the "can Datadog land this on community-owned directories?" question is the only real objection to this batch.

1. devflow/mergegate will show "N approvals missing". It is advisory, not a required check. Six of the twenty most recently merged PRs in this repo merged while mergegate was pending or failed:

• Automate post-tag wheel publishing handoff #3077 reached a terminal failure (1 issue: CODEOWNERS missing) and was merged ~15 hours later regardless.
• CloudZero UI update #3022 merged with 1 pending: 11 approvals missing.
• cloudsmith: add billable_only filter to analytics metrics (default: True) #3024, Pre-sort map keys inside list items in _tests.yaml fixtures #3027, Feat/re prometheus endpoint gauges and dbkeys deprecation #3078, Bump ping to 1.0.3 #3081 likewise merged with mergegate still in_progress.

2. There is now a direct precedent for the solely-externally-owned case, not just the co-owned one. #2925 (the original dataflows PR) only ever touched directories that already carried a @DataDog/... team, so it did not settle the harder case that this batch is made of. #3022 does: it modified /cloudzero/, whose CODEOWNERS line is nine external individuals and no Datadog team, and it was approved by Wyrine, sarah-witt, and steveny91 — none of them CloudZero owners — then merged. reviewDecision was APPROVED, which GitHub would not report if CODEOWNERS review were enforced.

What is enforced on master is GitHub's ordinary "at least one approving review" rule, satisfied by any reviewer with write access. The practical requirement for this PR is therefore one Datadog approval, not sign-off from the maintainers in the table above.

Caveat, stated honestly: GET /repos/DataDog/integrations-extras/branches/master/protection returns 404 for non-admin tokens, so the classic branch-protection config could not be read directly. The conclusions above are inferred from ruleset queries (only an org-level tag ruleset exists; rules/branches/master is empty) and from the merge behaviour of 20 recent PRs. Someone with admin access can confirm.

None of this changes the fact that this PR is a draft and no external maintainer has been notified. Marking it ready is what sends the review requests, and that should stay a deliberate step.

[philjlee]

CI note: run / Validate — confirmed pre-existing, not caused by this PR

Now that CI has run on all four batch-2 PRs, here is the observed result. The run / Validate outcome tracks exactly one thing: whether the PR happens to touch a directory whose generated config/model files are already out of sync on master.

PR	Files	Dirs already drifting on master	run / Validate
#3094	4	2 (gatekeeper, kepler)	fail
#3095	24	6	fail
#3096	43	16	fail
#3097	10	0	pass

#3097 is the natural control: same generator, same file shape, zero drifting directories, and run / Validate is green. That isolates the failure to pre-existing drift rather than to anything about dataflows.yaml.

Two further confirmations:

1. The string dataflow does not appear anywhere in the validation output of the failing runs. (It appears 5 times in each log, all in git fetch branch-listing noise.) No validator names a dataflows.yaml file. The failing steps are only "Validate default configuration files" and "Validate configuration models".

2. master emits the identical errors. The last Validate repository run on master — run 30528431295 at 9c197fe, the exact SHA these branches are based on — already reports, verbatim:

   ##[error]File `conf.yaml.example` is not in sync, run "ddev validate config kepler -s"
   ##[error]File `defaults.py` is not in sync, run "ddev validate models gatekeeper -s"
   ##[error]File `instance.py` is not in sync, run "ddev validate models gatekeeper -s"
   

   That master run has 182 is not in sync errors across 39 integration directories, and 10 of the last 10 Validate repository runs on master failed. Because PRs run with TARGET=changed, touching any of those 39 directories pulls the drift into scope.

This PR adds one new file per directory and changes nothing that ddev validate config/models inspects. Fixing the repo-wide codegen drift is out of scope here.