Add css obfuscation test [APMSP-2764] by Eldolfin · Pull Request #6648 · DataDog/system-tests

Eldolfin · 2026-03-31T10:59:56Z

requires feat!: integrate obfuscation to the stats exporter [APMSP-2764] libdatadog#1819 to me merged, a libdatadog bump and one tracer updating libdd-data-pipeline to test it

btw I don't get the fail in test_the_tests because the new scenario TRACE_STATS_COMPUTATION_OBFUSCATION_DISABLED has agent_env variables that the existing TRACE_STATS_COMPUTATION doesn't so they're not equivalent. fixed

TODO

add a missing obfuscation_version test
add a obfuscation_version = 0 test
~~add a test to verify peer tag obfuscation/aggregation client side:~~ (later in another PR) feat: use ip quantization when aggregating peer tags for trace stats libdatadog#1944
enable tests in manifest once enabled in dd-trace-py

Motivation

Test that libraries obfuscate spans before sending them to /stats

Changes

Add one testcase (is this considered a new feature ?)
Fix 2 small issues I found while looking at the repo

Workflow

⚠️ Create your PR as draft ⚠️
Work on you PR until the CI passes
Mark it as ready for review
- Test logic is modified? -> Get a review from RFC owner.
- Framework is modified, or non obvious usage of it -> get a review from R&P team

🚀 Once your PR is reviewed and the CI green, you can merge it!

🛟 #apm-shared-testing 🛟

Reviewer checklist

Anything but tests/ or manifests/ is modified ? I have the approval from R&P team
A docker base image is modified?
- the relevant build-XXX-image label is present
A scenario is added, removed or renamed?
- Get a review from R&P team

github-actions · 2026-03-31T11:00:33Z

CODEOWNERS have been resolved as:

.github/workflows/run-end-to-end.yml                                    @DataDog/system-tests-core
manifests/cpp.yml                                                       @DataDog/dd-trace-cpp
manifests/cpp_httpd.yml                                                 @DataDog/dd-trace-cpp
manifests/cpp_nginx.yml                                                 @DataDog/dd-trace-cpp
manifests/dotnet.yml                                                    @DataDog/apm-dotnet @DataDog/asm-dotnet
manifests/golang.yml                                                    @DataDog/dd-trace-go-guild
manifests/java.yml                                                      @DataDog/asm-java @DataDog/apm-java
manifests/java_otel.yml                                                 @DataDog/system-tests-core
manifests/nodejs_otel.yml                                               @DataDog/system-tests-core
manifests/php.yml                                                       @DataDog/apm-php @DataDog/asm-php
manifests/python.yml                                                    @DataDog/apm-python @DataDog/asm-python
manifests/python_lambda.yml                                             @DataDog/apm-python @DataDog/asm-python
manifests/python_otel.yml                                               @DataDog/apm-python @DataDog/asm-python
manifests/ruby.yml                                                      @DataDog/ruby-guild @DataDog/asm-ruby
tests/stats/test_stats.py                                               @DataDog/system-tests-core
utils/_context/_scenarios/__init__.py                                   @DataDog/system-tests-core
utils/_context/_scenarios/endtoend.py                                   @DataDog/system-tests-core
utils/_context/containers.py                                            @DataDog/system-tests-core
utils/proxy/mocked_response.py                                          @DataDog/system-tests-core

datadog-datadog-prod-us1 · 2026-03-31T11:26:34Z

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 45e38cc | Docs | Datadog PR Page | Give us feedback!}

VianneyRuhlmann · 2026-04-14T12:37:13Z

+            "DD_TRACE_TRACER_METRICS_ENABLED": "true",  # java
+        },
+        agent_env={
+            "DD_APM_SQL_OBFUSCATION_MODE": "normalize_only",


I don't see anything about disabling the obfuscation in the spec is it done by setting the obfuscation version to 0 (if not maybe we need a scenario for the missing/version 0 case)

It's not a setting from the CSS spec but to avoid breaking changes in the obfuscation behavior when switching from agent obfuscation to client obfuscation I decided that following as much of the config as possible would be preferable.

https://github.com/DataDog/datadog-agent/blob/main/pkg/config/config_template.yaml#L1633

Sorry I wasn't really clear what I meant was:

Agent obfuscation config doesn't seem to be covered in the current spec, maybe it should be updated (CC @ichinaski)

Does this scenario implies Agent version 0 or missing if not maybe we want to run a scenario for this case.

Yes, I think this is a shortcoming of the spec currently, because with CSS disabled the agent would have followed the obfuscation config.

No it's exactly like TRACE_STATS_COMPUTATION but with config such as obfuscation makes the 4 spans different so that they end up in different buckets so it's visible. I'll add a similar test with a missing version 👍🏽

VianneyRuhlmann

LGTM, the idea for a version 0 scenario can be addressed in a follow-up PR if needed

…ternal queries SQLite internally executes SELECT QUOTE(?) which naturally contains '?' as a parameter placeholder. Narrowing the resource filter from startswith('SELECT') to startswith('SELECT * FROM users') excludes these internal queries from the no-obfuscation assertions.

…ries Django ORM generates queries like 'SELECT ? FROM app_customuser...' that don't match the test's expected resource pattern. Filter to startswith('SELECT * FROM users') to only capture queries from the /rasp/sqli endpoint used in setup.

Eldolfin · 2026-06-05T15:55:46Z

    def test_obfuscation(self):
-        stats_count = 0
        hits = 0
        top_hits = 0
-        resource = "SELECT * FROM users WHERE id = ?"
-        # wait for 10 seconds to be sure all the buckets are flushed (better than be flaky)
-        for s in interfaces.agent.get_stats(resource):
-            stats_count += 1
+        for s in interfaces.agent.get_stats():
+            if s["Type"] != "sql" or "?" not in s["Resource"]:
+                continue
            logger.debug(f"asserting on {s}")
            hits += s["Hits"]
            top_hits += s["TopLevelHits"]
-            assert s["Type"] == "sql", "expect 'sql' type"
-        assert stats_count <= 4, (
-            "expect <= 4 stats"
-        )  # Normally this is exactly 2 but in certain high load this can flake and result in additional payloads where hits are split across two payloads
        assert hits == top_hits >= 4, "expect at least 4 'OK' hits and top level hits across all payloads"



should be restored

Eldolfin · 2026-06-05T15:56:00Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 45e38cceca

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-06-05T16:00:28Z

+
+
+@features.client_side_stats_supported
+@scenarios.trace_stats_computation_future_obfuscation_version


Regenerate the scenario map for new stats tests

For future PRs that touch only tests/stats/test_stats.py, CI selects impacted end-to-end scenarios from tests/test_the_test/scenarios.json via utils/scripts/compute_libraries_and_scenarios.py; I checked this commit's map with rg "Obfuscation|OBFUSCATION" and none of the new Test_Client_Stats_*Obfuscation* nodeids are present. As a result, these new TRACE_STATS_COMPUTATION_* scenarios are not added to inputs.scenarios, so the workflow steps added for them will not run on test-file-only changes. Please regenerate/update the scenario map with the new nodeids.

Useful? React with 👍 / 👎.

chatgpt-codex-connector · 2026-06-05T16:00:28Z

      if: steps.build.outcome == 'success' && !cancelled() && contains(inputs.scenarios, '"TRACE_STATS_COMPUTATION"')
      run: ./run.sh TRACE_STATS_COMPUTATION
+    - name: Run TRACE_STATS_COMPUTATION_OBFUSCATION_DISABLED scenario
+      if: always() && steps.build.outcome == 'success' && contains(inputs.scenarios, '"TRACE_STATS_COMPUTATION_OBFUSCATION_DISABLED"')


Preserve failure gating for obfuscation scenario steps

In this workflow section, the existing scenario steps use the default successful-previous-steps gating plus !cancelled(), but these new steps add always() and omit !cancelled(). GitHub Actions treats always() as a status override, so when an earlier scenario in this job fails or a run is cancelled, these obfuscation scenarios can still start instead of skipping like the surrounding scenario steps, wasting CI time and making cancellations less responsive. Use the same steps.build.outcome == 'success' && !cancelled() && ... pattern unless these are intentionally post-failure cleanup steps.

Useful? React with 👍 / 👎.

Eldolfin changed the title ~~Oscarld/add css obfuscation test [APMSP-2764]~~ Add css obfuscation test [APMSP-2764] Mar 31, 2026

Eldolfin force-pushed the oscarld/add-css-obfuscation-test branch from 4871555 to 0cf62ff Compare March 31, 2026 11:09

VianneyRuhlmann reviewed Mar 31, 2026

View reviewed changes

Comment thread utils/build/docker/python/flask/app.py

Comment thread tests/stats/test_stats.py Outdated

Eldolfin force-pushed the oscarld/add-css-obfuscation-test branch from 0cf62ff to 714e99a Compare April 2, 2026 11:32

VianneyRuhlmann reviewed Apr 14, 2026

View reviewed changes

VianneyRuhlmann approved these changes Apr 27, 2026

View reviewed changes

Eldolfin mentioned this pull request May 15, 2026

CSS v1.2.0: Phase A — parametric coverage additions #6938

Open

Eldolfin force-pushed the oscarld/add-css-obfuscation-test branch from 1c3b465 to 38567bc Compare June 2, 2026 16:25

Eldolfin changed the base branch from main to oscarld/fix-test-minimal-number-of-scenarios-check-agent-env June 4, 2026 11:25

Base automatically changed from oscarld/fix-test-minimal-number-of-scenarios-check-agent-env to main June 4, 2026 12:52

Eldolfin added 16 commits June 4, 2026 14:57

Add CSS obfuscation test

8de63e3

fix: fmt + don't assert the number of hits

1a547c7

feat: add tests for CSS obfuscation where agent version > sdk version

59964e3

fix: assert header is >= 1 for later

7091a66

fix: fmt

f2423d9

feat: add css obfuscation test with config

ae08cbb

fix: ci, rename obfuscation_disabled

be8a29c

fix: disabled test wrong env and quotes in values

47c6904

fix: remove useless fixmes

cc4a3e2

fix: assert len(sql_stats) == 4

76be75f

fix: revert assert len(sql_stats) == 4 because they should be aggregated

c5b16ba

feat: add tests for obfuscation_version = 0 and missing

b9fdb19

fix: fmt

35a38e2

feat: enable css obfuscation tests for python

cb6da35

fix: fmt

bd96e87

fix(ci): add new scenarios to run-end-to-end workflow

1fcdc3a

Eldolfin force-pushed the oscarld/add-css-obfuscation-test branch from 9d3f272 to 1fcdc3a Compare June 4, 2026 13:03

fix: ci

ad24599

Eldolfin added 4 commits June 4, 2026 17:37

fix(manifests): disable css obfuscation tests for all but python

e58ae7a

fix: allow more than 4 entries when no obfuscation is done

cf97552

Eldolfin commented Jun 5, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Jun 5, 2026

View reviewed changes



		@features.client_side_stats_supported
		@scenarios.trace_stats_computation_future_obfuscation_version

Conversation

Eldolfin commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

TODO

Motivation

Changes

Workflow

Reviewer checklist

Uh oh!

github-actions Bot commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-datadog-prod-us1 Bot commented Mar 31, 2026 • edited by datadog-datadog-prod-us1-2 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

VianneyRuhlmann Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

Eldolfin Apr 14, 2026

Choose a reason for hiding this comment

Uh oh!

VianneyRuhlmann Apr 27, 2026

Choose a reason for hiding this comment

Uh oh!

Eldolfin Apr 28, 2026

Choose a reason for hiding this comment

Uh oh!

VianneyRuhlmann left a comment

Choose a reason for hiding this comment

Uh oh!

Eldolfin Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

Eldolfin commented Jun 5, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Jun 5, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Eldolfin commented Mar 31, 2026 •

edited

Loading

github-actions Bot commented Mar 31, 2026 •

edited

Loading

datadog-datadog-prod-us1 Bot commented Mar 31, 2026 •

edited by datadog-datadog-prod-us1-2 Bot

Loading