[claude_code] Improve ECS field mappings: adopt gen_ai.* fields and fix event.category assignments

[jamiehynds]

Summary

To improve cross-integration correlation and take advantage of Elastic's ECS GenAI fieldset, we can map several of the claude_code integration's custom fields onto their ECS equivalents. The current pipeline is in good shape — it already sets event.kind, event.action, event.outcome, event.duration, the categorization set, and populates related.user. The items below are incremental refinements, mostly enabled by the recent addition of gen_ai.* fields to ECS.

Context: Claude Code exports telemetry over OTLP but uses its own attribute naming rather than the OTel GenAI semantic conventions on the metrics/logs this integration ingests, so the ECS alignment has to be done in the ingest pipeline.

1. Map custom fields to ECS gen_ai.* (not currently set)

Currently mapped as	Better ECS field	Notes
claude_code.events.model	gen_ai.request.model (+ gen_ai.response.model)
claude_code.events.input_tokens	gen_ai.usage.input_tokens	long → integer
claude_code.events.output_tokens	gen_ai.usage.output_tokens	long → integer
claude_code.events.cache_read_tokens	gen_ai.usage.cache_read.input_tokens
claude_code.events.cache_creation_tokens	gen_ai.usage.cache_creation.input_tokens
claude_code.events.request_id	gen_ai.response.id
claude_code.events.tool_name / mcp_tool_name	gen_ai.tool.name
claude_code.events.tool_use_id	gen_ai.tool.call.id
claude_code.events.tool_input_flattened / tool_parameters_flattened	gen_ai.tool.call.arguments	flattened → flattened
claude_code.events.query_source	gen_ai.operation.name	loose mapping
(constant)	gen_ai.provider.name = "anthropic"	add as constant_keyword

2. Map custom fields to ECS core (not currently set)

Currently mapped as	Better ECS field	Notes
claude_code.events.error	error.message	Currently copied to event.reason only; error.message is the canonical home. Keep event.reason if desired.
claude_code.events.error_code	error.code
claude_code.events.error_type	error.type
claude_code.events.organization.id	organization.id
claude_code.events.user.email	user.email	Already appended to related.user; the core user.email field is not set.
claude_code.events.user.id	user.id	Already appended to related.user; the core user.id field is not set.
claude_code.events.user.account_id / account_uuid	user.domain / user.id

3. ECS categorization refinements

3a. Additional signals for the categorization set

event.action, event.outcome (from success), event.kind, and event.duration are already handled in the pipeline. Remaining opportunities:

Signal	ECS field	Mapping	Status
source	event.provider	user, system	not set
decision / decision_type	event.outcome	accept→success, reject→failure	⚠️ verify — outcome is currently driven off success, so tool_decision rejects may not register as failure

3b. Fix event.category per event type

Event	Current	Proposed	Rationale
api_request	web / info	api / info	Outbound API call to Anthropic; api is the purpose-built ECS value.
api_error	web / error	api / error	Failed API call.
api_retries_exhausted	web / error	api / error	Failed API call.
api_refusal	web / denied	api / denied	Model refusal on an API response, not a web-server log.
tool_decision	process / info	iam (or configuration) / allowed|denied	Authorization/permission decision, not a process event.
user_prompt	process / start	(leave category empty)	No fitting closed-vocab value; rely on event.kind + event.action.
skill_activated	process / info	(leave category empty)	Same as user_prompt.
hook_registered	process / info	configuration (marginal)	Registration is a config action; execution is the process.
hook_execution_start	process / info	process / start	Refine type.
hook_execution_complete	process / info	process / end	Refine type.
plugin_loaded	package / info	library (marginal)	package implies install/remove; runtime load is closer to library.
plugin_installed	package / info	✅ package — leave as-is	Install genuinely fits package.
tool_result	process / info	process default, ideally derived from tool_name	Spans process (Bash), file (Read/Write), web (WebFetch).
auth	authentication / info	✅ no change	Correct.
permission_mode_changed	configuration / change	✅ no change	Correct.
mcp_server_connection	network / connection	✅ no change	Correct.

Note: event.category is a closed vocabulary. Where nothing fits (e.g. user_prompt, skill_activated), leave it empty and rely on event.kind + event.action rather than the current process.

4. Explicitly kept custom (no ECS equivalent)

cost_usd, cost_usd_micros (ECS GenAI has no cost field), session.id, prompt.id (no gen_ai.conversation.id), effort, speed, prompt_length, prompt_text, and all feature-specific fields (hook_*, mcp_*/server_*, plugin.*, skill.*, marketplace.*, *_path_count, permission-mode fields, terminal.type, tool_*_size_bytes, raw tool_input/tool_parameters).

Acceptance criteria

• GenAI fields mapped to gen_ai.* (Section 1), originals retained
• error.message / error.code / error.type, organization.id, and core user.email/user.id populated (Section 2)
• event.category corrected per event type (3b) — api_* → api, tool_decision → iam, empty where nothing fits
• event.provider populated from source; permission-decision outcome handling verified (3a)
• Docs/fields.yml and ingest pipeline updated; sample events re-tested

References

• ECS Gen AI fields
• Generative AI fields now available in ECS (Elastic blog)
• ECS event.category allowed values
• ECS event.type allowed values
• Claude Code monitoring / OTel telemetry

[infra-vault-gh-plugin-prod]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[github-actions]

tl;dr: This is actionable and should be implemented in the claude_code events ingest pipeline, with one schema caveat: several proposed gen_ai.* names are not present in the local ECS schema and need target-ECS confirmation before adding them as ECS imports.

Recommendation

Implement this as a focused enhancement to packages/claude_code/data_stream/events: keep the existing claude_code.events.* fields for compatibility, add ECS set/copy_from processors after the current type conversions, revise the categorization script, then regenerate/update pipeline expected outputs and docs.

The issue is accurate for the current package: GenAI values are only retained under claude_code.events.*, api_* events are categorized as web, permission decisions are categorized as process, and several core ECS copies are missing or partial.

Findings

Area	Evidence	Impact
Attribute namespacing	packages/claude_code/data_stream/events/elasticsearch/ingest_pipeline/default.yml:208-212 renames all expanded OTLP attributes into claude_code.events.	Custom originals are preserved, but ECS gen_ai.* fields are never populated.
Current categories	default.yml:91-171 maps api_request, api_error, api_retries_exhausted, and api_refusal to web; tool_decision, user_prompt, hook_registered, and skill_activated to process; plugin_loaded to package.	Matches the inaccurate category assignments called out in the issue.
Current ECS mappings	default.yml:339-377 sets event.action, maps only duration_ms to event.duration, derives outcome only from success/some API errors, sets event.provider to constant claude-code, and copies claude_code.events.error to event.reason.	Missing total_duration_ms, event.sequence, organization.id, direct user.*, error.*, decision/status outcomes, and GenAI mappings.
Token/model/request fields	fields.yml:90-117 defines model, token counts, query_source, and request_id only below claude_code.events.*; docs show the same at docs/README.md:299-323.	Add ECS copies while retaining these originals.
Tool fields	fields.yml:41-62 defines tool names/IDs and flattened arguments only below claude_code.events.*; docs show these at docs/README.md:337-344.	Add ECS copies for supported gen_ai.tool.* paths.
Core fields	fields.yml:132-141 defines custom error, error_type, and error_code; fields.yml:313, 348-350 document custom organization/user fields.	Add error.message, error.type, error.code, organization.id, user.email, and user.id where supported.
ECS imports	packages/claude_code/data_stream/events/fields/ecs.yml:1-38 imports event/user/error basics but no gen_ai.* or organization.id.	ecs.yml must be expanded for any ECS fields the pipeline starts producing.

Local ECS schema check against packages/osquery_manager/schemas/ecs.json showed these proposed paths are present: gen_ai.request.model, gen_ai.response.model, gen_ai.usage.input_tokens, gen_ai.usage.output_tokens, gen_ai.response.id, gen_ai.tool.name, gen_ai.tool.call.id, and gen_ai.operation.name.

The same local schema did not include gen_ai.usage.cache_read.input_tokens, gen_ai.usage.cache_creation.input_tokens, gen_ai.tool.call.arguments, or gen_ai.provider.name, so those exact names should be verified against the target ECS/package-spec version before adding them as external ECS fields.

Verification

Static checks from the workspace:

$ grep -RIn "gen_ai" packages/claude_code || true

$ python - <<'PY'  # summarize current categories and expected outputs
api_request: event.category=web
api_error: event.category=web
api_refusal: event.category=web
tool_decision: event.category=process
user_prompt: event.category=process
skill_activated: event.category=process
hook_registered: event.category=process
plugin_loaded: event.category=package
test-api-request.json-expected.json: gen_ai=False, category=['web'], duration=2493000000, outcome=unknown
test-tool-decision-reject.json-expected.json: gen_ai=False, category=['process'], duration=None, outcome=unknown
test-tool-result-mcp.json-expected.json: gen_ai=False, category=['process'], duration=120000000, outcome=success
test-hook-execution-complete.json-expected.json: gen_ai=False, category=['process'], duration=None, outcome=unknown
PY

$ if command -v elastic-package >/dev/null; then elastic-package version; else echo "elastic-package not installed"; fi
elastic-package not installed

Schema-name check:

gen_ai.request.model: present
gen_ai.response.model: present
gen_ai.usage.input_tokens: present
gen_ai.usage.output_tokens: present
gen_ai.usage.cache_read.input_tokens: not present
gen_ai.usage.cache_creation.input_tokens: not present
gen_ai.response.id: present
gen_ai.tool.name: present
gen_ai.tool.call.id: present
gen_ai.tool.call.arguments: not present
gen_ai.operation.name: present
gen_ai.provider.name: not present

I could not run elastic-package test pipeline in this environment because elastic-package is not installed.

Detailed Action Plan

1. Update packages/claude_code/data_stream/events/elasticsearch/ingest_pipeline/default.yml after the conversion block at default.yml:222-337:
   • copy claude_code.events.model to gen_ai.request.model and, if desired for correlation, gen_ai.response.model;
   • copy token fields to supported gen_ai.usage.* paths;
   • copy request_id to gen_ai.response.id;
   • copy tool_name or mcp_tool_name to gen_ai.tool.name with deterministic precedence;
   • copy tool_use_id to gen_ai.tool.call.id;
   • copy query_source to gen_ai.operation.name.
2. For proposed paths absent from the local ECS schema (cache_read/cache_creation, tool.call.arguments, provider.name), first confirm target ECS support. If unsupported in this package’s ECS version, either defer those fields, keep only the existing claude_code.events.* originals, or add package-owned fields only if that is acceptable for the integration.
3. Add core ECS copies in the same pipeline section:
   • claude_code.events.event.sequence -> event.sequence;
   • claude_code.events.organization.id -> organization.id;
   • claude_code.events.user.email and user.id -> ECS user.*;
   • claude_code.events.error, error_type, error_code -> error.message, error.type, error.code.
4. Extend duration handling at default.yml:345-350 so total_duration_ms also populates event.duration when duration_ms is absent. This is visible today in test-hook-execution-complete.json-expected.json:31, where total_duration_ms exists but event.duration is missing at :48-60.
5. Revise ecs_categorization at default.yml:85-183:
   • api_request, api_error, api_retries_exhausted, api_refusal -> event.category: [api];
   • tool_decision -> iam or configuration and set event.type from decision/decision_type where possible;
   • remove event.category for user_prompt and skill_activated if no closed-vocabulary category fits;
   • consider hook_registered -> configuration and plugin_loaded -> library if those are accepted ECS categories for this use case;
   • optionally derive tool_result category from tool_name for Bash/file/web tools instead of always process.
6. Improve outcome/provider logic near default.yml:352-371:
   • map success=true/false, status=connected/failed, and decision or decision_type=accept/reject to event.outcome;
   • decide whether the issue’s proposed source -> event.provider should replace the current constant claude-code at default.yml:368-371, or whether claude-code should remain as provider and source should stay custom.
7. Update packages/claude_code/data_stream/events/fields/ecs.yml for all newly emitted ECS fields and update fields.yml/docs/README.md only where new package-owned fields or documented behavior changes are needed.
8. Update pipeline expected outputs under packages/claude_code/data_stream/events/_dev/test/pipeline/, especially:
   • test-api-request.json-expected.json:7-28 and :52-64 for GenAI fields and api category;
   • test-tool-decision-reject.json-expected.json:50-61 for category/outcome;
   • test-tool-result-mcp.json-expected.json:57-69 for tool GenAI fields;
   • test-mcp-server-connection-failed.json-expected.json:49-60 for status=failed outcome and error.* copies;
   • test-hook-execution-complete.json-expected.json:31 / :48-60 for total_duration_ms -> event.duration.
9. Run elastic-package test pipeline -p packages/claude_code --data-streams events in a dev environment with elastic-package available, then update snapshots as needed.

Related Items

Type	Item	Relevance
Issue	#19967	Current issue; exact claude_code gen_ai issue search only found this item.
PR search	"claude_code" "gen_ai" is:pr	Returned total_count: 0; I did not find an existing PR implementing this.
File	packages/claude_code/data_stream/events/elasticsearch/ingest_pipeline/default.yml	Main implementation surface for namespacing, categorization, ECS copies, and outcomes.
File	packages/claude_code/data_stream/events/fields/ecs.yml	Needs ECS imports for new ECS fields.
File	packages/claude_code/data_stream/events/fields/fields.yml	Defines retained custom fields and any package-owned additions.
File	packages/claude_code/data_stream/events/_dev/test/pipeline/*-expected.json	Existing snapshots lock in the current legacy behavior and should be updated.

---

What is this? | From workflow: Issue Triage

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.

[jamiehynds]

@trisch-mecould you please review the proposed updates to our Claude Code ECS mappings? Main things are the gen_ai.* mappings and the event.category adjustments. (cc @efd6)

[efd6]

@jamiehynds Note that while we should absolutely be able to use error.message for things like claude_code.events.error, due to the way that success is checked in elastic-package testing, we cannot; any value in error.message is seen as a test failure.

[efd6]

Also, event.provider is currently set, to "claude-code".

[jamiehynds]

Thanks @efd6 - I think both are ok. Am I right in saying gen_ai.provider.name is set to anthropic?

[efd6]

@jamiehynds, yes that's correct.