feat(insights): correlate CSP vulnerabilities/misconfigurations with entity store#417
Open
CAWilson94 wants to merge 1 commit into
Open
Conversation
…entity store `generate-entity-ai-insights --correlate-with-entity-store` now reuses the entity-store host/user identities (the same source as the anomaly/DED correlation) for the generated vulnerability and misconfiguration docs, and stamps each host entity's `host.id` onto them. This makes the doc's computed EUID (`host:<host.id>`) equal the host entity's `entity.id`, so findings correlate to real entities on the v2 entity flyout / AI summary (host-only, EUID-based), not just to faked host names. Falls back to faked names when the entity store has no usable identities. Co-authored-by: Cursor <cursoragent@cursor.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the generate-entity-ai-insights --correlate-with-entity-store flow so Cloud Security Posture (CSP) vulnerability and misconfiguration documents can correlate to real entity-store host identities (via host.id → EUID matching), rather than only correlating anomaly (DED) data.
Changes:
- Add optional
hostIdsupport to vulnerability and misconfiguration document generators, stampinghost.idwhen provided. - Extend insights generation to optionally source host/user identities from the entity store (reusing the same identity-fetch helpers used by DED correlation).
- Update CLI help text to reflect that correlation now applies to anomalies, vulnerabilities, and misconfigurations.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
src/generators/create_vulnerability.ts |
Adds hostId param and conditionally sets host.id in generated vulnerability docs. |
src/generators/create_misconfigurations.ts |
Adds hostId param and conditionally sets host.id in generated misconfiguration docs. |
src/commands/misc/insights.ts |
Adds entity-store identity correlation for CSP docs and threads hostId through generated documents. |
src/commands/misc/index.ts |
Updates --correlate-with-entity-store option description to include CSP docs. |
Comment on lines
+101
to
+103
| } else { | ||
| ({ usersData, hostsData } = correlated); | ||
| } |
|
|
||
| const hostsData = hostIdentities | ||
| .filter((host) => Boolean(host.name || host.id)) | ||
| .map((host) => ({ hostname: host.name, hostId: host.id })); |
CAWilson94
added a commit
to elastic/kibana
that referenced
this pull request
Jul 7, 2026
…regenerate nudge (#276038) ## Summary Persists the **Entity AI Summary** so it no longer regenerates on every flyout open/click-away. Instead of writing to the entity-store `entity.attributes.summary` field (POC #269344), the summary is now stored as an **append-only document in the entity metadata datastream** (`.entities.v2.metadata.security_*`, base #271806), per [security-team#17968](elastic/security-team#17968). ## What changed - **Write** — generation persists a single immutable `ai_summary.*` doc via `asInternalUser` (server-derived `generated_by`; no per-user metadata write privilege needed). Structural caps applied (max 4 highlights, 3 recommended actions). - **Read** — gated `GET` (`asCurrentUser`): with metadata read access the persisted summary is shown; without it, the flyout gracefully falls back to on-demand generation (nothing persisted). - **Removed** the never-merged `entity.attributes.summary` field + `includeSummary` listing param. - **Staleness nudge** — if the entity's risk score drifts from the snapshot taken at generation time, a warning callout prompts regeneration. - **Prompt** — promoted the improved entity-highlights prompt (MITRE / Kill Chain mapping, synthesize-don't-list, anti-fabrication) as the single default. - **Tests/docs** — Scout API permissions matrix + `check_privileges` coverage, unit tests, and README privilege docs. Updated Security AI prompts Integration SO: elastic/integrations#19999 <details> <summary> <strong>Prompt and Summary Content</strong> </summary> - [Customer feedback flagged that the summary was **too long to be a summary**.](elastic/security-team#16484 (comment)) Rather than just trimming length, I reworked the prompt to change the *shape* of the output while keeping , and focussing, its value - helping an analyst quickly see **what needs their attention and how to move forward with an investigation**, not handing them a raw data dump to interpret themselves. - **Risk score** stays high-level: describe the dominant threat pattern, and only name a specific rule/alert when it clearly accounts for the majority of the score - instead of listing every contributing rule. - **Anomalies** are summarized as the single most significant pattern rather than listed, and mapped to a MITRE ATT&CK tactic or Lockheed Martin Kill Chain phase only when the ML results clearly correspond to a known technique (the mapping is omitted when anomalies are ambiguous or benign, to avoid guessing). - **Criticality / vulnerabilities** state the level / most critical finding and *why it matters*, not raw values. - **Reinforced with hard guardrails:** 1 sentence per highlight (2 only when an anomaly needs the extra clause for a MITRE/Kill Chain mapping), a concise highlights budget, up to 3 prioritized recommendations, no fabricated values, and consistent formatting (backticks for technical values, 2-decimal rounding). - The result is shorter *because* it's focused - not truncated. This directly addresses design/product feedback that the risk score and anomalies should be summarized at a high level rather than enumerated. Prompt quality is inherently subjective and hard to unit-test, so I validated it by desk-testing against seeded rich data (risk, criticality, anomalies, vulnerabilities); the telemetry below is there to keep tuning it from real usage post-merge. </details> <details> <summary> <strong>Telemetry</strong> </summary> - Generating an entity AI summary emits an `entity_ai_summary_persisted` EBT event with `entityType`, `spaceId`, and `highlightsCount` / `recommendedActionsCount`. The two counts are the **raw number of items the model produced, captured client-side before structural capping** — not the number persisted (the doc is always capped to `MAX_ENTITY_SUMMARY_HIGHLIGHTS` / `MAX_ENTITY_SUMMARY_RECOMMENDED_ACTIONS`). - **Why it's useful:** comparing these raw counts against the caps tells us how often, and by how much, the prompt overshoots the intended summary length — a data-driven feedback loop for tuning the prompt (e.g. if the model routinely returns 7 highlights against a cap of 4, the prompt is asking for too much). The counts are captured on the client because the server only ever receives the already-capped content and so cannot observe overshoot itself. No summary text or entity identifiers are included in the event. - **How it connects to the prompt change:** the prompt was reworked to be shorter by being more focused (per the customer feedback that it was too long); this telemetry closes the loop by letting us track overshoot over time and keep tuning the prompt and caps from real usage rather than guesswork. </details> ## Demo Latest Demo! 🔥 https://github.com/user-attachments/assets/471bb53e-ce6d-46dd-838b-4bd2e2e63c26 ## Notes for reviewers - **Generation is currently client-side** (the client POSTs the generated summary). This is transitional — it reuses the assistant's client-side anonymization/replacements context. Moving generation fully server-side is a tracked follow-up (also required to move staleness computation server-side so Agent Builder / external clients share it). - **The summary body (highlights + recommended actions) is English-only.** This matches all AI-generated content in the Security solution (no assistant prompt does locale-based generation) and is the same as pre-persistence behavior — **not a regression**. All UI chrome (headers, buttons, staleness callout + message) is i18n'd. **- Entity AI Summary runtime helpers:** (staleness detection + caps) are exported from a dedicated `@kbn/entity-store/common/entity_summary` subpath, not the top-level common barrel. The barrel is the page-load entry (always-loaded primitives only); this logic is used only by the lazy flyout and server, so keeping it in the barrel pushed the entityStore page-load bundle over budget. The subpath code-splits it out - fixing the size regression structurally rather than raising the limit - mirroring the existing euid_helpers / entity_maintainers modules. Types stay in the barrel (zero page-load cost). ## Testing **1. Prereqs:** Entity Store v2 enabled, an Enterprise license, and a configured AI connector. **2. Seed data** (uses generator PR [security-documents-generator#417](elastic/security-documents-generator#417) for entity-correlated vulnerabilities): ```bash # Entities + risk score + asset criticality + alerts yarn start risk-score-v2 --entity-kinds host,idp_user --hosts 20 --no-setup # Layer anomalies + vulnerabilities + misconfigurations, correlated to those entities yarn start generate-entity-ai-insights --correlate-with-entity-store -h 20 -u 20 -a 20 ``` **3. Verify persistence:** - Open a host/user entity flyout → AI summary → **Generate**. - Close and reopen the flyout → the **same** summary shows, with no regeneration, plus a "Generated by … on …" line. - As a second user (with metadata read) → the same summary + original generated-by is shown. **4. Verify staleness nudge:** - For an entity with an already generated summary - change the asset criticality, then re-open the flyout for that summary and verify → a warning callout appears ("Entity risk has changed…") → click **Regenerate summary** → callout clears. **5. Verify gated-read fallback:** - As a user **without** read on `.entities.v2.metadata.security_*` → the flyout falls back to on-demand generation and does **not** persist. Mappings - Using `GET .entities.v2.metadata.security_default/_mapping` from dev tools, after some reviews the latest mappings (Tuesday July 7th): ``` "Ai_summary": { "properties": { "anomaly_job_ids": { "type": "keyword" }, "generated_at": { "type": "date", "format": "epoch_millis" }, "generated_by": { "type": "keyword" }, "highlights": { "type": "object", "enabled": false }, "recommended_actions": { "type": "keyword", "index": false, "doc_values": false }, "staleness": { "properties": { "enabled_signals": { "type": "keyword" }, "snapshot": { "properties": { "risk_score": { "type": "float" } } } } }, "variant_id": { "type": "keyword" } } }, ``` --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> Co-authored-by: Alex Prozorov <alex.prozorov@elastic.co>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
generate-entity-ai-insights --correlate-with-entity-storepreviously only correlated anomaly data with the entity store; the generated vulnerability and misconfiguration docs were written for faked host/user names, so they never surfaced on real entities.This change makes those CSP docs correlate to entity-store entities:
host.idonto the vulnerability/misconfiguration docs, so the doc's computed EUID (host:<host.id>) equals the host entity'sentity.id.Why
The v2 entity flyout / AI summary matches vulnerabilities to a host by EUID (
host.id), host-only. Matching onhost.namealone (previous behaviour) never lit up on v2 host entities. This makes findings show on real entities for desk testing the Entity AI Summary.Usage
Then open a host entity's flyout — the Vulnerabilities section and AI summary populate.
Test plan
risk-score-v2thengenerate-entity-ai-insights --correlate-with-entity-store; confirm vuln docs carry the entity'shost.id/host.name.--correlate-with-entity-store: behaviour unchanged (faked names).Made with Cursor