[checkpoint] Fix incorrect field handling #19996
Draft
ash-darin wants to merge 7 commits into
Draft
Conversation
checkpoint.uid is not a user id but an object id of the currently handled Object/Rule/Policy. Use the already available field checkpoint.object as this is not populated in the context where checkpoint.uid appears.
Set @timestamp from event.end. Some events are updates to previous events but have the timestamp of the original event which may be hours in the past. timestamp is preserved as event.created further up.
remove field renaming until a more complete solution can be found
UID is not USER ID but UNIQUE ID.
user.id is never set. Remove all other references to it.
use grok to catch more variants of email fields
ReviewersBuildkite won't run for external contributors automatically; you need to add a comment:
NOTE: https://github.com/elastic/integrations/blob/main/.buildkite/pull-requests.json contains all those details. |
Timestamps are always UNIX, test for that format first.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed commit message
Correct the handling of various fields:
prevent UID fields from being treated as user IDs
handle email Adresses in a more concise matter
ignore original timestamp for events that are updates for previous events.
Please explain:
checkpoint.uidis not a user id. It is a UniqueID of whatever object is currently created/modified/deleted. This may be rules, objects or policies. This removes the renaming touser.idtill a better handling can be determined.checkpoint.usercheck_incident_uidis not a user id. It is a UniqueID of the incident. This removes the renaming todestination.user.idtill a better handling can be determined.Remove the handling of various *.id fields as these are never set, among other things due to the changes above.
Grok Email fields to cover the case an email is specified as "user name (user.name@domain.name)"
Use foreach to do an append. The pipeline does not cover the case of multiple e-mail addresses yet but use foreach anyway.
Set
@timestamptoevent.end. This is important as events may be 'updates' i.e. indications that they are still happening, The timestamp of those events is then still the one of the start of the event, even if those are hours in the past. Looking at "the last hour" would then miss all events that inform about events that started more than that in the past, even if the event came in just a few minutes ago. The original timestamp is preserved asevent.createdby the pipeline already.Checklist
changelog.ymlfile.Author's Checklist
How to test this PR locally
Related issues
Screenshots