[ti_misp] Fix daily_refetch never re-triggering after the first poll

[kcreddy]

Proposed commit message

[ti_misp] Fix daily_refetch never re-triggering after the first poll

The daily_refetch feature relies on a `last_daily_refetch` cursor value
that should persist across polls and only advance once 24h has elapsed.
It never did: httpjson evaluates cursor value templates with an empty
cursor context (updateCursor() builds an emptyTransformContext(), so
`.cursor` is `{}`), which means the `last_daily_refetch` template could
never read its own previous value. It fell through to writing "now" on
every poll, so the 24h gate (last_daily_refetch + 24h < now) was pushed
into the future each cycle and the re-fetch window never rewound to
initial_interval. Decay scores were never re-ingested.

Fix it entirely within the package. The request transform runs with the
real loaded cursor, so it decides whether this poll is a re-fetch and
writes the resulting marker into url.params.daily_refetch_at. That value
round-trips into .last_response.url.params, which IS available when the
cursor value template is evaluated, so the cursor field now reads the
marker back from there instead of from its own (empty) prior state. The
marker is carried across pagination and falls back to "now" so the
cursor is never stored empty.

The extra url param is sent to MISP but ignored: restSearch reads its
filters from the POST JSON body, not the query string. This mirrors the
existing url.params.timestamp workaround.

Verified with elastic-package lint/format and the threat_attributes
system tests (including daily-refetch). The 24h re-trigger itself is not
covered by an automated test.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Screenshots

[infra-vault-gh-plugin-prod]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[kcreddy]

whitespace change due to elastic-package build

[github-actions]

✅ Elastic Docs Style Checker (Vale)

No issues found on modified lines!

---

The Vale linter checks documentation changes against the Elastic Docs style guide. To use Vale locally or report issues, refer to Elastic style guide for Vale.

[elastic-vault-github-plugin-prod]

✅ All changelog entries have the correct PR link.

[vera-review-bot]

Severity: 🔵 Low confidence: low path: packages/ti_misp/data_stream/threat_attributes/fields/fields.yml:297

Stray trailing blank line added to fields.yml is unrelated to the bugfix; remove it to keep the diff clean (the same stray line was also added to elasticsearch/transform/latest_ioc/fields/fields.yml).

Details

This PR is a focused daily_refetch bugfix, but it adds an empty trailing line at the end of this fields.yml with no functional effect. An identical stray blank line is added to packages/ti_misp/elasticsearch/transform/latest_ioc/fields/fields.yml. It looks like an accidental editor artifact rather than an intended change, and elastic-package format normalizes trailing whitespace, so it is best dropped to keep the diff scoped to the fix.

Recommendation:

Remove the trailing blank line so the file ends right after the last field description:

        - name: attribute
          type: flattened
          description: >
            List of attributes of the object in which the attribute is attached.

---

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[vera-review-bot]

Review summary

Issues found across the latest commits dfdf00c — 1 low

• 🔵 Stray trailing blank line added to fields.yml is unrelated to the bugfix (link) (Unresolved)

A new commit triggers another review — at most once every 15 minutes. I skip the PR while it's approved or has merge conflicts.

🤖 AI-Generated Review | Vera Review Bot | 📚 Knowledge base: integration-skills

⚠️ Automated review — verify suggestions before applying.

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

Package ti_misp 👍(1) 💚(0) 💔(1)

Expand to view

Data stream	Previous EPS	New EPS	Diff (%)	Result
threat_attributes	3424.66	2188.18	-1236.48 (-36.11%)	💔

To see the full report comment with /test benchmark fullreport

[infra-vault-gh-plugin-prod]

💚 Build Succeeded

• Buildkite Build
• Commit: dfdf00c

History

• 💔 Build #45640 failed 82eeb2d
• 💔 Build #45639 failed 7584702

cc @kcreddy

[efd6]

unfortunate

[mergify]

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

• Queue this pull request

[elastic-vault-github-plugin-prod]

Package ti_misp - 1.44.2 containing this change is available at https://epr.elastic.co/package/ti_misp/1.44.2/