[FEATURE REQUEST]: Hybrid scan - performance validation & optimization for delete dataset

[sezruby]

Feature requested

#171 enables to utilize Hybrid scan with delete dataset as well as append dataset.
However, the feature still requires some validation and optimization before GA.
So I created the temporary config below to make it switchable until then.

  val INDEX_HYBRID_SCAN_DELETE_ENABLED = "spark.hyperspace.index.hybridscan.delete.enabled"
  val INDEX_HYBRID_SCAN_DELETE_ENABLED_DEFAULT = "false"

This issue is for tracking this config.

Acceptance criteria

Success criteria

• delete-only dataset TPCH
• delete-only dataset TPCH - 100K files
• append & delete dataset TPCH

Additional context

Add any other context or screenshots about the feature request here.