Feature/789 show messages to ingest on hyperdrive jobs

[jozefbakus]

Implements: #789

[filiphornak]

The code is well-written, at least the scala part. However, there is one inconsistency when reading the last commit file, which may throw an exception even when the method returns the Try monad.

Also, reading offsets from kafka and the latest checkpoint file are not parallelized due to the current implementation of the kafka cache. However, this would require a proper consumer pool implementation because calls on ordinary kafka consumer are not thread-safe or finding alternative how to request offsets from kafka without a consumer.

[filiphornak]

This is a suggestion for future projects because it would require reworking or unification of the entire architecture. I noticed that some services use Futures as output, and some don't even when they are blocking.

case class Offset(beginning: Long, end: Long)

def getBeginningEndOffset(topic: String, consumerProperties: Properties): Future[Map[Long, Offset]] =
  for {
    kafkaConsumer   <- consumerPool.getConsumer(consumerProperties)
    parts           <- listTopicPartitions(kafkaConsumer, topic)
    futureBeginning  = getBeginningOffsets(kafka, parts)
    futureEnd        = getEndOffsets(kafka, parts)
    begin           <- futureBegin
    end             <- futureEnd
  } yield begin.map { case (part, bOff) => part -> Offset(bOff, end(part)) }  // This might be done in safer manner

This might improve performance.

[jozefbakus]

I agree with you. Please create an issue for this one. With this PR I just follow style that was already introduced

[filiphornak]

Will do!

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
5 Code Smells

No Coverage information
0.0% Duplication

[filiphornak]

I'm pleased with this explanation and your changes, so I approve this PR. Also, I will create issues for the things we discussed earlier.

[filiphornak]

Will do!