Skip to content

#802: Adding some log messages in hyperdrive trigger#805

Merged
filiphornak merged 16 commits intodevelopfrom
feature/802-add-log-messages
Apr 18, 2023
Merged

#802: Adding some log messages in hyperdrive trigger#805
filiphornak merged 16 commits intodevelopfrom
feature/802-add-log-messages

Conversation

@filiphornak
Copy link
Collaborator

@filiphornak filiphornak commented Mar 23, 2023

I added some logging messages for the hyperdrive trigger. It will give us insight into what's happening with skipping specific workflows.

implicit override def executionContext: ExecutionContext = scala.concurrent.ExecutionContext.Implicits.global

private trait HasUpdateJob {
trait HasUpdateJob {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not private ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually don't know, but when I ran it locally, it complained that the mockito framework could not mock private static classes. I used the same command as in the GitHub actions script.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I reverted it and will test it in the pipeline

schedulerInstanceService
.registerNewInstance()
.map { id =>
.map(wireTap { id =>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't like it, I see this idea. But I would leave as it was. It is visible that what is returned and what is a side effect on first scan

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I will remove it. But, IMO, this looks clearer when you see that there is no change in value, just execution of side-effect.

Akka had something similar in their stream. Even native scala contains .tap, or .tapEach. And it could be easily implemented on any Functor type with cats.

import cats._

implicit class FunctorOps[F[_], A](val fa: F[A]) extends AnyVal {
  def tap[A, U](f: A => U)(implicit func: Functor[F]): F[A] =
    func.map(fa) { elm =>
      f(elm)
      elm
    }
}

Because, to me, this looks much cleaner

schedulerInstanceId match {
  case Some(id) => Future.successful(id)
  case _        => 
    schedulerInstanceService
      .registerNewInstance()
      .tap(id => schedulerInstanceId = Some(id))
      .omComplete {
        case Success(id) =>
          logger.info(s"Successfully assigned new (SchedulerId=${id}) to scheduler")
        case Failure(e) =>
          logger.error("Failed to get new SchedulerId", e)
      }
}

case Some(joinedDagDefinition) =>
for {
hasInQueueDagInstance <- dagInstanceRepository.hasInQueueDagInstance(joinedDagDefinition.workflowId)
hasInQueueDagInstance <- dagInstanceRepository
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can rewrite this and have logging after each "flatmap"
_ = logger.trace("bla")

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I know about it, and I will change it back. However, I don't like it because the = is for a map if scala had something for specifying side effects in for comprehension.

I would instead use something like https://typelevel.org/cats/typeclasses/arrow.html or a better example with precisely the same thing https://medium.com/virtuslab/arrows-monads-and-kleisli-part-ii-12ffd4da8bc9.

To me, writing it that way looks much cleaner, and also, using cats would add several additional structures, like IO, Functor (which in turn can be used to simplify all the .map(_.map(..._.map(f)...)) operations), Validated, Writer, etc...

}
fut.onComplete {
case Success(_) => logger.debug(s"Executing job. Job instance = $jobInstance")
case Success(_) => logger.debug(s"Executing job. (JobId={})", jobInstance)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

JobInstance instead of JobId

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The same bellow

private def updateJob(jobInstance: JobInstance): Future[Unit] = {
logger.info(
s"Job updated. ID = ${jobInstance.id} STATUS = ${jobInstance.jobStatus} EXECUTOR_ID = ${jobInstance.executorJobId}"
"(JobId={}). Job updated. ID = {} STATUS = {} EXECUTOR_ID = {}",
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unnecessary job id twice

@sonarqubecloud
Copy link

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@filiphornak filiphornak merged commit 13c068e into develop Apr 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add more debug log messages to flow activating, deactivating and exection of workflow

2 participants