ci: slack notification on pipeline failure

Why?

In order to set up CI on-call for vdk,
visible and reliable notifications are required.
The notifications should alert on pipeline failure
and provide the necessary information for troubleshooting.

What?

Call an incoming webhook for a custom slack application on
pipeline failure. This sends a message to the corresponding channel.

How was this tested?

Ran in pipeline and sent notification to debug channel.
Note that the env variable for the notification hook will have to be changed
after this is merged.

https://gitlab.com/vmware-analytics/versatile-data-kit/-/pipelines/914912201

What kind of change is this

Feature/non-breaking

Signed-off-by: Dilyan Marinov <mdilyan@vmware.com>

Author: Dilyan Marinov

.gitlab-ci.yml

@@ -22,6 +22,7 @@ stages:
   - release
   - release_image
   - end
+  - notify
 
 build-check-prerequisites:
   stage: build
@@ -37,3 +38,13 @@ end:
   only:
     refs:
       - external_pull_requests
+
+notify_failure:
+  stage: notify
+  script:
+    - ./cicd/notify.sh "The pipeline for the $CI_COMMIT_BRANCH branch has failed.\nPlease take the necessary steps to fix it. https://gitlab.com/vmware-analytics/versatile-data-kit/-/pipelines/$CI_PIPELINE_ID" $VDK_SLACK_NOTIFICATION_HOOK
+  rules:
+    - if: '$CI_COMMIT_BRANCH != "main"'
+      when: never
+    - if: '$CI_COMMIT_BRANCH == "main"'
+      when: on_failure

cicd/notify.sh

@@ -0,0 +1,5 @@
+#!/bin/bash -e
+# Copyright 2021-2023 VMware, Inc.
+# SPDX-License-Identifier: Apache-2.0
+
+curl -X POST -H 'Content-type: application/json' --data "{\"text\":\"$1\"}" $2