Support for configmaps (as env var & volume) in the template fields in the KubernetesPodOperator

[vizeit]

Description

Currently configmaps are not supported in the template fields in the KubernetesPodOperator. It will be helpful to add this support to dynamically attach configmaps to a pod

Use case/motivation

Currently configmaps are not supported in the template fields in the KubernetesPodOperator. It will be helpful to add this support to dynamically attach configmaps to a pod

I have made the code changes locally and also added unittests. I have also run all the Kubernetes provider unit tests after the changes

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[RNHTTR]

Do KubernetesPodOperator's existing parameters volumes and configmaps not suffice?

[vizeit]

@RNHTTR exisiting parameters configmaps and volumes for configmaps do not suffice. If the configmaps are dynamically created in a DAG with the name of configmaps template based, k8 operator currently fails to attach configmaps because the name is not rendered. To mount configmaps as Volumes, V1ConfigMapVolumeSource type will contain the name that is not supported in the current k8 operator recursive template rendering.
Here are the steps for dynamically created configmaps that currently do not work,

1. Create a DAG with the following tasks
   A. Create a configmap with your data. name the configmap with a jinja template e.g. run id with removed special characters. The configmap name will require to be template based because there could be multiple instances of the DAG running at the same time
   B. Create a k8 operator and attach configmap created in task A either as an environment variable or as a volume
   C. Delete the configmap from task A
2. Run the DAG multiple times with different inputs so that multiple instances of the DAG are created
3. observe that task B fails in all the instances. The reason of failure is that k8 operator does not render the configmap name and is unable to mount/process it

[vizeit]

Also to add, I was able to extend k8 operator to accommodate configmap as a volume with templated values. I also extended k8 operator last year in my project with pvcs having templated names before the pvcs were added to template fields in the k8 provider package. The reason I think it will help to add configmap template functionality rather in the k8 provider package is because decorator also uses k8 operator and I am not sure I can easily add template fields to k8 decorator without creating custom provider package for k8 that has extended k8 operator class. I hope these details will help you to understand better

[RNHTTR]

Sounds good. I assigned the Issue to you. Feel free to open a PR! The Contributors Quickstart Guide might be helpful, and feel free to reach out on slack.

The reason I think it will help to add configmap template functionality rather in the k8 provider package is because decorator also uses k8 operator and I am not sure I can easily add template fields to k8 decorator without creating custom provider package for k8 that has extended k8 operator class.

Wouldn't the template fields be added to the KubernetesPodOperator in the Kubernetes provider package?

[vizeit]

Yes, the change here for this feature will only be in the KubernetesPodOperator class in the Kubernetes provider package. I meant to say in my previous comment that if I wanted to incorporate additional template fields in the k8 decorator myself in my project, I could not do it unless I first created custom extended k8 operator class with those additional template fields and then creating custom k8 decorator in a custom provider package. I hope this clarifies