Is Subject.doAs thread-safe? #731
Description
Question
Is Subject.doAs thread-safe? If AccessController.getContext is called from multiple threads concurrently, where each call to AccessController.getContext is wrapped in Subject.doAs, are the correct contexts returned?
Motivation
If Subject.doAs is thread-safe, then it could be used in the InProcessLauncher to wrap around the call to submit a spark job, in order to ensure that two concurrent spark jobs with different keytabs don't accidentally use the wrong user, because UserGroupInformation.getCurrentUser takes that information from a static field in UserGroupInformation in the absence of a AccessControlContext.
Practical application
Submitting multiple spark jobs from InProcessLauncher (using separate threads, but sharing the same process) with different keytabs.
Alternatives could be
- Using SparkLauncher (separate process for each spark-submit).
- Impersonation approach
"Literature" Review
https://docs.oracle.com/en/java/javase/12/docs/api/java.base/java/security/AccessController.html
Links to blogs, docs, stackoverflow, etc.
Experiments
Experiment program was written. Two spark jobs with different keytabs were started in parallel. A Subject.doAs was wrapped around each spark job submission.
Log output (anonymized)
2022-12-01 16:59:44 INFO SparkSubmitService:36 - Starting application SparkSubmitConfigScala(first-941deb95-6875-4deb-99cc-70a14b03fe95,None,Map(spark.yarn.principal -> user-1, spark.yarn.keytab -> user-1.keytab, spark.yarn.archive -> hdfs:///user/user-1/spark-2.4.4-jars.zip),None,Some(yarn),Some(cluster),/opt/spark/examples/jars/spark-examples_2.11-2.4.4.jar,org.apache.spark.examples.SparkPi,Map(),Buffer(101),List(),List(),List())
2022-12-01 16:59:44 INFO SparkSubmitService:36 - Starting application SparkSubmitConfigScala(second-941deb95-6875-4deb-99cc-70a14b03fe95,None,Map(spark.yarn.keytab -> user-2.keytab, spark.yarn.archive -> hdfs:///user/user-2/spark-2.4.4-jars.zip, spark.yarn.principal -> user-2),None,Some(yarn),Some(cluster),/opt/spark/examples/jars/spark-examples_2.11-2.4.4.jar,org.apache.spark.examples.SparkPi,Map(),Buffer(102),List(),List(),List())
2022-12-01 16:59:44 INFO SparkSubmitService:90 - Logging in as user-1 with keytab user-1.keytab
2022-12-01 16:59:44 INFO SparkSubmitService:90 - Logging in as user-2 with keytab user-2.keytab
2022-12-01 16:59:44 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2022-12-01 16:59:44 INFO SparkSubmitService:94 - Entered privileged action as user-1@EXAMPLE.COM (auth:KERBEROS). Waiting...
2022-12-01 16:59:44 INFO SparkSubmitService:94 - Entered privileged action as user-2@EXAMPLE.COM (auth:KERBEROS). Waiting...
2022-12-01 16:59:44 INFO SparkSubmitService:95 - Current user before barrier is user-1@EXAMPLE.COM (auth:KERBEROS)
2022-12-01 16:59:44 INFO SparkSubmitService:95 - Current user before barrier is user-2@EXAMPLE.COM (auth:KERBEROS)
2022-12-01 16:59:44 INFO SparkSubmitService:97 - ugi is user-2@EXAMPLE.COM (auth:KERBEROS)
2022-12-01 16:59:44 INFO SparkSubmitService:97 - ugi is user-1@EXAMPLE.COM (auth:KERBEROS)
2022-12-01 16:59:44 INFO SparkSubmitService:98 - Current user is user-2@EXAMPLE.COM (auth:KERBEROS)
2022-12-01 16:59:44 INFO SparkSubmitService:98 - Current user is user-1@EXAMPLE.COM (auth:KERBEROS)
2022-12-01 16:59:44 INFO SparkSubmitService:99 - Login user is root (auth:KERBEROS)
2022-12-01 16:59:44 INFO SparkSubmitService:99 - Login user is root (auth:KERBEROS)
2022-12-01 16:59:45 INFO Client:54 - Kerberos credentials: principal = user-1, keytab = user-1.keytab
2022-12-01 16:59:45 INFO Client:54 - Kerberos credentials: principal = user-2, keytab = user-2.keytab
2022-12-01 16:59:45 WARN DomainSocketFactory:117 - The short-circuit local reads feature cannot be used because libhadoop cannot be loaded.
2022-12-01 16:59:45 INFO RequestHedgingRMFailoverProxyProvider:146 - Looking for the active RM in [rm1, rm2]...
2022-12-01 16:59:45 INFO RequestHedgingRMFailoverProxyProvider:146 - Looking for the active RM in [rm1, rm2]...
2022-12-01 16:59:45 INFO RequestHedgingRMFailoverProxyProvider:170 - Found active RM [rm1]
2022-12-01 16:59:45 INFO RequestHedgingRMFailoverProxyProvider:170 - Found active RM [rm1]
2022-12-01 16:59:45 INFO Client:54 - Requesting a new application from cluster with 54 NodeManagers
2022-12-01 16:59:45 INFO Client:54 - Requesting a new application from cluster with 54 NodeManagers
2022-12-01 16:59:45 INFO Client:54 - Verifying our application has not requested more than the maximum memory capability of the cluster (344064 MB per container)
2022-12-01 16:59:45 INFO Client:54 - Verifying our application has not requested more than the maximum memory capability of the cluster (344064 MB per container)
2022-12-01 16:59:45 INFO Client:54 - Will allocate AM container, with 5120 MB memory including 1024 MB overhead
2022-12-01 16:59:45 INFO Client:54 - Will allocate AM container, with 5120 MB memory including 1024 MB overhead
2022-12-01 16:59:45 INFO Client:54 - Setting up container launch context for our AM
2022-12-01 16:59:45 INFO Client:54 - Setting up container launch context for our AM
2022-12-01 16:59:45 INFO Client:54 - Setting up the launch environment for our AM container
2022-12-01 16:59:45 INFO Client:54 - Setting up the launch environment for our AM container
2022-12-01 16:59:45 INFO Client:54 - Preparing resources for our AM container
2022-12-01 16:59:45 INFO Client:54 - Preparing resources for our AM container
2022-12-01 16:59:46 INFO Client:54 - To enable the AM to login from keytab, credentials are being copied over to the AM via the YARN Secure Distributed Cache.
2022-12-01 16:59:46 INFO Client:54 - To enable the AM to login from keytab, credentials are being copied over to the AM via the YARN Secure Distributed Cache.
2022-12-01 16:59:46 INFO Client:54 - Uploading resource file:user-1.keytab -> hdfs:///user/user-1/.sparkStaging/application_1668609319046_131486/user-1.keytab
2022-12-01 16:59:46 INFO Client:54 - Uploading resource file:user-2.keytab -> hdfs:///user/user-2/.sparkStaging/application_1668609319046_131487/user-2.keytab
2022-12-01 16:59:46 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs:/user/user-1/spark-2.4.4-jars.zip
2022-12-01 16:59:46 INFO Client:54 - Source and destination file systems are the same. Not copying hdfs:/user/user-2/spark-2.4.4-jars.zip
2022-12-01 16:59:46 INFO Client:54 - Uploading resource file:/opt/spark/examples/jars/spark-examples_2.11-2.4.4.jar -> hdfs:///user/user-2/.sparkStaging/application_1668609319046_131487/spark-examples_2.11-2.4.4.jar
2022-12-01 16:59:46 INFO Client:54 - Uploading resource file:/opt/spark/examples/jars/spark-examples_2.11-2.4.4.jar -> hdfs:///user/user-1/.sparkStaging/application_1668609319046_131486/spark-examples_2.11-2.4.4.jar
2022-12-01 16:59:46 INFO Client:54 - Uploading resource file:/tmp/spark-38b0908d-103d-47fc-bec9-f136daa84b8b/__spark_conf__1935887187816665210.zip -> hdfs:///user/user-1/.sparkStaging/application_1668609319046_131486/__spark_conf__.zip
2022-12-01 16:59:46 INFO Client:54 - Uploading resource file:/tmp/spark-38b0908d-103d-47fc-bec9-f136daa84b8b/__spark_conf__2044477698681732827.zip -> hdfs:///user/user-2/.sparkStaging/application_1668609319046_131487/__spark_conf__.zip
2022-12-01 16:59:46 WARN Client:66 - spark.yarn.am.extraJavaOptions will not take effect in cluster mode
2022-12-01 16:59:46 WARN Client:66 - spark.yarn.am.extraJavaOptions will not take effect in cluster mode
2022-12-01 16:59:46 INFO SecurityManager:54 - Changing view acls to: root,user-1
2022-12-01 16:59:46 INFO SecurityManager:54 - Changing view acls to: root,user-2
2022-12-01 16:59:46 INFO SecurityManager:54 - Changing modify acls to: root,user-2
2022-12-01 16:59:46 INFO SecurityManager:54 - Changing modify acls to: root,user-1
2022-12-01 16:59:46 INFO SecurityManager:54 - Changing view acls groups to:
2022-12-01 16:59:46 INFO SecurityManager:54 - Changing view acls groups to:
2022-12-01 16:59:46 INFO SecurityManager:54 - Changing modify acls groups to:
2022-12-01 16:59:46 INFO SecurityManager:54 - Changing modify acls groups to:
2022-12-01 16:59:46 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root, user-1); groups with view permissions: Set(); users with modify permissions: Set(root, user-1); groups with modify permissions: Set()
2022-12-01 16:59:46 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root, user-2); groups with view permissions: Set(); users with modify permissions: Set(root, user-2); groups with modify permissions: Set()
2022-12-01 16:59:46 INFO HadoopFSDelegationTokenProvider:54 - getting token for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-932281088_18, ugi=user-1@EXAMPLE.COM (auth:KERBEROS)]]
2022-12-01 16:59:46 INFO HadoopFSDelegationTokenProvider:54 - getting token for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_210644297_19, ugi=user-2@EXAMPLE.COM (auth:KERBEROS)]]
2022-12-01 16:59:46 INFO DFSClient:1045 - Created HDFS_DELEGATION_TOKEN token 14691181 for user-1 on ha-hdfs
2022-12-01 16:59:46 INFO DFSClient:1045 - Created HDFS_DELEGATION_TOKEN token 14691180 for user-2 on ha-hdfs
2022-12-01 16:59:47 INFO HadoopFSDelegationTokenProvider:54 - getting token for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_210644297_19, ugi=user-2@EXAMPLE.COM (auth:KERBEROS)]]
2022-12-01 16:59:47 INFO HadoopFSDelegationTokenProvider:54 - getting token for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_-932281088_18, ugi=user-1@EXAMPLE.COM (auth:KERBEROS)]]
2022-12-01 16:59:47 INFO DFSClient:1045 - Created HDFS_DELEGATION_TOKEN token 14691182 for user-2 on ha-hdfs
2022-12-01 16:59:47 INFO DFSClient:1045 - Created HDFS_DELEGATION_TOKEN token 14691183 for user-1 on ha-hdfs
2022-12-01 16:59:47 INFO HadoopFSDelegationTokenProvider:54 - Renewal interval is 86400055 for token HDFS_DELEGATION_TOKEN
2022-12-01 16:59:47 INFO HadoopFSDelegationTokenProvider:54 - Renewal interval is 86400088 for token kms-dt
2022-12-01 16:59:47 INFO HadoopFSDelegationTokenProvider:54 - Renewal interval is 86400077 for token kms-dt
2022-12-01 16:59:47 INFO HadoopFSDelegationTokenProvider:54 - Renewal interval is 86400129 for token HDFS_DELEGATION_TOKEN
2022-12-01 16:59:48 INFO Client:54 - Submitting application application_1668609319046_131487 to ResourceManager
2022-12-01 16:59:48 INFO Client:54 - Submitting application application_1668609319046_131486 to ResourceManager
2022-12-01 16:59:48 INFO YarnClientImpl:274 - Submitted application application_1668609319046_131486
2022-12-01 16:59:48 INFO Client:54 - Application report for application_1668609319046_131486 (state: ACCEPTED)
2022-12-01 16:59:48 INFO Client:54 -
client token: N/A
diagnostics: [Thu Dec 01 14:59:48 +0000 2022] Application is Activated, waiting for resources to be assigned for AM. Details : AM Partition = hd ; Partition Resource = <memory:571392, vCores:144> ; Queue's Absolute capacity = 100.0 % ; Queue's Absolute used capacity = 25.0 % ; Queue's Absolute max capacity = 100.0 % ;
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: hd
start time: 1669906788033
final status: UNDEFINED
tracking URL: http://localhost:8088/proxy/application_1668609319046_131486/
user: user-1
2022-12-01 16:59:48 INFO Application$:22 - Finishing job 1
2022-12-01 16:59:48 INFO YarnClientImpl:274 - Submitted application application_1668609319046_131487
2022-12-01 16:59:48 INFO Client:54 - Application report for application_1668609319046_131487 (state: ACCEPTED)
2022-12-01 16:59:48 INFO Client:54 -
client token: Token { kind: YARN_CLIENT_TOKEN, service: }
diagnostics: AM container is launched, waiting for AM container to Register with RM
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: default
start time: 1669906788785
final status: UNDEFINED
tracking URL: http://localhost:8088/proxy/application_1668609319046_131487/
user: user-2
2022-12-01 16:59:48 INFO Application$:26 - Finishing job 2
2022-12-01 16:59:48 INFO ShutdownHookManager:54 - Shutdown hook called
2022-12-01 16:59:48 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-38b0908d-103d-47fc-bec9-f136daa84b8b
2022-12-01 16:59:48 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-4e0e9b29-2e53-4d76-a5e3-11b0b6361335
2022-12-01 16:59:48 INFO ShutdownHookManager:54 - Deleting directory /tmp/spark-d36e58bb-69e2-4898-ad4d-9bde0982f0cb
Conclusion
Yes, and Subject.doAs can indeed be used to separate keytabs like in https://github.com/AbsaOSS/hyperdrive-trigger/pull/744/files