[KYUUBI #3677][K8S] Bump Kubernetes Client Version to 6.1.1

[zwangsheng]

Why are the changes needed?

Bump Kubernetes Client Version to 6.1.1.
To keep up with apache/spark#37990 dependence kubernetes client changes.

Close #3677

How was this patch tested?

• Add some test cases that check the changes thoroughly including negative and positive cases if possible

• Add screenshots for manual tests if appropriate

• Run test locally before make a pull request

[zwangsheng]

Fail with some flaky test, cc @ulysses-you @pan3793 @yaooqinn

[codecov-commenter]

Codecov Report

Merging #3691 (d84d1c1) into master (8760eee) will decrease coverage by 0.05%.
The diff coverage is 8.82%.

@@             Coverage Diff              @@
##             master    #3691      +/-   ##
============================================
- Coverage     53.48%   53.44%   -0.05%     
  Complexity       13       13              
============================================
  Files           560      560              
  Lines         30412    30417       +5     
  Branches       4081     4084       +3     
============================================
- Hits          16266    16255      -11     
- Misses        12645    12656      +11     
- Partials       1501     1506       +5     

Impacted Files	Coverage Δ
...kyuubi/engine/KubernetesApplicationOperation.scala	21.62% <0.00%> (-0.92%)	⬇️
...scala/org/apache/kyuubi/util/KubernetesUtils.scala	82.14% <100.00%> (+0.66%)	⬆️
.../kyuubi/server/mysql/constant/MySQLErrorCode.scala	13.84% <0.00%> (-6.16%)	⬇️
...ache/kyuubi/server/mysql/MySQLCommandHandler.scala	77.77% <0.00%> (-4.05%)	⬇️
...g/apache/kyuubi/operation/BatchJobSubmission.scala	75.27% <0.00%> (-2.20%)	⬇️
...ache/kyuubi/server/mysql/MySQLGenericPackets.scala	76.59% <0.00%> (-2.13%)	⬇️
...rg/apache/kyuubi/ctl/cmd/log/LogBatchCommand.scala	59.09% <0.00%> (-1.52%)	⬇️
...mon/src/main/scala/org/apache/kyuubi/Logging.scala	41.25% <0.00%> (-1.25%)	⬇️
...org/apache/kyuubi/operation/ExecuteStatement.scala	79.26% <0.00%> (ø)
...n/scala/org/apache/kyuubi/engine/ProcBuilder.scala	79.01% <0.00%> (+0.61%)	⬆️
... and 1 more

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[zwangsheng]

For there are some it test fail, i'd like find cause. And hope to have this PR first to help locate the problem. #3728

[ulysses-you]

@zwangsheng seems the failed tests is related ?

[zwangsheng]

@zwangsheng seems the failed tests is related ?

2/10/31 10:09:18 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[2] at isEmpty at KyuubiSparkUtil.scala:48), which has no missing parents
22/10/31 10:09:18 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 6.6 KiB, free 127.2 MiB)
22/10/31 10:09:18 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 3.5 KiB, free 127.2 MiB)
22/10/31 10:09:18 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on fv-az256-952:41265 (size: 3.5 KiB, free: 127.2 MiB)
22/10/31 10:09:18 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1478
22/10/31 10:09:18 INFO DAGScheduler: Submitting 1 missing tasks from ResultStage 0 (MapPartitionsRDD[2] at isEmpty at KyuubiSparkUtil.scala:48) (first 15 tasks are for partitions Vector(0))
22/10/31 10:09:18 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks resource profile 0
22/10/31 10:09:33 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
22/10/31 10:09:48 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
22/10/31 10:10:03 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
22/10/31 10:10:18 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
22/10/31 10:10:33 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
22/10/31 10:10:48 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
22/10/31 10:11:03 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
22/10/31 10:11:18 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
22/10/31 10:11:33 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
22/10/31 10:11:37 ERROR SparkSQLEngine: The Engine main thread was interrupted, possibly due to `createSpark` timeout. The `kyuubi.session.engine.initialize.timeout` is (180000 ms)  and submitted at 1667210917099.
java.lang.InterruptedException
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1040)
	at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345)
	at scala.concurrent.impl.Promise$DefaultPromise.tryAwait(Promise.scala:242)
	at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:258)
	at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:187)
	at org.apache.spark.util.ThreadUtils$.awaitReady(ThreadUtils.scala:334)
	at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:929)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2214)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2235)
	at org.apache.spark.SparkContext.runJob(SparkContext.scala:2254)
	at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:492)
	at org.apache.spark.sql.execution.SparkPlan.executeTake(SparkPlan.scala:445)
	at org.apache.spark.sql.Dataset.$anonfun$isEmpty$1(Dataset.scala:604)
	at org.apache.spark.sql.Dataset.$anonfun$isEmpty$1$adapted(Dataset.scala:603)
	at org.apache.spark.sql.Dataset.$anonfun$withAction$1(Dataset.scala:3706)
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$5(SQLExecution.scala:103)
	at org.apache.spark.sql.execution.SQLExecution$.withSQLConfPropagated(SQLExecution.scala:163)
	at org.apache.spark.sql.execution.SQLExecution$.$anonfun$withNewExecutionId$1(SQLExecution.scala:90)
	at org.apache.spark.sql.SparkSession.withActive(SparkSession.scala:775)
	at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:64)
	at org.apache.spark.sql.Dataset.withAction(Dataset.scala:3704)
	at org.apache.spark.sql.Dataset.isEmpty(Dataset.scala:603)
	at org.apache.kyuubi.engine.spark.KyuubiSparkUtil$.$anonfun$initializeSparkSession$1(KyuubiSparkUtil.scala:48)
	at org.apache.kyuubi.engine.spark.KyuubiSparkUtil$.$anonfun$initializeSparkSession$1$adapted(KyuubiSparkUtil.scala:41)
	at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
	at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
	at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49)
	at org.apache.kyuubi.engine.spark.KyuubiSparkUtil$.initializeSparkSession(KyuubiSparkUtil.scala:41)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine$.createSpark(SparkSQLEngine.scala:206)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine$.main(SparkSQLEngine.scala:271)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine.main(SparkSQLEngine.scala)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:955)
	at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:165)
	at org.apache.spark.deploy.SparkSubmit$$anon$1.run(SparkSubmit.scala:163)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:163)
	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1043)
	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1052)
	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

Seems ask for executor resource timeout.

I'd like revert and run ci again,

[zwangsheng]

Close for we need to bump 6.4.1