[SPARK-25930][K8s] Fix scala string detection in k8s tests

[skonto]

What changes were proposed in this pull request?

• Issue is described in detail in SPARK-25930. Since we rely on the std output, pick always the last line which contains the wanted value. Although minor, current implementation breaks tests.

How was this patch tested?

manually. rm -rf ~/.m2 and then run the tests.

[skonto]

@srowen pls review. We missed that, kind of tricky. Making mvn silent is not that easy from what I checked, now I think this is robust enough.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/4725/

[SparkQA]

Test build #98409 has finished for PR 22931 at commit 913b4c4.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/4725/

[shaneknapp]

test this please

[vanzin]

This is what make-distribution.sh uses, perhaps do the same here?

(It also has a slight advantage of not hardcoding the version of the plugin.)

SCALA_VERSION=$("$MVN" help:evaluate -Dexpression=scala.binary.version $TRIMMED_ARGS 2>/dev/null\
    | grep -v "INFO"\
    | grep -v "WARNING"\
    | tail -n 1)

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/4726/

[SparkQA]

Test build #98410 has finished for PR 22931 at commit 913b4c4.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/4726/

[skonto]

@vanzin ok let me try that. I dont need $TRIMMED_ARGS

[shaneknapp]

btw next k8s test that runs will actually have logs archived for the build!

here's the integration test log from this run, which wasn't archived...
integration-tests.log

[skonto]

@vanzin modified it. Much better.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/4727/

[SparkQA]

Test build #98412 has finished for PR 22931 at commit bf85974.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

OK. Does this need to go in branch 2.4 too?

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/4727/

[shaneknapp]

sorry, ignore amplab's report. the build passed, but my hacking on the integration test reports was what caused the failure.

[skonto]

@srowen yes.

[SparkQA]

Test build #4412 has finished for PR 22931 at commit bf85974.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[srowen]

Merged to master/2.4