[SPARK-11328] [SQL] Improve error message when hitting this issue.

[nongli]

The issue is that the output commiter is not idempotent and retry attempts will
fail because the output file already exists. It is not safe to clean up the file
as this output committer is by design not retryable. Currently, the job fails
with a confusing file exists error. This patch is a stop gap to tell the user
to look at the top of the error log for the proper message.

This is difficult to test locally as Spark is hardcoded not to retry. Manually
verified by upping the retry attempts.

[marmbrus]

add to whitelist

[SparkQA]

Test build #46619 has finished for PR 9942 at commit d19b5c3.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[cloud-fan]

we can remove this line, scala will throw un-catched exception

[yhuai]

earlier logs => earlier logs or stage page?

[yhuai]

oh, https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/WriterContainer.scala#L406 is another place we create writer.

[yhuai]

Seems the jira number is not the right one?

[yhuai]

ah it is actually https://issues.apache.org/jira/browse/SPARK-11328

[naveenminchu]

May be its better idea to wrap current exception instance into SparkException
Example: SparkException("some error message", e)

[nongli]

I explicitly did not do that because that makes it really seem like the root cause is "file already exists". This error would be in the middle of a big stack trace. We can go either way though.

[SparkQA]

Test build #46989 has finished for PR 9942 at commit c4375ec.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):\n * class ChainedIterator extends UnsafeSorterIterator\n * class UnsafeCartesianRDD(left : RDD[UnsafeRow], right : RDD[UnsafeRow], numFieldsOfRight: Int)\n