[SPARK-21738] Thriftserver doesn't cancel jobs when session is closed

[mgaido91]

What changes were proposed in this pull request?

When a session is closed the Thriftserver doesn't cancel the jobs which may still be running. This is a huge waste of resources.
This PR address the problem canceling the pending jobs when a session is closed.

How was this patch tested?

The patch was tested manually.

[gatorsmile]

ok to test

[SparkQA]

Test build #80705 has finished for PR 18951 at commit ac7f0d4.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[gatorsmile]

LGTM

cc @cloud-fan @jiangxb1987 @wangyum @debugger87 @jerryshao

[jiangxb1987]

LGTM

[jerryshao]

LGTM based on my limited knowledge on area 😄 .

[gatorsmile]

Thanks! Merging to master.

[dongjoon-hyun]

Hi, @gatorsmile and @mgaido91 .
Can we have this fix in branch-2.2, too?

[mgaido91]

@dongjoon-hyun I think there is no problem, should I submit a new PR on branch-2.2 too then?

[dongjoon-hyun]

Thank you, @mgaido91 . Please! :)

[gatorsmile]

This issue is not a regression. That is why I did not merge it to branch-2.2

[dongjoon-hyun]

I see. @gatorsmile . Sorry, @mgaido91 and @gatorsmile .

[doruchiulan]

What happens when a connection is not gracefully closed ? (connection.close() not called, process exits). Does sessions somehow listen to underneath Jetty connection and somehow close themselves ?

[RamakrishnaChilaka]

Can someone please update on @doruchiulan 's statement? I want to know the same thing.

[cloud-fan]

cc @yaooqinn

[yaooqinn]

FYI, org.apache.thrift.server.TServerEventHandler#deleteContext