[SPARK-9853][Core] Optimize shuffle fetch of contiguous partition IDs#19788

Closed

yucai wants to merge 88 commits into

apache:masterfrom

yucai:shuffle_fetch_opt

Commits on Nov 20, 2017

[SPARK-9853][Core] Optimize shuffle fetch of contiguous partition IDs
yucai
committed

Commits on Nov 21, 2017

Simplify length in convertMapStatuses
yucai
committed

Commits on Nov 24, 2017

Simplify totalSize in convertMapStatuses
yucai
committed
Modify for external shuffle service
yucai
committed
Solve mima issue
yucai
committed
Fix ExternalShuffleBlockResolverSuite
yucai
committed

Commits on Feb 1, 2018

Merge remote-tracking branch 'origin/master' into 19788.1
yucai
committed
Remove unnecessary logDebug
yucai
committed
Improve indent
yucai
committed
Using ContinuousShuffleBlockId
yucai
committed
Support ContinuousShuffleBlockId in external shuffle service and fix tests issue
yucai
committed
Fix tests failure
yucai
committed
Add comments for getBlockData in ExternalShuffleBlockResolver
yucai
committed
Update as per internal review
yucai
committed
Minor fix
yucai
committed
Fix conflict for SPARK-22982
yucai
committed
Rename ContinuousShuffleBlockId.length to ContinuousShuffleBlockId.numBlocks
yucai
committed

Commits on Feb 2, 2018

Rename length to numBlocks
yucai
committed
Fix mima issue
yucai
committed

Commits on Feb 11, 2018

Commits on Feb 22, 2018

Commits on Mar 1, 2018

Use serializerRelocatable: Boolean
yyu29
committed
Keep FetchFailed unchanged
yucai
committed

Commits on Mar 3, 2018

Merge remote-tracking branch 'upstream/master' into shuffle_fetch_opt
yucai
committed

Commits on Mar 4, 2018

minor
yucai
committed

Commits on Oct 22, 2018

Merge remote-tracking branch 'origin/master' into pr19788
yucai
committed

Commits on Oct 24, 2018

add spark.shuffle.continuousBlockBatchFetch
yucai
committed

Commits on Jan 2, 2019

Commits on Jan 4, 2019

external shuffle service
yucai
committed
remove spark.shuffle.continuousBlockBatchFetch
yucai
committed
improve encoders
yucai
committed
rename shuffleBlocksBatchFetch
yucai
committed
address.executorId == blockManager.blockManagerId.executorId
yucai
committed
enhance support for failure fetch
yucai
committed

Commits on Jan 5, 2019

DownloadCallback's onFailure
yucai
committed
Fix MapOutputTrackerSuite
yucai
committed
Fix tests
yucai
committed

Commits on Jan 6, 2019

minor
yucai
committed
Fix ExternalShuffleIntegrationSuite
yucai
committed

Commits on Jan 7, 2019

Fix ShuffleBlockFetcherIteratorSuite
yucai
committed
Fix ExternalShuffleServiceSuite
yucai
committed

Commits on Jan 8, 2019

rename to ShuffleBlockBatchId
yucai
committed
Fix retry scenario
yucai
committed
add tests for ShuffleBlockFetcherIteratorSuite
yucai
committed
enable optimization only when AE enables
yucai
committed

Commits on Jan 9, 2019

add testBatchFetchThreeSort
yucai
committed
add tests for OpenBlocks and StreamHandle backward compatibility
yucai
committed
Fix MapOutputTrackerSuite
yucai
committed

Commits on Jan 10, 2019

minor
yucai
committed
Improve corrupt retry for external shuffle service
yucai
committed
improve getBlockData
yucai
committed
minor
yucai
committed

Commits on Jan 11, 2019

minor
yucai
committed
fix all java style issue
yucai
committed
fix ExternalShuffleBlockHandlerSuite
yucai
committed

Commits on Jan 14, 2019

decrease 0-size array creation
yucai
committed

Commits on Jan 15, 2019

Dont change StreamHandle
yucai
committed
address bug
yucai
committed
fix java style
yucai
committed
don't use batchId in refetch
yucai
committed
java style
yucai
committed
return blockIds when failure
yucai
committed
Merge remote-tracking branch 'origin/master' into pr19788_server
yucai
committed

Commits on Jan 16, 2019

using ArrayShuffleBlockId to improve retry
yucai
committed
fetchContinuousShuffleBlocksInBatch == true, OpenBlocks could contain ShuffleBlockId only
yucai
committed
checkout MapOutputTracker.scala
yucai
committed
merge local in ShuffleBlockFetcherIterator
yucai
committed
revert MapOutputTrackerSuite.scala
yucai
committed

Commits on Jan 17, 2019

local merge
yucai
committed
remove ArrayShuffleBlockString
yucai
committed
use arrayBlockId in results always
yucai
committed
minor
yucai
committed
remove ShuffleBlockBatchId
yucai
committed
Merge remote-tracking branch 'origin/master' into pr19788_server
yucai
committed

Commits on Jan 19, 2019

simplify merge function
yucai
committed

Commits on Jan 20, 2019

minor
yucai
committed
Merge remote-tracking branch 'origin/master' into pr19788_server
yucai
committed

Commits on Jan 22, 2019

address comments
yucai
committed
minor
yucai
committed