[SPARK-36206][CORE] Support shuffle data corruption diagnosis via shuffle checksum#33451

Closed

Ngone51 wants to merge 48 commits into

apache:masterfrom

Ngone51:SPARK-36206

Commits on Aug 2, 2021

init
Ngone51
committed
update
Ngone51
committed
move comment
Ngone51
committed
fix ExternalBlockStoreClient
Ngone51
committed
mark Cause as private
Ngone51
committed
remove unused Cause import
Ngone51
committed
fix indents of BlockStoreClient.diagnoseCorruption
Ngone51
committed
verify -> verifying
Ngone51
committed
fix comment
Ngone51
committed
include checksumfile path
Ngone51
committed
remove throws
Ngone51
committed
use ByteStreams.skipFully
Ngone51
committed
check cheaper fields first
Ngone51
committed
use this.transportConf
Ngone51
committed
add todo for pushbased shuffle
Ngone51
committed
resolve magic number
Ngone51
committed
use fileName strip the suffix
Ngone51
committed
use Files.newDirectoryStream
Ngone51
committed
remove checksum file existence check
Ngone51
committed
combine ShuffleChecksumHelper&ShuffleCorruptionDiagnosisHelper
Ngone51
committed
fix tests
Ngone51
committed
diagnose corruption when the block corrupted twice
Ngone51
committed
send checksum algorithm together
Ngone51
committed
fix tests
Ngone51
committed
fix rat of ShuffleChecksumSupport
Ngone51
committed
add since for Cause
Ngone51
committed
add comment for fileName
Ngone51
committed
fix ExternalBlockStoreClient
Ngone51
committed
use the explicit checksum algorithm
Ngone51
committed
update comment of diagnoseCorruption
Ngone51
committed
improve error msg for different causes
Ngone51
committed
inline diagnosisResponse into diagnoseCorruption
Ngone51
committed
update warning msg
Ngone51
committed
fix ShuffleBlockFetcherIteratorSuite
Ngone51
committed
fix java lint
Ngone51
committed
fix tests
Ngone51
committed
swallow exception from diagnoseCorruption
Ngone51
committed
ensure test stability
Ngone51
committed
update version to 3.2.0
Ngone51
committed
fix import style
Ngone51
committed
add tests
Ngone51
committed
refactor bufIn
Ngone51
committed
use nano time
Ngone51
committed
Use Option
Ngone51
committed
move Cause
Ngone51
committed
fix tests
Ngone51
committed
fix
Ngone51
committed
address comment
Ngone51
committed