Support FP8 NVLS multimem reduction#826
Open
Binyang2014 wants to merge 5 commits into
Open
Conversation
multimem.ld_reduce on FP8 inputs can use the .acc::f16 instruction variant for FP16 accumulation. Thread AccumT through the NVLS zero-copy allreduce path so callers can select that behavior via the existing accum dtype dispatch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Probe whether the current CUDA target supports FP8 multimem reduction and use that result when selecting or launching NVLS FP8 algorithms. Keep the existing algorithm sizing and buffer behavior unchanged for this sub-PR. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Apply formatter output and mark the unused extras parameter in the NVLS block-pipeline guard path. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends NVLS-based allreduce to better support FP8 by adding an FP16 accumulation path for multimem load+reduce, and by probing/gating FP8 NVLS algorithm selection based on device/runtime support (plumbed through the NCCL algorithm selector).
Changes:
- Add an FP8 NVLS support probe (
isFp8NvlsSupported) and propagate anfp8NvlsSupportedflag through the NCCL algorithm selector config. - Add an optional FP16 accumulator variant for FP8 multimem load+reduce (via
.acc::f16instruction variants). - Thread FP8 NVLS support gating into NVLS allreduce algorithm implementations.
Reviewed changes
Copilot reviewed 14 out of 14 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| src/ext/nccl/nccl.cc | Computes FP8 NVLS support and passes it into the selector config. |
| src/ext/nccl/CMakeLists.txt | Adds collectives include path so NCCL extension can use collective utilities. |
| src/ext/nccl/algorithm_selector.hpp | Extends selector config with fp8NvlsSupported. |
| src/ext/nccl/algorithm_selector.cc | Gates NVLS selection for FP8 based on native FP8 and probed support. |
| src/ext/collectives/include/collective_utils.hpp | Declares FP8 dtype helpers and FP8 NVLS support probe API. |
| src/ext/collectives/include/allreduce/common.hpp | Extends multimem reduce helper to accept an accumulation type. |
| src/ext/collectives/include/allreduce/allreduce_nvls_zero_copy.hpp | Adds cached fp8NvlsSupported_ state to the algorithm class. |
| src/ext/collectives/include/allreduce/allreduce_nvls_warp_pipeline.hpp | Adds cached fp8NvlsSupported_ state to the algorithm class. |
| src/ext/collectives/include/allreduce/allreduce_nvls_block_pipeline.hpp | Adds cached fp8NvlsSupported_ state to the algorithm class. |
| src/ext/collectives/collective_utils.cu | Implements FP8 dtype checks and runtime FP8 NVLS support probing. |
| src/ext/collectives/allreduce/allreduce_nvls_zero_copy.cu | Adds AccumT plumbing for multimem load+reduce and runtime FP8 support gating. |
| src/ext/collectives/allreduce/allreduce_nvls_warp_pipeline.cu | Adds runtime FP8 NVLS support gating for this algorithm variant. |
| src/ext/collectives/allreduce/allreduce_nvls_block_pipeline.cu | Adds runtime FP8 NVLS support gating for this algorithm variant. |
| include/mscclpp/switch_channel_device.hpp | Adds FP16-accumulator multimem load+reduce variants for FP8 vector types. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Validation