Skip to content

cuda : add support for f16->f16 GGML_OP_SET_ROWS#25367

Merged
am17an merged 2 commits into
ggml-org:masterfrom
fairydreaming:set-rows-f16-cuda
Jul 8, 2026
Merged

cuda : add support for f16->f16 GGML_OP_SET_ROWS#25367
am17an merged 2 commits into
ggml-org:masterfrom
fairydreaming:set-rows-f16-cuda

Conversation

@fairydreaming

@fairydreaming fairydreaming commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Overview

This PR adds support for CUDA GGML_OP_SET_ROWS where src0 type is f16 and dst type is also f16. This will come in handy in models based on DeepSeek Sparse Attention (DeepSeek V3.2, V4, GLM 5.0, 5.1, 5.2) where we have to "sparsify" KQ mask based on lightning indexer indices. With f16 support in GGML_OP_SET_ROWS it will be possible to use f16 FA KQ masks without any extra casts, this will reduce memory usage and simplify code.

Additional information

set_rows_cuda() already supported f16 in src_t template argument internally, so I simply added two fully specialized set_rows_cuda<half, int32_t>() and set_rows_cuda<half, int64_t>() that omit unnecessary dst type cases (necessary as set_rows_cuda_quant() handles only float src0 type). Partial specialization would be more compact, but is not supported for functions.

GGML_OP_SET_ROWS tests for f16->f16 case are in #25344 that shall be merged first.

Requirements

@github-actions github-actions Bot added ggml changes relating to the ggml tensor library for machine learning CUDA Related to the CUDA backend labels Jul 6, 2026
@fairydreaming fairydreaming marked this pull request as ready for review July 8, 2026 05:27
@fairydreaming fairydreaming requested a review from a team as a code owner July 8, 2026 05:27
@am17an am17an added the merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge. label Jul 8, 2026

@ggerganov ggerganov left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the HIP quality check is not caused by these changes?

@am17an am17an merged commit ed8c261 into ggml-org:master Jul 8, 2026
21 of 22 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CUDA Related to the CUDA backend ggml changes relating to the ggml tensor library for machine learning merge ready A maintainer can use this label to indicate that they consider the changes final and ready to merge.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants