Skip to content

refactor: use Bytes instead of Vec<u8> in CarBlock#7023

Merged
hanabi1224 merged 1 commit into
mainfrom
hm/opt-car-block
May 8, 2026
Merged

refactor: use Bytes instead of Vec<u8> in CarBlock#7023
hanabi1224 merged 1 commit into
mainfrom
hm/opt-car-block

Conversation

@hanabi1224
Copy link
Copy Markdown
Contributor

@hanabi1224 hanabi1224 commented May 8, 2026

Summary of changes

Changes introduced in this pull request:

  • to improve the performance of iterating all CIDs in a Forest CAR with CarStream (e.g. Building indices of Forest CAR require iterating all CIDs without accessing data)
# main
Iterating lite_40320_5986246.forest.car.zst
loaded 120095076 blocks from lite_40320_5986246.forest.car.zst, took 4m 22s 936ms 105us 98ns

# PR
Iterating lite_40320_5986246.forest.car.zst
loaded 120095076 blocks from lite_40320_5986246.forest.car.zst, took 3m 49s 148ms 266us 6ns

Reference issue to close (if applicable)

Closes

Other information and links

Change checklist

  • I have performed a self-review of my own code,
  • I have made corresponding changes to the documentation. All new code adheres to the team's documentation standards,
  • I have added tests that prove my fix is effective or that my feature works (if possible),
  • I have made sure the CHANGELOG is up-to-date. All user-facing changes should be reflected in this document.

Outside contributions

  • I have read and agree to the CONTRIBUTING document.
  • I have read and agree to the AI Policy document. I understand that failure to comply with the guidelines will lead to rejection of the pull request.

Summary by CodeRabbit

  • Refactor

    • Internal improvements to data structure handling for enhanced efficiency.
    • Updated caching mechanism for better resource management.
    • Adjusted dependency configuration to support additional features.
  • Tests

    • Updated test implementations for consistency.

@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented May 8, 2026

Review Change Stack

Walkthrough

This PR migrates the CarBlock payload field from Vec<u8> to bytes::Bytes across the codebase. The change includes updating the core CarBlock struct definition, adding the bytes crate feature to build dependencies, updating cache layer types, and converting all construction sites to use explicit .into() conversions for byte payload handling.

Changes

CarBlock Bytes Type Migration

Layer / File(s) Summary
Core CarBlock Type Definition
src/utils/db/car_stream.rs
CarBlock struct field changes from Vec<u8> to Bytes; CarBlock::from_bytes parsing uses cursor.into_inner() directly as Bytes instead of to_vec().
Workspace Dependency & Parsing
Cargo.toml
Adds bytes feature to get-size2 workspace dependency to support bytes crate integration.
Cache Layer Type Updates
src/db/car/mod.rs
ZstdFrameCache LRU cache value type changes from Vec<u8> to Bytes; public signatures updated: get() returns Option<Option<Bytes>>, put() accepts HashMap<CidWrapper, Bytes>; test helper gen_index generates Bytes values.
Blockstore Implementation Updates
src/db/car/forest.rs
ForestCar::get cache-hit and collision-decode paths return Vec<u8> via to_vec() instead of cached value directly.
CarBlock Construction Site Updates
src/blocks/tipset.rs, src/chain/mod.rs, src/db/memory.rs, src/tool/subcommands/archive_cmd.rs, src/rpc/methods/state.rs
All sites constructing CarBlock use explicit .into() conversion for the data field: data: payload.into().
Task::Emit Payload Type Update
src/ipld/util.rs
Task::Emit variant payload type changes from Option<Vec<u8>> to Option<Bytes>; ChainStream and IpldStream emit handling wraps blockstore data with .into() for conversion; tipset header block enqueue sites pass .into() to Emit tasks.
Test Assertion & Validation Updates
src/db/car/forest.rs, src/db/parity_db/gc.rs, src/tool/subcommands/car_cmd.rs, src/utils/db/car_stream/tests.rs, src/utils/db/car_util.rs
Test assertions convert blockstore results to Bytes before comparison; car_cmd validate function compares using Bytes::from(block.data); test helpers construct CarBlock with .into() conversions.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

  • ChainSafe/forest#6901: Both PRs perform the same Vec to bytes::Bytes migration pattern across caching and blockstore components.
  • ChainSafe/forest#5841: Both PRs modify get-size dependency and LRU cache value types to bytes::Bytes.
  • ChainSafe/forest#5868: Both PRs modify Task::Emit and src/ipld/util.rs block data handling related to CAR block construction.

Suggested reviewers

  • LesnyRumcajs
  • akaladarshi
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the main refactoring: replacing Vec with Bytes in CarBlock across the codebase.
Docstring Coverage ✅ Passed Docstring coverage is 83.33% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch hm/opt-car-block
✨ Simplify code
  • Create PR with simplified code
  • Commit simplified code in branch hm/opt-car-block

Comment @coderabbitai help to get the list of available commands and usage tips.

Comment thread src/utils/db/car_stream.rs
@hanabi1224 hanabi1224 marked this pull request as ready for review May 8, 2026 08:42
@hanabi1224 hanabi1224 requested a review from a team as a code owner May 8, 2026 08:42
@hanabi1224 hanabi1224 requested review from LesnyRumcajs and sudo-shashank and removed request for a team May 8, 2026 08:42
Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/tool/subcommands/car_cmd.rs`:
- Line 120: The current validation masks DB/index errors by using
.ok().flatten(), so replace that silent conversion with proper error propagation
and context: call db.get(&block.cid) and propagate errors using ? plus
.context(...) (or .with_context(...)) to attach a helpful message, then flatten
the Option and map(Bytes::from) into a variable (e.g., stored) and use
anyhow::ensure!(stored == Some(block.data), "validation mismatch: ...") to
report mismatches; reference the db.get(&block.cid) call and the ensure!
comparison so you update the exact expression that currently reads
db.get(&block.cid).ok().flatten().map(Bytes::from) == Some(block.data).
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 08e2edf0-f5a4-4ad4-a965-2163fcd3a015

📥 Commits

Reviewing files that changed from the base of the PR and between 9259f45 and 95bd88b.

⛔ Files ignored due to path filters (1)
  • Cargo.lock is excluded by !**/*.lock
📒 Files selected for processing (14)
  • Cargo.toml
  • src/blocks/tipset.rs
  • src/chain/mod.rs
  • src/db/car/forest.rs
  • src/db/car/mod.rs
  • src/db/memory.rs
  • src/db/parity_db/gc.rs
  • src/ipld/util.rs
  • src/rpc/methods/state.rs
  • src/tool/subcommands/archive_cmd.rs
  • src/tool/subcommands/car_cmd.rs
  • src/utils/db/car_stream.rs
  • src/utils/db/car_stream/tests.rs
  • src/utils/db/car_util.rs

Comment thread src/tool/subcommands/car_cmd.rs
@codecov
Copy link
Copy Markdown

codecov Bot commented May 8, 2026

Codecov Report

❌ Patch coverage is 67.79661% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 64.24%. Comparing base (8f2b32b) to head (95bd88b).
⚠️ Report is 2 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
src/rpc/methods/state.rs 0.00% 12 Missing ⚠️
src/ipld/util.rs 69.23% 4 Missing ⚠️
src/db/parity_db/gc.rs 66.66% 0 Missing and 2 partials ⚠️
src/tool/subcommands/archive_cmd.rs 0.00% 1 Missing ⚠️
Additional details and impacted files
Files with missing lines Coverage Δ
src/blocks/tipset.rs 84.88% <100.00%> (+0.08%) ⬆️
src/chain/mod.rs 76.85% <100.00%> (ø)
src/db/car/forest.rs 83.87% <100.00%> (+0.09%) ⬆️
src/db/car/mod.rs 85.71% <100.00%> (ø)
src/db/memory.rs 88.81% <100.00%> (ø)
src/tool/subcommands/car_cmd.rs 77.93% <100.00%> (+0.46%) ⬆️
src/utils/db/car_stream.rs 71.42% <100.00%> (-0.36%) ⬇️
src/utils/db/car_util.rs 88.88% <100.00%> (ø)
src/tool/subcommands/archive_cmd.rs 29.30% <0.00%> (ø)
src/db/parity_db/gc.rs 35.20% <66.66%> (+2.14%) ⬆️
... and 2 more

... and 13 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 9259f45...95bd88b. Read the comment docs.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@hanabi1224 hanabi1224 enabled auto-merge May 8, 2026 09:50
@hanabi1224 hanabi1224 added this pull request to the merge queue May 8, 2026
Merged via the queue into main with commit 7a988ea May 8, 2026
40 checks passed
@hanabi1224 hanabi1224 deleted the hm/opt-car-block branch May 8, 2026 10:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants