Skip to content

Object store failing during bulk delete in S3 #277

@Omega359

Description

@Omega359

Describe the bug

This started recently and I'm uncertain as to the cause. I have a rather large S3 directory (10TB) that I delete using the following code:

let object_store_s3_path = &object_store::path::Path::from(s3_path);
let stream = object_store.list(Some(object_store_s3_path)).map_ok(|m| m.location).boxed();
object_store.delete_stream(stream).try_collect::<Vec<Path>>().await?;

The error I started receiving this weekend is:

Got invalid DeleteObjects response: unknown variant `Code`, expected `Deleted` or `Error`

Caused by:
    0: Got invalid DeleteObjects response: unknown variant `Code`, expected `Deleted` or `Error`
    1: unknown variant `Code`, expected `Deleted` or `Error`

Using version 0.11.0.

Deleting the directory via other tooling works as expected.

To Reproduce

For me just attempting to delete a large directory seems to trigger it.

Expected behavior

Additional context

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions