InconsistentStateException in AzureBlobStorage when trying to migrate from 3.x to 7

[krmr]

Using migration grain storage with GrainMigrationMode.ReadDestinationWithFallback_WriteBoth an InconsistentStateException in AzureBlobStorage when trying to write state to the original source. This is because blob storage returns 409 with an error code BlobAlreadyExists
Note that this does not happen on the first call when the grain state is migrated but only when running the application again later.

From looking into the underlying code this seems to be caused by the source Etag to be null in this situation. At least my understanding is that after the initial migration only the destination grain will be read from blob. This leaves the source Etag null which seems to cause the exception.

More background:

• .Net 8.0 on windows
• Orleans version: 3.8.0-preview5
• using Azurite for Azure Blob Storage

Here the stack trace:

Orleans.Persistence.Migration.MigrationGrainStorage[102203]
      Error from storage provider MigrationGrainStorage.HelloWorld.Grains.HelloArchiveGrain,HelloWorld.Grains.archive during WriteState for grain Type=HelloWorld.Grains.HelloArchiveGrain,HelloWorld.Grains.archive Pk=*grn/HelloWorld.Grains.HelloArchiveGrain/0-0xB8D1C56B Id=GrainReference:*grn/HelloWorld.Grains.HelloArchiveGrain/0 Error=

      Exc level 0: Orleans.Storage.InconsistentStateException: Blob storage condition not Satisfied.  BlobName: HelloWorld.Grains.HelloArchiveGrain,HelloWorld.Grains.archive-GrainReference=0000000000000000000000000000000003fffffff6440cfb.json, Container: grainstate, CurrentETag:
         at Orleans.Storage.AzureBlobGrainStorage.DoOptimisticUpdate[TResult](Func`1 updateOperation, BlobClient blob, String currentETag) in /_/src/Azure/Orleans.Persistence.AzureStorage/Providers/Storage/AzureBlobStorage.cs:line 224
         at Orleans.Storage.AzureBlobGrainStorage.WriteStateAndCreateContainerIfNotExists(String grainType, GrainReference grainId, IGrainState grainState, Byte[] contents, String mimeType, BlobClient blob) in /_/src/Azure/Orleans.Persistence.AzureStorage/Providers/Storage/AzureBlobStorage.cs:line 198
         at Orleans.Storage.AzureBlobGrainStorage.WriteStateAsync(String grainType, GrainReference grainId, IGrainState grainState) in /_/src/Azure/Orleans.Persistence.AzureStorage/Providers/Storage/AzureBlobStorage.cs:line 137
         at Orleans.Persistence.Migration.MigrationGrainStorage.WriteStateAsync(String grainType, GrainReference grainReference, IGrainState grainState) in /_/src/Orleans.Persistence.Migration/MigrationGrainStorage.cs:line 181
         at Orleans.Core.StateStorageBridge`1.WriteStateAsync() in /_/src/Orleans.Runtime/Storage/StateStorageBridge.cs:line 97
      Exc level 1: Azure.RequestFailedException: The specified blob already exists.

A repository to reproduce the issue is https://github.com/krmr/OrleansStorageMigrationRepro (which is based on Samples/3.0/HelloWorld).

Steps to reproduce the issue:

1. start Azurite
2. start SiloHost and wait until it is up and running
3. run OrleansClient – this should work just fine
4. stop both OrleansClient and SiloHost
5. start SiloHost and wait until it is up and running
6. run OrleansClient – this time the exception will be shown