Storage - SocketException Socket closed

[fondberg]

Hi fellow Google cloud java users,
We are seeing Socket closed quite often.
The StorageOptions.Builder().connectTimeout and StorageOptions.Builder().readTimeout are not set so I guess the default of 20 seconds is used.
I didn't want to change them as 20 seconds should be plenty.

Does anyone have an idea about this?

java.net.SocketException: Socket closed
    at java.net.SocketInputStream.read
    at java.net.SocketInputStream.read
    at sun.security.ssl.InputRecord.readFully
    at sun.security.ssl.InputRecord.read
    at sun.security.ssl.SSLSocketImpl.readRecord
    at sun.security.ssl.SSLSocketImpl.readDataRecord
    at sun.security.ssl.AppInputStream.read
    at java.io.BufferedInputStream.fill
    at java.io.BufferedInputStream.read1
    at java.io.BufferedInputStream.read
    at sun.net.www.http.HttpClient.parseHTTPHeader
    at sun.net.www.http.HttpClient.parseHTTP
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream0
    at sun.net.www.protocol.http.HttpURLConnection.getInputStream
    at java.net.HttpURLConnection.getResponseCode
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode
    at com.google.api.client.http.javanet.NetHttpResponse.<init>
    at com.google.api.client.http.javanet.NetHttpRequest.execute
    at com.google.api.client.http.HttpRequest.execute
    at com.google.cloud.storage.spi.DefaultStorageRpc.write
    at com.google.cloud.storage.BlobWriteChannel$1.run
    at java.util.concurrent.Executors$RunnableAdapter.call
    at com.google.cloud.RetryHelper.doRetry
    at com.google.cloud.RetryHelper.runWithRetries
    at com.google.cloud.RetryHelper.runWithRetries
    at com.google.cloud.storage.BlobWriteChannel.flushBuffer
    at com.google.cloud.BaseWriteChannel.close
    at com.spotify.buildartifactarchiver.storage.GcsUploaderClient.storeInputStream
    at com.spotify.buildartifactarchiver.storage.ArtifactUploader.uploadData
    at com.spotify.buildartifactarchiver.ArtifactArchiver.archiveRequest
    at com.spotify.buildartifactarchiver.ArtifactArchiver.archiveBuild
    at com.spotify.buildartifactarchiver.web.ArtifactArchiverPubSubResource.messageHandler
    at com.spotify.google.cloud.pubsub.client.Puller.lambda$pullBatch$11
    at java.util.concurrent.CompletableFuture.uniWhenComplete
    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire
    at java.util.concurrent.CompletableFuture.postComplete
    at java.util.concurrent.CompletableFuture.complete
    at com.spotify.google.cloud.pubsub.client.PubsubFuture.succeed
    at com.spotify.google.cloud.pubsub.client.PubsubFuture.lambda$wrap$9
    at java.util.concurrent.CompletableFuture.uniWhenComplete
    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire
    at java.util.concurrent.CompletableFuture.postComplete
    at java.util.concurrent.CompletableFuture.complete
    at com.spotify.google.cloud.pubsub.client.PubsubFuture.succeed
    at com.spotify.google.cloud.pubsub.client.Pubsub.lambda$requestJavaNet$8
    at java.util.concurrent.ThreadPoolExecutor.runWorker
    at java.util.concurrent.ThreadPoolExecutor$Worker.run
    at java.lang.Thread.run
com.google.cloud.storage.StorageException: Socket closed
    at com.google.cloud.storage.spi.DefaultStorageRpc.translate
    at com.google.cloud.storage.spi.DefaultStorageRpc.write
    at com.google.cloud.storage.BlobWriteChannel$1.run
    at java.util.concurrent.Executors$RunnableAdapter.call
    at com.google.cloud.RetryHelper.doRetry
    at com.google.cloud.RetryHelper.runWithRetries
    at com.google.cloud.RetryHelper.runWithRetries
    at com.google.cloud.storage.BlobWriteChannel.flushBuffer
    at com.google.cloud.BaseWriteChannel.close
    at com.spotify.buildartifactarchiver.storage.GcsUploaderClient.storeInputStream
    at com.spotify.buildartifactarchiver.storage.ArtifactUploader.uploadData
    at com.spotify.buildartifactarchiver.ArtifactArchiver.archiveRequest
    at com.spotify.buildartifactarchiver.ArtifactArchiver.archiveBuild
    at com.spotify.buildartifactarchiver.web.ArtifactArchiverPubSubResource.messageHandler
    at com.spotify.google.cloud.pubsub.client.Puller.lambda$pullBatch$11
    at java.util.concurrent.CompletableFuture.uniWhenComplete
    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire
    at java.util.concurrent.CompletableFuture.postComplete
    at java.util.concurrent.CompletableFuture.complete
    at com.spotify.google.cloud.pubsub.client.PubsubFuture.succeed
    at com.spotify.google.cloud.pubsub.client.PubsubFuture.lambda$wrap$9
    at java.util.concurrent.CompletableFuture.uniWhenComplete
    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire
    at java.util.concurrent.CompletableFuture.postComplete
    at java.util.concurrent.CompletableFuture.complete
    at com.spotify.google.cloud.pubsub.client.PubsubFuture.succeed
    at com.spotify.google.cloud.pubsub.client.Pubsub.lambda$requestJavaNet$8
    at java.util.concurrent.ThreadPoolExecutor.runWorker
    at java.util.concurrent.ThreadPoolExecutor$Worker.run
    at java.lang.Thread.run

[mziccard]

Hi @fonzy2013, thanks for the report.

Maybe @Capstan has some insights on this. I can confirm that this exception showed once in our integration tests recently. I have never experienced it before though. @Capstan can this be related to some recent change on your side?

@aozarov I believe we could retry this kind of exceptions just want to hear some more thoughts on this :)

[aozarov]

I think it should be safe to add SocketException as a retryable IOException provided the call is idempotent. Basically to add it to here.

[fondberg]

sounds great! This would make it much cleaner for us using the library. Is there anything I can do to help out with debugging the root cause of this?

[mziccard]

@fonzy2013 I am working on a PR to retry SocketException and "insufficient data written" exceptions right now. I'll cc you to it!

Is there anything I can do to help out with debugging the root cause of this?

I did some digging in the java code that is throwing these exceptions. I am quite sure this has nothing to do with our library code. I suspect these exceptions are related to network link disruption or the server resetting the connection. @Capstan is the most qualified person to confirm or deny my hypothesis.

[fondberg]

@mziccard that sounds great! Thanks a million for all your help.
Looking forward to it ending up in a release!

[mziccard]

@fonzy2013 gcloud-java 0.2.8 is now out and includes this fix!

[fondberg]

great! Thanks!