Increasing failure rate when downloading packages - 504 (Gateway Time-out)

[ssmith80]

Prerequisites

• Write a descriptive title.
• Make sure you are able to repro it on the latest version
• Search the existing issues.

Steps to reproduce

Our Azure DevOps release pipeline installs the Az modules, and we have noticed a considerable increase in the frequency of 504 Gateway Time-out errors in the past few days. It has failed repeatedly in the last hour.

Expected behavior

We can pull the Az modules package in a timely manner.

Actual behavior

The process fails, with the following output:
‌Install-PSResource : ‌Response status code does not indicate success: 504 (Gateway Time-out).

Error details

‌Install-PSResource : ‌Response status code does not indicate success: 504 (Gateway Time-out). Request sent: https:www.powershellgallery.comapiv2FindPackagesById()?$filter=NormalizedVersion+ge+%274.0.1%27+and+Id+eq+%27Az.Accounts%27&$inlinecount=allpages&$skip=0&$orderby=NormalizedVersion+desc&id=%27Az.Accounts%27‌

Environment data

Azure DevOps

Version

?? Azure DevOps

Visuals

No response

[ssmith80]

We are still experiencing this with high frequency...

[alerickson]

Hi, the request from the error looks it's formatted strangely. Is this error consistent or intermittent?
Can you share the output of:
Get-PSResourceRepository -Name PSGallery

[boblodgett]

We are seeing slowness and timeouts as well. Seems something may have started around the 17th. Requests are not taking 900k MS before they timeout this is just our general AWS monitor for all our packages in PSG.

[alerickson]

Hi @boblodgett we've had a few reports of this now. We're seeing failures and high amounts of specific exceptions but are still investigating the underlying issues. Our CPU is a bit high, so we've scaled up the instances we're using. Please report back here if the issue persists or resolves. We'll be monitoring our logs to see if these exceptions continue but it's helpful to have more reports with timestamps from users as well.

[boblodgett]

The duration has been normal since around 1PM on 6/20 PT.

[ssmith80]

We had a much better day yesterday with no more 504s so this seems to have resolved the issue, thanks. I will report back should it start occurring again.

[pbo-top]

We had several pipelines failing this morning (7-8am CET) when they tried to install modules, and the PowerShell Gallery website was really slow and gave 504 in the browser randomly. Now, 9am it seems more responsive again. So still problems with stability.

[ssmith80]

This is happening again this morning, and preventing us from deploying to production. Our release pipelines have been trying to pull from PSGallery for over an hour, some of which are timing out with 504s. I am concerned that this seems to happen too frequently for my liking - it is such a crucial step to our deployment. Please advise on a) fixing it short term and b) what are the longer term plans to stop this reoccuring?