Publishing is a long running task, but recently we saw issues with a pod that was slowly degrading and slowing publishing tasks even further.
We should track the time taken for a publishing job and report an exception to Sentry (but not raise the exception in the task) when publishing takes too long.
Similar functionality is implemented here for tracking slow change application: https://github.com/learningequality/studio/blob/unstable/contentcuration/contentcuration/viewsets/sync/endpoint.py#L181
A threshold of 1 hour should be sufficient for now.
Publishing is a long running task, but recently we saw issues with a pod that was slowly degrading and slowing publishing tasks even further.
We should track the time taken for a publishing job and report an exception to Sentry (but not raise the exception in the task) when publishing takes too long.
Similar functionality is implemented here for tracking slow change application: https://github.com/learningequality/studio/blob/unstable/contentcuration/contentcuration/viewsets/sync/endpoint.py#L181
A threshold of 1 hour should be sufficient for now.