HuggingFace download issues

[investigating] We are aware of 504s being returned by models that reach out to HuggingFace during setup. We believe this is likely related to the disruption visible at [https://status.huggingface.co/](https://status.huggingface.co/ "https://status.huggingface.co/"). We are monitoring and will update when we can identify the HuggingFace CDN is back up/

[monitoring] HuggingFace is reporting recovery, which we are also seeing on our end. However, we still see minor degradation, which we believe is due to thundering herd once the main issue was resolved. We are still keeping an eye out, but believe we are recovering

[monitoring] We have seen most HuggingFace calls start succeeding, except for a small subset that are being downloaded through [cas-bridge.xethub.hf.co](http://cas-bridge.xethub.hf.co "cas-bridge.xethub.hf.co"). Most models are now succeeding to spin up

[monitoring] Still seeing signs of improvement, most models are able to spin up with a few still waiting their turn from HuggingFace. Models should spin up after enough tries if they are failing for HuggingFace related reasons.

[resolved] We have not seen elevated rates of HuggingFace/model setup errors for a couple hours now, so we believe this incident is cleared