CC's - Elevated deployment and function invocations failures – Incident details

All systems operational

Elevated deployment and function invocations failures

Resolved
Degraded performance
Started 3 months agoLasted about 9 hours

Affected

第三方

Degraded performance from 6:06 AM to 2:59 PM, Operational from 2:59 PM to 3:20 PM

Vercel → Serverless Functions

Degraded performance from 6:06 AM to 2:59 PM, Operational from 2:59 PM to 3:20 PM

Updates
  • Resolved
    Resolved

    This incident has been resolved.

  • Monitoring
    Monitoring

    We have rolled out a second mitigation for elevated Build errors and are seeing recovery. All builds are now excluding the Dubai region (dxb1) from their deployment targets as a temporary measure. We will provide additional updates as they become available.

  • Update
    Update

    We have rolled out a first mitigation for elevated Build errors. Builds that use Middleware are now excluding the Dubai region (dxb1) from their deployment targets as a temporary measure, and should complete successfully again. We are now working on a mitigation for Builds that are using Edge Functions.

  • Update
    Update

    We are currently deploying a mitigation for elevated Build errors. Builds that use Middleware or Edge Functions will exclude the Dubai region (dxb1) from their deployment targets as a temporary measure. We will provide additional updates as they become available.

  • Update
    Update

    We are still seeing elevated errors in Builds in all regions, because Middleware and Edge Functions may be deployed globally. Builds that don't use Middleware and Edge Functions are not impacted. We are continuing to work on a fix for this issue.

  • Update
    Update

    The dxb1 Edge traffic is currently being rerouted to the nearest Edge region (bom1) to mitigate the impact. We will provide additional updates as they become available.

  • Update
    Update

    We have rolled out mitigations and are seeing recovery. If you are still seeing build failures and are using Dubai (dxb1) as your primary Vercel Functions region, you can switch to another region as a workaround.

  • Identified
    Identified

    Starting from 5:00 am UTC, we have started seeing failures to deploy and invoke functions in Dubai region (dxb1). Deployments with Middleware Functions are also impacted in all regions, because Middleware Functions are deployed globally for production deployments. Our team is actively investigating the issue.