Notice #468793
Recovering

Azure FrontDoor Update

Began:

Certain limitations remain in place due to the outage on Azure Front Door that began on 10/29. Specific operations on the Azure Management API data plane (WAF Modifications, CDN, Routes, and Profiles) will be limited or restricted until further notice.

For complete details on what happened and actions taken, see the Azure Status History page here: https://azure.status.microsoft/en-us/status/history/

  • Main M3 Site
  • M3 Support/Ticket System
  • Core Cloud Systems
    • Cloud Core Segments 1-5
    • Cloud Core Segments 6-10
    • Cloud Core Segments 11-15
    • Cloud Core Segments 16-20
    • Cloud Core Segments 21-26
    • Cloud Core Segments 27-35
    • Cloud Core Segments 36-40
    • Cloud Core Segments 41-45
    • Cloud Core Segments 45-50
  • DNS
  • Mail Gateways
  • Shared Web Servers
  • Shared DB Servers
  • Shared Mail Servers
  • LiveChat
  • M3 Status Page
  • Dedicated Servers
  • Core Routers
  • Core Switches
  • Segment Switches
  • Firewall Stack
  • IDS/IPS
  • DDOS Protection
  • Network Storage
  • Azure East
  • Azure East2
  • Azure West
  • Azure West2
  • Azure Other
  • M3 Monitoring System
  • SMS/TEXT Notification System
  • Other
Previous Incidents

[Resolved] Azure Frontdoor Connectivity

Began: Ended: Duration:
  • Azure Other

We are investigating intermittent alerts on some customers connections via Azure Frontdoor globally. We will provide an update as more information becomes available.

Microsoft is aware of the issue affecting Frontdoor and is actively working on resolution. More information will be provided shortly.

Microsoft has provided the following recent update: We are experiencing Azure Front Door (AFD) issues resulting in a loss of availability of some services. We suspect that an inadvertent configuration change as the trigger event for this issue. We are taking two concurrent actions where we are blocking all changes to the AFD services and at the same time rolling back to our last known good state. We do not have an ETA for when the rollback will be completed but will provide updates as more information becomes available.

Microsoft has confirmed that an inadvertent configuration change was the trigger event for this issue and are continuing with the rollback. No ETA is available at this time.

Current status:

Initiated the deployment of the 'last known good' configuration. This is expected to be fully deployed in about 30 minutes from which point customers will start to see initial signs of recovery. Once this is completed, the next stage is to start to recover nodes while we route traffic through these healthy nodes.

Customer configuration changes will remain blocked during this time as we work towards mitigation. We will communicate to customers when this block is reverted.

We are seeing some gradual recovery as the roll-back is underway.

Microsoft initiated the deployment of the ‘last known good’ configuration, which has now successfully been completed. Customers may have begun to see initial signs of recovery. They are currently recovering nodes and routing traffic through healthy nodes, and as they make progress in this workstream, customers will continue to see improvement.

At this stage, they anticipate full mitigation within the next four hours as they continue to recover nodes. We will provide another update on progress within two hours, or sooner if warranted.

Most sites affected by this FrontDoor issue have recovered but we are still seeing some intermittent performance issues. They are currently recovering nodes and routing traffic through healthy nodes, and as they make progress in this workstream, customers will continue to see improvement over the next hour.

Azure has now resolved the incident. Thanks for your patience.

No further notices from the past 7 days.