Cost of change
The Cost of Upstream API Failures: Lessons from the OpenAI Outage
An unexpected OpenAI outage halted new Workspace Agent runs for over nine hours. What this means for engineering budgets, roadmaps, and API dependency design.
Published November 13, 2026 5 min read By the Breakwatch team 1 source
Late on November 11, 2026, an issue began at OpenAI. According to the OpenAI Status Page, a full outage prevented users from starting new Workspace Agent runs. The incident started at 11:50 PM and lasted until November 12 at 09:32 AM. For nearly ten hours, any application relying on this specific feature was unable to process new agent requests. The vendor eventually resolved the issue, noting that "All affected services are operating normally again."
This event highlights a common reality in modern software development. We build products on top of powerful APIs provided by other companies. When those APIs experience downtime, our products fail. The end user does not care that a vendor caused the issue. They only see that your application is broken. For engineering leaders, these incidents represent a significant, often unmeasured cost to the business.
The Financial Impact of Unnoticed Failures
When a critical dependency fails, the immediate cost is obvious because features stop working. However, the hidden costs are much higher and directly impact the engineering budget. If an upstream outage goes unnoticed by your monitoring tools, your application will throw generic errors. Your on-call engineers will be paged for a system failure. They will spend hours debugging internal databases, network configurations, and recent deployments before realizing the issue originates outside your infrastructure.
This wasted time is expensive. It drains the engineering budget and delays the features promised on the product roadmap. A nine hour outage often means an entire day of lost productivity for the engineers who must investigate the alerts, communicate with customer support, and monitor the vendor status page. This context switching destroys momentum.
Furthermore, if your product has Service Level Agreements with enterprise customers, an upstream outage can trigger financial penalties. You are held responsible for the uptime of the vendors you choose. Customer success teams spend hours answering tickets and apologizing for a problem they cannot fix. Trust is eroded, and the cost of acquiring and retaining customers increases.
Synchronous Dependencies and User Experience
Modern applications are distributed systems. You control your code, but you do not control the network or the third party servers. The OpenAI incident highlights a specific vulnerability regarding partial platform degradation. A single feature experienced a full outage while other components might have remained operational.
If your system assumes that an API is either entirely up or entirely down, partial outages will cause unpredictable behavior. When a user clicked a button to start a Workspace Agent run during that nine hour window, what happened in your application? If the application was designed with a tight synchronous coupling, the user stared at a loading spinner until the request timed out. This is a poor user experience that leads to immediate frustration.
Engineering leaders must advocate for asynchronous designs where possible. If a request can be queued, the user can be told that their task is pending. The system can then retry the request in the background once the vendor resolves the outage. This shifts the failure from a hard error to a delayed processing event, preserving the user experience as much as possible.
Architectural Patterns for API Resilience
You cannot prevent a vendor from having an outage. You can only control how your application responds to that outage. The first line of defense is a robust timeout strategy. If the vendor API hangs, your application must not hang with it. Set strict timeouts on all external network calls to ensure your servers do not exhaust their connection pools waiting for a response that will never arrive.
The second line of defense is the circuit breaker pattern combined with exponential backoff. If an API is consistently returning errors, stop sending requests immediately. This prevents your system from hammering a struggling vendor API and allows your application to fail fast.
When implementing retries, always use an exponential backoff strategy. A simple implementation ensures that your application waits progressively longer between each attempt.
async function callVendorApiWithRetry(endpoint, maxRetries = 3) {
let attempt = 0;
while (attempt < maxRetries) {
try {
const response = await fetch(endpoint);
if (!response.ok) {
throw new Error('API request failed');
}
return await response.json();
} catch (error) {
attempt++;
if (attempt >= maxRetries) throw error;
const backoffTime = Math.pow(2, attempt) * 1000;
await new Promise(resolve => setTimeout(resolve, backoffTime));
}
}
}This generic pattern protects both your infrastructure and the vendor infrastructure. It gives the upstream service time to recover without being overwhelmed by automated retry storms from thousands of clients.
What Engineering Leaders Must Check Today
Engineering leaders must ask their teams specific questions about their API integrations. Do not wait for the next outage to find the weak points in your system. Start by auditing your error handling for external calls. Verify that every call to a vendor API is wrapped in a mechanism that gracefully handles network failures and unexpected HTTP status codes.
Next, review your user experience during a failure. If an API call fails, what does the user see? A generic error page is unacceptable. The application should explain that a specific feature is temporarily unavailable while allowing the user to continue using other parts of the product.
Examine your alerting rules. Your monitoring system should alert the engineering team when an upstream API error rate spikes, not just when your own servers crash. You need to know a vendor is failing before your customers report it.
Finally, ensure you have a clear incident response playbook for third party outages. The playbook must define who communicates with users, who monitors the vendor status updates, and how the engineering team verifies that the system has recovered once the vendor resolves the issue.
Managing the Risk of Shared Infrastructure
Building software today means accepting dependency risk. The goal is not to eliminate dependencies entirely, as that would mean building every service from scratch. The goal is to manage the risk they introduce. This requires a shift in engineering culture. Developers must treat external APIs with the same skepticism they apply to user input. Every response must be validated, and every failure must be anticipated.
When evaluating a new vendor, engineering leaders should review the vendor historical uptime and their incident communication practices. A vendor that provides clear and timely updates during an outage is far more valuable than a vendor that remains silent. Transparency from your vendors allows you to be transparent with your own users.
Upstream outages will happen. The November incident preventing new Workspace Agent runs is just one example of the unpredictable nature of third party dependencies. Your responsibility is to ensure your application degrades gracefully and your team knows exactly how to respond when the inevitable occurs.
Breakwatch maps which files in your repos call each vendor API, watches those vendors' changelogs, docs and API specs, and alerts you when a change affects your code. By understanding exactly where your dependencies live, you can build more resilient software and protect your engineering roadmap from unexpected disruptions.