Yesterday the Mastra platform experienced a severe incident, playing out in two parts.
Starting early morning PT, our builds system was down. Then much more severely, around 2:50pm PT yesterday, our infrastructure provider Railway announced a security breach. This means that Mastra platform users’ environment variables may have been accessed by third parties, though we do not have evidence of breach of our users at this time.
The most challenging part about this incident is the quandary our users were caught in. When Railway announced the security incident, their builds system – and thus our builds system – were down.
What this meant is that for several hours, users could not rotate their keys without taking down their production application. We recommended that users monitor for signs of unauthorized use, but did not recommend key rotation until builds were back up, instead recommending users monitor for signs of unauthorised access.
Over the next few weeks, we are accelerating pre-planned control plane work to allow hosting on other providers, as well as creating a failover system to avoid prolonged outages in the future. We are considering moving off of Railway entirely, or limiting Railway to a subset of free tier users.
Over the next few days we expect we will get additional details from Railway which we will communicate. We are deeply sorry for this disruption. Please reach out with any questions and we will respond.
Note this is a platform incident; it does not affect Mastra open-source.
The full timeline is below.
Timeline:
-
02:11 PT - We first reported an Incident reported internally
-
02:27 PT -After initial investigation, we opened an incident on our status page
Build failures due to an upstream provider outage The upstream provider has identified the issue and is working on a fix. -
8:15 PT - We provided an update on our status page
The upstream provider is still working on a fix for this outage. -
13:32 PT - We provided an update on our status page
The upstream provider continues to work on a fix for this outage, and reports expecting builds to come back online within the hour. In the meantime, we have begun work on a fallback deployment system to avoid future deployment disruptions. -
14:59 PT - Railway status page acknowledges security breach. They communicate this to us over Slack DMs.
-
16:38 PT - After investigating with the Railway team and related providers, we posted an update to our status page to warn customers and advise them to monitor their api keys and related systems for suspicious activity.
Our infrastructure provider, Railway, is being affected by a security incident: https://blog.railway.com/p/security-disclosure-wednesday-october-21st. We are being advised by Railway that they will have builds up in the next hour. Currently environment variables are NOT propagating to production due to the outage. Therefore, we are not recommending rolling any keys that would cause a production outage for you unless you see suspicious activity. -
18:14 PT We shared additional context and instructions on our status page:
Mastra platform users should check the logs of services with keys stored on Mastra platform for signs of unauthorized usage: - Check access logs. - Check if any new users or tokens were created that you don't recognize. - Check the balance and activity from API tokens from AI inference providers that you have configured. -
20:59 PT - We confirmed that builds in Railway once again running, and we shared an initial report on our status page.
Railway has given us the green light to start rebuilding. We are testing internal projects and then will be rotating Mastra platform specific keys from each project. -
21:18 PT We shared an updated report with platform users over shared Slack channels and email informing them of our latest understanding of the incident and with updated instructions on how and when to rotate keys to avoid any additional security or downtime risk.
-
22:44 PT - We moved the incident to monitoring and updated our status page. The team continues to work on supporting customers rotating their keys.
We are operational. We are keeping the status in Monitoring for now as we inspect our systems.
