Rendered at 20:11:10 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
tristanj 1 days ago [-]
Github is struggling because AI-boosted coding increased the number of commits 14x in the past year, and the pace is still accelerating. The site is struggling to keep up.
Github's COO confirms it here: https://x.com/kdaigle/status/2040164759836778878
Platform activity is surging. There were 1 billion commits in 2025. As of three months ago, it was 275 million per week, on pace for 14 billion this year if growth remains linear (spoiler: it won't.)
GitHub’s reliability issues started earlier than the boom in agentic coding they talk about. I have comments on here from 2025 talking about massive outages with actions and issues.
fooey 1 hours ago [-]
When you're the uncontested market leader, you don't budget operations to be perfectly reliable, you budget to not be too unreliable. All those 9's are expensive after all.
The AI boom caught them napping though, and they're slipping well into the too unreliable category
chocrates 1 hours ago [-]
GitHub uptime is cyclical. It's bad because they won't focus on the core, and then the right users complain and GitHub focuses on core again before it repeats. This is just the latest round exacerbated by vibe coding
jjice 1 hours ago [-]
I want to believe that it's justification, but looking at the historical uptime [0] (early data is actually overly positive on this graph since it had to scrape historical data instead of realtime), I can't be convinced of this anymore. They do have more load - that's for sure. I just don't believe it's an excuse for their god awful uptime.
I would take what C-level execs say with a grain of salt. Remember these folks hire people with plenty of system design experience. They likely also have the money for hardware resources.
If load was an issue these things would have been solved already. Or they are purposely causing millions of dollars of productivity loss daily by not solving this.
Something more serious is at play. And I want to hear from the actual underlings.
Illniyar 2 hours ago [-]
More load does not just mean needing to increase capacity, which would be simple for Microsoft to do.
Additional load reveals limits and breaklines that were never known before - your temporary integer IDs are suddenly too small, you reach the maximum file descriptors in a Linux distro, your hash function suddenly has repeating conflicts, your DB index is suddenly oversaturated.
Having massive exponential growth of your traffic would probably require constant refactoring to meet the ever changing system limits.
browningstreet 3 hours ago [-]
> They likely also have the money for hardware resources.
Amazing you can say this with a straight face in these DC building times.
Have you tried getting Azure east instances anytime lately?
srmatto 3 hours ago [-]
Infra is a cost center for a business unless your a PaaS and even then maybe still. Cost centers are usually ferreted out for savings, not investment.
roughly 2 hours ago [-]
Infra is a cost center right up until its absence begins to undercut your value proposition to customers, and then it becomes an investment. GitHub seems well on the wrong side of that equation right now.
geodel 42 minutes ago [-]
In general true. But most of paying customers, including enterprise have made a bet that they cut down expense and resources on operational experience, system management knowledge to minimum because cloud .
So even if these cloud/SAAS solutions perform poorly it is not like customers have wherewithal to create alternate solutions.
pan69 1 days ago [-]
> I would take what C-level execs say with a grain of salt.
100% agree.
> Remember these folks hire people with plenty of system design experience.
Not my overall experience 30 years in the industry.
> They likely also have the money for hardware resources.
You'll always be asked to do more with less, at least, that's my own experience.
joshspankit 4 hours ago [-]
Also my experience
fooey 1 hours ago [-]
I think owning GitHub is a prestige play, not necessarily thought of as a profit center.
Now however, it's competing for resources directly with the corporations core strategic plans.
GitHub having issues is embarrassing, but the shareholders don't really care.
nobodysaidso 3 hours ago [-]
>They likely also have the money for hardware resources.
Hey man, in case you hadn't heard, the hardware industry is having a lot of trouble keeping up. "Expand our capacity" isn't as easy as it was 2 years ago.
layer8 8 minutes ago [-]
Turns out that the cloud doesn’t scale that easily by itself after all?
vb-8448 2 hours ago [-]
I'm curios what is the bottleneck then? Because github(ms) has enough resources to hire competent people and provision needed hw to solve scaling the issues.
c0rruptbytes 2 hours ago [-]
need competent people to hire competent people first - it seems like their only directive is to raise $MSFT not make good tech
(talking about leadership, not my lovely msft engineers reading this)
lallysingh 2 hours ago [-]
Probably contention on databases.
moralestapia 2 hours ago [-]
Yeah, aren't they super geniuses with infra that scales with a button, etc.?
aleph_minus_one 2 hours ago [-]
> infra that scales with a button
That is how Microsoft markets Azure. :-)
evanelias 1 days ago [-]
My sense is that a lot of this activity isn't even legitimate use. As one example: my company has a GitHub app, and last night some bot added my app to 1700 repos. Then the bot immediately started rapid-firing commits which each affected hundreds of files, triggering a deluge of GitHub webhooks to my servers.
The repo names all matched other GitHub apps, so my sense is this bot had added 1700 GitHub apps to 1700 repos, with that number rapidly increasing at the upper bound of whatever GitHub's rate limit is.
My systems caught it quickly and auto-booted the bot, but the whole situation is ridiculous.
digitalsushi 1 days ago [-]
there's this attack on youtube where people who dont want their real names known will autoblock comments with their real names; bots will spam comments with every word and note the ones that were not visible.
kinda reminds me of it. not my most valuable contribution to forensics, admittedly.
meerita 1 hours ago [-]
I can't believe they've been asleep all these months, not foreseeing that AI would generate more commits and Git usage and, consequently, not preparing themselves to scale.
bhouston 1 days ago [-]
> Github is struggling because AI-boosted coding increased the number of commits 14x in the past year, and the pace is still accelerating.
Sure, but how is OpenAI, and Claude handling the increased usage? They are handling it a lot better than Github. Just because you have fast growth does not mean you can not handle it.
Instead you have to realize that you are entering into a high growth phase and design around that.
It has been clear that we are in a high growth phase for around a year now and Github still hasn't adapted to that properly.
At some point you have to admit that Github ready for the moment.
traviswingo 1 days ago [-]
GitHub is on Azure. Azure is notorious for being terrible. There was a post here recently from an ex-Azure engineer explaining all of this in a multi-part series.
Isn't a lot of OpenAI infra also on Azure? They seem to be faring better than GitHub.
bhouston 1 days ago [-]
I bet OpenAI has failovers to other data centers. I think they have a variety of data center provides all over the place.
prewett 1 days ago [-]
There's already a noun for this: "resources"
moralestapia 2 hours ago [-]
Q. how can someone publish an article like that, with their name, etc...
Don't they sign NDAs? Do they not care? Are they not enforceable?
Very curious to know.
ornornor 2 hours ago [-]
Maybe they didn’t take the severance part that makes the NDA valid? IIRC for an NDA to be enforceable there has to be a monetary consideration to it. Or maybe it’s for no competes?
stronglikedan 2 hours ago [-]
Open AI and Claude were not nearly as mature as Github when the boom started. That's a lot of legacy infra to scale up.
maccard 2 hours ago [-]
For a very long time this year Claude was down multiple times a week. Can’t speak for OpenAI
SamBam 1 days ago [-]
OpenAI and Anthropic are spending hundreds of billions of dollars that they may never recoup forcing the building massive new data centers in the face of widespread protest.
You think GitHub ought to match that?
bhouston 1 days ago [-]
> OpenAI and Anthropic are spending hundreds of billions of dollars that they may never recoup forcing the building massive new data centers in the face of widespread protest.
So has Github's owner Microsoft. It's build out is on par with OpenAI and Anthropic.
> You think GitHub ought to match that?
Microsoft is.
The issue is that the software stack is not stable. That is clearly the issue here.
It isn't a hardware build out issue. It is related to the architecture of Github.
tekno45 1 days ago [-]
OpenAI and Claude don't accept commits and host PRs? How are they the same?
dumberquestions 1 days ago [-]
They're getting increasing demand for tokens at an accelerating pace.
pixelesque 1 days ago [-]
Is that the same as storing state in a DB and on a filesystem though?
Like, I get there are other features they have, and things like history, but I don't think it's quite the same.
Isn't most (not all) of the LLM's resources stateless compute to a large degree?
(Not defending MS at all though, I currently can't create a PR on the required branch for my work (won't load the branch list) and their availability is really bad recently).
bhouston 1 days ago [-]
I am pretty sure OpenAI and Claude actually have a higher hardware cost per request that Microsoft. Github is actually most just a standard REST API app, although with some caching and git specific operations in the background. It is not as hardware intensive as GPU-based inference per quest with huge contexts and low latency.
Also Github's feature offering has been incredibly stable for the last 6 years. There isn't an excuse for it to be unstable like this.
pixelesque 1 days ago [-]
Yes, but come on...
That's due to the need for ultra-fast memory and inference compute, it's not the same thing.
fragmede 1 days ago [-]
Fundamentally, there's incoming requests, an application server, and a database. The devil is in the nature of those requests, and how do you process those requests and stick them in the database. And then serve them back out again. Different application, different database, but fundamentally, it's the same problem. Google managed to do this very well. Microsoft has not.
logicchains 1 days ago [-]
They're undergoing the same kind of load. Almost every AI commit that's putting pressure on GitHub's servers was written on OpenAI and Anthropic's servers.
simonw 1 days ago [-]
Scaling ChatGPT and scaling GitHub are very different problems.
For LLMs, prompt handling is effectively stateless. They do try to route follow-up prompts to the same cluster to benefit from prompt caching, but those can be effectively sharded. They also need to write results to storage but that's on a per-customer basis, so also easy to shard.
GitHub is a shared space, where commits and issues and PRs need to become instantly available to all readers across all geographies. They have a very different shape of scaling challenge to the LLM vendors.
bhouston 1 days ago [-]
> GitHub is a shared space, where commits and issues and PRs need to become instantly available to all readers across all geographies. They have a very different shape of scaling challenge to the LLM vendors.
Github data is accessible by all (if open source), but they should be partition-able by individual repository (and their related forks.). Thus while there is more shared state across users, it isn't fully shared state.
And they have been working on this semi-shared state design for over 10 years now.
remus 1 days ago [-]
> Sure, but how is OpenAI, and Claude handling the increased usage?
By spending tens of billions in investor cash. It is not clear what a sustainable business model looks like for these companies at the moment.
> They are handling it a lot better than Github.
Are they? Anthropic has fairly frequent issues.
There's high growth and then there's demand increasing 10x in a year. For a mature business like github that's a massive, and I can't think of many big businesses who would be able to scale smoothly to accommodate that kind of demand.
theappsecguy 1 days ago [-]
Well they heavily limit free tier usage and for paid users they are charging a tiered pricing model that ramps up very aggressively. Github is a lot more generous in this sense and it has a lot more surface area in terms of the functionality it serves
1attice 4 hours ago [-]
This would require foresight, budget, and attention. MS currently has none of these.
Free Users use GitHub => when a company needs source control => all their coders use GitHub => company buys expensive enterprise contract.
If there’s less of an advantage to using free GitHub then that funnel works less effectively.
hei-lima 1 days ago [-]
GH isn't good enough to charge that much for its core services. 90% of its user base would just find better or cheaper alternatives, and the only ones remaining would be legacy enterprise codebases that are just impossible or too expensive to migrate.
kypro 22 hours ago [-]
80/20 rule would suggest that you don't need to apply $1 to all users, just those who create an excessive number of AI slop repos or make an excessive number of commits.
Could do similar for GH actions usage.
imjonse 3 hours ago [-]
I would have thought it's public repos that got the most AI slop recently as a form of gathering karma and boost CVs. It would be nice to know if private repo commits increased instead.
fourseventy 1 days ago [-]
Github is owned by Microsoft, its not like its some boostrapped startup who has a closet of servers that are melting.
platevoltage 1 hours ago [-]
And on top of that, Microsoft is one of the companies that created this problem in the first place.
pluc 1 days ago [-]
Nobody at Microsoft could've predicted AI was going to be so popular!
blipvert 2 hours ago [-]
Like the way that they shipped ‘95 without a browser?
adzuci 1 days ago [-]
I think that data is a bit dated and misleading, but I would love to see if AI slop is a contributing factor here once the dust settles and how Github will address it. Looking at the return to mean in https://areyougoingexponential.rhys.dev/kdaigle makes me think the COO spike could be a Karpathy wiki or an agent too.
bluerooibos 1 days ago [-]
> Github is struggling because AI-boosted coding increased the number of commits 14x in the past year,
OK so then use the additional revenue to add more resources and employees? What the f*k are they doing over there?
flexagoon 1 days ago [-]
> OK so then use the additional revenue to add more resources and employees?
What revenue? I'd assume the people pushing thousands of AI-generated commits are not the people paying for GitHub Enterprise.
bluerooibos 1 days ago [-]
> What revenue
I'd assume at least a portion of them are paying for GitHub Pro, and by now at least a small % would have converted to Enterprise.
If not, then they need to fix their operational model - I don't understand people acting like this is some insurmountable challenge given Microsoft reported over $331 billion in total revenue and an operating income exceeding $155 billion in 2026.
roryirvine 1 days ago [-]
Changing the operating model can be surprisingly tough for an established business. Imagine the outcry if they started charging per git push, for example.
Clearly, something needs to change. But if they're not careful they might provoke a reaction from their existing userbase that's even more negative that the response to these outages.
oceanician 3 hours ago [-]
Sure, but max 100 pushes per day per account? Or should it be higher?
bluerooibos 1 days ago [-]
> can be tough
Of course it can - everything is tough at that scale, but their parent company has billions in profit. They've had a year to come up with a plan/fix and have failed miserably.
thevinter 1 days ago [-]
What revenue do AI-boosted commits provide? Am I missing something?
bluerooibos 1 days ago [-]
You're missing the fact that Microsoft reported over $331 billion in total revenue and an operating income exceeding $155 billion - this is a fixable problem.
If they're really that cash-strapped at GitHub, then are they planning to just continue allowing this to happen and drown, or fix the issues? This has been ongoing for over a year.
rurban 1 days ago [-]
So why did all the Azure and Github engineers leave then? There's only management and outsourced cheap labor there, and management has no idea. Cheap labor even less
1 days ago [-]
pavel_lishin 1 days ago [-]
Is there additional revenue from a bunch of people pushing their vibe-coded stuff up to Github?
0xbadcafebee 1 days ago [-]
If that's the superficial cause, then the root cause is a lack of limits. Any production system in the world will go down from too much load if you don't limit the load. You have to do performance testing to determine what the system's peak utilization is, and put limits in place to prevent exceeding that limit.
There are multiple kinds of limits you need. First you need the raw i/o limits on things like networking, load balancer requests per second, etc so you don't slow to a crawl on high load; you can throttle max i/o so demanding users don't steal performance for others, or even throttle network traffic to the client IP and internal services. Then you need additional limits at app layer, like number of requests you can process that make DB calls (as most DBs have a connection and request limit), or limit API requests to X/minute. Then you need more business logic limits, like max number of concurrent logged-in users, max number of repos a user can have, max number of requests for free users, priority for paid requests, etc.
When you hit a limit, you have a couple choices. You can shed load (dropping calls immediately, which results in 503 errors for users, or even network disconnects), but it makes a very poor user experience, and can cause larger issues/conflicts in a distributed system. You can instead keep a max number of active sessions, and adjust that number over time, to ensure that (on average) your users have a stable experience; this requires implementing a "waiting room" for new sessions when old ones expire or log out; still not great for users, but it prevents everyone from getting random errors, which means the majority of users have a good experience.
There are way more kinds of limits to implement. Each business has to analyze its entire system to identify performance bottlenecks and implement limits to ensure the majority of users continue to have good service. Without those limits, you get a shit experience for everyone, your brand is tarnished, and customers run for the hills.
Finally, the performance testing helps you plan how much infrastructure you need to serve a given number of users. If you plan properly, you know X infra allows for Y number of users. Once you exceed Y users, you need to start throttling or stopping new user signup (and requests per minute) until you can increase capacity. Continuing to allow signups and additional usage, without the capacity to support it, results in what we see today.
But competent engineers know all this. If they had competent engineers, and it gets this bad anyway, the problem is management. Microsoft likely already made back what they acquired GitHub for. They probably figured they could keep riding the gravy train until the train went off the rails, and either eventually sell GitHub or write it off. Such is business.
thex10 1 hours ago [-]
I really enjoyed this read, as I am starting to grapple with some of this now. Do you have any resources for learning more about this stuff and these tradeoffs?
darkr 2 hours ago [-]
> until the train went off the rails
Ironically the core application is written in rails, which may be part of the issue
TZubiri 1 days ago [-]
Are these disproportionately in the free tier? Or is the increased storage load similar across the paid tiers?
Might be worth considering a /GB/MO fee. Let's face it, VCS is a (glorified) storage service.
cute_boi 1 days ago [-]
I think Github should start charging who goes beyond fair usages.
I just checked no 1 vibe coder https://github.com/steipete and he has like 279,440. Compared this to Taylor who works on huge project like Laravel and is only around 8k commits.
ProofHouse 1 days ago [-]
The scam that anthropic pulled, is they remove free tier (bad idea for Github honestly), but blame it on AI vibe code abusers. Instead of for example finding the 1-5% of absolutely aggregous users like Steipete and restricting them.
logicallee 1 days ago [-]
it's a bit of a stretch to call removing a free tier a scam. as you can see a lot of people in this thread are saying Github should start charging.
cute_boi 1 days ago [-]
It is just an excuse. If they can't support paid member or don't have infra they should start charging these slop projects who are committing relentlessly.
andyjohnson0 25 minutes ago [-]
Github used to be built on mysql / redis / ruby on rails / C / shell, running on dedicated hardware. Microsoft left it like that after they acquired the company.
Eventually, though, they decided to migrate the whole thing to Azure. And they were far enough through that to be basically committed... when AI coding started hitting them with much higher workloads.
I personally think the reliability problems are more to do with the reliability of the Azure migration. But both factors are likely relevant.
alightsoul 2 hours ago [-]
Lovable only uses GitHub for storing projects. And requires people to provide their own GitHub account. Lots of vibe coders with no technical background are now having lovable push commits to GitHub
To be fair, before the aquisition, GitHub was super stable, but also it felt like they rarely changed or added anything.
After Microsoft acquired them, they started rolling out new features at an incredible pace. And if you look at the breakdown of the downtime, most of the downtime is in features that didn't exist before the acquisition.
Though not all of the downtime is in new features. The core is also less stable than ever.
nrr 58 minutes ago [-]
My (actuarial) read on this is that pre-Microsoft GitHub acknowledged that the complexity inherent in their technology stack imparted a lot of change risk, and that probably informed their decisions around shipping new features.
My (not-so-actuarial 1990's-experience-colored) feeling is that this is Microsoft doing what Microsoft does best, namely including a lot of seemingly compelling features in a product to get people hooked and to keep competitors in a position of trying to keep up.
ornornor 2 hours ago [-]
I’ve noticed the same thing with computers, whenever I install MS windows on them uptime tanks ;)
flexagoon 1 days ago [-]
Note that most of the downtime in that graph comes from Github Actions, which simply didn't exist before the Microsoft acquisition.
The downtime also starts happening in 2020, which is a while after the acquisition. 2020 was the year many people started learning to code while being stuck at home.
joshdavham 1 days ago [-]
Microsoft acquiring GitHub is not a valid root cause analysis explaining Github’s degraded SLA.
ryandvm 7 hours ago [-]
Mmmm - I don't do root cause analysis for free. Best I can do is correlation.
cwnyth 1 days ago [-]
According to that graph, performance drops widely after Microsoft's acquisition before AI enters the scene. If not Microsoft, how else do you explain it, or is the graph somehow misleading?
lordgilman 1 days ago [-]
You're the one making the argument! You need to show causality instead of correlation.
cwnyth 1 days ago [-]
I've made no argument at all. I'm asking a question to a person who made the claim that "Microsoft acquiring GitHub is not a valid root cause analysis."
lezojeda 2 hours ago [-]
[dead]
ciupicri 1 days ago [-]
On the other hand I think usage increased and GitHub operates now at a different scale.
traviswingo 1 days ago [-]
Microsoft acquires GitHub and subsequently moves all infrastructure to Azure
preciousoo 1 days ago [-]
No one hates msft more than me, but there’s no way you think this one data point is useful to explain anything at all
Tbarlow 1 hours ago [-]
14x commit growth in one year is brutal for any infrastructure. Scaling isn't just adding hardware.
solfox 1 days ago [-]
Many microsoft services are down/failing today, including sites hosted on Azure. I'm guessing it's a larger MS outage.
joshdavham 1 days ago [-]
So perhaps we should be asking Azure engineers, not Guthub engineers?
The page shows all green, which coincidentally is also how it looks when Azure is down. That page is for SLA/billing purposes, it won't show live incidents as they occur.
palerdot 1 days ago [-]
> That page is for SLA/billing purposes, it won't show live incidents as they occur.
lol ... That is absolutely hilarious ... I'm not sure if it is /s, but if this is true, enterprise stupidity has reached a new level.
jdelsman 1 days ago [-]
How sure are we that their status page is accurate? AWS's status page often lags actual issues.
jaggederest 3 hours ago [-]
AWS's status page would be green after a microsingularity swallowed the solar system. Azure's status page is very similar.
qrush 1 days ago [-]
I got an "it is unacceptable" from their CPO on 8/7, and that they are "working around the clock on it".
I really think something deeper is going wrong there, and they're not being honest with their paying customers (and enterprises) about it.
VCFundedGenYer 1 days ago [-]
I think it's two things:
- GitHub attempting (and seemingly failing) to move to Azure infrastucture for its website backend
- AI generated code wrecking the site due to the volume of activities.
zeafoamrun 1 days ago [-]
So you're asking github employees to violate their NDA?
sharts 1 days ago [-]
Frankly yes. Do it in a way that doesn’t violate outright or cover your tracks. Customers deserve to know the truth of the type of company they are dealing with.
zeafoamrun 24 hours ago [-]
So someone should potentially get fired and face legal action to satisfy your curiosity which you'll get sated anyway when you read the postmortem in a few days?
ornornor 2 hours ago [-]
It’s not like OP is putting a gun to anyone’s head!! If they want to share we’d appreciate but there is no obligation.
Are you also shocked that you can stand in the middle of the highway and get hit by a car and die? You can do it if someone asks you but you’re not forced to.
watt 2 hours ago [-]
It's not the postmortem du jour that we want, the whole pattern is a mystery. Do they push releases every day and bring down the site every day by botched rollouts? What really is it why they can't GH keep the core service running (e.g. pushing commits and pulling changes)?
stickfigure 2 hours ago [-]
We can only ask.
Don't act shocked.
Also, gritty insider information is never the same as the official word.
2 hours ago [-]
iJohnDoe 1 hours ago [-]
It's bad karma to sling mud about outages or problems. Cloudflare used to sling mud back in the day and then they went through some really bad outages afterwards.
the_sleaze_ 1 hours ago [-]
Github moved to azure -> infrastructure problems
Taikhoom10 2 hours ago [-]
and yet people do not leave, hence the power of switching costs. (Paying Customers).
If you are doing 1000 commits a day, whats the point of git?
Does the AI ever look back at the shit trail it left behind?
Diffs are no longer diffs, they look like largescale delete and rewrite
odo1242 1 days ago [-]
AI definitely does look at git commit history in my experience
boldlybold 1 hours ago [-]
It certainly looks at the commit history more than I ever have. I find it quite cool when it points out interesting hints from the history.
pluc 1 days ago [-]
Microsoft. Azure. AI.
Pick one!
lexoj 1 days ago [-]
Also the fact that its written in Ruby on Rails probably doesn’t make it easier, performance wise.
t-writescode 1 days ago [-]
Honestly. They’re already performing miracles with RoR.
How many non Ruby apps and all the databases and systems behind could climb their traffic 14x when they probably only perf-tested for a sustained 2-3x and, and paid for a profitable amount of rack space rather than a “lol, VC money, we’ll 10x later” amount of infra.
the_sleaze_ 1 hours ago [-]
Systems engineers generally don't run performance tests at 2x or any specific scale.
You load test and see how much it takes until the system falls over - basically you DDoS yourself and see what the numbers were when things broke, and what broke first second so on. Maybe in a sandbox maybe against prod.
It's very telling that issues began almost to the day of the Microsoft acquisition.
Platform activity is surging. There were 1 billion commits in 2025. As of three months ago, it was 275 million per week, on pace for 14 billion this year if growth remains linear (spoiler: it won't.)
20% of all GitHub accounts were created in the past 6 months https://x.com/kdaigle/status/2082604368399159542
The AI boom caught them napping though, and they're slipping well into the too unreliable category
[0] https://damrnelson.github.io/github-historical-uptime/
If load was an issue these things would have been solved already. Or they are purposely causing millions of dollars of productivity loss daily by not solving this.
Something more serious is at play. And I want to hear from the actual underlings.
Additional load reveals limits and breaklines that were never known before - your temporary integer IDs are suddenly too small, you reach the maximum file descriptors in a Linux distro, your hash function suddenly has repeating conflicts, your DB index is suddenly oversaturated.
Having massive exponential growth of your traffic would probably require constant refactoring to meet the ever changing system limits.
Amazing you can say this with a straight face in these DC building times.
Have you tried getting Azure east instances anytime lately?
So even if these cloud/SAAS solutions perform poorly it is not like customers have wherewithal to create alternate solutions.
100% agree.
> Remember these folks hire people with plenty of system design experience.
Not my overall experience 30 years in the industry.
> They likely also have the money for hardware resources.
You'll always be asked to do more with less, at least, that's my own experience.
Now however, it's competing for resources directly with the corporations core strategic plans.
GitHub having issues is embarrassing, but the shareholders don't really care.
Hey man, in case you hadn't heard, the hardware industry is having a lot of trouble keeping up. "Expand our capacity" isn't as easy as it was 2 years ago.
(talking about leadership, not my lovely msft engineers reading this)
That is how Microsoft markets Azure. :-)
The repo names all matched other GitHub apps, so my sense is this bot had added 1700 GitHub apps to 1700 repos, with that number rapidly increasing at the upper bound of whatever GitHub's rate limit is.
My systems caught it quickly and auto-booted the bot, but the whole situation is ridiculous.
kinda reminds me of it. not my most valuable contribution to forensics, admittedly.
Sure, but how is OpenAI, and Claude handling the increased usage? They are handling it a lot better than Github. Just because you have fast growth does not mean you can not handle it.
Instead you have to realize that you are entering into a high growth phase and design around that.
It has been clear that we are in a high growth phase for around a year now and Github still hasn't adapted to that properly.
At some point you have to admit that Github ready for the moment.
Edit: Found the thread: https://news.ycombinator.com/item?id=47616242
Don't they sign NDAs? Do they not care? Are they not enforceable?
Very curious to know.
You think GitHub ought to match that?
So has Github's owner Microsoft. It's build out is on par with OpenAI and Anthropic.
> You think GitHub ought to match that?
Microsoft is.
The issue is that the software stack is not stable. That is clearly the issue here.
It isn't a hardware build out issue. It is related to the architecture of Github.
Like, I get there are other features they have, and things like history, but I don't think it's quite the same.
Isn't most (not all) of the LLM's resources stateless compute to a large degree?
(Not defending MS at all though, I currently can't create a PR on the required branch for my work (won't load the branch list) and their availability is really bad recently).
Also Github's feature offering has been incredibly stable for the last 6 years. There isn't an excuse for it to be unstable like this.
That's due to the need for ultra-fast memory and inference compute, it's not the same thing.
For LLMs, prompt handling is effectively stateless. They do try to route follow-up prompts to the same cluster to benefit from prompt caching, but those can be effectively sharded. They also need to write results to storage but that's on a per-customer basis, so also easy to shard.
GitHub is a shared space, where commits and issues and PRs need to become instantly available to all readers across all geographies. They have a very different shape of scaling challenge to the LLM vendors.
Github data is accessible by all (if open source), but they should be partition-able by individual repository (and their related forks.). Thus while there is more shared state across users, it isn't fully shared state.
And they have been working on this semi-shared state design for over 10 years now.
By spending tens of billions in investor cash. It is not clear what a sustainable business model looks like for these companies at the moment.
> They are handling it a lot better than Github.
Are they? Anthropic has fairly frequent issues.
There's high growth and then there's demand increasing 10x in a year. For a mature business like github that's a massive, and I can't think of many big businesses who would be able to scale smoothly to accommodate that kind of demand.
No, they are not.
https://status.claude.com/
https://www.githubstatus.com/
Charge $1/month per repo. If your project isn't worth that, run your own git or use another free forge.
Seems like github is too popular for itself.
Free Users use GitHub => when a company needs source control => all their coders use GitHub => company buys expensive enterprise contract.
If there’s less of an advantage to using free GitHub then that funnel works less effectively.
Could do similar for GH actions usage.
OK so then use the additional revenue to add more resources and employees? What the f*k are they doing over there?
What revenue? I'd assume the people pushing thousands of AI-generated commits are not the people paying for GitHub Enterprise.
I'd assume at least a portion of them are paying for GitHub Pro, and by now at least a small % would have converted to Enterprise.
If not, then they need to fix their operational model - I don't understand people acting like this is some insurmountable challenge given Microsoft reported over $331 billion in total revenue and an operating income exceeding $155 billion in 2026.
Clearly, something needs to change. But if they're not careful they might provoke a reaction from their existing userbase that's even more negative that the response to these outages.
Of course it can - everything is tough at that scale, but their parent company has billions in profit. They've had a year to come up with a plan/fix and have failed miserably.
If they're really that cash-strapped at GitHub, then are they planning to just continue allowing this to happen and drown, or fix the issues? This has been ongoing for over a year.
There are multiple kinds of limits you need. First you need the raw i/o limits on things like networking, load balancer requests per second, etc so you don't slow to a crawl on high load; you can throttle max i/o so demanding users don't steal performance for others, or even throttle network traffic to the client IP and internal services. Then you need additional limits at app layer, like number of requests you can process that make DB calls (as most DBs have a connection and request limit), or limit API requests to X/minute. Then you need more business logic limits, like max number of concurrent logged-in users, max number of repos a user can have, max number of requests for free users, priority for paid requests, etc.
When you hit a limit, you have a couple choices. You can shed load (dropping calls immediately, which results in 503 errors for users, or even network disconnects), but it makes a very poor user experience, and can cause larger issues/conflicts in a distributed system. You can instead keep a max number of active sessions, and adjust that number over time, to ensure that (on average) your users have a stable experience; this requires implementing a "waiting room" for new sessions when old ones expire or log out; still not great for users, but it prevents everyone from getting random errors, which means the majority of users have a good experience.
There are way more kinds of limits to implement. Each business has to analyze its entire system to identify performance bottlenecks and implement limits to ensure the majority of users continue to have good service. Without those limits, you get a shit experience for everyone, your brand is tarnished, and customers run for the hills.
Finally, the performance testing helps you plan how much infrastructure you need to serve a given number of users. If you plan properly, you know X infra allows for Y number of users. Once you exceed Y users, you need to start throttling or stopping new user signup (and requests per minute) until you can increase capacity. Continuing to allow signups and additional usage, without the capacity to support it, results in what we see today.
But competent engineers know all this. If they had competent engineers, and it gets this bad anyway, the problem is management. Microsoft likely already made back what they acquired GitHub for. They probably figured they could keep riding the gravy train until the train went off the rails, and either eventually sell GitHub or write it off. Such is business.
Ironically the core application is written in rails, which may be part of the issue
Might be worth considering a /GB/MO fee. Let's face it, VCS is a (glorified) storage service.
I just checked no 1 vibe coder https://github.com/steipete and he has like 279,440. Compared this to Taylor who works on huge project like Laravel and is only around 8k commits.
Eventually, though, they decided to migrate the whole thing to Azure. And they were far enough through that to be basically committed... when AI coding started hitting them with much higher workloads.
I personally think the reliability problems are more to do with the reliability of the Azure migration. But both factors are likely relevant.
After Microsoft acquired them, they started rolling out new features at an incredible pace. And if you look at the breakdown of the downtime, most of the downtime is in features that didn't exist before the acquisition.
Though not all of the downtime is in new features. The core is also less stable than ever.
My (not-so-actuarial 1990's-experience-colored) feeling is that this is Microsoft doing what Microsoft does best, namely including a lot of seemingly compelling features in a product to get people hooked and to keep competitors in a position of trying to keep up.
The downtime also starts happening in 2020, which is a while after the acquisition. 2020 was the year many people started learning to code while being stuck at home.
lol ... That is absolutely hilarious ... I'm not sure if it is /s, but if this is true, enterprise stupidity has reached a new level.
https://x.com/mariorod1/status/2085800861469495465
I really think something deeper is going wrong there, and they're not being honest with their paying customers (and enterprises) about it.
- GitHub attempting (and seemingly failing) to move to Azure infrastucture for its website backend
- AI generated code wrecking the site due to the volume of activities.
Are you also shocked that you can stand in the middle of the highway and get hit by a car and die? You can do it if someone asks you but you’re not forced to.
Don't act shocked.
Also, gritty insider information is never the same as the official word.
https://s-1.vercel.app/posts/why-dropbox-is-a-obvious-pe-tar...
HN Search: azure capacity - https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
Azure Capacity Crunch Extends into 2026 Amid Data Center Constraints - https://windowsforum.com/threads/azure-capacity-crunch-exten... - October 9th, 2025
https://news.ycombinator.com/item?id=47427779
Does the AI ever look back at the shit trail it left behind?
Diffs are no longer diffs, they look like largescale delete and rewrite
Pick one!
How many non Ruby apps and all the databases and systems behind could climb their traffic 14x when they probably only perf-tested for a sustained 2-3x and, and paid for a profitable amount of rack space rather than a “lol, VC money, we’ll 10x later” amount of infra.
You load test and see how much it takes until the system falls over - basically you DDoS yourself and see what the numbers were when things broke, and what broke first second so on. Maybe in a sandbox maybe against prod.
It's very telling that issues began almost to the day of the Microsoft acquisition.