Yeah, clickbait title, I know. It's the only thing anyone pays attention to, so I'm running with it. Ruby Central has published a new blog entry about the whole Rubygems fiasco. Unfortunately, again, it's full of corporate speak and weak excuses that don't hold up under scrutiny.
Before I go further, let me explain where I'm coming from and why I'm qualified to speak confidently about this. For the past 9 years or so, my work has focused less on writing code and more on architecture, infrastructure, and SRE. Prior to my current role, my most recent position was as the technical lead for the team that handles the deployments at Shopify. That team also maintains Shipit, the software which rubygems.org uses to deploy the site. Prior to joining that team, I was an SRE technical lead at Shopify. Total coincidence that that company's name comes up a lot around this whole situation. I only mention it to establish ethos — not only do I have insights the kinds of systems a lot of folks get to blissfully ignore when they deploy via a simple git push or merging a pull request, but I also maintained the specific system that Ruby Central uses to deploy rubygems.org.
I was originally going to write up a critique of the full post but in the spirit of a teacher wheeling in the TV cart to watch Bill Nye (that is, I have other things I actually need to be doing, such as eating dinner, so I need to do the minimum I can get away with here), I'm just going to write about the most glaring part of it from my perspective.
Since the beginning, I've been saying with my entire barrel chest that Ruby Central could've forked the repos. If they wanted to own the code, they could own a fork.
So my biggest frustration with this whole situation is their insistence that they must own the GitHub repos to maintain supply-chain security. It's at best a misunderstanding of how deploying code works and at worst it's a lie to mask accountability for taking over the Rubygems GitHub repos.
Back to the point, Ruby Central's insistence that they need to own the code continues in this post with the following:
While the RubyGems.org service is deployed from rubygems/rubygems.org repository, it also needs at least 2 more private repositories in the same organization for infrastructure-as-code, and CDN deployments. Additionally, the deployment process has access controls gated on GitHub team membership in the same organization.
Again, at best a misunderstanding and at worst a lie. This is only a valid excuse if the deployment process was immutable and/or those 2 private repos could not be transferred to Ruby Central's own GitHub organization.
If those 2 private repos are only used for the purpose of deployment to rubygems.org and Ruby Central directly controls the infrastructure that serves the rubygems.org domain, it makes perfect sense that those specific repos be owned by Ruby Central. With the benefit of hindsight, it actually doesn't make sense that they were added to the rubygems organization to begin with.
This can be fixed by replicating the team structure within the RubyCentral organization and transfer ownership of those 2 private repos to that org. With a few additional updates, a fork would continue working.
The following paragraph is equally misleading:
On the client tools side, while the repos could have been forked, as long as individuals could publish the gems from any source they want, no forked repository could have been the canonical and secure source of the tools.
This doesn't make any sense. People publishing gems to a different source has no bearing on your supply-chain security.
rubygems.org pulls gems from rubygems.org with only two exceptions:
avo-advanced, marked as optionalBoth of these exceptions are explicit. Everything else necessarily comes from rubygems.org. Someone publishing gems to a non-rubygems.org source doesn't harm rubygems.org's supply chain. Bear in mind, you must specify the source you pull your gems from in a Gemfile. Even rubygems.org has to do this to use themselves as a gem source. There is no default for Bundler and the default for the gem CLI is rubygems.org, so rubygems.org cannot accidentally pull gems from a non-rubygems.org source.
If Ruby Central had forked the repo, they could have continued to own the entire process of deploying to the rubygems.org infrastructure that they own and control. The repo has no deployment automation in GitHub Actions, likely because they're using Shipit.
In fact, one of the 2 private repos they mentioned above is likely their custom wrapper for Shipit to adapt it to their infrastructure. Since they mentioned that it uses GitHub team membership (that page even has one of my old teammates as the last person to modify that file — that's nostalgic) as an authorization gate.
There has been nothing technical stopping Ruby Central from forking the repos. I used to maintain the software that deploys their software. The way they deploy could have been adapted to a fork. It would've been less of an undertaking than dealing with the fallout of what they ended up doing.
The most charitable thing I can attribute this to is having lost a lot of institutional knowledge over the past few years as people have left the organization and the remaining team don't actually know how to adapt the system to a structure that would give them what they want. That's a real problem, certainly, but to say it couldn't be deployed from a fork feels like they're trying to intentionally mislead folks. Shipit is flexible and deploys damn near everything at Shopify, including infrastructure. It can do what you're telling people it can't.
", "contentMap":{ "en-US":"Yeah, clickbait title, I know. It's the only thing anyone pays attention to, so I'm running with it. Ruby Central has published a new blog entry about the whole Rubygems fiasco. Unfortunately, again, it's full of corporate speak and weak excuses that don't hold up under scrutiny.
Before I go further, let me explain where I'm coming from and why I'm qualified to speak confidently about this. For the past 9 years or so, my work has focused less on writing code and more on architecture, infrastructure, and SRE. Prior to my current role, my most recent position was as the technical lead for the team that handles the deployments at Shopify. That team also maintains Shipit, the software which rubygems.org uses to deploy the site. Prior to joining that team, I was an SRE technical lead at Shopify. Total coincidence that that company's name comes up a lot around this whole situation. I only mention it to establish ethos — not only do I have insights the kinds of systems a lot of folks get to blissfully ignore when they deploy via a simple git push or merging a pull request, but I also maintained the specific system that Ruby Central uses to deploy rubygems.org.
I was originally going to write up a critique of the full post but in the spirit of a teacher wheeling in the TV cart to watch Bill Nye (that is, I have other things I actually need to be doing, such as eating dinner, so I need to do the minimum I can get away with here), I'm just going to write about the most glaring part of it from my perspective.
Since the beginning, I've been saying with my entire barrel chest that Ruby Central could've forked the repos. If they wanted to own the code, they could own a fork.
So my biggest frustration with this whole situation is their insistence that they must own the GitHub repos to maintain supply-chain security. It's at best a misunderstanding of how deploying code works and at worst it's a lie to mask accountability for taking over the Rubygems GitHub repos.
Back to the point, Ruby Central's insistence that they need to own the code continues in this post with the following:
While the RubyGems.org service is deployed from rubygems/rubygems.org repository, it also needs at least 2 more private repositories in the same organization for infrastructure-as-code, and CDN deployments. Additionally, the deployment process has access controls gated on GitHub team membership in the same organization.
Again, at best a misunderstanding and at worst a lie. This is only a valid excuse if the deployment process was immutable and/or those 2 private repos could not be transferred to Ruby Central's own GitHub organization.
If those 2 private repos are only used for the purpose of deployment to rubygems.org and Ruby Central directly controls the infrastructure that serves the rubygems.org domain, it makes perfect sense that those specific repos be owned by Ruby Central. With the benefit of hindsight, it actually doesn't make sense that they were added to the rubygems organization to begin with.
This can be fixed by replicating the team structure within the RubyCentral organization and transfer ownership of those 2 private repos to that org. With a few additional updates, a fork would continue working.
The following paragraph is equally misleading:
On the client tools side, while the repos could have been forked, as long as individuals could publish the gems from any source they want, no forked repository could have been the canonical and secure source of the tools.
This doesn't make any sense. People publishing gems to a different source has no bearing on your supply-chain security.
rubygems.org pulls gems from rubygems.org with only two exceptions:
avo-advanced, marked as optionalBoth of these exceptions are explicit. Everything else necessarily comes from rubygems.org. Someone publishing gems to a non-rubygems.org source doesn't harm rubygems.org's supply chain. Bear in mind, you must specify the source you pull your gems from in a Gemfile. Even rubygems.org has to do this to use themselves as a gem source. There is no default for Bundler and the default for the gem CLI is rubygems.org, so rubygems.org cannot accidentally pull gems from a non-rubygems.org source.
If Ruby Central had forked the repo, they could have continued to own the entire process of deploying to the rubygems.org infrastructure that they own and control. The repo has no deployment automation in GitHub Actions, likely because they're using Shipit.
In fact, one of the 2 private repos they mentioned above is likely their custom wrapper for Shipit to adapt it to their infrastructure. Since they mentioned that it uses GitHub team membership (that page even has one of my old teammates as the last person to modify that file — that's nostalgic) as an authorization gate.
There has been nothing technical stopping Ruby Central from forking the repos. I used to maintain the software that deploys their software. The way they deploy could have been adapted to a fork. It would've been less of an undertaking than dealing with the fallout of what they ended up doing.
The most charitable thing I can attribute this to is having lost a lot of institutional knowledge over the past few years as people have left the organization and the remaining team don't actually know how to adapt the system to a structure that would give them what they want. That's a real problem, certainly, but to say it couldn't be deployed from a fork feels like they're trying to intentionally mislead folks. Shipit is flexible and deploys damn near everything at Shopify, including infrastructure. It can do what you're telling people it can't.
" }, "mediaType":"text/html", "attachment":[], "url":["https://jgaskins.blog/ruby-central-doesnt-know-how-their-deployment-infra-works"], "type":"Note", "id":"https://jgaskins.blog/objects/-O8lT6176jw" }