X open-sources its algorithm, lets users check shadowbans
What's the story
X, the social media platform formerly known as Twitter, has open-sourced its "For You" algorithm and core ranking engine. The move comes as part of a larger effort to make more of its codebase publicly available on GitHub under the Apache v2 license. The company is also introducing a feature that will allow users to check if their accounts or posts have been affected by any of these ranking systems.
Open-source initiative
Codebase now 10-15x larger than before
The source code for the "For You" timeline, which is the default feed on X, is now available on GitHub under the Apache v2 license.
The company has also expanded its previous open-source efforts to include more details such as model configuration, filter, and core ranking system details.
This expansion makes the codebase roughly 10 to 15 times larger than before.
Insight
Core ranking code is part of the release
Keith Coleman, X's VP of Product, said in an interview with TechCrunch that the open-source release includes the core ranking code that pulls and ranks posts for individual users.
He also said it includes systems that filter out potentially problematic content.
Some systems like the ranker and score can even be run outside the company.
User empowerment
New transparency tool for users
Along with the repository, X is also releasing tools that will let users check if and how its ranking systems have affected their account or posts.
A new transparency tool will be available on an "Under the Hood" page in the app's settings.
Users who have posted 10 or more times over the past month can download their aggregate stats as a JSON file from this page.
Research access
External researchers previewed the open-source code
Before launch, X had previewed its open-source codebase to external researchers familiar with recommendation systems.
These researchers were able to get the "score," or numerical value calculated for every post, up and running outside of X.
This was a major milestone in X's transparency efforts, according to Coleman.
Public participation
Developers can submit updates to improve algorithm
From the GitHub repository, developers will be able to submit updates, known as pull requests. These will be considered by X engineers for incorporation into its algorithm.
Although not all additions will make the cut, Coleman is excited about this public participation in improving the algorithm.
However, some systems that use Grok to predict rule-violating posts are not included in this release to prevent misuse by bad actors.