jsulz
@jsulz.com
I like pretty things, functional things, funny things, food things, and computer things.
Former: Storage infra things 🤗 @hf.co, Devops things @lexblog.bsky.social and devex/cloud infra things at @pantheon.io
Former: Storage infra things 🤗 @hf.co, Devops things @lexblog.bsky.social and devex/cloud infra things at @pantheon.io
Sometimes I am struck by the weirdness of our dystopian present.
November 18, 2025 at 8:57 PM
Sometimes I am struck by the weirdness of our dystopian present.
The Hub is on 100% on Xet. 🚀
A little over a year ago, @hf.co acquired XetHub to unlock the next phase of growth in models and datasets. huggingface.co/blog/xethub-...
In April, there were 1,000 Hugging Face repos on Xet. Now every repo (over 6M) on the Hub is on Xet.
A little over a year ago, @hf.co acquired XetHub to unlock the next phase of growth in models and datasets. huggingface.co/blog/xethub-...
In April, there were 1,000 Hugging Face repos on Xet. Now every repo (over 6M) on the Hub is on Xet.
October 3, 2025 at 3:16 PM
The Hub is on 100% on Xet. 🚀
A little over a year ago, @hf.co acquired XetHub to unlock the next phase of growth in models and datasets. huggingface.co/blog/xethub-...
In April, there were 1,000 Hugging Face repos on Xet. Now every repo (over 6M) on the Hub is on Xet.
A little over a year ago, @hf.co acquired XetHub to unlock the next phase of growth in models and datasets. huggingface.co/blog/xethub-...
In April, there were 1,000 Hugging Face repos on Xet. Now every repo (over 6M) on the Hub is on Xet.
A sneaky part of making this all work is our backward compatibility with Git LFS. This allows us to roll out a significant protocol change without forcing workflow changes
We call this the Git LFS Bridge internally, and like our migration process, it's power is in its simplicity.
We call this the Git LFS Bridge internally, and like our migration process, it's power is in its simplicity.
July 15, 2025 at 3:16 PM
A sneaky part of making this all work is our backward compatibility with Git LFS. This allows us to roll out a significant protocol change without forcing workflow changes
We call this the Git LFS Bridge internally, and like our migration process, it's power is in its simplicity.
We call this the Git LFS Bridge internally, and like our migration process, it's power is in its simplicity.
You can see over the past few months some of the biggest migrations show up in our cluster throughput.
Each spike corresponds to a significant migration (where we download from LFS and upload to Xet) with the baseline steadily increasing to just shy of 100 Gb/s
Each spike corresponds to a significant migration (where we download from LFS and upload to Xet) with the baseline steadily increasing to just shy of 100 Gb/s
July 15, 2025 at 3:16 PM
You can see over the past few months some of the biggest migrations show up in our cluster throughput.
Each spike corresponds to a significant migration (where we download from LFS and upload to Xet) with the baseline steadily increasing to just shy of 100 Gb/s
Each spike corresponds to a significant migration (where we download from LFS and upload to Xet) with the baseline steadily increasing to just shy of 100 Gb/s
The engine behind moving from Git LFS to Xet is our migration process. It's simple, powerful, and has moved well over a dozen PB just by itself. Here's a high level view of how it works.
July 15, 2025 at 3:16 PM
The engine behind moving from Git LFS to Xet is our migration process. It's simple, powerful, and has moved well over a dozen PB just by itself. Here's a high level view of how it works.
Meanwhile, our migrations have pushed throughput to numbers that are bonkers.
In June, we hit upload speeds of 577Gb/s (crossing 500Gb/s for the first time).
In June, we hit upload speeds of 577Gb/s (crossing 500Gb/s for the first time).
June 26, 2025 at 2:48 PM
Meanwhile, our migrations have pushed throughput to numbers that are bonkers.
In June, we hit upload speeds of 577Gb/s (crossing 500Gb/s for the first time).
In June, we hit upload speeds of 577Gb/s (crossing 500Gb/s for the first time).
It's been a bit since I took a step back and looked at our progress to migrate @hf.co from Git LFS to Xet, but every time I do it's mind boggling.
A month ago there were 5,500 users/orgs on Xet with 150K repos and 4PB. Today?
🤗 700,000 users/orgs
📈 350,000 repos
🚀 15PB
A month ago there were 5,500 users/orgs on Xet with 150K repos and 4PB. Today?
🤗 700,000 users/orgs
📈 350,000 repos
🚀 15PB
June 26, 2025 at 2:48 PM
It's been a bit since I took a step back and looked at our progress to migrate @hf.co from Git LFS to Xet, but every time I do it's mind boggling.
A month ago there were 5,500 users/orgs on Xet with 150K repos and 4PB. Today?
🤗 700,000 users/orgs
📈 350,000 repos
🚀 15PB
A month ago there were 5,500 users/orgs on Xet with 150K repos and 4PB. Today?
🤗 700,000 users/orgs
📈 350,000 repos
🚀 15PB
Along with the rest of the crucial services of the internet, it is back up.
But I'll never know how often she was chasing squirrels for those few hours.
But I'll never know how often she was chasing squirrels for those few hours.
June 12, 2025 at 10:42 PM
Along with the rest of the crucial services of the internet, it is back up.
But I'll never know how often she was chasing squirrels for those few hours.
But I'll never know how often she was chasing squirrels for those few hours.
Learned GCP was out by seeing my dog door monitor was broken (backed by a cloud SQL instance).
How am I going to replay/track the events of her going in and out the dog door during this outage?
These are the important questions.
How am I going to replay/track the events of her going in and out the dog door during this outage?
These are the important questions.
June 12, 2025 at 7:15 PM
Learned GCP was out by seeing my dog door monitor was broken (backed by a cloud SQL instance).
How am I going to replay/track the events of her going in and out the dog door during this outage?
These are the important questions.
How am I going to replay/track the events of her going in and out the dog door during this outage?
These are the important questions.
Awk is for kids and adults alike, just like A&W's root beer floats.
June 1, 2025 at 6:34 PM
Awk is for kids and adults alike, just like A&W's root beer floats.
I misspelled avocado once while grocery shopping. Now I only buy avacardos.
May 31, 2025 at 2:04 PM
I misspelled avocado once while grocery shopping. Now I only buy avacardos.
Continuing to move all the LFS bytes into Xet storage on Hugging Face! Currently up to:
🤗 5,500 users and orgs with Xet access
🚀 150,000 Xet-backed models and datasets
🤯 4+ PB managed by Xet
How much more to go? If the Hub's top storage users are any indication: many bytes
🤗 5,500 users and orgs with Xet access
🚀 150,000 Xet-backed models and datasets
🤯 4+ PB managed by Xet
How much more to go? If the Hub's top storage users are any indication: many bytes
May 21, 2025 at 6:52 PM
Continuing to move all the LFS bytes into Xet storage on Hugging Face! Currently up to:
🤗 5,500 users and orgs with Xet access
🚀 150,000 Xet-backed models and datasets
🤯 4+ PB managed by Xet
How much more to go? If the Hub's top storage users are any indication: many bytes
🤗 5,500 users and orgs with Xet access
🚀 150,000 Xet-backed models and datasets
🤯 4+ PB managed by Xet
How much more to go? If the Hub's top storage users are any indication: many bytes
And moving all these bytes is no joke. Our content-addressed-store (CAS) is doing a lot of hard work, hitting up to 150 Gb/s as we migrate repos from LFS to Xet.
April 30, 2025 at 4:15 PM
And moving all these bytes is no joke. Our content-addressed-store (CAS) is doing a lot of hard work, hitting up to 150 Gb/s as we migrate repos from LFS to Xet.
We've also updated our repo graph which shows how Xet-backed repos share bytes with each other.
Here you can see how different versions of the Qwen, Llama, and Phi models are grouped together.
Interactive graph here: huggingface.co/spaces/xet-t...
Here you can see how different versions of the Qwen, Llama, and Phi models are grouped together.
Interactive graph here: huggingface.co/spaces/xet-t...
April 30, 2025 at 4:15 PM
We've also updated our repo graph which shows how Xet-backed repos share bytes with each other.
Here you can see how different versions of the Qwen, Llama, and Phi models are grouped together.
Interactive graph here: huggingface.co/spaces/xet-t...
Here you can see how different versions of the Qwen, Llama, and Phi models are grouped together.
Interactive graph here: huggingface.co/spaces/xet-t...
Come find BERT island. Or see how datasets relate in practice, and how model libraries or tasks can tie repos together.
It's a byte-level map of the Hub.
The result is a beautiful visualization from Saba Noorassa and @reverius42.bsky.social that I’ve already lost way too much time to.
It's a byte-level map of the Hub.
The result is a beautiful visualization from Saba Noorassa and @reverius42.bsky.social that I’ve already lost way too much time to.
April 9, 2025 at 3:19 PM
Come find BERT island. Or see how datasets relate in practice, and how model libraries or tasks can tie repos together.
It's a byte-level map of the Hub.
The result is a beautiful visualization from Saba Noorassa and @reverius42.bsky.social that I’ve already lost way too much time to.
It's a byte-level map of the Hub.
The result is a beautiful visualization from Saba Noorassa and @reverius42.bsky.social that I’ve already lost way too much time to.
This graph shows requests per second (rps) to our content-addressed store (CAS) right as the release went live (h/t to @rajatarya.com for the screenshot)
yellow = GETs; dashed line = launch time.
I think it's pretty easy to spot when Xet started to send the first bytes to excited downloaders 👀
yellow = GETs; dashed line = launch time.
I think it's pretty easy to spot when Xet started to send the first bytes to excited downloaders 👀
April 7, 2025 at 2:39 PM
This graph shows requests per second (rps) to our content-addressed store (CAS) right as the release went live (h/t to @rajatarya.com for the screenshot)
yellow = GETs; dashed line = launch time.
I think it's pretty easy to spot when Xet started to send the first bytes to excited downloaders 👀
yellow = GETs; dashed line = launch time.
I think it's pretty easy to spot when Xet started to send the first bytes to excited downloaders 👀
If you go to any model in the collection, you'll see the Xet logo supporting the many TB of tensor files.
Every request to download these files comes to our infrastructure.
Every request to download these files comes to our infrastructure.
April 7, 2025 at 2:39 PM
If you go to any model in the collection, you'll see the Xet logo supporting the many TB of tensor files.
Every request to download these files comes to our infrastructure.
Every request to download these files comes to our infrastructure.
Thanks to the Meta team for launching on Xet!
April 5, 2025 at 8:24 PM
Thanks to the Meta team for launching on Xet!
With the models on our infrastructure, we can peer in and see how well our dedupe performs across the Llama 4 family.
On average, we're seeing ~25% dedupe, providing huge savings to the community who iterate on these state-of-the-art models. Here's a few selected models and how they perform on Xet.
On average, we're seeing ~25% dedupe, providing huge savings to the community who iterate on these state-of-the-art models. Here's a few selected models and how they perform on Xet.
April 5, 2025 at 8:05 PM
With the models on our infrastructure, we can peer in and see how well our dedupe performs across the Llama 4 family.
On average, we're seeing ~25% dedupe, providing huge savings to the community who iterate on these state-of-the-art models. Here's a few selected models and how they perform on Xet.
On average, we're seeing ~25% dedupe, providing huge savings to the community who iterate on these state-of-the-art models. Here's a few selected models and how they perform on Xet.
March 26, 2025 at 12:24 PM
These are the kinds of challenges you only see when you move from theory to practice.
There's nothing more satisfying than working on infrastructure for months and seeing requests funnel through and take off like a rocket 🚀
There's nothing more satisfying than working on infrastructure for months and seeing requests funnel through and take off like a rocket 🚀
March 18, 2025 at 2:18 PM
These are the kinds of challenges you only see when you move from theory to practice.
There's nothing more satisfying than working on infrastructure for months and seeing requests funnel through and take off like a rocket 🚀
There's nothing more satisfying than working on infrastructure for months and seeing requests funnel through and take off like a rocket 🚀
The first migrations routed ~6% of @hf.co download traffic through Xet infrastructure. Real requests gave us an opportunity to see pods experiencing load imbalances and overhead from streaming entire blocks for partial requests.
We fixed these issues on the fly without any major disruption.
We fixed these issues on the fly without any major disruption.
March 18, 2025 at 2:18 PM
The first migrations routed ~6% of @hf.co download traffic through Xet infrastructure. Real requests gave us an opportunity to see pods experiencing load imbalances and overhead from streaming entire blocks for partial requests.
We fixed these issues on the fly without any major disruption.
We fixed these issues on the fly without any major disruption.
You can apply for yourself, or your entire organization.
Head over to your account settings for more information or join anywhere you see the Xet logo on a repository you know.
Head over to your account settings for more information or join anywhere you see the Xet logo on a repository you know.
March 12, 2025 at 10:36 PM
You can apply for yourself, or your entire organization.
Head over to your account settings for more information or join anywhere you see the Xet logo on a repository you know.
Head over to your account settings for more information or join anywhere you see the Xet logo on a repository you know.
All good. Just communicate through GitHub comments. (somewhere, an email angel 📧 😇 lost its wings)
February 26, 2025 at 4:45 PM
All good. Just communicate through GitHub comments. (somewhere, an email angel 📧 😇 lost its wings)
Here's what you can expect with this step:
✅ First off, no action needed - this migration is helping us test and scale the infrastructure before a broader rollout
👀🔎 But you can play "spot the Xet logo" - if you see our logo on a file in a repo, that's a file we're serving now!
Download away 🌐
✅ First off, no action needed - this migration is helping us test and scale the infrastructure before a broader rollout
👀🔎 But you can play "spot the Xet logo" - if you see our logo on a file in a repo, that's a file we're serving now!
Download away 🌐
February 21, 2025 at 3:22 AM
Here's what you can expect with this step:
✅ First off, no action needed - this migration is helping us test and scale the infrastructure before a broader rollout
👀🔎 But you can play "spot the Xet logo" - if you see our logo on a file in a repo, that's a file we're serving now!
Download away 🌐
✅ First off, no action needed - this migration is helping us test and scale the infrastructure before a broader rollout
👀🔎 But you can play "spot the Xet logo" - if you see our logo on a file in a repo, that's a file we're serving now!
Download away 🌐