On July 7, 2026, Hugging Face announced that you can now mount a Hugging Face Bucket or any Hub repo into a SkyPilot job with a single hf:// URL and your existing HF_TOKEN. That’s the most concrete change the blog post offers: data stays on the Hub, compute lands wherever GPUs are free, and you don’t pay a cent for egress.
Key Takeaways
- SkyPilot can schedule jobs across 20+ clouds, Kubernetes, Slurm, and on‑prem clusters.
- Hugging Face Storage charges no egress, eliminating the typical $0.09/GB outbound fee on AWS.
- The new
hf://scheme supports read‑write buckets and read‑only model or dataset repos via MOUNT or COPY modes. - Underlying Xet‑backed dedup means only changed chunks are transferred, speeding incremental checkpointing.
- Authentication is a single
HF_TOKENthat works across all clouds, removing per‑vendor key management.
Zero Egress Storage with SkyPilot and Hugging Face
What makes this integration noteworthy is that the data path stops deciding where you run. Teams that reserve H100s on one hyperscaler, keep a neocloud block on another, and spin up an on‑prem rack can now point a single job spec at any of those resources, and the model or dataset will stream straight from its Hub bucket without ever leaving the provider’s network.
How SkyPilot Handles Multi‑Cloud GPU Jobs
SkyPilot’s scheduler treats every supported environment as a node in a global pool. When you submit a job, SkyPilot queries the availability of GPUs across 20+ clouds, Kubernetes clusters, Slurm farms, and even private racks. The first place with a free slot gets the job, and the same hf:// mount works there without modification.
Unified Compute, Distributed Resources
Because the compute layer is abstracted, you don’t have to maintain separate CI pipelines for AWS, GCP, Azure, or Nebius. The same Docker image, same Python entrypoint, and same --secret HF_TOKEN flag travel with the workload. That’s a relief for anyone who’s ever tangled with per‑cloud IAM policies.
Mounting Hugging Face Buckets with hf://
The integration adds a new store: hf backend to SkyPilot’s existing list of object stores (S3, GCS, Azure, R2, etc.). In a SkyPilot YAML you can declare a mount like this:
file_mounts:
/checkpoints:
source: hf://buckets/my-org/qwen-sft
store: hf
mode: MOUNT
/base-model:
source: hf://Qwen/Qwen3.5-4B
store: hf
mode: MOUNT
/data:
source: hf://datasets/my-org/my-dataset@main
store: hf
mode: MOUNT
When the job starts, SkyPilot invokes the hf‑mount FUSE driver, which presents the bucket or repo as a local directory. Reads are lazy: the driver pulls only the bytes your code actually accesses, caching them on the node. The same behaviour applies whether you choose MOUNT or MOUNT_CACHED; both keep a local cache.
Copy vs. Mount
If you prefer a full pre‑download, you can switch the mode to COPY. That triggers a one‑time huggingface_hub download before the job runs, but it also means you need enough disk space to hold the entire dataset or checkpoint.
Why Zero Egress Matters for AI Teams
Object storage is regional by design. A bucket in AWS’s us‑east‑1 zone is a different endpoint than a bucket in GCP’s europe‑west1. Pulling a model from an AWS bucket into a GCP GPU instance normally incurs an egress charge – the blog cites roughly $0.09/GB when data leaves AWS.
Hugging Face’s storage layer sidesteps that fee entirely. The bucket lives on Xet, a content‑addressable backend that serves data from the nearest edge node, but it never charges for outbound traffic. For a 200 GB model, that translates to a potential savings of $18 per transfer, and for large training runs that can add up quickly.
Deduplication at Scale
The Xet‑backed dedup feature ensures that incremental checkpoints only move the chunks that changed. If a checkpoint differs from its predecessor by 5 GB, only those 5 GB travel over the network. That cuts both bandwidth and cost, especially when you iterate dozens of times.
Performance and Cost Implications
Because reads happen at the filesystem layer, a training script can start processing a large file before the whole file finishes downloading. The GPU stays busy, and you avoid the idle period that a full COPY would cause. The blog notes that the biggest benefit shows up on the first epoch, when nothing is cached yet.
- Reduced idle time → higher GPU utilisation.
- No egress charges → direct cost savings on multi‑cloud workloads.
- Chunk‑level transfer → faster checkpointing for iterative experiments.
- Single token authentication → less operational overhead.
These points matter whether you’re a startup racing to train a 10‑billion‑parameter model or a research lab fine‑tuning a 4‑billion‑parameter checkpoint across several regions.
Getting Started: A Quick Walkthrough
First, make sure your environment has HF_TOKEN set. Then, create a SkyPilot job file that references the hf:// mounts you need. Finally, launch the job with:
sky launch -c my-job.yaml --secret HF_TOKEN
SkyPilot will pick the nearest free GPU, mount the bucket or repo, and start your script. If you need to write checkpoints back to the bucket, just point the output path to the mounted /checkpoints directory.
Practical Tips
Keep your bucket names short – they become part of the local path. Use the @revision syntax for datasets if you want reproducibility. And remember that the cache lives on the node; if you spin up a new instance, the cache starts empty.
What This Means For You
For developers, the biggest win is simplicity. You no longer have to script a multi‑step data copy before launching a GPU job, and you don’t need to juggle separate access keys for each cloud. That means less boilerplate, fewer failing pipelines, and faster iteration cycles.
For builders of AI platforms, the integration opens a path to truly vendor‑agnostic compute. You can advertise “run anywhere, pay nothing for data transfer,” and back it up with a concrete hf:// mount that works on AWS, GCP, Azure, or your own Kubernetes cluster. That could be a differentiator when pitching to enterprises that already split their GPU spend across multiple providers.
Looking ahead, the question is whether other model hubs will adopt a similar zero‑egress model, or if Hugging Face will remain the sole player offering this frictionless multi‑cloud experience.
Historical Context
SkyPilot first entered the market as a cloud‑agnostic scheduler for AI workloads. Early versions supported only the classic object stores – S3, GCS, Azure Blob – and required users to copy data into a cloud‑specific bucket before a job could start. Those copies incurred egress fees whenever the compute node lived in a different provider than the storage endpoint. Over time, the community requested a tighter coupling with model hubs, because the majority of large‑scale training still starts from a public repository.
The Hugging Face storage backend arrived as a response to that demand. By embedding the Xet‑based deduplication engine inside the Hub, Hugging Face gave the community a way to keep data on a single, globally reachable endpoint while still benefiting from edge caching. The July 7, 2026 announcement marks the first public release where SkyPilot can address that endpoint directly, using the hf:// scheme instead of a cloud‑specific URL.
From a timeline perspective, the new feature builds on two prior milestones: the introduction of SkyPilot’s store abstraction in version 1.4, and Hugging Face’s rollout of zero‑egress buckets in early 2025. The combination reduces the number of moving parts a team needs to manage, and it reflects a broader industry trend toward “data‑centric compute” – where the location of the data no longer drives scheduling decisions.
Competitive Landscape
Several cloud providers offer their own egress‑free storage options, but they typically tie the service to a single ecosystem. For example, a bucket in AWS’s S3 can be accessed without charge from within AWS, yet pulling it from GCP still incurs the $0.09/GB fee. The same pattern repeats across Azure and other public clouds. Those solutions solve the intra‑cloud problem but leave multi‑cloud teams with the same cost dilemma.
By contrast, the Hugging Face approach decouples storage from any single provider. The Xet backend distributes content globally, and because the Hub never bills for outbound traffic, the cost model stays flat regardless of where the GPU lands. That gives SkyPilot users a unique advantage: they can chase the cheapest GPU spot, move between providers on the fly, and keep the data‑transfer bill at zero.
Other open‑source projects have experimented with similar ideas – mounting remote filesystems via FUSE, using rsync‑style sync tools, or relying on shared NFS clusters. Those methods often require either a persistent network connection or a manual cache warm‑up. The Hugging Face driver, however, implements lazy loading natively, which means the first read triggers an on‑demand fetch while the rest of the job proceeds uninterrupted.
Key Questions Remaining
- Will the zero‑egress model scale to petabyte‑size datasets without degrading latency? The current implementation relies on edge caches; long‑tail objects could still experience download delays.
- How will future changes to Xet’s pricing or caching policies affect the cost‑free guarantee? Hugging Face has not announced any plans to alter the egress model, but market pressures could prompt revisions.
- Can other model hubs replicate the
hf://protocol, or will interoperability require a new standard? The current scheme is tightly coupled to Hugging Face’s token system.
What Happens Next
Early adopters are expected to benchmark the integration on many workloads – from single‑node fine‑tuning to multi‑node large‑scale training. Those tests will reveal how much idle time drops when the mount replaces a full copy, and they will surface any edge‑case failures in the lazy‑load driver. SkyPilot’s roadmap includes adding explicit health checks for the mount point, as well as tighter telemetry around chunk‑level transfer rates.
Meanwhile, Hugging Face plans to extend the bucket API to support write‑through semantics, which would let jobs push checkpoints directly back to the Hub without an extra copy step. If that feature lands, the end‑to‑end workflow could become fully zero‑copy: data streams in, computation runs, and results stream out, all while staying within the same zero‑egress boundary.
Developers who follow the integration closely should watch for announcements around Xet cache warm‑up and for any updates to the hf:// driver that improve concurrency handling. Those refinements will determine whether the partnership becomes the default choice for multi‑cloud AI pipelines, or whether teams will still fall back to provider‑specific storage when latency becomes a concern.
Sources: Hugging Face Blog, original report
