Automatic Chunking for Large File Downloads

AI and ML workloads, container images, and other large binaries are growing in size, and traditional single-stream downloads don't scale well with them. Transfer time is tied directly to file size, and very large files are more prone to timeouts and slow delivery.

Automatic chunking addresses this by splitting large file transfers into multiple parts and fetching them in parallel, instead of pulling the file as one sequential stream. This reduces transfer time for large files without requiring any configuration or client-side changes.

How It Works

When Artifactory retrieves a large file that qualifies (1 GB or larger) for Automatic Chunking, it:

1. Divides the file into multiple parts using HTTP range requests.
2. Downloads the parts concurrently.
3. Reassembles and verifies the complete file with checksum validation before it caches and serves the file.

This preserves Artifactory's standard integrity guarantee. Artifactory always retrieves and verifies the full file before making it available for download. Automatic chunking changes how fast the file arrives, not the trust model around it.

Availability

As of Artifactory 7.173.13, Automatic Chunking is generally available for Smart Remote repositories on SaaS and self-managed environments, for all package types. Since automatic chunking applies only to files of 1 GB or larger, Artifactory downloads smaller files as a single stream, as before.

Benefits

• Faster large-file transfers: Parallel chunk fetching reduces download time for large files compared to a single sequential stream.
• Lower timeout risk: Particularly relevant for multi-gigabyte AI and ML model files and other large binaries that are prone to timing out on a single long-lived connection.
• Resilient to interruptions: If a download fails partway through, Artifactory retries only the failed parts instead of restarting the entire file transfer.
• No action required: Works automatically for qualifying repositories, and you don't need client or configuration changes.

Related Topics

• Smart Remote Repositories
• Remote Repositories
• Get Started with Repositories