Caching and Proxying Packages Using Remote Repositories

This use case describes how you can use remote repositories to address the challenges of managing packages from remote public registries.

Challenges

Organizations face multiple challenges when managing access to remote packages on public registries, including:

Public registries may be unavailable at times or suffer from latency issues.
Administrators lack an efficient method to moderate developer access to public registries without impacting productivity.
Particular package versions may become unavailable on the public registry.
Network overuse, which leads to reduced speed and increased cost.
Multiple users keep copies of the same files, which overloads the local storage system.
Inconsistent package versions among developers.

The Artifactory Solution: Remote Repositories

To address these challenges, JFrog Artifactory offers remote repositories, which can cache and proxy packages from a remote public or private registry. Remote repositories provide multiple benefits to developers and DevOps engineers, including fast, convenient, and secure access to remote resources.

For more information about these benefits, see:

Benefits for developers
Benefits for DevOps engineers

Benefits of Remote Repositories for Developers

Developers working in an IDE need to be able to call packages easily and insert them into their code. Packages must always be available to avoid delays or other issues when compiling a build.

Remote repositories offer the following benefits for developers:

Builds become much faster because the required dependencies are cached and available for use.
Cached dependencies are available even when public registries are down or experiencing latency.

Imagine the following scenario:

Anne is a developer and early riser who begins work early and starts running builds. When she requires a new dependency, Artifactory pulls it from a public registry and caches it for later use.

When other developers on the team begin work later that morning, they soon discover that their build times have improved greatly because the dependencies they require are cached and available in the remote repository.

Later that day, a sudden storm causes intermittent outages in the company’s connection to the outside world. However, the development team can continue their work without interruption secure in the knowledge that the dependencies they require are available for use in the remote repository.

Benefits of Remote Repositories for DevOps Engineers

DevOps engineers build the integrations that enable the rest of the organization to operate. They set up integrations between applications, build automation, and pipelines, and keep things running.

Remote repositories enable DevOps engineers to ensure that the developers in their organization download from a particular public registry via the Artifactory remote repository. You can block direct downloads from the public registry at the network level.

Remote repositories make downloads:

Moderated, using include and exclude patterns, regular expressions, security filters, domain filters, and so on
Traceable
Subject to security scans to ensure license compliance and block or send alerts regarding malicious software and other vulnerabilities

Additional benefits for DevOps engineers include:

Improved performance thanks to caching
Reduced storage costs due to storage optimization, because checksum-based storage keeps only one instance of each binary, regardless of the number of requests from different developers
Reduced bandwidth and consumption

Imagine the following scenario:

Terry is a DevOps engineer who faces a situation where developers download software from public registries in an uncontrolled manner and without moderation. As a result, multiple versions of the same dependencies exist, and in some cases, this introduces security vulnerabilities into the organization’s software.

To correct this situation, Terry sets up a remote repository to serve as a proxy for the public registry. At the same time, he blocks direct developer access to the registry, ensuring that everyone must use the remote repository to download the software they require. Using a remote repository, the DevOps engineer can track who downloads each package.

To keep the organization secure and compliant, he can also use JFrog Curation to limit which packages developers can access on remote registries. After he brings permitted packages into the remote repository, he can use JFrog Xray to scan all packages for possible vulnerabilities. These multiple layers of security help prevent vulnerabilities and possible malware from entering the organization.

Remote Repositories Workflow

The workflow with remote repositories in Artifactory is as follows:

A developer calls for a particular software dependency typically while working in an IDE.
JFrog pulls the dependency from the relevant public registry, for example, Maven Central and Docker Hub, and caches it in a remote repository.
When the next developer needs the same dependency, JFrog serves it directly from the cache.

Artifactory can perform authentication with remote registries using tokens and TLS certificates.

Artifactory supports the following methods for creating remote repositories:

Smart Remote Repositories

A Smart Remote repository is a special kind of remote repository that proxies a local, remote, or Federated repository from another Artifactory endpoint or Edge service. In addition to the benefits of remote repositories, Smart Remote repositories offer the following additional benefits:

Remote download statistics
Synchronized properties
Remote repository browsing
Source absence detection

Virtual Repositories

Virtual repositories in Artifactory can benefit developers and DevOps engineers in addition to, or in use with, remote repositories. A virtual repository is a collection of local, remote, and other virtual repositories that you access through a single URL.