S3 Binary Storage Templates

You need to configure the S3 provider with parameters specific to your can account (but leave all other parameters with the recommended values).

You need to configure the S3 provider with parameters specific to your account (but can leave all other parameters with the recommended values).

S3 Direct Upload Template (Recommended)

This template uploads binaries directly to the S3 bucket. This mechanism generally allows for faster uploads, since there is less contention for disk IO, particularly when Artifactory is hosted on AWS. The direct template can be used for both single-node and high-availability (HA) configurations. A shared mount is not needed for HA. For more information, see Configure Artifactory to Use S3 Storage.

The S3 direct <chain="s3-storage-v3-direct"/> (or eventual-less) template shown below allows directly uploading to S3, bypassing the eventual upload mechanism.

<config version="2">
    <chain template="s3-storage-v3-direct"/>    
    <provider id="s3-storage-v3" type="s3-storage-v3">
       <endpoint>s3.amazonaws.com</endpoint>
       <bucketName>bucketName</bucketName>
       <path>pathPrefix</path>
       <region>s3Region</region>
       <identity>yourIdentity</identity>
       <credential>yourCredentials</credential>
       <awsSdkV2>true</awsSdkV2> <!-- Supported from Artifactory version 7.125.4 -->
    </provider>
</config>

While you don't need to configure anything else in your binarystore.xml, the following example shows what the s3-storage-v3 template looks like under the hood. This example sets S3 with your credentials. The default settings for maxConnections, connectionTimeout, and cache-fs maxCacheSize configuration are also shown in this example.

<config version="2">
   <chain>
        <provider id="cache-fs" type="cache-fs">
            <provider id="s3-storage-v3" type="s3-storage-v3"/>
        </provider>
   </chain>
   <provider id="s3-storage-v3" type="s3-storage-v3">
       <endpoint>s3.amazonaws.com</endpoint>
       <bucketName>bucketName</bucketName>
       <path>pathPrefix</path>
       <region>s3Region</region>
       <identity>yourIdentity</identity>
       <credential>yourCredentials</credential>
       <maxConnections>50</maxConnections>
       <connectionTimeout>10000</connectionTimeout>
       <awsSdkV2>true</awsSdkV2> <!-- Supported from Artifactory version 7.125.4 -->
   </provider>
   <provider type="cache-fs" id="cache-fs">
       <maxCacheSize>5000000000</maxCacheSize>
   </provider>
</config>

S3 Eventual Upload Template

If you are uploading on a system where the S3 upload speed is slow (for example, when Artifactory is hosted self-hosted), you may want to use the S3 Eventual Upload template <chain="s3-storage-v3"/>. The Eventual Upload template also allows you to upload when S3 is down or experiencing network issues. The S3 eventual template is shown below.

<chain> <!-- template="s3-storage-v3" -->
    <provider id="cache-fs" type="cache-fs">
        <provider id="eventual" type="eventual">
            <provider id="retry" type="retry">
                <provider id="s3-storage-v3" type="s3-storage-v3"/>
            </provider>
        </provider>
    </provider>
</chain>

The following snippet shows an example of S3 eventual upload mechanism.

<config version="2">
   <chain template="s3-storage-v3"/>
   <provider id="s3-storage-v3" type="s3-storage-v3">
       <endpoint>s3.amazonaws.com</endpoint>
       <bucketName>bucketName</bucketName>
       <path>pathPrefix</path>
       <region>s3Region</region>
       <identity>yourIdentity</identity>
       <credential>yourCredentials</credential>
       <awsSdkV2>true</awsSdkV2> <!-- Supported from Artifactory version 7.125.4 -->
   </provider>
</config>

S3 Cluster Binary Provider

This is the setting used for the S3 Official SDK Template library when configuring filestore sharding for an HA cluster. It is based on the sharding and dynamic provider logic that synchronizes the cluster-file-system.

When using the cluster-s3-storage-v3 template, data is temporarily stored on the file system of each node using the Eventual-Cluster Binary Provider, and is then passed on to your S3 object storage for persistent storage.

Each node has its own local filestore (just like in the cluster file system binary provider) and is connected to all other cluster nodes via dynamically allocated Remote Binary Providers using the Sharding-Cluster Binary Provider.

📘
Note
JFrog recommends using the S3 Direct Upload Template instead of the S3 Cluster Binary Provider, as described here.

cluster-s3-storage-v3 template configuration

Because you must configure the s3 provider with parameters specific to your account (but can leave all other parameters with the recommended values), if you choose to use the cluster-s3-storage-v3 template, your binarystore.xml configuration file should look like the example below.

<config version="2">
    <chain template="cluster-s3-storage-v3"/>
    <provider id="s3-storage-v3" type="s3-storage-v3">
       <endpoint>s3.amazonaws.com</endpoint>
       <bucketName>bucketName</bucketName>
       <path>pathPrefix</path>
       <region>s3Region</region>
       <identity>yourIdentity</identity>
       <credential>yourCredentials</credential>
       <signatureExpirySeconds>600</signatureExpirySeconds>
       <awsSdkV2>true</awsSdkV2> <!-- Supported from Artifactory version 7.125.4 -->
   </provider>
</config>

What's in the cluster-s3-storage-v3 template?

While you don't need to configure anything else in your binarystore.xml, this is what the cluster-s3-storage-v3 template looks like under the hood.

<config version="2">
    <chain> <!-- template="cluster-s3-storage-v3" -->
      <provider id="cache-fs-eventual-s3" type="cache-fs">
         <provider id="sharding-cluster-eventual-s3" type="sharding-cluster">
            <sub-provider id="eventual-cluster-s3" type="eventual-cluster">
               <provider id="retry-s3" type="retry">
                  <provider id="s3-storage-v3" type="s3-storage-v3"/>
               </provider>
            </sub-provider>
           <dynamic-provider id="remote-s3" type="remote"/>
         </provider>
      </provider>
    </chain>
    <provider id="cache-fs-eventual-s3" type="cache-fs">
        <maxCacheSize>5000000000</maxCacheSize>
    </provider>
    <provider id="sharding-cluster-eventual-s3" type="sharding-cluster">
         <redundancy>2</redundancy>
         <lenientLimit>1</lenientLimit>
         <property name="zones" value="local,remote"/>
    </provider>
    <provider id="eventual-cluster-s3" type="eventual-cluster">
         <maxWorkers>5</maxWorkers>
         <dispatcherInterval>1000</dispatcherInterval>
         <checkPeriod>15000</checkPeriod>
         <addStalePeriod>300000</addStalePeriod>
         <zone>local</zone>
    </provider>
    <provider id="remote-s3" type="remote">
         <checkPeriod>15000</checkPeriod>
         <connectionTimeout>5000</connectionTimeout>
         <socketTimeout>15000</socketTimeout>
         <maxConnections>200</maxConnections>
         <connectionRetry>2</connectionRetry>
         <zone>remote</zone>
    </provider>
</config>

S3 Sharding

You can implement sharding with multiple S3 buckets. The s3-sharding template is available with Artifactory to configure sharding with S3. A sub-binary provider, state-aware-s3, template is used with the s3-sharding template to implement sharding across multiple S3 buckets.

⚠️
Warning
Sharding rules such as redundancy and write/read mechanisms are applicable on the new data added to the filestore post the changes to the filestore configuration. For existing data in the filestore, you must copy the data to other new S3 shards.

S3-sharding Template

The s3-sharding template uses the same parameters as Amazon S3 official template except for UrlPresigning, which is not supported.

<chain> <!-- template="s3-sharding" -->
        <provider id="cache-fs" type="cache-fs">
            <provider type="sharding" id="sharding">
                <sub-provider type="state-aware-s3" id="s3-shard1"/>
                <sub-provider type="state-aware-s3" id="s3-shard2"/>
            </provider>
        </provider>
</chain>

State-Aware-S3 Binary Provider

The provider is aware if it's underlying S3 bucket is functioning or not. It can also recover from errors (the parent provider is responsible for recovery) with the addition of the checkPeriod field.

📘
Note
This binary provider is not independent and will always be used with S3 sharding.

Parameter	Description
type	state-aware-s3
checkPeriod	Default: 15000 ms The minimum time to wait between trying to re-activate the provider if it had fatal errors at any point.
writeEnabled	Default: true Enables/disables the write operations for the binary provider. If set to false, the state-aware-s3 provider will continue to serve read requests, so Artifactory can continue to read binaries from that provider. In addition, the garbage collection can continue to clean the deleted binaries from the provider. (Only applicable under a sharding provider.)
zone	The name of the sharding zone the provider is part of (only applicable under a sharding provider).

S3 Sharding Examples

The following examples show different ways to implement S3 sharding.

S3 Sharding Example 1

In the following sample configuration, the filestore is implemented with two S3 shards, one region, and one redundancy.

<config version="2">

 <chain>
        <provider id="cache-fs" type="cache-fs">
            <provider type="sharding" id="sharding">                
                <sub-provider type="state-aware-s3" id="s3-shard1"/>
                <sub-provider type="state-aware-s3" id="s3-shard2"/>
            </provider>
        </provider>
 </chain>

 <provider id="sharding" type="sharding">
      <redundancy>1</redundancy>
 </provider>
   
 <provider id="s3-shard1" type="state-aware-s3">
      <endpoint>http://s3.amazonaws.com</endpoint>
      <bucketName>data1212</bucketName>
      <path>yon1220d</path>
      <region>us-east-1</region>
      <provider.id>aws-s3</provider.id>
      <identity>AK...-accessKeyId</identity>
      <credential>ePE...-secretAccessKey</credential>
      <enableSignedUrlRedirect>true</enableSignedUrlRedirect>
      <signedUrlExpirySeconds>3600</signedUrlExpirySeconds>
      <testConnection>false</testConnection>
      <awsSdkV2>true</awsSdkV2> <!-- Supported from Artifactory version 7.125.4 -->      
 </provider>
 <provider id="s3-shard2" type="state-aware-s3">
      <endpoint>http://s3.amazonaws.com</endpoint>
      <bucketName>test-tomers-bucket</bucketName>
      <path>yon1220t</path>
      <region>us-east-1</region>
      <provider.id>aws-s3</provider.id>
      <identity>AK...-accessKeyId</identity>
      <credential>ePE...-secretAccessKey</credential>
      <enableSignedUrlRedirect>true</enableSignedUrlRedirect>
      <signedUrlExpirySeconds>3600</signedUrlExpirySeconds>
      <testConnection>false</testConnection>
      <awsSdkV2>true</awsSdkV2> <!-- Supported from Artifactory version 7.125.4 -->     
 </provider>
</config>

S3 Sharding Example 2

In the following sample configuration, the filestore is implemented with two S3 shards, one region, and two redundancy.

<config version="2">
 <chain> <!-- template="s3-sharding" -->
        <provider id="cache-fs" type="cache-fs">
            <provider type="sharding" id="sharding">               
                <sub-provider type="state-aware-s3" id="s3-shard1"/>
                <sub-provider type="state-aware-s3" id="s3-shard2"/>
            </provider>
        </provider>
    </chain>   

   <provider id="sharding" type="sharding">
      <redundancy>2</redundancy>
   </provider>
   <provider id="s3-shard1" type="state-aware-s3">
      <endpoint>http://s3.amazonaws.com</endpoint>
      <bucketName>bucket1</bucketName>
      <path>path1</path>
      <region>us-east-1</region>
      <provider.id>aws-s3</provider.id>
      <identity>AK...-accessKeyId</identity>
      <credential>ePE...-secretAccessKey</credential>
      <enableSignedUrlRedirect>true</enableSignedUrlRedirect>
      <signedUrlExpirySeconds>3600</signedUrlExpirySeconds>
      <testConnection>false</testConnection>
      <awsSdkV2>true</awsSdkV2> <!-- Supported from Artifactory version 7.125.4 -->      
   </provider>
   <provider id="s3-shard2" type="state-aware-s3">
      <endpoint>http://s3.amazonaws.com</endpoint>
      <bucketName>bucket2</bucketName>
      <path>path2</path>
      <region>us-east-1</region>
      <provider.id>aws-s3</provider.id>
      <identity>AK...-accessKeyId</identity>
      <credential>ePE...-secretAccessKey</credential>
      <enableSignedUrlRedirect>true</enableSignedUrlRedirect>
      <signedUrlExpirySeconds>3600</signedUrlExpirySeconds>
      <testConnection>false</testConnection>
      <awsSdkV2>true</awsSdkV2> <!-- Supported from Artifactory version 7.125.4 -->      
   </provider>
</config>

S3 Sharding Example 3

In the following sample configuration, the filestore is implemented with five S3 shards, two region, and three redundancy.

<?xml version="1.0" encoding="UTF-8"?>
<config version="5">
 <chain> <!-- template="s3-sharding" -->
        <provider id="cache-fs" type="cache-fs">
            <provider type="sharding" id="sharding">                
                <sub-provider type="state-aware-s3" id="s3-shard1" />
                <sub-provider type="state-aware-s3" id="s3-shard2" />
                <sub-provider type="state-aware-s3" id="s3-shard3" />
                <sub-provider type="state-aware-s3" id="s3-shard4" />
                <sub-provider type="state-aware-s3" id="s3-shard5" />
            </provider>
        </provider>
    </chain>

   <provider id="sharding" type="sharding">
      <redundancy>3</redundancy>
   </provider>
    <provider id="s3-shard1" type="state-aware-s3">
      <endpoint>http://s3.amazonaws.com</endpoint>
      <bucketName>data120</bucketName>
      <path>yond</path>
      <region>us-east-1</region>
      <provider.id>aws-s3</provider.id>
      <identity>AK...-accessKeyId</identity>
      <credential>ePE...-secretAccessKey</credential>
      <enableSignedUrlRedirect>true</enableSignedUrlRedirect>
      <signedUrlExpirySeconds>3600</signedUrlExpirySeconds>
      <testConnection>false</testConnection>
      <awsSdkV2>true</awsSdkV2> <!-- Supported from Artifactory version 7.125.4 -->
   </provider>
   <provider id="s3-shard2" type="state-aware-s3">
      <endpoint>http://s3.amazonaws.com</endpoint>
      <bucketName>data125</bucketName>
      <path>yont</path>
      <region>us-east-1</region>
      <provider.id>aws-s3</provider.id>
      <identity>AK...-accessKeyId</identity>
      <credential>ePE...-secretAccessKey</credential>
      <writeEnabled>false</writeEnabled>
      <enableSignedUrlRedirect>true</enableSignedUrlRedirect>
      <signedUrlExpirySeconds>3600</signedUrlExpirySeconds>
      <testConnection>false</testConnection>
      <awsSdkV2>true</awsSdkV2> <!-- Supported from Artifactory version 7.125.4 -->
   </provider>
   <provider id="s3-shard3" type="state-aware-s3">
      <endpoint>http://s3.amazonaws.com</endpoint>
      <bucketName>data121</bucketName>
      <path>yonb</path>
      <region>us-west-1</region>
      <provider.id>aws-s3</provider.id>
      <identity>AK...-accessKeyId</identity>
      <credential>ePE...-secretAccessKey</credential>
      <enableSignedUrlRedirect>true</enableSignedUrlRedirect>
      <signedUrlExpirySeconds>3600</signedUrlExpirySeconds>
      <testConnection>false</testConnection>
      <awsSdkV2>true</awsSdkV2> <!-- Supported from Artifactory version 7.125.4 -->
   </provider>
   <provider id="s3-shard4" type="state-aware-s3">
      <endpoint>http://s3.amazonaws.com</endpoint>
      <bucketName>data122</bucketName>
      <path>yonb2</path>
      <region>us-east-1</region>
      <provider.id>aws-s3</provider.id>
      <identity>AK...-accessKeyId</identity>
      <credential>ePE...-secretAccessKey</credential>
      <enableSignedUrlRedirect>true</enableSignedUrlRedirect>
      <signedUrlExpirySeconds>3600</signedUrlExpirySeconds>
      <testConnection>false</testConnection>
      <awsSdkV2>true</awsSdkV2> <!-- Supported from Artifactory version 7.125.4 -->
   </provider>
   <provider id="s3-shard5" type="state-aware-s3">
      <endpoint>http://s3.amazonaws.com</endpoint>
      <bucketName>data123</bucketName>
      <path>yonb3</path>
      <region>us-west-1</region>
      <provider.id>aws-s3</provider.id>
      <identity>AK...-accessKeyId</identity>
      <credential>ePE...-secretAccessKey</credential>
      <enableSignedUrlRedirect>true</enableSignedUrlRedirect>
      <signedUrlExpirySeconds>3600</signedUrlExpirySeconds>
      <testConnection>false</testConnection>
      <awsSdkV2>true</awsSdkV2> <!-- Supported from Artifactory version 7.125.4 --> 
   </provider>
</config>

Updated 2 days ago