Configure Synchronization in Access Federation

Configure which security entities an Access service should synchronize to its target services. At the same time, you can optionally set a number of parameters that govern the synchronization process. The configuration parameters are specified in the YAML configuration file that is uploaded to the different Access services.

📘
Note
Federated Repositories use the dedicated URL named federatedRepoUrlBase which is only for Federated Repositories and cannot be used for geosynchronization of other features. To configure geo-synchronization, you will need to set it up through the Access YAML Configuration.

Access Federation Parameters

The following sections in the YAML configuration file govern access federation:

inbound: The inbound parameters map service ids from a source Access service to service ids to which the receiving Access service is attached. To determine the service ID of an Artifactory service, use the JFConnect GET Service ID REST API endpoint.
outbound: The outbound parameters define parameters that govern the synchronization of security entities outwardly from the source service to a target service.

The format of the federation YAML file parameters is as follows with the default values.

federation:
  outbound:
    entity-types-to-sync:
    - users
    - groups
    - permissions
    - tokens
    exclude-users:
    - <username 1>
    - <username 2>
    buffer-wait-millis: 30000
    buffer-max-size: 500
    consider-stale-hours: 168
    maximum-future-time-diff-millis: 60000
    timeout-millis: 3000
    number-of-retries: 3
    max-stored-events: -1
    auto-full-sync-recovered-servers: false
 
    servers:
    - name: "<target-name>"
      url: "<target-url>"
    - name: "<target-name>"
      url: "<target-url>"
    ...
    - name: "<target-name>"
      url: "<target-url>"
      permission-filters: # include and exclude patterns filtering permissions to federate per server
          include-patterns:
            - ".*a.*"
            - ".*b.*"
          exclude-patterns:
            - ".*aa.*"

  inbound:
    service-id-mapping:
    - from: <Source Artifactory | Xray | Distribution Service ID1>
      to: <Target Artifactory | Xray | Distribution Service ID1>
    - from: <Source Artifactory | Xray | Distribution Service ID2>
      to: <Target Artifactory | Xray | Distribution Service ID2>
    ...
    - from: <Source Artifactory | Xray | Distribution Service IDn>
      to: <Target Artifactory | Xray | Distribution Service IDn>
    allow-partial-entity-sync: true

Parameter	Values
entity-types-to-sync	Specifies which entities should be synchronized to the target servers configured. Possible values are: users groups permissions tokens 📘 Implicit synchronization To maintain consistency in the hierarchical relationship of entities between the different Access services in a circle of trust, the following rules apply to the synchronization of security entities: For any group that is synchronized from a source Access service to a target, all the users in that group are also synchronized. For any permission that is synchronized from a source Access service to a target all the users and groups associated with that permission are also synchronized. 📘 LDAP, SAML, Crowd etc. When using 3rd party authentication providers such as LDAP, SAML, and Crowd, only the Users defined in these providers can be federated, NOT configurations. Note also that for federation to work, these users must be created AFTER logging in to Artifactory. 📘 Cases where the entity is ignored The entity is ignored (that is, not synchronized) in the following cases: One of the groups associated with the federated user on the source node is not defined on the target node. In such cases, the user will fail to federate to the target node. A user or group associated with a permission on the source node is not defined on the target node. In such cases, the permission will fail to federate to the target node due to the missing user or group.
exclude-users	Optional Specific users that should be excluded from synchronization
buffer-wait-millis	Optional [Default: 30000] The time (ms) between synchronization attempts. For example: `buffer-wait-millis = 30000` means that the source Access service will wait 30 seconds to attempt to initiate a sync with the target Access service. Note: this amount of time is a best effort, and will not necessarily be achieved.
buffer-max-size	Optional [Default: 500] The maximum number of entities that can be accumulated between synchronization attempts. For Example: buffer-max-size = 500 means that whenever 500 security entity updates have been accumulated, the source Access service will attempt to sync with the target Access.
consider-stale-hours	Optional [Default: 168 (one week)] The length of time a receiving server can fail to respond or respond with an error before it is considered to be "stale" and unable to be synchronized with updates. Once a server is deemed to be "stale", it can only be brought back into operation using the REST API endpoint described in Updating a Stale Server.
maximum-future-time-diff-millis	Optional [Default: 60000 If an Access service is updated, and another update is subsequently synchronized over to it, the Access service will only accept the subsequent change if its timestamp is at least `maximum-future-time-diff-millis` later than the previous update. For more details, please refer to Handling Synchronization Conflicts.
timeout-millis	Optional [Default: 3000] The length of time after synchronizing an update that an Access service will wait for acknowledgment from the receiving Access service before retrying to synchronize the update. If no acknowledgment is received within this period of time, the sending server will retry synchronizing the update to the target server the number of times specified in the `number-of-retries` parameter.
number-of-retries	Optional [Default: 3] The number of times an Access service will attempt to synchronize an update to a distant service until. If an acknowledgment is not received from the target server after this number of retries the synchronization is deemed to have failed.
max-stored-events	Optional [Default: -1] The threshold number of events stored in the database. By default, the value is -1, which means that unlimited number of events are stored
auto-full-sync-recovered-servers	Optional [Default: false] Whether to automatically fully synchronize stale services.
servers	The logical name and URL of target Access services to which this Access service can synchronize entities
service-id-mapping	When a service such as Mission Control, Xray, or Distribution selects an Artifactory instance as its authentication provider, that service's service ID is automatically registered with the corresponding Access service. This field maps the service IDs of services being synchronized over to this Access service. The mapping is automatically generated by Access and need not be entered manually except for a few limitations described below. For a synchronization target, the "from" and "to" fields define the mapping of service IDs. The from: field should specify the service type with a wildcard character. For example, for Artifactory, specify `jfrt@, f`or Xray, specify `jfxr@.` The to: field should specify the specific service to which the source service (in the from: field) is being mapped. 📘 Limitations An access service can only serve one service of each supported type, i.e., one each of Artifactory, Mission Control, Xray, and Distribution. If you remove a registered service (for example, by changing the Artifactory that is used as the authentication provider), you need to manually remove the corresponding service ID registered with Access before you can add a new service of the same type. From version 5.4, the Artifactory service ID format changed. If you need to map a service ID whose format is from below Artifactory version 5.4 to a format that is from Artifactory version 5.4 and above, or vice versa, then you have to manually write the service ID. When upgrading Artifactory, the service ID format remains that of the originally installed version. For example, if the original version of Artifactory that you installed was version 5.2, and you have upgraded through the versions to version 5.11, the service ID format of your Artifactory installation will still be that of version 5.2.
allow-partial-entity-sync	Optional [Default: false] Whether to allow advanced conflict resolution to merge and resolve conflicted entities.
permission-filter	Optional You can select to filter permissions with an `include-pattern` or `exclude-pattern` filter that filters federated permissions per server.

📘
Watch your whitespace
The YAML file format is sensitive to whitespace, so you need to ensure that all required whitespace characters are in place. If the YAML format is invalid, permissions will not be synchronized successfully and you will not receive error messages.

Handle Synchronization Conflicts in Access Federation

Depending on the synchronization topology you have configured, an Access server may receive multiple updates from different sources at short time intervals. Successive updates may even conflict. For example, in a full mesh topology, an Access server may have a user's password updated to one value sync'ed over from one source, but that same user's password may be updated to a different value synced over from another source.

To resolve such conflicts, an Access server will finalize upon the update that has the latest timestamp. However, since synchronization of server clocks may not always be completely accurate, there could still be an inconsistency when an Access server receives conflicting updates. To overcome the possible inaccuracy in the synchronization of server clocks, an Access server uses a time buffer specified in the maximum-future-time-diff-millis parameter. Access will not update an entity if the update arrives sooner than maximum-future-time-diff-millis milliseconds from the previous update.

Consider the following example using the default value of maximum-future-time-diff-millis = 60000.

On Access Service 1, the password for User1 is changed to "abc" at time 09:02:00:000.

However, at approximately the same time, User1 is synchronized over to Access Service 1 from Access Service 2 which has set the password to be "def".

If the change synchronized over from Access Service 2 has a timestamp that is between 09:02:00:000 and 09:02:59:999, then User1's password will be "abc" because the password synchronized from Access Service 2 is ignored.

If the change synchronized over from Access Service 2 has a timestamp that is 09:03:00:000 or later, then User1's password will be "def" because the password synchronized from Access Service 2 is deemed to be later than the one set directly on Access Server 1 so it will be accepted.

Conflict Resolution in Access Federation

Starting from Artifactory 7.77.3, Access Federation has implemented advanced conflict resolution methodology.

Previously, when the source JPD publishes a full entity to the target server, the target server replaced its own entity with the new entity from the source JPD without managing any conflicts. If a user is added to group A in node A and to group B in node B at the same time, the result might not be as expected as the groups might get overridden upon sync between the nodes. Now, only the delta of changes are exchanged between the JPDs without overriding the data.

Set the property, federation.inbound.allow-partial-entity-sync, as true in the Access Configuration YAML file to enable conflict resolution. The default is false.

federation:
   .
   .
   inbound:
    .
    .
    allow-partial-entity-sync: true

Access Federation stores any changes done to an entity in a JPD in the federation buffer. When the synchronization between JPDs occur, if the same entity has been updated in different JPDs, the timestamp of the entity is checked against the entry in the database. If the entity has been modified, the changes that has occurred in the entity between the JPD is merged and inserted to the database. The updated entity is then synchronized to the JPDs.

Conflict resolution kicks in during the following cases.

Group change
Password change
Permission change.

Conflict resolution is not applied in cases of deleted entities. You may encounter a race condition when you delete a permission in one JPD and update the same permission in another JPD at the same time.

Conflict Resolution Example

Consider an Access Federation setup with two JPDs - JPD A and JPD B. The same user is updated at both JPD A and JPD B.

The user is added to group A in JPD A and a sync event is added to the federation buffer of JPD A, waiting to be published.
The same user is added to group B in JPD B and a sync event is added to the federation buffer of JPD B for the user, before JPD A has synced its changes.
JPD B publishes changes to JPD A.
JPD A receives the published changes from JPD B. JPD A updates received entities by matching the modified timestamp before and after the published changes.
If there is modification, the entity is fetched again from the database, conflicts are merged and the entity is updated in the database.
After JPD A publishes the changes, the previous steps are repeated on JPD B.

Synchronize Data in Access Federation

Once your access services are configured with their corresponding YAML files, you are ready to synchronize data from the source to the target Access services.

There are two way to synchronize data:

Invoke a Full Broadcast from the Access Federation Menu to each federation source (available from 7.42.1)
Apply a REST API Endpoint on the Source Service to Invoke a Full Broadcast

Invoke a Full Broadcast from the Access Federation Menu

The purpose of a full broadcast is when you are first setting up a federation. You can also use this functionality to resolve a "stale-server" status.

📘
Resource-heavy Operation
Because a "full broadcast" is resource-heavy, you should avoid repeated Broadcast clicks whenever possible.

To trigger a full broadcast from a specific Access Federation source, navigate to Topology | Access Federation.
Select the source you wish to broadcast from, and from three-dot menu on the right, select Broadcast.

You are prompted to confirm the full broadcast and resync.
Click OK to begin the broadcast or Cancel.

Apply a REST API Endpoint on the Source Service to Invoke a Full Broadcast

Description: Synchronize data from a source to a target Access service

Notes: Requires Artifactory Enterprise Plus

Since: 5.11

Security: Requires an admin user.

Usage: PUT /api/v1/system/federation/[targetServer]/full_broadcast

Sample Usage:

curl -uaccess-admin:password http://localhost:8040/access/api/v1/system/federation/access-2/full_broadcast -XPUT

📘
The access-admin user
REST API calls on an Access service need to be authenticated using the Admin user of the Access service on which you are making the call.
Don't confuse this with any Admin user defined in Artifactory. Those can be used to make REST API calls on Artifactory. This call is made on the Access service to revive it after a long period of inactivity, not on Artifactory.

The following example shows how you can configure a 1-way replication of permissions by synchronizing permissions settings from a source Access service in your circle of trust to a target.

📘
Map Permission Targets with repositories of different types
By default, federation only works for permission targets where the permissions are applied on resources of the same type (local-to-local and remote-to-remote).
Mapping repositories with different types (remote-to-local or local-to-remote) requires editing the Artifactory system properties.
To enable local permissions on remote repositories:
artifactory.apply.local.repos.permissions.on.remote.repos=true
To enable remote permissions on local repositories:
artifactory.apply.remote.repos.permissions.on.local.repos=true

Step 1: Obtain Service IDs

The first step is to obtain the service IDs of the Artifactory services involved in the permission synchronization relationship. Note that we need the Artifactory service IDs, although, in fact, the permissions are managed and synchronized by the corresponding Access services.

For each of the source and target Artifactory services run the Get Service ID REST API endpoint. In this example, we are assuming you are applying this call directly on the machine that the Artifactory service is running (i.e. on localhost):

curl -uadmin:password -v http://localhost:8081/artifactory/api/system/service_id
  
On the source service, the response is:
jfrt@01c9s2y40bcs960sdxwy4302yc
  
On the target service, the response is:
jfrt@01c9wadx1fvh3d1egg7wne0e0g

✅
Plan and map out your synchronization
We strongly recommend having the service ID of each Artifactory cluster involved ready for use, and creating a diagram that illustrates the permission synchronization topology you are implementing while assigning a logical name for service ID. In this example, we will assign the following logical names to the source and target service IDs:
access-1: jfrt@01c9s2y40bcs960sdxwy4302yc
access-2: jfrt@01c9wadx1fvh3d1egg7wne0e0g

Step 2: Inbound Configuration

Load the following YAML file on the target Access service to create the service ID mapping as described in Uploading the YAML File. Note that since the target service is receiving permissions (and not synchronizing anything out), there is no need to include the entity-types-to-sync field.

federation:\n
    inbound:\n 
        service-id-mapping:\n
            - from: jfrt@*\n
            - to: jfrt@01c9wadx1fvh3d1egg7wne0e0g\n

Don't forget to restart the Access service after placing the YAML file in the designated location.

Step 3: Outbound Configuration

📘
Outbound configuration goes on the source Access instance
Note that the Outbound configuration should be applied to the source Access instance from which you are synchronizing security entities.

In this example, we want to synchronize all security entities using default values except for the following settings that we want to modify:

timeout-millis = 4000

number-of-retries: 5

Load the following YAML file on the source Access service as described in Uploading the YAML file to specify the above parameters and set access-2 as the synchronization target for access-1 (this example assumes the access-2 service IP address is 35.230.92.254):

federation:\n
    outbound:\n
        timeout-millis: 3000\n
        number-of-retries: 3\n
        servers:\n
            name : "access-2"\n
            url : "http://35.230.92.254:8040/access"\n

Don't forget to restart the Access service after placing the YAML file in the designated location.

Step 4: Invoke Synchronization in Access Federation

Once your federation is setup, you are ready to synchronize data by applying the Federation REST API endpoint described in Synchronizing Data on access-1:

curl -uaccess-admin:password http://localhost:8040/access/api/v1/system/federation/access-2/full_broadcast -XPUT