Troubleshooting

Use this guide to diagnose and resolve common issues with Federated repositories.

This topic covers the following troubleshooting scenarios:

• Federation Members out of Sync
• Files Not Downloadable After Replication
• Replication Delays Between Members
• Clock Synchronization Issues
• Migration Does Not Complete
• RTFS Enable or Disable Fails on Some HA Nodes
• Certificate Errors with RTFS
• Exhausted Event Queues
• Receiving-Only Troubleshooting

Federation Members out of Sync

Symptom: The system triggers an out-of-sync notification for one or more Federation members.

Cause: Network failures or configuration changes to Federation members.

To resolve out-of-sync Federation members:

1. Push the updated configuration to the target Federation member using one of these methods:

   • REST API: Run the Synchronize Federated Member Configuration REST API:

     HTTP

     POST http://localhost:<PORT>/artifactory/api/federation/configSync/<REPOSITORY_KEY>

     Where:

     • <PORT>: The port number of your Artifactory instance
     • <REPOSITORY_KEY>: The key of the Federated repository

     For example:

     HTTP

     POST http://localhost:8081/artifactory/api/federation/configSync/federated-libs-release

   • Platform UI: In the Administration module, click Repositories. Use the filter to show Federated repositories, select the relevant member, and from the Actions menu, select Push Configuration.

2. If pushing configuration does not resolve the issue, try auto-healing (requires Artifactory 7.71.1+ on all JPDs).

3. If auto-healing does not resolve the issue, perform a Full Sync.

Files Not Downloadable After Replication

Symptom: A file that was replicated from one Federation member to another cannot be downloaded by users from the target member.

Cause: The metadata has not yet arrived at the target member. Binary content and metadata are replicated separately.

Resolution:

• Verify the file is displayed in the tree browser or the database of the target member.
• Starting from version 7.74, SaaS customers can confirm arrival by looking for the following log message:
  Text

  "Finished the handle mirror event for repo <REPO_KEY> and message: <MESSAGE>"

Replication Delays Between Members

Symptom: Significant synchronization delays between Federation members.

Cause: Insufficient workers, thread executors, network bandwidth, or database performance.

To resolve replication delays:

1. Check the Federation Dashboard or Legacy UI Monitoring to identify delayed members.
2. Review and tune the following in Best Practices:
   • Number of metadata workers (artifactory.persistentQueue.trigger.number.of.threads)
   • Number of thread executors (artifactory.federated.mirror.events.bulk.max.pools), which must be equal to or greater than the number of metadata workers
   • Number of binary import workers (numberOfRemoteImporters)
   • Timeout thresholds for heavy loads, including the socket timeout for larger repositories (artifactory.mirror.http.client.socket.timeout.mili, Artifactory 7.38.17+ self-managed)
3. Evaluate your network infrastructure between sites.

Clock Synchronization Issues

Symptom: Unexpected conflict resolution results. The last update doesn't take precedence as expected.

Cause: Server clocks of the machines running Federation members are not synchronized.

Resolution:

Synchronize the server clocks of all machines running Federation members. See Prerequisites.

Migration Does Not Complete

Symptom: The migration from legacy Federation to the Artifactory Federation Service (RTFS) does not complete as expected.

Cause: Repositories were not processed due to unexpected errors during migration.

To resolve a migration that does not complete:

1. Retry the migration using the --stateful-run option (requires Artifactory 7.117.1+):

   Bash

   jf federation-migrator migrate_rtfs <BASE_URL> <SCOPED_TOKEN> --stateful-run

   Where:

   • <BASE_URL>: The base URL of your JFrog Platform instance
   • <SCOPED_TOKEN>: A scoped access token with permission to perform the migration

   For example:

   Bash

   jf federation-migrator migrate_rtfs https://mycompany.jfrog.io <SCOPED_TOKEN> --stateful-run

2. If the issue persists, contact JFrog Support.

3. If needed, perform a rollback to the legacy Federation service.

RTFS Enable or Disable Fails on Some HA Nodes

Symptom: The API or jf federation-migrator returns an error, but RTFS already changed on one HA node.

Cause: The node that handled the request applied the change locally. Other HA nodes did not receive it. Artifactory does not roll back the successful node.

To resolve RTFS enable or disable failures on HA nodes:

1. Open artifactory-service-federation.log on the node that handled the request. The ERROR and WARN messages appear only on that node. They don't appear on the other HA nodes.
2. Don't compare GET /artifactory/api/federation/rtfs across nodes. That status comes from the shared database and looks the same everywhere.
3. Migrating to RTFS: The federation health-check job (default 80 seconds, artifactory.federation.health.check.job.sec) auto-reconnects a broken RTFS gRPC client. It does not enable RTFS on lagging nodes. If nodes remain out of alignment after the health-check interval, remediate those nodes manually (retry migration or restart lagging nodes), then confirm RTFS is healthy. See RTFS Database Requirements.
4. Disabling RTFS or returning to legacy: Manually restart every other HA node. Until you do, those nodes get no federation events and federation can stay slow. If a node still fails after restart, investigate that node (likely a local issue) and contact JFrog Support if needed.

For the full log message reference, see Enable or Disable RTFS in an HA Cluster.

Certificate Errors with RTFS

Symptom: Errors when using self-signed certificates or certificates signed by a custom CA with the Artifactory Federation Service.

Cause: Running an Artifactory version earlier than 7.133.4.

Resolution:

Upgrade to Artifactory version 7.133.4 or later. Self-signed certificates and certificates signed by a custom CA are supported starting from this version.

Exhausted Event Queues

Symptom: Federation moves into an error state after synchronization continues to fail.

Cause: The event queue has exceeded the maximum number of retry attempts (default: 6).

To resolve exhausted event queues:

1. If running Artifactory 7.71.1+ on all JPDs, the auto-healing mechanism should automatically reset failed events.
2. Perform manual recovery using the Federation Recovery REST API.
3. If events have been purged, perform a Full Sync.

Receiving-Only Troubleshooting

Symptom: Repeated log entries on Docker or OCI federated repositories configured as receiving-only. Busy repositories can generate thousands of these entries per day. The activity group is often DOCKER_TAG_ORPHAN_LAYER_CLEANUP or appears in federation mirror sync responses (Error while handling mirror sync response).

Cause: The Docker orphan-layer cleanup background job queues cleanup after tag overwrites (including overwrites replicated from a bidirectional sender). It retries every 90 seconds for up to ~2 hours per manifest. Receiving-only mode blocks direct deletes on the local member, so cleanup can't complete locally.

Log format depends on version:

• Before the RTFS-3995 fix: ERROR - Failed in beforeDelete due to exception with Direct delete on federated repository '<REPO_KEY>' is blocked: local member is in receiver mode
• After the RTFS-3995 fix: WARN - Delete not allowed: Direct delete on federated repository '<REPO_KEY>' is blocked: local member is in receiver mode

To address repeated cleanup log entries on receiving-only members:

1. Confirm federation health. The connection state should be HEALTHY and the member mode should be Receiving_Only.
2. Treat the messages as expected log noise, not a permissions issue, while federation is otherwise healthy.
3. Perform tag overwrites on the bidirectional (sender) member so orphan-layer cleanup and subsequent deletions originate from the hub.
4. Monitor spoke storage if tag churn is high.
5. Upgrade to the Artifactory release that includes RTFS-3995 to reduce log severity from ERROR to WARN.

Note: A future update will prevent the cleanup job from scheduling work on receiving-only members.

For the full behavior model, see Docker and OCI Repositories on Receiving-Only Members.

Frequently Asked Questions

This section provides answers to frequently asked questions.

Related Topics

• Federation Dashboard
• Monitoring Connection States
• Best Practices
• System Properties Reference
• Error Reference