Troubleshooting

Use this guide to diagnose and resolve common issues with Federated repositories.

This topic covers the following troubleshooting scenarios:

• Federation Members out of Sync
• Files Not Downloadable After Replication
• Replication Delays Between Members
• Clock Synchronization Issues
• Migration Does Not Complete
• RTFS Enable or Disable Fails on Some HA Nodes
• Certificate Errors with RTFS
• Exhausted Event Queues
• Receiving-Only Troubleshooting

Federation Members out of Sync

Symptom: The system triggers an out-of-sync notification for one or more Federation members.

Cause: Network failures or configuration changes to Federation members.

To resolve out-of-sync Federation members:

1. Push the updated configuration to the target Federation member using one of these methods:

   • REST API: Run the Synchronize Federated Member Configuration REST API:

     HTTP

     POST http://localhost:<PORT>/artifactory/api/federation/configSync/<REPOSITORY_KEY>

     Where:

     • <PORT>: The port number of your Artifactory instance
     • <REPOSITORY_KEY>: The key of the Federated repository

     For example:

     HTTP

     POST http://localhost:8081/artifactory/api/federation/configSync/federated-libs-release

   • Platform UI: In the Administration module, click Repositories. Use the filter to show Federated repositories, select the relevant member, and from the Actions menu, select Push Configuration.

2. If pushing configuration does not resolve the issue, try auto-healing (requires Artifactory 7.71.1+ on all JPDs).

3. If auto-healing does not resolve the issue, perform a Full Sync.

Files Not Downloadable After Replication

Symptom: A file that was replicated from one Federation member to another cannot be downloaded by users from the target member.

Cause: The metadata has not yet arrived at the target member. Binary content and metadata are replicated separately.

Resolution:

• Verify the file is displayed in the tree browser or the database of the target member.
• Starting from version 7.74, SaaS customers can confirm arrival by looking for the following log message:
  Text

  "Finished the handle mirror event for repo <REPO_KEY> and message: <MESSAGE>"

Replication Delays Between Members

Symptom: Significant synchronization delays between Federation members.

Cause: Insufficient workers, thread executors, network bandwidth, or database performance.

To resolve replication delays:

1. Check the Federation Dashboard or Legacy UI Monitoring to identify delayed members.
2. Review and tune the following in Best Practices:
   • Number of metadata workers (artifactory.persistentQueue.trigger.number.of.threads)
   • Number of thread executors (artifactory.federated.mirror.events.bulk.max.pools), which must be equal to or greater than the number of metadata workers
   • Number of binary import workers (numberOfRemoteImporters)
   • Timeout thresholds for heavy loads, including the socket timeout for larger repositories (artifactory.mirror.http.client.socket.timeout.mili, Artifactory 7.38.17+ self-managed)
3. Evaluate your network infrastructure between sites.

Clock Synchronization Issues

Symptom: Unexpected conflict resolution results. The last update doesn't take precedence as expected.

Cause: Server clocks of the machines running Federation members are not synchronized.

Resolution:

Synchronize the server clocks of all machines running Federation members. See Prerequisites.

Migration Does Not Complete

Symptom: The migration from legacy Federation to the Artifactory Federation Service (RTFS) does not complete as expected.

Cause: Repositories were not processed due to unexpected errors during migration.

To resolve a migration that does not complete:

1. Retry the migration using the --stateful-run option (requires Artifactory 7.117.1+):

   Bash

   jf federation-migrator migrate_rtfs <BASE_URL> <SCOPED_TOKEN> --stateful-run

   Where:

   • <BASE_URL>: The base URL of your JFrog Platform instance
   • <SCOPED_TOKEN>: A scoped access token with permission to perform the migration

   For example:

   Bash

   jf federation-migrator migrate_rtfs https://mycompany.jfrog.io <SCOPED_TOKEN> --stateful-run

2. If the issue persists, contact JFrog Support.

3. If needed, perform a rollback to the legacy Federation service.

RTFS Enable or Disable Fails on Some HA Nodes

Symptom: The API or jf federation-migrator returns an error, but RTFS already changed on one HA node.

Cause: The node that handled the request applied the change locally. Other HA nodes did not receive it. Artifactory does not roll back the successful node.

To resolve RTFS enable or disable failures on HA nodes:

1. Open artifactory-service-federation.log on the node that handled the request. The ERROR and WARN messages appear only on that node. They don't appear on the other HA nodes.
2. Don't compare GET /artifactory/api/federation/rtfs across nodes. That status comes from the shared database and looks the same everywhere.
3. Migrating to RTFS: The federation health-check job (default 80 seconds, artifactory.federation.health.check.job.sec) auto-reconnects a broken RTFS gRPC client. It does not enable RTFS on lagging nodes. If nodes remain out of alignment after the health-check interval, remediate those nodes manually (retry migration or restart lagging nodes), then confirm RTFS is healthy. See RTFS Database Requirements.
4. Disabling RTFS or returning to legacy: Manually restart every other HA node. Until you do, those nodes get no federation events and federation can stay slow. If a node still fails after restart, investigate that node (likely a local issue) and contact JFrog Support if needed.

For the full log message reference, see Enable or Disable RTFS in an HA Cluster.

Certificate Errors with RTFS

Symptom: Errors when using self-signed certificates or certificates signed by a custom CA with the Artifactory Federation Service.

Cause: Running an Artifactory version earlier than 7.133.4.

Resolution:

Upgrade to Artifactory version 7.133.4 or later. Self-signed certificates and certificates signed by a custom CA are supported starting from this version.

Exhausted Event Queues

Symptom: Federation moves into an error state after synchronization continues to fail.

Cause: The event queue has exceeded the maximum number of retry attempts (default: 6).

To resolve exhausted event queues:

1. If running Artifactory 7.71.1+ on all JPDs, the auto-healing mechanism should automatically reset failed events.
2. Perform manual recovery using the Federation Recovery REST API.
3. If events have been purged, perform a Full Sync.

Receiving-Only Troubleshooting

Content Differs Between the Source and a Receiving-Only Member

Symptom: A receiving-only member holds artifacts that don't exist on the source, is missing artifacts that exist on the source, or serves a different binary for the same artifact path. Federation reports the member as HEALTHY and synchronized.

Cause: Someone uploaded, deleted, or modified artifacts directly on the receiving-only member. A receiving-only member never sends events to the Federation, so these local changes stay on that JPD and are never reflected on the source. Federation monitoring doesn't flag the difference, because Federation events themselves are flowing normally.

To resolve content differences on a receiving-only member:

1. Identify the local changes by comparing the receiving-only member with the source. Check the Artifactory request log on the receiving-only JPD for deploy and delete requests against the Federated repository that didn't originate from Federation.
2. If the local artifacts should exist across the Federation, deploy them on the sending member so that Federation distributes them to all members.
3. Run a Full Sync from the sending member to reconcile the receiving-only member against the source. Full Sync may remove locally added artifacts and restore locally deleted ones.
4. Prevent recurrence by restricting write permissions on the Federated repository on the receiving-only JPD so that only Federation introduces changes.

Artifactory doesn't block local writes on a receiving-only member. For more information, see Best Practice: Keep Receiving-Only Members Read-Only.

Blocked-Delete Log Entries on Docker or OCI Repositories

Symptom: Repeated log entries on Docker or OCI Federated repositories configured as receiving-only. Busy repositories can generate thousands of these entries per day. The activity group is often DOCKER_TAG_ORPHAN_LAYER_CLEANUP, or the entries appear in Federation mirror sync responses (Error while handling mirror sync response).

Failed in beforeDelete due to exception
Direct delete on federated repository '<REPO_KEY>' is blocked: local member is in receiver mode

Cause: In earlier Artifactory versions, receiving-only mode blocked direct deletes on the local member. The Docker orphan-layer cleanup job queues cleanup after tag overwrites, including overwrites replicated from the sender, and retries every 90 seconds for up to about two hours per manifest. Each attempt failed against the delete block.

To resolve blocked-delete log entries:

1. Upgrade to an Artifactory version that no longer blocks local deletes on receiving-only members. Orphan-layer cleanup then completes on the receiving-only member as it does on any other member, and the entries stop.
2. Until you upgrade, treat the entries as expected log noise rather than a permissions problem, as long as the connection state is HEALTHY and the member mode is Receiving_Only.
3. Monitor storage on spoke JPDs with high tag churn, because orphan layers persist until cleanup can run.

For the full behavior model, see Docker and OCI Repositories on Receiving-Only Members.

Frequently Asked Questions

This section provides answers to frequently asked questions.

Related Topics

• Federation Dashboard
• Monitoring Connection States
• Best Practices
• System Properties Reference
• Error Reference