Recovery and Auto-Healing

Artifactory features an auto-healing mechanism that ensures the reliability and consistency of Federated repositories by performing recovery actions at regular intervals.

When Recovery Is Needed

In certain cases, it may not be possible to maintain near real-time synchronization of all artifact events (create, update, delete) and binary tasks among Federation members. Examples include short-term networking issues between the JPDs, Artifactory upgrades, a user-initiated synchronization pause, and so on.

If synchronization continues to fail after reaching the maximum number of retry events, event sync is paused and the Federation moves into an error state. One way to recover is to call the Full Sync API, but this can be time-consuming if the Federated repositories contain a large number of artifacts.

How Auto-Healing Works

Auto-healing performs the following recovery actions:

• Recovering exhausted queues: The mechanism checks for queues that have exceeded the maximum number of attempts to send events to other Federation members. Such queues are considered exhausted. The auto-healing mechanism resets the failed events automatically, enabling the system to retry synchronization with the target mirror.
• Recovering exhausted binary tasks: In cases where binary tasks have reached their maximum retry attempts (default: 10 retries), the auto-healing mechanism invokes the Replay Failed Binary Tasks API, resetting the retry count and enabling the system to retry the task.

flowchart TD
    check["Auto-healing job runs\n(every interval)"]

    check --> queueCheck{"Exhausted event queue?"}
    queueCheck -->|Yes| resetQueue["Reset failed events"]
    resetQueue --> retry["Retry synchronization"]

    check --> taskCheck{"Binary task exceeded\nmax retries?"}
    taskCheck -->|Yes| resetTask["Reset retry count"]
    resetTask --> retryTransfer["Retry binary transfer"]

    check --> driftCheck["Configuration Sync job\n(hourly)"]
    driftCheck --> driftFound{"Configuration\ndrift detected?"}
    driftFound -->|Yes| markOOS["Flag member\nERROR_OUT_OF_SYNC"]

    retry --> healthy["Member is Healthy"]
    retryTransfer --> healthy
    driftFound -->|No| healthy
    markOOS --> fullSyncNote["Full Sync may be\nrequired to recover"]

    classDef component fill:#41b375,stroke:#389b65,stroke-width:1px,color:#fff;
    classDef secondary fill:#ffffff,stroke:#41b375,stroke-width:1.5px,color:#41b375;
    class check,resetQueue,retry,resetTask,retryTransfer,driftCheck,healthy component;
    class queueCheck,taskCheck,driftFound,markOOS,fullSyncNote secondary;

    linkStyle default stroke:#8e9aaf,stroke-width:1.5px;

Email Notifications

All administrators registered for Artifactory's mail service receive notifications when auto-healing takes place. Example notification:

"Auto Healing" task has recognized a mirror in an exhausted state:
'http://localhost:11552/artifactory/generic-fed-9ac0b669-5760-4ff3-a209-c6b8c7826928'
-> 'http://localhost:55295/artifactory/generic-fed-9ac0b669-5760-4ff3-a209-c6b8c7826928'.
Recovery attempt is now in progress...

System Properties

Federation recovery and auto-healing are controlled using the following properties in artifactory.system.properties:

Configuration Drift Detection

In addition to event queue and binary task recovery, RTFS continuously monitors for configuration drift between members. The Configuration Sync job (rtfs.repo.config.sync.job.interval.cron, default: hourly) performs three types of checks:

1. RTFS-to-Artifactory alignment: Validates that the list of Federated repositories in RTFS matches the repositories in the local Artifactory. Missing configurations are imported automatically. Mismatches are flagged as ERROR_OUT_OF_SYNC.
2. Local state sync: Ensures that member enabled/disabled flags and tenant base URLs in RTFS match the state reported by the local Artifactory and Access service.
3. Remote member config sync: Compares repository configuration on remote members against the local state, detecting and flagging inconsistencies.

Configuration drift metrics (jfrtfs_inconsistent_repo_configs_to_remote_rtfs_count, jfrtfs_inconsistent_repo_configs_to_local_rt_count) are available in the Open Metrics Reference.

Manual Recovery Using the REST API

Use the Federation Recovery REST API to perform recovery manually. This API can be used when auto-healing has been disabled or when you want to perform recovery immediately without waiting for the auto-healing interval.

Disabled-by-System Repositories and Metadata Negotiation

The standalone Artifactory Federation Service (RTFS) improves the federation flow by decoupling from metadata negotiation to reduce protocol overhead and simplify the connection handshake. The negotiation code is no longer part of RTFS. Federation continues to operate as expected without that step.

Under legacy Federation, metadata negotiation could mark repositories DISABLED_BY_SYSTEM when negotiation failed. That status is stored in the database and may persist after you migrate to RTFS or upgrade to newer self-hosted versions (for example, 7.133.x and later). It does not mean RTFS is disabling federation because of a new negotiation failure.

If customers report repositories disabled by system after RTFS migration:

1. Confirm migration to RTFS completed successfully.
2. Review member status in Federation Dashboard or RTFS monitoring APIs.
3. Try auto-healing, or use the Federation Recovery REST API where applicable.
4. If repositories remain disabled by system, a Full Sync is unlikely to clear this state. The repository status value in the database may need to be corrected manually. Contact JFrog Support for guidance before changing database values.

The federated.negotiation.enabled property applies to legacy Federation only and is not used with RTFS.

Frequently Asked Questions

This section provides answers to frequently asked questions.

Related Topics

• Trigger a Full Sync
• Federation Dashboard
• System Properties Reference