Member States and Health

Every member in a federated repository has a state that reflects its current synchronization health. The state is continuously monitored by the Federation Service (RTFS) and displayed in the Federation Status dashboard. Understanding member states is the primary way administrators identify and resolve synchronization issues.

How States Work

Members transition through states as RTFS processes events, runs full sync operations, and performs auto-healing recovery. The state lifecycle follows a general pattern:

A new member starts in PENDING_FS (waiting for its initial full sync).
After a successful full sync, the member moves to HEALTHY.
If synchronization errors occur, the member may transition to an ERROR state.
The auto-healing mechanism attempts to recover error states automatically.
If auto-healing cannot resolve the issue, a full sync may be required, returning the member to PENDING_FS.

stateDiagram-v2
    [*] --> PENDING_FS: New member
    PENDING_FS --> FULL_SYNC_PREPARING: Full sync starts
    FULL_SYNC_PREPARING --> FULL_SYNC_RUNNING: File list ready
    FULL_SYNC_RUNNING --> HEALTHY: Sync complete
    HEALTHY --> ERROR: Sync error
    ERROR --> HEALTHY: Auto-healing
    ERROR --> PENDING_FS: Full sync required
    HEALTHY --> READ_ONLY: Receiving-only mode enabled
    READ_ONLY --> HEALTHY: Bidirectional sync restored

    classDef component fill:#41b375,stroke:#389b65,color:#fff
    class PENDING_FS,FULL_SYNC_PREPARING,FULL_SYNC_RUNNING,HEALTHY,READ_ONLY component

State Severity

When the Federation dashboard displays the status of a repository, it shows the most severe state among all its members. States are ordered by severity from lowest (healthiest) to highest.

Severity	State	Meaning
0	READ_ONLY	Receiver-only mode. Events are received but not sent.
1	HEALTHY	Fully synchronized. Events are processing normally.
3	FULL_SYNC_RUNNING	Full synchronization in progress.
3	FULL_SYNC_PREPARING	Preparing file lists for full sync (shown on the dashboard as Full Sync Running).
4	PENDING_FS	Waiting for full sync to begin.
5	ERROR_GENERIC	General synchronization error.
5	ERROR_INACTIVE	Target member unreachable (shown on the dashboard as Error).
6	ERROR_OUT_OF_SYNC	Queued events are purged before sync. Full sync is required.

For the complete list of all member states, queue states, binary task states, heartbeat statuses, and full sync phases, see Member State Reference.

Disabled Members

An administrator can disable a Federation member using the sync toggle in the Federation tab. Disabling a member:

Stops all event processing for that member.
Doesn't remove the member from the Federation topology.
Can be re-enabled at any time, after which a Full Sync may be required to reconcile missed events.

DISABLED isn't a member state enum value. It is determined by the member's enabled flag and displayed as a distinct status in the dashboard.

Auto-Healing

The Federation Service includes an auto-healing mechanism that periodically checks for members in error states and attempts recovery. Auto-healing can recover ERROR_GENERIC and ERROR_INACTIVE states automatically. ERROR_OUT_OF_SYNC typically requires a full sync.

For details on auto-healing configuration and behavior, see Recovery and Auto-Healing.

Frequently Asked Questions

This section provides answers to frequently asked questions about Federation member states and health.

plusFAQs

Q: What does PENDING_FS mean for a Federation member?

A: PENDING_FS means the member is waiting for a full sync to begin. Once the full sync completes successfully, the member transitions to HEALTHY.

Q: How does a member recover from an ERROR state?

A: The auto-healing mechanism periodically checks for members in error states and attempts recovery automatically. ERROR_OUT_OF_SYNC typically requires a full sync to recover.

Q: What happens when I disable a Federation member?

A: Disabling a member stops all event processing for it, but it isn't removed from the Federation topology. You can re-enable it at any time, though a Full Sync may be required to reconcile missed events.

Q: Which member state is the most severe?

A: ERROR_OUT_OF_SYNC is the most severe state. It means queued events were purged before syncing, so a full sync is required. See State Severity for the complete ranking.

Q: Why isn't DISABLED listed as a member state?

A: DISABLED isn't a member state enum value. It's determined by the member's enabled flag and displayed as a distinct status in the Federation dashboard.