Replay Failed Binary Tasks

When binary tasks reach their maximum retry count, they enter an EXHAUSTED_SUBMISSION or EXHAUSTED_STORAGE state and stop retrying automatically. You can use the Replay Failed Binary Tasks API to reset the retry count and re-queue failed tasks for processing.

This topic covers the following tasks:

List Failed Binary Tasks
Replay Failed Tasks
Delete Failed Tasks

Who can perform this task: Platform Administrators

Prerequisites:

Admin access to the JFrog Platform
The root cause of the failure has been identified and resolved (network issue, storage problem, member unavailability)

When to Use

Replay failed binary tasks when:

The Federation Dashboard or Monitoring Binaries Tasks view shows a high count of Failed binary tasks.
Member state reference shows EXHAUSTED_SUBMISSION or EXHAUSTED_STORAGE tasks.
Auto-healing has not resolved the failures (for example, if the root cause was a prolonged outage that has since been resolved).

List Failed Binary Tasks

Before replaying, review the list of failed tasks to understand the scope.

To list failed binary tasks:

Send the following request:

GET /api/federation/binaries/failures/list?repoKey=<REPO_KEY>

Where:

<REPO_KEY>: The Federated repository key.

Example:

curl -u admin:password \
  "https://myartifactory.example.com/artifactory/api/federation/binaries/failures/list?repoKey=libs-release-federated"

Replay Failed Tasks

To replay failed binary tasks:

Reset the retry count and re-queue failed binary tasks for processing:

PUT /api/federation/binaries/failures/replay?repoKey=<REPO_KEY>&sha1=<SHA1>

Where:

<REPO_KEY>: The Federated repository key.
<SHA1>: (Optional) Specific binary SHA1 to replay. If omitted, all failed tasks for the repository are replayed.

Replay all failed tasks for a repository:

curl -X PUT -u admin:password \
  "https://myartifactory.example.com/artifactory/api/federation/binaries/failures/replay?repoKey=libs-release-federated"

Replay a specific failed task:

curl -X PUT -u admin:password \
  "https://myartifactory.example.com/artifactory/api/federation/binaries/failures/replay?repoKey=libs-release-federated&sha1=da39a3ee5e6b4b0d3255bfef95601890afd80709"

Delete Failed Tasks

To delete failed binary tasks:

Remove failed tasks without replaying (for example, if the binary is no longer needed):
```
DELETE /api/federation/binaries/failures/delete?repoKey=<REPO_KEY>&sha1=<SHA1>
```
Where:
- <REPO_KEY>: The Federated repository key.
- <SHA1>: (Optional) Specific binary SHA1 to delete. If omitted, all failed tasks for the repository are deleted.
Example:
```
curl -X DELETE -u admin:password \
  "https://myartifactory.example.com/artifactory/api/federation/binaries/failures/delete?repoKey=libs-release-federated"
```

Verification

After replaying:

Monitor the Binaries Tasks view. The In Progress count should increase temporarily as replayed tasks are processed.
The Failed count should decrease as tasks complete successfully.
If tasks fail again after replay, investigate the root cause before replaying a second time.

Auto-Healing and Binary Tasks

The auto-healing mechanism automatically replays exhausted binary tasks at configurable intervals. Manual replay is needed only when:

Auto-healing is disabled.
You want immediate recovery without waiting for the next auto-healing cycle.
You want to replay tasks for a specific repository or binary.

Related Properties

Property	Default	Description
`rtfs.binary.tasks.max.retry.per.task`	10	Maximum retries before a task is marked as exhausted.
`rtfs.binary.tasks.tenant.penalty.enabled`	true	Enable tenant penalty to throttle failing tenants.
`rtfs.binary.tasks.tenant.penalty.duration`	PT1S	Penalty duration applied to failing tenants.

Frequently Asked Questions

This section provides answers to frequently asked questions.

plusFAQs

Q: When should I replay failed binary tasks manually instead of waiting for auto-healing?

A: Replay manually when auto-healing is disabled, when you want immediate recovery without waiting for the next cycle, or when you need to target a specific repository or binary.

Q: Can I replay a specific binary instead of all failed tasks?

A: Yes. Pass the optional sha1 parameter with the binary's SHA1 to replay just that task. If you omit it, all failed tasks for the repository are replayed.

Q: What happens if I delete a failed task instead of replaying it?

A: The task is removed from the failure queue without retrying. Use this when the binary is no longer needed rather than when you want it recovered.

Q: How many times does a binary task retry before it's marked exhausted?

A: Up to rtfs.binary.tasks.max.retry.per.task retries, which defaults to 10. After that, the task enters an EXHAUSTED_SUBMISSION or EXHAUSTED_STORAGE state and stops retrying automatically.