Load Healer Automatic Load Protection
Protect Artifactory from overload by monitoring worker-thread consumption and throttling offending activities with HTTP 429 when the node enters an outage state.
Since: Available in Artifactory 7.161.
Load Healer protects JFrog Artifactory from overload caused by a single activity monopolizing the available worker threads. Most Artifactory outages occur when the working thread pool is exhausted and new requests are rejected or time out.
Load Healer continuously monitors how worker threads are consumed and identifies workloads or activities that threaten to overload the node. When Load Healer triggers protection, it limits the offending activity to a safe portion of the available worker threads, so other requests can continue running uninterrupted.
To do this, Load Healer classifies incoming requests into activities, such as downloads for a package type, uploads to a repository, calls to a specific API, or traffic from a single user. A single request can belong to multiple activities. For example, a request may be classified as both limitUser and limitModule.
Load Healer tracks how many worker threads each activity consumes relative to maxThreads. If one activity grows large enough to put the node at risk, Load Healer can reject only that activity with HTTP 429 Too Many Requests, which helps prevent the entire node from becoming unresponsive.
On JFrog SaaS deployments, Load Healer is enabled by default. On self-managed deployments, Load Healer runs in Simulation mode by default, where it observes overload conditions and records what would have been throttled but never blocks requests.
Load Healer supports the following modes.
| Mode | Returns HTTP 429? | When it acts | Typical use |
|---|---|---|---|
| Simulation (self-managed default) | No | Never | Observe load patterns and identify activities that would trigger protection without blocking requests. |
| Reactive | Yes | Only when the node enters an OUTAGE state because worker threads are exhausted and request latency is high. | Protect the system as a last resort during overload conditions. |
Activities Tracked by Load Healer
A single request can be classified into multiple activities. For example, one request may count toward both limitUser and limitModule.
| Activity | Description | Threshold |
|---|---|---|
xrayBlockUnscanned | Requests waiting for JFrog Xray to clear unscanned artifacts. | 80% |
xrayDownloadBlocked | Downloads waiting for an Xray decision. | 80% |
curationCachedPackageStatus | Requests waiting for JFrog Curation to approve a cached artifact. | 80% |
parallelUploadPreRepoKey | Concurrent uploads to a single repository. | 80% |
parallelDownloadUploadPerPackageType | Concurrent download and upload activity for a package type. | 85% |
qrlLowPriorityRequests | Low-priority database-bound requests. | 80% |
limitActivity | General upload and download activity. | 90% |
limitModule | Requests targeting a specific /api/{module} endpoint, such as /api/npm. | 90% |
limitEndPoint | Requests targeting a specific endpoint. | 80% |
limitUser | Requests from a single user. | 80% |
You can enable Reactive mode, return to Simulation mode, or verify Reactive mode during an outage:
Enable Reactive Mode
Once Simulation mode consistently identifies recurring overload conditions, enable Reactive mode to protect the instance during future overload events. For more information on reading overload logs and deciding when to enable Reactive mode, see Artifactory Overloading Troubleshooting.
You can set the mode as a JVM system property or in artifactory.system.properties.
To enable Reactive mode:
-
Set the following property.
-Dartifactory.load.healer.mode=reactive -
Restart Artifactory.
Return to Simulation Mode
Return to Simulation mode when you want Load Healer to observe overload conditions without blocking requests.
To return to Simulation mode:
-
Set the following property.
-Dartifactory.load.healer.mode=simulation -
Restart Artifactory.
Verify Reactive Mode
After you enable Reactive mode and restart Artifactory, you can confirm that Load Healer engages when the node enters an OUTAGE state. An OUTAGE state means all Tomcat working threads are busy and the Tomcat connection queue is full.
To verify Reactive mode:
-
When the node enters an OUTAGE state, open the Artifactory logs and confirm that the Reactive mode management virtual thread has started.
Starting reactive mode management virtual thread -
Confirm that Load Healer reports which activities switched to Reactive throttling.
Setting activities to reactive mode: {<list of activities>}
By default, Reactive mode remains active for approximately 20 minutes before the affected activities automatically return to their previous mode.
Reactive Mode Behavior
Reactive mode significantly improves Artifactory's resilience to sudden load spikes by preventing a single activity from exhausting the available worker threads.
When Load Healer intervenes, only the activity contributing to the overload receives an HTTP 429 Too Many Requests response. Other requests continue to process normally, which helps keep the instance responsive.
If client applications or automation handle HTTP 429 responses by retrying with an appropriate backoff strategy, temporary spikes are smoothed out and operations typically complete successfully. If clients don't retry, some builds or automated processes may fail during periods of high load.
You can reduce disruption when Reactive mode is active:
- Clients and automation can retry
HTTP 429 Too Many Requestsresponses using exponential backoff or another appropriate retry strategy. - Gradual increases in concurrency are less likely to trigger Load Healer protection than sudden bursts of highly concurrent requests.
- Recurring overload events usually require investigating the reported activities to identify the underlying workload or dependency before increasing system capacity. For more information, see Artifactory Overloading Troubleshooting.
Related Topics
Updated about 6 hours ago