Load Healer (Automatic Load Protection)
Protect Artifactory from overload by monitoring worker-thread consumption and throttling offending activities with HTTP 429 when the node enters an outage state.
Since: Available in Artifactory 7.161.15.
Load Healer protects Artifactory from overload caused by a single activity monopolizing the available worker threads. Most Artifactory outages occur when the working thread pool is exhausted and new requests are rejected or time out.
Load Healer continuously monitors how worker threads are consumed and identifies workloads or activities that threaten to overload the node. When protection is triggered, Load Healer limits the offending activity to a safe portion of the available worker threads, allowing other requests to continue running uninterrupted.
To do this, Load Healer classifies incoming requests into activities, such as downloads for a package type, uploads to a repository, calls to a specific API, or traffic from a single user. A single request can belong to multiple activities. For example, a request may be classified as both limitUser and limitModule.
Load Healer tracks how many worker threads each activity consumes relative to maxThreads. If one activity grows large enough to put the node at risk, Load Healer can reject only that activity with HTTP 429 (Too Many Requests), preventing the entire node from becoming unresponsive.
On JFrog SaaS deployments, Load Healer is enabled by default. On self-managed deployments, Load Healer runs in Simulation mode by default, where it observes overload conditions and records what would have been throttled but never blocks requests.
Load Healer supports the following modes.
| Mode | Returns HTTP 429? | When it acts | Typical use |
|---|---|---|---|
| Simulation (self-managed default) | No | Never | Observe load patterns and identify activities that would trigger protection without blocking requests. |
| Reactive | Yes | Only when the node enters an OUTAGE state because worker threads are exhausted and request latency is high. | Protect the system as a last resort during overload conditions. |
Activities Tracked by Load Healer
A single request can be classified into multiple activities. For example, one request may count toward both limitUser and limitModule.
| Activity | Description | Threshold |
|---|---|---|
xrayBlockUnscanned | Requests waiting for JFrog Xray to clear unscanned artifacts. | 80% |
xrayDownloadBlocked | Downloads waiting for an Xray decision. | 80% |
curationCachedPackageStatus | Requests waiting for JFrog Curation to approve a cached artifact. | 80% |
parallelUploadPreRepoKey | Concurrent uploads to a single repository. | 80% |
parallelDownloadUploadPerPackageType | Concurrent download and upload activity for a package type. | 85% |
qrlLowPriorityRequests | Low-priority database-bound requests. | 80% |
limitActivity | General upload and download activity. | 90% |
limitModule | Requests targeting a specific /api/{module} endpoint, such as /api/npm. | 90% |
limitEndPoint | Requests targeting a specific endpoint. | 80% |
limitUser | Requests from a single user. | 80% |
Identify Overload Causes from Logs
By default, self-managed deployments require no configuration. Load Healer automatically starts in Simulation mode. During startup, verify the following log entry:
Load Healer mode auto-defaulted to SIMULATION (on-prem deployment)When all worker threads are busy, Artifactory logs every activity consuming more than 20% of the available threads.
all threads are busy (300 out of 300), activities that take above 20% of threads: {...}These log entries identify which activity saturated the node and provide valuable information for troubleshooting.
If the same activity repeatedly appears during overload events:
- Investigate the workload or dependency causing the overload.
- Reduce excessive client concurrency where possible (for example, when
limitUserappears). - Investigate external dependencies such as JFrog Xray or JFrog Curation when related activities appear.
- Consider enabling Reactive mode to protect the instance from future outages.
Enable Reactive Mode
Once Simulation mode consistently identifies recurring overload conditions, enable Reactive mode to protect the instance during future overload events.
-
Set the following JVM system property.
-Dartifactory.load.healer.mode=reactive -
Restart Artifactory.
To return to monitor-only behavior, set the mode back to Simulation and restart Artifactory.
-Dartifactory.load.healer.mode=simulationVerify Reactive Mode
After restarting in Reactive mode, the next time the node enters an OUTAGE (All tomcat working threads are busy and the tomcat connection queue is full) state, look for the following log entries.
The virtual thread responsible for managing Reactive mode starts.
Starting reactive mode management virtual threadLoad Healer then reports which activities have been switched to Reactive throttling.
Setting activities to reactive mode: {<list of activities>}By default, Reactive mode remains active for approximately 20 minutes before the affected activities automatically return to their previous mode.
What to Expect When Turning On Reactive Mode
Reactive mode significantly improves Artifactory's resilience to sudden load spikes by preventing a single activity from exhausting the available worker threads.
When Load Healer intervenes, only the activity contributing to the overload receives an HTTP 429 (Too Many Requests) response. Other requests continue to be processed normally, helping keep the instance responsive.
If client applications or automation handle HTTP 429 responses by retrying with an appropriate backoff strategy, temporary spikes are smoothed out and operations typically complete successfully. If clients do not retry, some builds or automated processes may fail during periods of high load.
To minimize disruptions:
- Configure clients and automation to retry
HTTP 429 (Too Many Requests)responses using exponential backoff or another appropriate retry strategy. - Avoid generating sudden bursts of highly concurrent requests. Instead, gradually increase concurrency to reduce the likelihood of triggering Load Healer protection.
- If repeated overload events occur, investigate the reported activities to identify the underlying workload or dependency before increasing system capacity.
Related Topics
Updated 8 days ago