Bulk Mirroring Configuration
Speed up Federation sync with parallel bulk mirroring and thread pool tuning.
Bulk event mirroring, introduced in Artifactory version 7.63.2, optimizes Federation performance by collecting events into bulks and sending them to the target mirror for parallel processing.
How Bulk Mirroring Works
Before bulk event mirroring was introduced, artifacts were copied to target mirrors sequentially one event at a time per repository. This limited the number of events that could be processed in a reasonable amount of time and sometimes resulted in HTTP network bottlenecks, especially between geographically distant members.
Bulk event mirroring bypasses potential HTTP network bottlenecks by collecting events into bulks, each dedicated to a single repository, and sending them to the target mirror. The target mirror assigns a dedicated thread pool to each bulk. Each pool contains multiple threads that upload artifacts to the local Artifactory in parallel. This parallel processing enables events to be uploaded much faster.
In addition to improving the standard event flow, bulk events also improve the performance of Full Sync operations. Bulk mirroring works with a cache mechanism, also introduced in 7.63.2, to greatly reduce the load on Artifactory during Full Sync operations.
Event Queue Properties
The following properties control how the Event Queue processes bulk mirroring events.
| Property | Description | Default |
|---|---|---|
artifactory.federated.mirror.events.bulk.exclude.repo.types | Excludes specific package types from bulk event mirroring. Use * for all, or comma-separated list (for example, docker,composer,npm) | (none) |
artifactory.federated.mirror.events.bulk.max.pools | Number of available event processing thread pools (executors) | 4 (maximum: 5) |
artifactory.federated.mirror.events.bulk.threadsPerPool | Number of threads per pool (executor) | 4 (maximum: 5) |
artifactory.federated.mirror.events.batch.message.bulk.size | Number of Event Queue events sent with each replication message to the target mirror | 100 |
artifactory.federated.mirror.events.bulk.executor.poll.timeout.minutes | Time new requests wait if all processing pools are occupied | 2 minutes |
For example, to raise the pool and thread limits toward their maximum in artifactory.system.properties:
artifactory.federated.mirror.events.bulk.max.pools=5
artifactory.federated.mirror.events.bulk.threadsPerPool=5Bulk Property Fetching
Bulk fetching of event properties from the database was introduced in Artifactory release 7.100.x as an additional performance enhancement.
| Property | Description | Default |
|---|---|---|
artifactory.federated.bulk.properties.enabled | Enable bulk property fetching | false (disabled) |
Full Sync Event Properties
The following properties control how Full Sync events are processed. See the Scaling and Sizing Guide for thread pool sizing guidance.
| Property | Description | Default |
|---|---|---|
artifactory.federated.mirror.events.bulk.fullSync.threadsPerPool | Threads per pool for Full Sync events. Full Sync events take priority in a single dedicated executor. | 10 (max: determined by available processors) |
artifactory.federated.mirror.events.bulk.fullSync.executor.wait.timeout.minutes | Wait time for Full Sync calls for other repositories while the pool works on a particular repository | 2 minutes |
artifactory.full.sync.batch.retry.count | Number of retries for a failing batch during Full Sync | 5 |
artifactory.full.sync.batch.retry.quiet.period.seconds | Wait time between batch retries (requires Artifactory 7.77.3+) | 5 seconds |
In-Memory Sorting for Full Sync
Full Sync queries can take a long time to perform and under extreme circumstances can sometimes fail to complete. If you determine that your database isn't sufficiently robust, you can sort queries in memory instead of in the database. The sort is performed according to depth, path, and name.
In-memory sorting is required when performing a Full Sync from a file.
The following properties control in-memory sorting for Full Sync operations.
| Property | Description | Default |
|---|---|---|
artifactory.federated.repo.full.sync.enabled.inMemorySorter | Enables in-memory sorting for legacy Federation (in-Artifactory processing). | false (sorting performed in the database) |
rtfs.sync.in.memory.sort.enabled | Enables in-memory sorting for RTFS Full Sync operations. Set under rtfs.persistence.federationProperties in system.yaml or Helm values. | true |
artifactory.federated.repo.full.sync.limit.entries.for.split | Determines how many artifacts in the file list are sorted at one time for legacy Federation. Relevant only when in-memory sorting is enabled. | 400,000 |
rtfs.sync.amount.of.entries.in.file | Determines how many artifacts in the file list are sorted at one time for RTFS. Set under rtfs.persistence.federationProperties. Relevant only when rtfs.sync.in.memory.sort.enabled is true. | 400,000 |
Related Topics
Updated about 7 hours ago