Rotate the RabbitMQ Password, Erlang Cookie and User in Xray (Self-Hosted)

Step 0 — Prepare (Linux)

Rules

1. One cookie and one password per cluster. Generate them once and copy the same values to every node. RabbitMQ shares users and passwords across the whole cluster, so you run rabbitmqctl change_password and add_user once, on any node.
2. Change every copy. The password is stored in system.yaml and in every rabbitmq.conf on the node. Every time RabbitMQ starts, Xray's start-up script sets the password of default_user to default_pass from rabbitmq.conf. Because the password is shared across the cluster, one node with an old default_pass resets the password for the whole cluster the next time it starts.
3. Do not paste the new values into tickets, chat or source control.
4. Make the password, cookie and user changes as separate steps, and verify each one before the next. Deleting guest is part of the procedure, not optional.
5. Stop and restart RabbitMQ only through its service, with shared.rabbitMq.autoStop: true. Without this setting, systemctl stop xray and systemctl restart xray on a node that runs both Xray and RabbitMQ stop or restart only Xray, and RabbitMQ keeps running. The cookie change then fails. Step 0.1 sets it. The RabbitMQ package for split mode (xray-rabbitmq.service) sets it by default.

0.1 Set autoStop. On every node that runs RabbitMQ, add this to system.yaml (for example /opt/jfrog/xray/var/etc/system.yaml):

shared:
  rabbitMq:
    autoStop: true

Without it, systemctl stop xray and systemctl restart xray stop or restart only Xray, and RabbitMQ keeps running. RPM, Debian and archive installations where Xray and RabbitMQ run on the same node do not set it by default. On split-mode RabbitMQ nodes, the RabbitMQ package sets it; check that it is there.

Check:

sudo grep -n autoStop /opt/jfrog/xray/var/etc/system.yaml

No restart is needed now; the setting is used the next time the service is stopped or restarted.

0.2 Plan a maintenance window and announce it. Scans and other Xray processing stall during the change.

0.3 Generate the new values once, on a trusted host, with shell history turned off:

set +o history
RMQ_PASSWORD=$(openssl rand -hex 24)
RMQ_COOKIE=$(openssl rand -hex 32)

Store both in your secret manager, and use the same two values on every node. The examples use the same $RMQ_PASSWORD for the new xray_admin user.

In the shell on each node, set the new values and the current values (the checks use them):

set +o history
RMQ_PASSWORD='<new password>'
RMQ_COOKIE='<new cookie>'
OLD_PASS='<current password>'
OLD_COOKIE='<current cookie>'

The installer default for the password and the cookie is JFXR_RABBITMQ_COOKIE, and the default user is guest. Many installations set their own values during installation, so do not assume the default. Read the current password from default_pass in rabbitmq.conf, and the current cookie from the .erlang.cookie file.

Set the RabbitMQ folder. It is .../rabbitmq on RabbitMQ 3 and .../rabbitmq4 on RabbitMQ 4. Check it, then set it in the same shell:

grep JF_PRODUCT_RABBITMQ_HOME /opt/jfrog/xray/app/bin/xray.default
RMQ_HOME=/opt/jfrog/xray/app/third-party/rabbitmq

Use the folder printed by the first command. The commands in this tab use $RMQ_HOME for the server cookie and for rabbitmqctl.

0.4 Record the current settings. The paths are listed in A.0.

• Current RabbitMQ user (<user> in this tab). Default: guest. Check shared.rabbitMq.username in system.yaml.
• Active node (multi-node only): the node named in shared.rabbitMq.active.node.name in system.yaml.
• Cluster nodes, users and vhosts: rabbitmqctl cluster_status, rabbitmqctl list_users, rabbitmqctl list_vhosts.

0.5 Back up. Copy, do not move. The backups contain secrets; keep them readable by root only. On nodes that run RabbitMQ, back up system.yaml, both rabbitmq.conf files, both cookie files and the RabbitMQ data folder. For example:

sudo tar -czpf /var/tmp/xray-rmq-backup.tgz \
  /opt/jfrog/xray/var/etc/system.yaml \
  /opt/jfrog/xray/var/etc/rabbitmq.conf \
  /opt/jfrog/xray/app/bin/rabbitmq/rabbitmq.conf \
  /opt/jfrog/xray/.erlang.cookie \
  $RMQ_HOME/.erlang.cookie \
  /opt/jfrog/xray/var/data/rabbitmq
sudo chmod 600 /var/tmp/xray-rmq-backup.tgz

Leave out /opt/jfrog/xray/var/etc/rabbitmq.conf if it does not exist. On split-mode Xray nodes, back up system.yaml.

0.6 Check that the system is healthy before you start, and again after each step:

• Xray health, on a node that runs Xray. All services must show "state" : "HEALTHY":

  Bash

  curl -s http://localhost:8082/router/api/v1/system/health

• Xray is connected to RabbitMQ:

  Bash

  rabbitmqctl list_connections user

• The current password is correct (prints Success):

  Bash

  rabbitmqctl authenticate_user <user> "$OLD_PASS"

After a restart, the Xray services can take a few minutes to report HEALTHY again.

In this tab, rabbitmqctl <command> means sudo -u xray -i $RMQ_HOME/sbin/rabbitmqctl <command> (see A.0).

A.0 Paths and Users

RPM, Debian and archive installations use the same RabbitMQ scripts and the same folder layout under the installation folder. The RabbitMQ package for split mode installs only RabbitMQ, with the same layout and its own service. The commands in this tab use the default RPM/Debian paths; replace /opt/jfrog/xray with your installation folder.

In this tab, "the service" means xray on nodes that run Xray and RabbitMQ, and xray-rabbitmq on split-mode RabbitMQ nodes.

Split mode, Xray nodes. They run xray.service without RabbitMQ and connect to all RabbitMQ nodes through shared.rabbitMq.urls. Only system.yaml matters on these nodes. One user name and password are used for every URL in the list; this works because RabbitMQ users are shared across the cluster.

shared:
  rabbitMq:
    urls:
      - amqp://<rabbitmq-node-1>:5672/
      - amqp://<rabbitmq-node-2>:5672/
    username: "xray_admin"
    password: "<RMQ_PASSWORD>"

system.yaml and encryption. On nodes that run Xray, Xray encrypts the password in system.yaml after it starts, so you will not see the new value in plain text afterwards. On split-mode RabbitMQ nodes, the value stays in plain text; make sure only the service user and root can read the file.

How to run rabbitmqctl. Run it as the service user, with a login shell (-i):

sudo -u xray -i $RMQ_HOME/sbin/rabbitmqctl cluster_status

• rabbitmqctl reads the cookie from the service user's home folder. As root, or as any other user, it uses a different cookie and cannot connect.
• Without -i, it runs in your current folder. If the service user cannot read that folder, it fails with File operation error: eacces.

In this tab, rabbitmqctl <command> means sudo -u xray -i $RMQ_HOME/sbin/rabbitmqctl <command>.

What happens on every start. The service starts RabbitMQ as the service user. The start-up script:

• creates a cookie file only if none exists, so the cookie files you write are kept;
• writes the entries of shared.rabbitMq.node.rabbitmqConf from system.yaml into rabbitmq.conf;
• sets the password of default_user to default_pass from rabbitmq.conf.

A.1 Password

Where each part runs: system.yaml on every node (split mode: every Xray node and every RabbitMQ node); rabbitmq.conf on every node that runs RabbitMQ; change_password once per cluster.

A.1.1 On every node, edit system.yaml as the service user, so its owner does not change:

sudo -u xray vi /opt/jfrog/xray/var/etc/system.yaml

Set the password under shared.rabbitMq:

shared:
  rabbitMq:
    password: "<RMQ_PASSWORD>"

Change only the password: line inside the rabbitMq: block. shared.database also has a password: line, and on split-mode Xray nodes the rabbitMq: block also contains the urls: list; leave both unchanged. Check:

sudo grep -n -A6 'rabbitMq:' /opt/jfrog/xray/var/etc/system.yaml

A.1.2 On every node that runs RabbitMQ, set default_pass to the new password in each rabbitmq.conf that exists:

• /opt/jfrog/xray/app/bin/rabbitmq/rabbitmq.conf (always present; RabbitMQ reads this one)
• /opt/jfrog/xray/var/etc/rabbitmq.conf (newer installations only)

default_pass = <RMQ_PASSWORD>

Edit the files as the service user, or run, for each file:

sudo -u xray sed -i "s|^default_pass *=.*|default_pass = $RMQ_PASSWORD|" /opt/jfrog/xray/app/bin/rabbitmq/rabbitmq.conf
sudo -u xray sed -i "s|^default_pass *=.*|default_pass = $RMQ_PASSWORD|" /opt/jfrog/xray/var/etc/rabbitmq.conf

Check:

sudo grep -n '^default_' /opt/jfrog/xray/app/bin/rabbitmq/rabbitmq.conf /opt/jfrog/xray/var/etc/rabbitmq.conf

A.1.3 Change the password in RabbitMQ. Run once per cluster, on any RabbitMQ node:

rabbitmqctl change_password <user> "$RMQ_PASSWORD"
rabbitmqctl authenticate_user <user> "$RMQ_PASSWORD"

Expect Success. Existing Xray connections keep working, but new connections fail until Xray restarts. Continue with A.1.4 straight away.

A.1.4 Restart the service, one node at a time, active node first:

sudo systemctl restart xray            # split-mode RabbitMQ node: sudo systemctl restart xray-rabbitmq

After each restart, check that the new password still works:

rabbitmqctl authenticate_user <user> "$RMQ_PASSWORD"

If it fails, that node still has the old default_pass. Fix its rabbitmq.conf (A.1.2), then run A.1.3 again. Split mode: then restart xray on every Xray node, one at a time.

A.1.5 Verify

• The new password works and the old one is rejected:

  Bash

  rabbitmqctl authenticate_user <user> "$RMQ_PASSWORD"
  rabbitmqctl authenticate_user <user> "$OLD_PASS"

  Expect Success for the first, an error for the second.

• The password survives a restart: restart the service once more and repeat the check above.

• rabbitmqctl cluster_status lists all nodes. The health check (0.6) shows HEALTHY, and Xray connections are back.

• There are no RabbitMQ authentication errors in the Xray logs. This prints nothing:

  Bash

  sudo grep -l ACCESS_REFUSED /opt/jfrog/xray/var/log/*.log

• A test scan completes.

If a check fails, see Troubleshooting (A.4), or go to Rollback (A.5).

A.2 Erlang Cookie

All nodes must be stopped at the same time. No node may run with the new cookie while another runs with the old one. In split mode, also stop the Xray nodes; they cannot work while RabbitMQ is down.

❗️

Start order in a cluster

• Stop the nodes one at a time and note which node stopped last.
• Start that node first, then start all other nodes right away.

A.2.1 Stop everything.

1. Split mode only: on every Xray node, run sudo systemctl stop xray.

2. On every node that runs RabbitMQ, one at a time, stop the service and note which node you stop last:

   Bash

   sudo systemctl stop xray            # split-mode RabbitMQ node: sudo systemctl stop xray-rabbitmq

3. Check that RabbitMQ is stopped. This must print RabbitMQ stopped:

   Bash

   pgrep -f beam.smp || echo "RabbitMQ stopped"

   If RabbitMQ is still running, autoStop is not set on that node (0.1). Do not change the cookie while RabbitMQ is running.

A.2.2 On every node that runs RabbitMQ, write the new cookie to both cookie files. Each file must be owned by the service user, with mode 400.

CLI cookie (used by rabbitmqctl):

F=/opt/jfrog/xray/.erlang.cookie
sudo chmod 600 $F
printf '%s' "$RMQ_COOKIE" | sudo tee $F > /dev/null
sudo chown xray:xray $F
sudo chmod 400 $F

Server cookie (used by RabbitMQ):

F=$RMQ_HOME/.erlang.cookie
sudo chmod 600 $F
printf '%s' "$RMQ_COOKIE" | sudo tee $F > /dev/null
sudo chown xray:xray $F
sudo chmod 400 $F

Check:

sudo ls -l /opt/jfrog/xray/.erlang.cookie $RMQ_HOME/.erlang.cookie

Expect both files listed as -r-------- and owned by xray. If both paths point to the same file, writing it once is enough. If only the server cookie is changed, RabbitMQ starts but rabbitmqctl cannot connect to it. If a file is owned by root, or unreadable by the service user, RabbitMQ cannot start.

A.2.3 On every node that runs RabbitMQ (split mode: also on Xray nodes where the key is present), set the cookie in system.yaml. The start-up script uses this value when it creates a cookie file, for example on a new node or after a reinstall.

shared:
  rabbitMq:
    erlangCookie:
      value: "<RMQ_COOKIE>"

A.2.4 Start the node that stopped last first, then immediately all other RabbitMQ nodes:

sudo systemctl start --no-block xray            # split-mode RabbitMQ node: xray-rabbitmq

On a single node, sudo systemctl start xray is enough. Wait until rabbitmqctl cluster_status lists all nodes under Running Nodes. Split mode: then run sudo systemctl start xray on every Xray node.

A.2.5 Verify

• Both cookie files hold the new value, on every node. Compare checksums, so you never print the value; all three must be the same:

  Bash

  sudo sha256sum /opt/jfrog/xray/.erlang.cookie $RMQ_HOME/.erlang.cookie
  printf '%s' "$RMQ_COOKIE" | sha256sum

• The old cookie is rejected. This command must fail with Erlang distribution failed:

  Bash

  sudo -u xray -i $RMQ_HOME/sbin/rabbitmq-diagnostics -n rabbit@$(hostname -s) --erlang-cookie "$OLD_COOKIE" status

• rabbitmqctl cluster_status lists all nodes and shows no network partitions.

• The password from A.1 still works: rabbitmqctl authenticate_user <user> "$RMQ_PASSWORD".

• rabbitmqctl list_queues -p xray_haq name messages consumers shows the expected queues with consumers, and the health check (0.6) shows HEALTHY.

• A test scan completes, and there are no errors in the Xray or RabbitMQ logs.

If a check fails, see Troubleshooting (A.4), or go to Rollback (A.5).

A.3 User — Replace guest with xray_admin and Delete guest

Before you start: A.1 and A.2 are done and verified, and $RMQ_PASSWORD is set in your shell. Xray restarts in A.3.3 and RabbitMQ restarts in A.3.4.

A.3.1 Create the new user. Run once, on any RabbitMQ node; users are shared across the cluster.

1. List the vhosts. Expect at least / and xray_haq:

   Bash

   rabbitmqctl list_vhosts

2. Create the user and give it full permissions on each vhost Xray uses:

   Bash

   rabbitmqctl add_user xray_admin "$RMQ_PASSWORD"
   rabbitmqctl set_user_tags xray_admin administrator
   rabbitmqctl set_permissions -p / xray_admin ".*" ".*" ".*"
   rabbitmqctl set_permissions -p xray_haq xray_admin ".*" ".*" ".*"

   If list_vhosts prints another vhost that Xray uses, run set_permissions for it too.

3. Check:

   Bash

   rabbitmqctl list_user_permissions xray_admin
   rabbitmqctl authenticate_user xray_admin "$RMQ_PASSWORD"

Permissions are stored in RabbitMQ and survive restarts.

A.3.2 Point Xray at the new user, on every node:

1. In system.yaml (split mode: every Xray node and every RabbitMQ node):

   YAML

   shared:
     rabbitMq:
       username: "xray_admin"
       password: "<RMQ_PASSWORD>"

2. On every node that runs RabbitMQ, in each rabbitmq.conf that exists:

   ini

   default_user = xray_admin
   default_pass = <RMQ_PASSWORD>

   For example:

   Bash

   sudo -u xray sed -i "s|^default_user *=.*|default_user = xray_admin|" /opt/jfrog/xray/app/bin/rabbitmq/rabbitmq.conf
   sudo -u xray sed -i "s|^default_user *=.*|default_user = xray_admin|" /opt/jfrog/xray/var/etc/rabbitmq.conf
   sudo grep -n '^default_' /opt/jfrog/xray/app/bin/rabbitmq/rabbitmq.conf /opt/jfrog/xray/var/etc/rabbitmq.conf

   This keeps the password of xray_admin in line with default_pass when RabbitMQ restarts.

A.3.3 Restart Xray and confirm it uses the new user. Run sudo systemctl restart xray on each node that runs Xray, one at a time, active node first. On nodes that run both Xray and RabbitMQ, this also restarts RabbitMQ (because of autoStop). Then:

rabbitmqctl list_connections user

All Xray connections must show xray_admin; none may show guest. Run a test scan, and check the Xray logs for RabbitMQ authentication errors. Do not continue until no connection uses guest.

A.3.4 Restrict guest to localhost, permanently. On every node that runs RabbitMQ, add this to system.yaml. The start-up script writes it into rabbitmq.conf on every start.

shared:
  rabbitMq:
    node:
      rabbitmqConf:
        - name: loopback_users.guest
          value: "true"

If rabbitmqConf already has entries, add this one to the list. Then restart the service, one node at a time, and check:

rabbitmqctl environment | grep loopback_users

Expect {loopback_users,[<<"guest">>]}.

A.3.5 Delete guest.

rabbitmqctl delete_user guest
rabbitmqctl list_users

guest must not be listed. After this, the RabbitMQ start-up log shows these lines on every start. They are expected and harmless:

• Failed to change password for user 'guest' / the user does not exist
• Failed to set permissions for vhost: xray_haq with user: guest

A.3.6 Verify

• From a different host, guest is refused. Each command must print 401 (this works only if port 15672 is reachable from that host):

  Bash

  curl -s -o /dev/null -w '%{http_code}\n' -u "guest:$OLD_PASS" http://<rabbitmq-host>:15672/api/whoami
  curl -s -o /dev/null -w '%{http_code}\n' -u "guest:$RMQ_PASSWORD" http://<rabbitmq-host>:15672/api/whoami
  curl -s -o /dev/null -w '%{http_code}\n' -u guest:JFXR_RABBITMQ_COOKIE http://<rabbitmq-host>:15672/api/whoami
  curl -s -o /dev/null -w '%{http_code}\n' -u guest:guest http://<rabbitmq-host>:15672/api/whoami

• rabbitmqctl list_users does not list guest, and rabbitmqctl list_connections user shows only xray_admin.

• After one more restart, both checks above still pass.

• A test scan completes.

A.3.7 After every Xray upgrade or installer re-run, check that the old settings did not come back. An upgrade can rewrite rabbitmq.conf from the installer's defaults.

rabbitmqctl list_users                                                                    # no guest
sudo grep -n 'loopback_users.guest' /opt/jfrog/xray/app/bin/rabbitmq/rabbitmq.conf     # true (or no line)
sudo grep -n '^default_user' /opt/jfrog/xray/app/bin/rabbitmq/rabbitmq.conf            # xray_admin
sudo grep -n -A6 'rabbitMq:' /opt/jfrog/xray/var/etc/system.yaml                      # username: xray_admin

If a value is back to the default, set it again (A.3.2, A.3.4) and restart.

A.4 Troubleshooting

A.5 Rollback

1. Stop Xray on every node (split mode: the Xray nodes first), then the service on every RabbitMQ node. Note which node you stop last. Check that pgrep -f beam.smp prints nothing. If RabbitMQ is still running, stop it with the cookie it runs with: sudo -u xray -i $RMQ_HOME/sbin/rabbitmqctl --erlang-cookie <cookie> shutdown.

2. From the backup (0.5), restore only system.yaml, every rabbitmq.conf, and both cookie files, with the same owner and mode.

3. Start the node you stopped last first, then the others immediately (systemctl start --no-block). The restored default_pass is applied on start.

4. If guest was already deleted, create it again:

   Bash

   rabbitmqctl add_user guest "$OLD_PASS"
   rabbitmqctl set_user_tags guest administrator
   rabbitmqctl set_permissions -p / guest ".*" ".*" ".*"
   rabbitmqctl set_permissions -p xray_haq guest ".*" ".*" ".*"

   Set username: guest and default_user = guest back, and remove the loopback_users.guest entry from shared.rabbitMq.node.rabbitmqConf.

5. Run the checks from A.1.5 and A.2.5 with the old values.

A.6 Clean Up

• Clear the values from your shell and turn history back on:

  Bash

  unset RMQ_PASSWORD RMQ_COOKIE OLD_PASS OLD_COOKIE
  set -o history

• When the change is confirmed, delete the backups that contain secrets (for example sudo rm /var/tmp/xray-rmq-backup.tgz), or move them to secure storage.

• Keep shared.rabbitMq.autoStop: true. It is the recommended setting.

• Record the change in your change-management system: date, layout (single node, multi-node or split mode), versions, and the result of each check.

Step 0 — Prepare (Docker Compose)

Rules

1. One cookie and one password per cluster. Generate them once and copy the same values to every node. RabbitMQ shares users and passwords across the whole cluster, so you run rabbitmqctl change_password and add_user once, on any node.
2. Change every copy. The password is stored in system.yaml and in every rabbitmq.conf on the node. Every time RabbitMQ starts, Xray's start-up script sets the password of default_user to default_pass from rabbitmq.conf. Because the password is shared across the cluster, one node with an old default_pass resets the password for the whole cluster the next time it starts.
3. Do not paste the new values into tickets, chat or source control.
4. Make the password, cookie and user changes as separate steps, and verify each one before the next. Deleting guest is part of the procedure, not optional.

0.1 Plan a maintenance window and announce it. Scans and other Xray processing stall during the change.

0.2 Generate the new values once, on a trusted host, with shell history turned off:

set +o history
RMQ_PASSWORD=$(openssl rand -hex 24)
RMQ_COOKIE=$(openssl rand -hex 32)

Store both in your secret manager, and use the same two values on every node. The examples use the same $RMQ_PASSWORD for the new xray_admin user.

In the shell on each node, set the new values and the current values (the checks use them):

set +o history
RMQ_PASSWORD='<new password>'
RMQ_COOKIE='<new cookie>'
OLD_PASS='<current password>'
OLD_COOKIE='<current cookie>'

The installer default for the password and the cookie is JFXR_RABBITMQ_COOKIE, and the default user is guest. Many installations set their own values during installation, so do not assume the default. Read the current password from default_pass in rabbitmq.conf, and the current cookie from the .erlang.cookie file. The paths are listed in B.0.

0.3 Record the current settings.

• Current RabbitMQ user (<user> in this tab). Default: guest. Check shared.rabbitMq.username in system.yaml.

• Active node (multi-node only): the node named in shared.rabbitMq.active.node.name in system.yaml.

• Cluster nodes, users and vhosts:

  Bash

  sudo docker exec xray_rabbitmq rabbitmqctl cluster_status
  sudo docker exec xray_rabbitmq rabbitmqctl list_users
  sudo docker exec xray_rabbitmq rabbitmqctl list_vhosts

0.4 Back up. Copy, do not move. The backups contain secrets; keep them readable by root only. Back up the whole $ROOT_DATA_DIR folder without var/log, and the .env file. When you restore, restore only the files this procedure changes; the same folder can also hold PostgreSQL data.

0.5 Check that the system is healthy before you start, and again after each step:

• Xray health, on a node that runs Xray. All services must show "state" : "HEALTHY":

  Bash

  curl -s http://localhost:8082/router/api/v1/system/health

• Xray is connected to RabbitMQ:

  Bash

  sudo docker exec xray_rabbitmq rabbitmqctl list_connections user

• The current password is correct (prints Success):

  Bash

  sudo docker exec xray_rabbitmq rabbitmqctl authenticate_user <user> "$OLD_PASS"

After a restart, the Xray services can take a few minutes to report HEALTHY again.

B.0 Projects and Paths

• Data folder (ROOT_DATA_DIR). It is set in the .env file next to the docker-compose*.yaml files. Go to the Compose installation folder and set it in your shell:

  Bash

  cd <compose-install-folder>
  export ROOT_DATA_DIR=$(grep ^ROOT_DATA_DIR= .env | cut -d= -f2)
  echo $ROOT_DATA_DIR

• Files, under $ROOT_DATA_DIR:

  File	Path	Notes
  system.yaml	$ROOT_DATA_DIR/var/etc/system.yaml	Xray encrypts the password in it after it starts.
  rabbitmq.conf	$ROOT_DATA_DIR/app/third-party/rabbitmq/rabbitmq.conf	Mounted into the container as /etc/rabbitmq/rabbitmq.conf.
  rabbitmq.conf (second copy)	$ROOT_DATA_DIR/var/etc/rabbitmq.conf	Exists on newer installations only.
  Erlang cookie	$ROOT_DATA_DIR/var/data/rabbitmq/.erlang.cookie	The only cookie file. The folder is mounted as /var/lib/rabbitmq.

  The RabbitMQ container is named xray_rabbitmq.

• Root or sudo. The data folder and the Docker socket belong to root. Run the commands with sudo, or as root. The cookie file is hidden; list it with sudo ls -la.

• Compose projects. By default, RabbitMQ and Xray run as two separate Compose projects. Check with sudo docker compose ls. Run the commands from the Compose installation folder:

  Action	Command
  RabbitMQ project	sudo docker compose -p xray-rabbitmq -f docker-compose-rabbitmq.yaml <command>
  Xray project	sudo docker compose -p xray <command>
  rabbitmqctl	sudo docker exec xray_rabbitmq rabbitmqctl <command>

  A plain docker compose restart restarts only one of the projects. Stop order: Xray, then RabbitMQ. Start order: RabbitMQ, then Xray. If PostgreSQL also runs from this folder, leave it running.

• Split mode. RabbitMQ nodes run only the RabbitMQ project. Xray nodes run only the Xray project and list every RabbitMQ node under shared.rabbitMq.urls, with one user name and password for all of them; RabbitMQ users are shared across the cluster. On Xray nodes, only system.yaml matters.

• One merged project (docker-compose-all.yaml). If Xray, RabbitMQ and PostgreSQL run from one file, use sudo docker compose -p <project> -f docker-compose-all.yaml <command> with the project name from docker compose ls. Where a step restarts only RabbitMQ, restart only the RabbitMQ service: sudo docker compose -p <project> -f docker-compose-all.yaml restart rabbitmq (check the service name in the file).

• Restarts and configV2.sh. A plain restart does not run configV2.sh, so the cookie and rabbitmq.conf stay as you wrote them. Do not run configV2.sh during this procedure; it copies a fresh rabbitmq.conf from the installer's template.

⚠️

Edit rabbitmq.conf in place

rabbitmq.conf is mounted into the container as a single file. sed -i and most editors write a new file in its place, and the container keeps reading the old one until it is recreated. Edit a copy, then write the copy's content back into the same file, as below. This also keeps the file's owner unchanged.

To edit rabbitmq.conf (repeat for the second copy, if it exists):

sudo cp $ROOT_DATA_DIR/app/third-party/rabbitmq/rabbitmq.conf $ROOT_DATA_DIR/app/third-party/rabbitmq/rabbitmq.conf.edit
sudo vi $ROOT_DATA_DIR/app/third-party/rabbitmq/rabbitmq.conf.edit
sudo sh -c "cat $ROOT_DATA_DIR/app/third-party/rabbitmq/rabbitmq.conf.edit > $ROOT_DATA_DIR/app/third-party/rabbitmq/rabbitmq.conf"
sudo rm $ROOT_DATA_DIR/app/third-party/rabbitmq/rabbitmq.conf.edit

Then check that the container sees the change:

sudo docker exec xray_rabbitmq grep -n '^default_' /etc/rabbitmq/rabbitmq.conf

B.1 Password

B.1.1 On every node (split mode: Xray nodes and RabbitMQ nodes), set the password in $ROOT_DATA_DIR/var/etc/system.yaml:

shared:
  rabbitMq:
    password: "<RMQ_PASSWORD>"

Change only the password: line inside the rabbitMq: block. Check with sudo grep -n -A6 'rabbitMq:' $ROOT_DATA_DIR/var/etc/system.yaml. Xray encrypts the value when it next starts.

B.1.2 On every node that runs RabbitMQ, set the password in both rabbitmq.conf files, in place (see B.0):

default_pass = <RMQ_PASSWORD>

Then check that the container sees the new value.

B.1.3 Change the password in RabbitMQ, once per cluster:

sudo docker exec xray_rabbitmq rabbitmqctl change_password <user> "$RMQ_PASSWORD"
sudo docker exec xray_rabbitmq rabbitmqctl authenticate_user <user> "$RMQ_PASSWORD"

Expect Success.

B.1.4 Restart RabbitMQ and then Xray straight away, active node first, then the other nodes without delay:

sudo docker compose -p xray-rabbitmq -f docker-compose-rabbitmq.yaml restart
sudo docker exec xray_rabbitmq rabbitmqctl await_startup
sudo docker exec xray_rabbitmq rabbitmqctl authenticate_user <user> "$RMQ_PASSWORD"
sudo docker compose -p xray restart

If await_startup fails because the container is still starting, wait a few seconds and run it again.

B.1.5 Verify

• The new password works and the old one is rejected:

  Bash

  sudo docker exec xray_rabbitmq rabbitmqctl authenticate_user <user> "$RMQ_PASSWORD"
  sudo docker exec xray_rabbitmq rabbitmqctl authenticate_user <user> "$OLD_PASS"

  Expect Success for the first, an error for the second.

• The password survives a restart: restart the RabbitMQ project once more and repeat the check above.

• The health check (0.5) shows HEALTHY, Xray connections are back, there are no RabbitMQ authentication errors in the Xray logs, and a test scan completes.

If a check fails, go to Rollback (B.5).

B.2 Erlang Cookie

All nodes must be stopped at the same time. No node may run with the new cookie while another runs with the old one. In a cluster, the node that stopped last starts first, then all others immediately. The other nodes wait for a peer for about 5 minutes and then fail to start, so do not wait between starts.

B.2.1 On every node, one at a time, stop Xray, then RabbitMQ, and note which node you stop last:

sudo docker compose -p xray down
sudo docker compose -p xray-rabbitmq -f docker-compose-rabbitmq.yaml down
sudo docker ps --filter name=xray_rabbitmq

The last command must list no container. With docker-compose-all.yaml: sudo docker compose -p <project> -f docker-compose-all.yaml down.

B.2.2 On every RabbitMQ node, write the new cookie. The file must keep the owner of the RabbitMQ data folder (the rabbitmq user inside the container, UID 999 by default), with mode 400:

F=$ROOT_DATA_DIR/var/data/rabbitmq/.erlang.cookie
sudo chmod 600 $F
printf '%s' "$RMQ_COOKIE" | sudo tee $F > /dev/null
sudo chown --reference=$ROOT_DATA_DIR/var/data/rabbitmq $F
sudo chmod 400 $F
sudo ls -ln $F

The owner must match the other files in $ROOT_DATA_DIR/var/data/rabbitmq. RabbitMQ does not start if it cannot read the file.

B.2.3 On every RabbitMQ node (split mode: also on Xray nodes where the key is present), set the cookie in $ROOT_DATA_DIR/var/etc/system.yaml. The installer uses this value when it creates a cookie file, for example when you add a node or reinstall.

shared:
  rabbitMq:
    erlangCookie:
      value: "<RMQ_COOKIE>"

B.2.4 Start RabbitMQ on the node that stopped last first, then immediately on all other nodes (up -d returns at once):

sudo docker compose -p xray-rabbitmq -f docker-compose-rabbitmq.yaml up -d

When RabbitMQ runs on all nodes, check the cluster, then start Xray on each node:

sudo docker exec xray_rabbitmq rabbitmqctl await_startup
sudo docker exec xray_rabbitmq rabbitmqctl cluster_status
sudo docker compose -p xray up -d

With docker-compose-all.yaml: sudo docker compose -p <project> -f docker-compose-all.yaml up -d.

B.2.5 Verify

• The cookie holds the new value on every node. Both checksums must be the same:

  Bash

  sudo sha256sum $ROOT_DATA_DIR/var/data/rabbitmq/.erlang.cookie
  printf '%s' "$RMQ_COOKIE" | sha256sum

• The old cookie is rejected. This command must fail with Erlang distribution failed:

  Bash

  sudo docker exec xray_rabbitmq rabbitmq-diagnostics -n rabbit@<node> --erlang-cookie "$OLD_COOKIE" status

  Use the node name that cluster_status prints.

• cluster_status lists all nodes with no network partitions, and the password from B.1 still works.

• sudo docker exec xray_rabbitmq rabbitmqctl list_queues -p xray_haq name messages consumers shows the expected queues with consumers, and the health check (0.5) shows HEALTHY.

• A test scan completes, and there are no errors in the Xray or RabbitMQ logs.

If a check fails, see Troubleshooting (B.4), or go to Rollback (B.5).

B.3 User — Replace guest with xray_admin and Delete guest

Before you start: B.1 and B.2 are done and verified, and $RMQ_PASSWORD is set in your shell. Xray restarts in B.3.3 and RabbitMQ restarts in B.3.4.

B.3.1 Create the new user, once, in the RabbitMQ container; users are shared across the cluster.

sudo docker exec xray_rabbitmq rabbitmqctl list_vhosts
sudo docker exec xray_rabbitmq rabbitmqctl add_user xray_admin "$RMQ_PASSWORD"
sudo docker exec xray_rabbitmq rabbitmqctl set_user_tags xray_admin administrator
sudo docker exec xray_rabbitmq rabbitmqctl set_permissions -p / xray_admin ".*" ".*" ".*"
sudo docker exec xray_rabbitmq rabbitmqctl set_permissions -p xray_haq xray_admin ".*" ".*" ".*"
sudo docker exec xray_rabbitmq rabbitmqctl list_user_permissions xray_admin
sudo docker exec xray_rabbitmq rabbitmqctl authenticate_user xray_admin "$RMQ_PASSWORD"

list_vhosts must show at least / and xray_haq. If it prints another vhost that Xray uses, run set_permissions for it too.

B.3.2 Point Xray at the new user, on every node:

1. In $ROOT_DATA_DIR/var/etc/system.yaml:

   YAML

   shared:
     rabbitMq:
       username: "xray_admin"
       password: "<RMQ_PASSWORD>"

2. On RabbitMQ nodes, in both rabbitmq.conf files, in place (see B.0):

   ini

   default_user = xray_admin
   default_pass = <RMQ_PASSWORD>

   This keeps the password of xray_admin in line with default_pass when RabbitMQ restarts.

B.3.3 Restart Xray and confirm it uses the new user. Restart only Xray, on each node, active node first; RabbitMQ keeps running:

sudo docker compose -p xray restart
sudo docker exec xray_rabbitmq rabbitmqctl list_connections user

All connections must show xray_admin; none may show guest. Run a test scan, and check the Xray logs for RabbitMQ authentication errors. Do not continue until no connection uses guest.

B.3.4 Restrict guest to localhost, permanently. Set the value in two places:

• in system.yaml, so the installer writes it into rabbitmq.conf every time configV2.sh runs, for example during an upgrade;
• in the current rabbitmq.conf files, because a plain restart does not re-read system.yaml.

1. On every RabbitMQ node, add this to $ROOT_DATA_DIR/var/etc/system.yaml:

   YAML

   shared:
     rabbitMq:
       node:
         rabbitmqConf:
           - name: loopback_users.guest
             value: "true"

   If rabbitmqConf already has entries, add this one to the list.

2. On every RabbitMQ node, change this line in both rabbitmq.conf files, in place (see B.0):

   ini

   loopback_users.guest = true

3. Restart RabbitMQ, one node at a time, and check:

   Bash

   sudo docker compose -p xray-rabbitmq -f docker-compose-rabbitmq.yaml restart
   sudo docker exec xray_rabbitmq rabbitmqctl await_startup
   sudo docker exec xray_rabbitmq rabbitmqctl environment | grep loopback_users

   Expect {loopback_users,[<<"guest">>]}.

B.3.5 Delete guest.

sudo docker exec xray_rabbitmq rabbitmqctl delete_user guest
sudo docker exec xray_rabbitmq rabbitmqctl list_users

guest must not be listed. After this, the RabbitMQ container log shows these lines on every start. They are expected and harmless: Failed to change password for user 'guest' / the user does not exist, and Failed to set permissions for vhost: xray_haq with user: guest.

B.3.6 Verify

• From a different host, guest is refused. Each command must print 401:

  Bash

  curl -s -o /dev/null -w '%{http_code}\n' -u "guest:$OLD_PASS" http://<rabbitmq-host>:15672/api/whoami
  curl -s -o /dev/null -w '%{http_code}\n' -u "guest:$RMQ_PASSWORD" http://<rabbitmq-host>:15672/api/whoami
  curl -s -o /dev/null -w '%{http_code}\n' -u guest:JFXR_RABBITMQ_COOKIE http://<rabbitmq-host>:15672/api/whoami
  curl -s -o /dev/null -w '%{http_code}\n' -u guest:guest http://<rabbitmq-host>:15672/api/whoami

• list_users does not list guest, and list_connections user shows only xray_admin.

• After one more restart of both projects, both checks above still pass.

• A test scan completes.

B.3.7 After every upgrade or configV2.sh run, check that guest is still absent, loopback_users.guest is not false, rabbitmq.conf has default_user = xray_admin, and system.yaml has username: xray_admin. If a value is back to the default, set it again (B.3.2, B.3.4) and restart.

B.4 Troubleshooting

B.5 Rollback

1. On every node, stop Xray, then RabbitMQ (docker compose ... down, as in B.2.1). Note which node you stop last.
2. From the backup (0.4), restore only system.yaml, both rabbitmq.conf files, the cookie file (with the same owner and mode) and .env. Do not restore the whole folder over PostgreSQL data. Write rabbitmq.conf back in place.
3. Start RabbitMQ on the node you stopped last first, then on the others, then Xray (docker compose ... up -d, as in B.2.4). The restored default_pass is applied on start.
4. If guest was already deleted, create it again, with sudo docker exec xray_rabbitmq rabbitmqctl ...: add_user guest "$OLD_PASS", set_user_tags guest administrator, and set_permissions on / and xray_haq.
5. Run the checks from B.1.5 and B.2.5 with the old values.

B.6 Clean Up

• Clear the values from your shell and turn history back on:

  Bash

  unset RMQ_PASSWORD RMQ_COOKIE OLD_PASS OLD_COOKIE
  set -o history

• When the change is confirmed, delete the backups that contain secrets (the copy of $ROOT_DATA_DIR and .env), or move them to secure storage.

• Record the change in your change-management system: date, layout (single node, multi-node or split mode), versions, and the result of each check.

📘

Installed with the JFrog Platform chart (jfrog/jfrog-platform)?

Follow the same steps, but read C.8 JFrog Platform Chart first. The default user, vhosts, resource names and secrets differ.

Step 0 — Prepare (Helm)

Rules

1. One cookie and one password per cluster. All RabbitMQ pods use the same cookie and password from the RabbitMQ secret. RabbitMQ shares users and passwords across the whole cluster, so you run rabbitmqctl add_user once, in pod 0.
2. Change every copy. The password is stored in the RabbitMQ secret and in your Helm values, and the two must match.
3. Do not paste the new values into tickets, chat or source control.
4. All three changes are made in one downtime window. Deleting guest is part of the procedure, not optional.

0.1 Plan a maintenance window and announce it. Scans and other Xray processing stall during the change.

0.2 Generate the new values once, on a trusted host, with shell history turned off:

set +o history
RMQ_PASSWORD=$(openssl rand -hex 24)
RMQ_COOKIE=$(openssl rand -hex 32)

Store both in your secret manager. The examples use the same $RMQ_PASSWORD for the new xray_admin user.

In the shell where you run kubectl and helm, set the new values and the current values (the checks use them):

set +o history
RMQ_PASSWORD='<new password>'
RMQ_COOKIE='<new cookie>'
OLD_PASS='<current password>'
OLD_COOKIE='<current cookie>'

The installer defaults are:

Many installations set their own values during installation, so do not assume the default. Read the current password and cookie from the secret <release>-rabbitmq:

kubectl -n <ns> get secret <release>-rabbitmq -o jsonpath='{.data.rabbitmq-password}' | base64 -d
kubectl -n <ns> get secret <release>-rabbitmq -o jsonpath='{.data.rabbitmq-erlang-cookie}' | base64 -d

0.3 Record the current settings, back up, and check that the system is healthy: see C.0.

On Helm, you make all three changes in one downtime window. Xray and RabbitMQ are stopped, you write the new password and cookie into the RabbitMQ secret, start RabbitMQ, and then one helm upgrade switches Xray to the new user.

C.0 to C.7 use the names of the Xray chart (jfrog/xray). For the JFrog Platform chart (jfrog/jfrog-platform), the steps are the same, but some names and defaults differ; see C.8 JFrog Platform Chart.

In the commands, replace <ns> with your namespace and <release> with your release name.

C.0 Before You Begin

How the chart handles the RabbitMQ credentials. Read this first; it explains each step.

Record your settings:

• Release name, namespace and chart version: helm list -n <ns>.

• Current values: helm get values <release> -n <ns>. Note rabbitmq.auth.username (default guest), rabbitmq.replicaCount, whether rabbitmq.auth.existingPasswordSecret or existingErlangSecret is set, and whether autoscaling (KEDA or HPA) is enabled for any Xray workload.

• The workloads and their replica counts:

  Bash

  kubectl -n <ns> get sts,deploy

  The RabbitMQ StatefulSet is <release>-rabbitmq. The Xray workloads are the StatefulSet <release> and, depending on the chart version and your values, Deployments such as <release>-server and <release>-ipa. Every Xray workload that connects to RabbitMQ is stopped in C.2.

• The cluster and the vhosts. Expect / and xray_haq:

  Bash

  kubectl -n <ns> exec <release>-rabbitmq-0 -c rabbitmq -- rabbitmqctl cluster_status
  kubectl -n <ns> exec <release>-rabbitmq-0 -c rabbitmq -- rabbitmqctl list_vhosts

Back up the values and the RabbitMQ secret:

helm get values <release> -n <ns> > values-backup.yaml
kubectl -n <ns> get secret <release>-rabbitmq -o yaml > rabbitmq-secret-backup.yaml

Prepare the values file. Put the new user name, password and cookie in a separate values file, rabbitmq-auth.yaml. Store it in your secret manager, do not commit it, and pass it on every helm upgrade from now on:

# rabbitmq-auth.yaml
rabbitmq:
  auth:
    username: xray_admin
    password: "<RMQ_PASSWORD>"
    erlangCookie: "<RMQ_COOKIE>"

The password and cookie in this file must be the same values you write into the secret in C.3. The values are used for the RabbitMQ definitions and when the secret is created again, for example on a new installation.

Health check. Run it before you start and after each step. All services must show "state" : "HEALTHY":

kubectl -n <ns> exec <release>-0 -c router -- curl -s http://localhost:8082/router/api/v1/system/health

C.1 Create the New User

RabbitMQ is still running. Create the new user once, in pod 0; users are shared across the cluster.

kubectl -n <ns> exec <release>-rabbitmq-0 -c rabbitmq -- rabbitmqctl add_user xray_admin "$RMQ_PASSWORD"
kubectl -n <ns> exec <release>-rabbitmq-0 -c rabbitmq -- rabbitmqctl set_user_tags xray_admin administrator
kubectl -n <ns> exec <release>-rabbitmq-0 -c rabbitmq -- rabbitmqctl set_permissions -p / xray_admin ".*" ".*" ".*"
kubectl -n <ns> exec <release>-rabbitmq-0 -c rabbitmq -- rabbitmqctl set_permissions -p xray_haq xray_admin ".*" ".*" ".*"
kubectl -n <ns> exec <release>-rabbitmq-0 -c rabbitmq -- rabbitmqctl authenticate_user xray_admin "$RMQ_PASSWORD"

Expect Success. If list_vhosts printed another vhost that Xray uses, run set_permissions for it too. Leave guest in place for now: Xray still uses it, and it is your fallback if you roll back.

C.2 Stop Xray

This starts the downtime window.

1. If KEDA autoscaling is enabled for an Xray workload, pause it so it does not scale the workload up again:

   Bash

   kubectl -n <ns> annotate scaledobject <scaledobject> autoscaling.keda.sh/paused-replicas="0" --overwrite

2. Scale every Xray workload that connects to RabbitMQ to 0, and wait until its pods are gone. For example:

   Bash

   kubectl -n <ns> scale sts/<release> --replicas=0
   kubectl -n <ns> get pods -w

   Do the same for each Xray Deployment, for example kubectl -n <ns> scale deploy/<release>-server --replicas=0.

C.3 Stop RabbitMQ, Change the Secret, Start RabbitMQ

1. Note the RabbitMQ replica count (for example 3), then scale RabbitMQ to 0 and wait until all its pods are gone:

   Bash

   kubectl -n <ns> get sts <release>-rabbitmq
   kubectl -n <ns> scale sts/<release>-rabbitmq --replicas=0
   kubectl -n <ns> get pods -w

2. Write the new password and cookie into the secret:

   Bash

   kubectl -n <ns> patch secret <release>-rabbitmq --type merge -p "{\"stringData\":{\"rabbitmq-password\":\"$RMQ_PASSWORD\",\"rabbitmq-erlang-cookie\":\"$RMQ_COOKIE\"}}"

3. Check the secret. Compare checksums, so the values are never printed; the first two lines must match, and so must the last two:

   Bash

   kubectl -n <ns> get secret <release>-rabbitmq -o jsonpath='{.data.rabbitmq-password}' | base64 -d | sha256sum
   printf '%s' "$RMQ_PASSWORD" | sha256sum
   kubectl -n <ns> get secret <release>-rabbitmq -o jsonpath='{.data.rabbitmq-erlang-cookie}' | base64 -d | sha256sum
   printf '%s' "$RMQ_COOKIE" | sha256sum

4. Start RabbitMQ with kubectl, not with helm upgrade, with the replica count from step 1, and wait until all pods are ready:

   Bash

   kubectl -n <ns> scale sts/<release>-rabbitmq --replicas=<replica count from step 1>
   kubectl -n <ns> rollout status sts/<release>-rabbitmq

   With podManagementPolicy: Parallel, all pods start together. With OrderedReady, pod 1 starts only after pod 0 is ready; if pod 0 waits for its peers, see Troubleshooting (C.9).

5. Check RabbitMQ:

   • kubectl -n <ns> exec <release>-rabbitmq-0 -c rabbitmq -- rabbitmqctl cluster_status lists all nodes and shows no network partitions.

   • kubectl -n <ns> exec <release>-rabbitmq-0 -c rabbitmq -- rabbitmqctl authenticate_user xray_admin "$RMQ_PASSWORD" prints Success.

   • The cookie is the same on every pod and matches the new value. Run the first command for each pod (-0, -1, -2, ...). The cookie file ends with a newline, so the second command uses '%s\n':

     Bash

     kubectl -n <ns> exec <release>-rabbitmq-0 -c rabbitmq -- sha256sum /opt/bitnami/rabbitmq/.rabbitmq/.erlang.cookie
     printf '%s\n' "$RMQ_COOKIE" | sha256sum

📘

If the release uses rabbitmq.auth.existingPasswordSecret or existingErlangSecret

Patch those secrets in step 2 instead of, or as well as, <release>-rabbitmq, with the same keys (rabbitmq-password, rabbitmq-erlang-cookie). With existingPasswordSecret, the chart reads the password for the RabbitMQ definitions from that secret, so it must hold the new password before you run helm upgrade.

C.4 Switch Xray to the New User with helm upgrade

RabbitMQ is running, so the pre-upgrade hook can reach it. Run the upgrade with the same chart version, your values, and rabbitmq-auth.yaml:

helm upgrade <release> jfrog/xray --version <chart version> -n <ns> --reuse-values -f rabbitmq-auth.yaml

If you manage your values in a file, use -f <your-values.yaml> -f rabbitmq-auth.yaml instead of --reuse-values.

The upgrade:

• changes the user name in the RabbitMQ and Xray pods to xray_admin;
• renders the RabbitMQ definitions with xray_admin and the new password;
• restarts the RabbitMQ pods one at a time (the cluster stays up, with the same cookie);
• sets the Xray workloads back to their configured replica counts, so Xray starts with the new user and the password from the secret.

Then:

1. Wait for the rollouts, and the same for each Xray Deployment:

   Bash

   kubectl -n <ns> rollout status sts/<release>-rabbitmq
   kubectl -n <ns> rollout status sts/<release>

2. If an Xray workload is still at 0 (for example, because autoscaling manages its replicas), scale it back to the count you recorded in C.0.

3. If you paused KEDA, remove the annotation: kubectl -n <ns> annotate scaledobject <scaledobject> autoscaling.keda.sh/paused-replicas-.

C.5 Verify

All checks must pass before you delete guest.

• The secret still holds the new values (repeat the check from C.3 step 3).

• xray_admin works on every RabbitMQ pod (-0, -1, -2, ...):

  Bash

  kubectl -n <ns> exec <release>-rabbitmq-0 -c rabbitmq -- rabbitmqctl authenticate_user xray_admin "$RMQ_PASSWORD"

• The cookie is the same on every pod (C.3 step 5), and the old cookie is rejected. This command must fail with Erlang distribution failed. Use a node name that cluster_status prints:

  Bash

  kubectl -n <ns> exec <release>-rabbitmq-0 -c rabbitmq -- rabbitmq-diagnostics -n rabbit@<release>-rabbitmq-1.<release>-rabbitmq-headless.<ns>.svc.cluster.local --erlang-cookie "$OLD_COOKIE" status

  If you never changed the cookie before, the old value is XRAYRABBITMQCLUSTER.

• cluster_status lists all nodes, with no network partitions.

• Xray uses the new user. All connections must show xray_admin. Do not continue until no connection uses guest.

  Bash

  kubectl -n <ns> exec <release>-rabbitmq-0 -c rabbitmq -- rabbitmqctl list_connections user

• kubectl -n <ns> exec <release>-rabbitmq-0 -c rabbitmq -- rabbitmqctl list_queues -p xray_haq name consumers shows the expected queues with consumers.

• The health check shows all services HEALTHY, and a test scan completes.

If a check fails, see Troubleshooting (C.9), or go to Rollback (C.10).

C.6 Delete guest

1. Delete the user. guest must not be listed afterwards:

   Bash

   kubectl -n <ns> exec <release>-rabbitmq-0 -c rabbitmq -- rabbitmqctl delete_user guest
   kubectl -n <ns> exec <release>-rabbitmq-0 -c rabbitmq -- rabbitmqctl list_users

   On Helm, delete guest rather than restricting it. After C.4 the RabbitMQ definitions contain only xray_admin, so guest is not created again when RabbitMQ restarts.

2. Restart RabbitMQ once, and check that only xray_admin is listed and connected:

   Bash

   kubectl -n <ns> rollout restart sts/<release>-rabbitmq
   kubectl -n <ns> rollout status sts/<release>-rabbitmq
   kubectl -n <ns> exec <release>-rabbitmq-0 -c rabbitmq -- rabbitmqctl list_users
   kubectl -n <ns> exec <release>-rabbitmq-0 -c rabbitmq -- rabbitmqctl list_connections user

3. Check that guest is refused. Run it from a pod that can reach the RabbitMQ service, for example the Xray router container. It must print 401; repeat it with the old password and with password:

   Bash

   kubectl -n <ns> exec <release>-0 -c router -- curl -s -o /dev/null -w '%{http_code}\n' -u "guest:$OLD_PASS" http://<release>-rabbitmq:15672/api/whoami

4. Run a test scan.

C.7 After Every Future helm upgrade

• Always pass rabbitmq-auth.yaml (or keep its keys in your values file). Without it, the user name and the RabbitMQ definitions go back to guest and the old password, and Xray can no longer connect.
• After the upgrade, check that helm get values <release> -n <ns> shows username: xray_admin, that the secret still matches the new values (C.3 step 3), that list_users does not list guest, and that list_connections user shows only xray_admin.
• To rotate the password or cookie again later, repeat C.2 to C.5 with the new values in both the secret and rabbitmq-auth.yaml. Do not change only the values file: the Xray chart keeps the old values in the secret.

C.8 JFrog Platform Chart

If Xray is installed with the JFrog Platform chart (jfrog/jfrog-platform), the procedure is the same as C.0 to C.7. The RabbitMQ settings are still under the top-level rabbitmq: key, so rabbitmq-auth.yaml has the same format. What differs are the resource names, the default user, the vhosts, and one extra secret:

You still change <release>-rabbitmq directly while RabbitMQ is scaled to 0 (C.3). A helm upgrade with a new cookie while RabbitMQ is running would restart the RabbitMQ pods one at a time with different cookies, and the pre-upgrade hook does not allow helm upgrade while RabbitMQ is scaled to 0.

Steps for the JFrog Platform chart. $RMQ_PASSWORD, $RMQ_COOKIE, $OLD_PASS and $OLD_COOKIE are set as in Step 0.

1. Back up and record (C.0):

   Bash

   helm get values <release> -n <ns> > values-backup.yaml
   kubectl -n <ns> get secret <release>-rabbitmq <release>-xray-rabbitmq-creds <release>-load-definition -o yaml > rabbitmq-secrets-backup.yaml
   kubectl -n <ns> exec <release>-rabbitmq-0 -c rabbitmq -- rabbitmqctl list_users
   kubectl -n <ns> exec <release>-rabbitmq-0 -c rabbitmq -- rabbitmqctl list_vhosts

   Expect the user admin, and the vhosts xray and xray_haq. Write rabbitmq-auth.yaml as in C.0.

2. Create the new user (C.1):

   Bash

   kubectl -n <ns> exec <release>-rabbitmq-0 -c rabbitmq -- rabbitmqctl add_user xray_admin "$RMQ_PASSWORD"
   kubectl -n <ns> exec <release>-rabbitmq-0 -c rabbitmq -- rabbitmqctl set_user_tags xray_admin administrator
   kubectl -n <ns> exec <release>-rabbitmq-0 -c rabbitmq -- rabbitmqctl set_permissions -p xray xray_admin ".*" ".*" ".*"
   kubectl -n <ns> exec <release>-rabbitmq-0 -c rabbitmq -- rabbitmqctl set_permissions -p xray_haq xray_admin ".*" ".*" ".*"
   kubectl -n <ns> exec <release>-rabbitmq-0 -c rabbitmq -- rabbitmqctl authenticate_user xray_admin "$RMQ_PASSWORD"

3. Stop Xray (C.2):

   Bash

   kubectl -n <ns> scale sts/<release>-xray --replicas=0

4. Stop RabbitMQ, change the secret, start RabbitMQ (C.3). Run these as separate steps.

   a. Note the RabbitMQ replica count (for example 3), then scale RabbitMQ to 0 and wait until all its pods are gone:

   Bash

   kubectl -n <ns> get sts <release>-rabbitmq
   kubectl -n <ns> scale sts/<release>-rabbitmq --replicas=0
   kubectl -n <ns> get pods -w

   b. Only when no RabbitMQ pod is left, write the new password and cookie into the secret, and run the secret check from C.3 step 3:

   Bash

   kubectl -n <ns> patch secret <release>-rabbitmq --type merge -p "{\"stringData\":{\"rabbitmq-password\":\"$RMQ_PASSWORD\",\"rabbitmq-erlang-cookie\":\"$RMQ_COOKIE\"}}"

   c. Start RabbitMQ with the replica count from step a, then run the RabbitMQ checks from C.3 step 5:

   Bash

   kubectl -n <ns> scale sts/<release>-rabbitmq --replicas=<replica count from step a>
   kubectl -n <ns> rollout status sts/<release>-rabbitmq

5. Run helm upgrade (C.4):

   Bash

   helm upgrade <release> jfrog/jfrog-platform --version <chart version> -n <ns> --reuse-values -f rabbitmq-auth.yaml
   kubectl -n <ns> rollout status sts/<release>-rabbitmq
   kubectl -n <ns> rollout status sts/<release>-xray

   The upgrade renders <release>-xray-rabbitmq-creds with xray_admin and the new password, renders the definitions, restarts the RabbitMQ pods one at a time, and starts Xray again.

6. Verify (C.5). In addition to the C.5 checks, confirm that Xray's secret has the new user and password. Expect xray_admin, then two matching checksums:

   Bash

   kubectl -n <ns> get secret <release>-xray-rabbitmq-creds -o jsonpath='{.data.username}' | base64 -d
   kubectl -n <ns> get secret <release>-xray-rabbitmq-creds -o jsonpath='{.data.password}' | base64 -d | sha256sum
   printf '%s' "$RMQ_PASSWORD" | sha256sum

   Run the health check in <release>-xray-0, container router. The old cookie to test with is secretcookie if you never changed it.

7. Delete admin (C.6). It is the platform chart's default user, with the well-known password password, and rabbitmq.conf allows it to log in from other hosts.

   Bash

   kubectl -n <ns> exec <release>-rabbitmq-0 -c rabbitmq -- rabbitmqctl delete_user admin
   kubectl -n <ns> rollout restart sts/<release>-rabbitmq
   kubectl -n <ns> rollout status sts/<release>-rabbitmq
   kubectl -n <ns> exec <release>-rabbitmq-0 -c rabbitmq -- rabbitmqctl list_users

   Expect only xray_admin. Then check that admin is refused. It must print 401; repeat it with the new password and with password:

   Bash

   kubectl -n <ns> exec <release>-xray-0 -c router -- curl -s -o /dev/null -w '%{http_code}\n' -u "admin:$OLD_PASS" http://<release>-rabbitmq:15672/api/whoami

8. After every future helm upgrade, follow C.7 with these names.

Rollback follows Rollback (C.10), with these changes: create admin again instead of guest, with permissions on xray and xray_haq, and run helm upgrade with jfrog/jfrog-platform.

C.9 Troubleshooting

C.10 Rollback

1. Stop Xray and RabbitMQ as in C.2 and C.3 step 1.
2. Write the old password and cookie back into the secret, as in C.3 step 2, with $OLD_PASS and $OLD_COOKIE. helm upgrade alone does not change the secret.
3. Start RabbitMQ with kubectl scale (C.3 step 4) and wait until it is ready.
4. If guest was already deleted (C.6), create it again, with kubectl -n <ns> exec <release>-rabbitmq-0 -c rabbitmq -- rabbitmqctl ...: add_user guest "$OLD_PASS", set_user_tags guest administrator, and set_permissions on / and xray_haq.
5. Run helm upgrade with your values from before the change (values-backup.yaml from C.0), without rabbitmq-auth.yaml. This sets the user name back to guest, renders the definitions with the old password, and starts Xray again.
6. Check that authenticate_user guest "$OLD_PASS" prints Success, that the connections show guest, and that Xray is HEALTHY.

C.11 Clean Up

• Clear the values from your shell and turn history back on:

  Bash

  unset RMQ_PASSWORD RMQ_COOKIE OLD_PASS OLD_COOKIE
  set -o history

• When the change is confirmed, delete the backups that contain secrets (values-backup.yaml and rabbitmq-secret-backup.yaml), or move them to secure storage.

• Record the change in your change-management system: date, layout (single node, multi-node or split mode), versions, and the result of each check.