From f60d9f3e3fff3fae8b023d57fb3e342905d9d43c Mon Sep 17 00:00:00 2001 From: Gabriel Mougard Date: Wed, 3 Apr 2024 20:06:44 +0200 Subject: [PATCH] doc: Add documentation on how to set up a dedicated Ceph cluster network Signed-off-by: Gabriel Mougard --- doc/.custom_wordlist.txt | 4 + doc/.wordlist.txt | 2 +- doc/explanation/microcloud.rst | 6 + doc/how-to/ceph_networking.rst | 127 ++++ doc/how-to/index.rst | 1 + doc/how-to/initialise.rst | 1 + doc/how-to/preseed.yaml | 5 + doc/images/ceph_network_full_setup.svg | 765 ++++++++++++++++++++++ doc/images/ceph_network_partial_setup.svg | 761 +++++++++++++++++++++ doc/images/ceph_network_usual_setup.svg | 634 ++++++++++++++++++ doc/tutorial/get_started.rst | 3 +- 11 files changed, 2307 insertions(+), 2 deletions(-) create mode 100644 doc/how-to/ceph_networking.rst create mode 100644 doc/images/ceph_network_full_setup.svg create mode 100644 doc/images/ceph_network_partial_setup.svg create mode 100644 doc/images/ceph_network_usual_setup.svg diff --git a/doc/.custom_wordlist.txt b/doc/.custom_wordlist.txt index c94d9715d..988ff141c 100644 --- a/doc/.custom_wordlist.txt +++ b/doc/.custom_wordlist.txt @@ -35,3 +35,7 @@ VLAN VM VMs ZFS +disaggregated +subnets +GbE +QSFP \ No newline at end of file diff --git a/doc/.wordlist.txt b/doc/.wordlist.txt index d8479de93..30f63cae6 100644 --- a/doc/.wordlist.txt +++ b/doc/.wordlist.txt @@ -36,4 +36,4 @@ UI Jira VM YAML -CephFS +CephFS \ No newline at end of file diff --git a/doc/explanation/microcloud.rst b/doc/explanation/microcloud.rst index 5a3827e07..9961f0beb 100644 --- a/doc/explanation/microcloud.rst +++ b/doc/explanation/microcloud.rst @@ -64,6 +64,12 @@ MicroCloud will still be usable, but you will see some limitations: As a result of this, network forwarding works at a basic level only, and external addresses must be forwarded to a specific cluster member and don't fail over. - There is no support for hardware acceleration, load balancers, or ACL functionality within the local network. +Dedicated internal network for Ceph +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +You can set up a dedicated network for Ceph to separate the internal Ceph traffic from the rest of the MicroCloud cluster traffic. + +See :ref:`howto-ceph-networking` for how to set up a dedicated internal network for Ceph. + Storage ------- diff --git a/doc/how-to/ceph_networking.rst b/doc/how-to/ceph_networking.rst new file mode 100644 index 000000000..c4824d1a3 --- /dev/null +++ b/doc/how-to/ceph_networking.rst @@ -0,0 +1,127 @@ +.. _howto-ceph-networking: + +How to configure Ceph networking +================================ + +When running :command:`microcloud init`, you are asked if you want to provide a custom subnet for the Ceph cluster. +Here is the question you will be asked: + +- ``What subnet (either IPv4 or IPv6 CIDR notation) would you like your Ceph internal traffic on? [default: 203.0.113.0/24]: `` + +You can choose to skip this question (just hit ``Enter``) and use the default value which is the subnet used for the internal MicroCloud traffic. +This is referred to as a *usual* Ceph networking setup. + +.. figure:: /images/ceph_network_usual_setup.svg + :alt: All the Ceph traffic is on the same network interface + :align: center + +Sometimes, you want to be able to use different network interfaces for some Ceph related usages. +Let's imagine you have machines with network interfaces that are tailored for high throughput and low latency data transfer, +like 100 GbE+ QSFP links, and other ones that might be more suited for management traffic, like 1 GbE or 10 GbE links. + +In this case, it would probably be ideal to set your Ceph internal (or cluster) traffic on the high throughput network interface. This is referred to as a *partially disaggregated* Ceph networking setup. + +.. figure:: /images/ceph_network_partial_setup.svg + :alt: The Ceph internal traffic uses a dedicated high throughput network interface. + :align: center + +To use a partially disaggregated Ceph networking setup with your MicroCloud, specify the corresponding subnets during the MicroCloud initialisation process. + +The following instructions build on the :ref:`get-started` tutorial and show how you can test setting up a MicroCloud with disaggregated Ceph networking inside a LXD setup. + +1. Create the dedicated network for Ceph: + + #. First, just like when you created an uplink network for MicroCloud so that the cluster members could have external connectivity, you will need to create a dedicated network for the Ceph cluster members to communicate with each other. Let's call it ``cephbr0``:: + + lxc network create cephbr0 + + #. Enter the following commands to find out the assigned IPv4 and IPv6 addresses for the networks and note them down:: + + lxc network get cephbr0 ipv4.address + lxc network get cephbr0 ipv6.address + +2. Create the network interfaces that will be used for the Ceph networking setup for each VM: + + #. Add the network device for the ``cephbr0`` network:: + + lxc config device add micro1 eth2 nic network=cephbr0 name=eth2 + lxc config device add micro2 eth2 nic network=cephbr0 name=eth2 + lxc config device add micro3 eth2 nic network=cephbr0 name=eth2 + lxc config device add micro4 eth2 nic network=cephbr0 name=eth2 + +3. Now, just like in the tutorial, start the VMs. + +4. On each VM, bring the network interfaces up and give them an IP address within their network subnet: + + #. For the ``cephbr0`` network, do the following for each VM:: + + # If the `cephbr0` gateway address is `10.0.1.1/24` (subnet should be `10.0.1.0/24`) + ip link set enp7s0 up + # `X` should be a number between 2 and 254, different for each VM + ip addr add 10.0.1.X/24 dev enp7s0 + +5. Now, you can start the MicroCloud initialisation process and provide the subnets you noted down in step 1.c when asked for the Ceph networking subnets. + +#. We will use ``cephbr0`` for the Ceph internal traffic. In a production setup, you'd choose the fast subnet for the internal Ceph traffic:: + + What subnet (either IPv4 or IPv6 CIDR notation) would you like your Ceph internal traffic on? [default: 203.0.113.0/24]: 10.0.1.0/24 + + Interface "enp7s0" ("10.0.1.3") detected on cluster member "micro2" + Interface "enp7s0" ("10.0.1.4") detected on cluster member "micro3" + Interface "enp7s0" ("10.0.1.2") detected on cluster member "micro1" + +7. The MicroCloud initialisation process will now continue as usual and the Ceph cluster will be configured with the networking setup you provided. +8. You can now inspect the Ceph network setup: + + a. Inspect the Ceph configuration file: + + .. terminal:: + :input: microceph.ceph config dump + :user: root + :host: micro1 + :scroll: + + WHO MASK LEVEL OPTION VALUE RO + global advanced cluster_network 10.0.1.0/24 * + global advanced osd_pool_default_crush_rule 2 + + b. Inspect your Ceph-related network traffic: + + .. terminal:: + :input: lxc launch ubuntu:22.04 u5 -s remote + :user: root + :host: micro1 + :scroll: + + Creating c1 + Starting c1 + + c. At the same time, observe the Ceph traffic on the ``enp7s0`` (or ``enp8s0`` in a fully disaggregated setup) interface (on any cluster member) using ``tcpdump``: + + .. terminal:: + :input: tcpdump -i enp7s0 + :user: root + :host: micro2 + :scroll: + + 17:48:48.600971 IP 10.0.1.4.6804 > micro1.48746: Flags [P.], seq 329386555:329422755, ack 245889462, win 24576, options [nop,nop,TS val 3552095031 ecr 3647909539], length 36200 + 17:48:48.601012 IP micro1.48746 > 10.0.1.4.6804: Flags [.], ack 329386555, win 24317, options [nop,nop,TS val 3647909564 ecr 3552095031], length 0 + 17:48:48.600971 IP 10.0.1.4.6804 > micro1.48746: Flags [P.], seq 329422755:329451715, ack 245889462, win 24576, options [nop,nop,TS val 3552095031 ecr 3647909563], length 28960 + 17:48:48.601089 IP 10.0.1.4.6804 > micro1.48746: Flags [P.], seq 329451715:329516875, ack 245889462, win 24576, options [nop,nop,TS val 3552095031 ecr 3647909563], length 65160 + 17:48:48.601089 IP 10.0.1.4.6804 > micro1.48746: Flags [P.], seq 329516875:329582035, ack 245889462, win 24576, options [nop,nop,TS val 3552095031 ecr 3647909563], length 65160 + 17:48:48.601089 IP 10.0.1.4.6804 > micro1.48746: Flags [P.], seq 329582035:329624764, ack 245889462, win 24576, options [nop,nop,TS val 3552095031 ecr 3647909563], length 42729 + 17:48:48.601204 IP micro1.48746 > 10.0.1.4.6804: Flags [.], ack 329624764, win 23357, options [nop,nop,TS val 3647909564 ecr 3552095031], length 0 + 17:48:48.601206 IP 10.0.1.4.6803 > micro1.33328: Flags [P.], seq 938255:938512, ack 359644195, win 24576, options [nop,nop,TS val 3552095031 ecr 3647909540], length 257 + 17:48:48.601310 IP micro1.48746 > 10.0.1.4.6804: Flags [P.], seq 245889462:245889506, ack 329624764, win 24576, options [nop,nop,TS val 3647909564 ecr 3552095031], length 44 + 17:48:48.602839 IP micro1.48746 > 10.0.1.4.6804: Flags [P.], seq 245889506:245889707, ack 329624764, win 24576, options [nop,nop,TS val 3647909566 ecr 3552095031], length 201 + 17:48:48.602947 IP 10.0.1.4.6804 > micro1.48746: Flags [.], ack 245889707, win 24576, options [nop,nop,TS val 3552095033 ecr 3647909564], length 0 + 17:48:48.602975 IP 10.0.1.4.6804 > micro1.48746: Flags [P.], seq 329624764:329624808, ack 245889707, win 24576, options [nop,nop,TS val 3552095033 ecr 3647909564], length 44 + 17:48:48.603028 IP 10.0.1.4.6803 > micro1.33328: Flags [P.], seq 938512:938811, ack 359644195, win 24576, options [nop,nop,TS val 3552095033 ecr 3647909540], length 299 + 17:48:48.603053 IP micro1.33328 > 10.0.1.4.6803: Flags [.], ack 938811, win 1886, options [nop,nop,TS val 3647909566 ecr 3552095031], length 0 + 17:48:48.604594 IP micro1.33328 > 10.0.1.4.6803: Flags [P.], seq 359644195:359709355, ack 938811, win 1886, options [nop,nop,TS val 3647909568 ecr 3552095031], length 65160 + 17:48:48.604644 IP micro1.33328 > 10.0.1.4.6803: Flags [P.], seq 359709355:359774515, ack 938811, win 1886, options [nop,nop,TS val 3647909568 ecr 3552095031], length 65160 + 17:48:48.604688 IP micro1.33328 > 10.0.1.4.6803: Flags [P.], seq 359774515:359839675, ack 938811, win 1886, options [nop,nop,TS val 3647909568 ecr 3552095031], length 65160 + 17:48:48.604733 IP micro1.33328 > 10.0.1.4.6803: Flags [P.], seq 359839675:359904835, ack 938811, win 1886, options [nop,nop,TS val 3647909568 ecr 3552095031], length 65160 + 17:48:48.604751 IP 10.0.1.4.6803 > micro1.33328: Flags [.], ack 359709355, win 24317, options [nop,nop,TS val 3552095035 ecr 3647909568], length 0 + 17:48:48.604757 IP micro1.33328 > 10.0.1.4.6803: Flags [P.], seq 359904835:359910746, ack 938811, win 1886, options [nop,nop,TS val 3647909568 ecr 3552095035], length 5911 + 17:48:48.604797 IP micro1.33328 > 10.0.1.4.6803: Flags [P.], seq 359910746:359975906, ack 938811, win 1886, options [nop,nop,TS val 3647909568 ecr 3552095035], length 65160 \ No newline at end of file diff --git a/doc/how-to/index.rst b/doc/how-to/index.rst index 3525d268b..81ff2ae38 100644 --- a/doc/how-to/index.rst +++ b/doc/how-to/index.rst @@ -11,6 +11,7 @@ These how-to guides cover key operations and processes in MicroCloud. Install MicroCloud Manage the snaps Initialise MicroCloud + Configure Ceph networking Add a machine Get support Contribute to MicroCloud diff --git a/doc/how-to/initialise.rst b/doc/how-to/initialise.rst index 5ceea16fa..d71c4f2e1 100644 --- a/doc/how-to/initialise.rst +++ b/doc/how-to/initialise.rst @@ -73,6 +73,7 @@ Complete the following steps to initialise MicroCloud: Wiping a disk will destroy all data on it. #. You can choose to optionally set up a CephFS distributed file system. +#. Select either an IPv4 or IPv6 CIDR subnet for the Ceph internal traffic. You can leave it empty to use the default value, which is the MicroCloud internal network (see :ref:`howto-ceph-networking` for how to configure it). #. Select whether you want to set up distributed networking (using MicroOVN). If you choose ``yes``, configure the distributed networking: diff --git a/doc/how-to/preseed.yaml b/doc/how-to/preseed.yaml index 62ccca0a7..b185f6f20 100644 --- a/doc/how-to/preseed.yaml +++ b/doc/how-to/preseed.yaml @@ -24,6 +24,11 @@ systems: - name: micro04 ovn_uplink_interface: eth1 +# `ceph` is optional and represents the Ceph global configuration +ceph: + internal_network: 10.0.1.0/24 + public_network: 10.0.2.0/24 + # `ovn` is optional and represents the OVN & uplink network configuration for LXD. ovn: ipv4_gateway: 192.0.2.1/24 diff --git a/doc/images/ceph_network_full_setup.svg b/doc/images/ceph_network_full_setup.svg new file mode 100644 index 000000000..e512d993a --- /dev/null +++ b/doc/images/ceph_network_full_setup.svg @@ -0,0 +1,765 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/doc/images/ceph_network_partial_setup.svg b/doc/images/ceph_network_partial_setup.svg new file mode 100644 index 000000000..9b37ce72b --- /dev/null +++ b/doc/images/ceph_network_partial_setup.svg @@ -0,0 +1,761 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/doc/images/ceph_network_usual_setup.svg b/doc/images/ceph_network_usual_setup.svg new file mode 100644 index 000000000..cfe19187d --- /dev/null +++ b/doc/images/ceph_network_usual_setup.svg @@ -0,0 +1,634 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + \ No newline at end of file diff --git a/doc/tutorial/get_started.rst b/doc/tutorial/get_started.rst index 8b73b74ab..d4c649d2a 100644 --- a/doc/tutorial/get_started.rst +++ b/doc/tutorial/get_started.rst @@ -269,6 +269,7 @@ Complete the following steps: #. Select all listed disks (these should be ``remote1``, ``remote2``, and ``remote3``). #. You don't need to wipe any disks (because we just created them). #. Select ``yes`` to optionally configure the CephFS distributed file system. + #. Leave the question empty for the IPv4 or IPv6 CIDR subnet address used for the Ceph internal network. #. Select ``yes`` to configure distributed networking. #. Select all listed network interfaces (these should be ``enp6s0`` on the four different VMs). #. Specify the IPv4 address that you noted down for your ``microbr0`` network as the IPv4 gateway. @@ -537,7 +538,7 @@ You can now inspect your cluster setup. total space: 29.67GiB used by: {} -#. Inspect the network setup: +#. Inspect the OVN network setup: .. terminal:: :input: lxc network list