Mirroring local clusters to the Cloud
This tutorial is a demonstration of mirroring locally with two clusters on docker.
Home and Remote
To undestand mirroring, we need to undestand what is a Home and a Remote cluster:
- Home cluster is the target cluster that will receive and consume data.
- Remote cluster is the source cluster that will send data.
Install Fluvio
Create a Cloud account
Using your web browser, navigate to https://infinyon.cloud/ui/signup, where this experimental feature is available.
After the account is created, you will be placed in the Dashboard. You may choose to create a cluster in the GUI. In this tutorial, we'll create a cluster using the CLI later.
Download fluvio binary
Use curl to download and install:
curl -fsS https://hub.infinyon.cloud/install/install.sh | bash
Make sure to add .fluvio/bin to the $PATHas specified in the installation script.
Login to InfinyOn Cloud
Login to InfinyOn Cloud:
fluvio cloud login --use-oauth2
Leave out --use-oauth2 if you prefer username/password method.
Provision a new Cluster
Let's provision a new cloud cluster
fluvio cloud cluster create
Check the result with:
fluvio cluster status
Next, we'll configure the cluster to receive traffic from the remote clusters.
Register Remote clusters on the Home
Use the remote CLI to register the remote clusters with the home cluster:
fluvio remote register edge-remote
List remote clusters to check their status:
fluvio remote list
It should show the following:
REMOTE SC STATUS SPU STATUS LAST SEEN ERRORS
edge-remote Waiting Waiting - -
Create the mirror topic
Mirror topics on the home clusters has multiple partitions, where each partition has a 1-to-1 relationship with the remote cluster.
Create a partition assignment file to define the remote devices:
echo '["edge-remote"]' > assignment_file.json
Apply the configuration file to create the topic:
fluvio topic create mirror-topic --mirror-apply assignment_file.json
List partitions to check the assignment:
fluvio partition list
It should display all partitions:
TOPIC PARTITION LEADER MIRROR REPLICAS RESOLUTION SIZE HW LEO LRS FOLLOWER OFFSETS
mirror-topic 0 5001 edge-remote [] Online 0 B 0 0 0 0 []
Generate Metadata for Remote Clusters from the Cloud Cluster
Each remote cluster requires a unique metadata file that gives the remote cluster the information to connect to the home cluster and the topic/mirror where the data is synchronized.
Generate a metadata file for the remote:
fluvio cloud remote export edge-remote --file edge-remote.json
Connect to the Home Cluster from the Remote
First, we'll start a local cluster:
fluvio cluster start
Then, connect to the home cluster:
fluvio home connect --file edge-remote.json
Let's check the partitions:
fluvio partition list
The remote device should show the following partition::
TOPIC PARTITION LEADER MIRROR REPLICAS RESOLUTION SIZE HW LEO LRS FOLLOWER OFFSETS
mirror-topic 0 5001 c7f891c9-f1f9-4d28-b181-9777d8d07731:0:router.infinyon.cloud:9005 [] Online 0 B 0 0 0 0 []
Also, check the home status with:
fluvio home status
It should show the following:
HOME ROUTE SC STATUS SPU STATUS LAST SEEN ERRORS
c7f891c9-f1f9-4d28-b181-9777d8d07731 router.infinyon.cloud:9003 Connected Connected 1s -
Producing and Consuming on Mirroring
Let's produce on the remote and consume from the home cluster.
Produce to remote cluster
Produce with:
fluvio produce mirror-topic
> A
Ok!
> B
Ok!
Consume from Home cluster
First, switch to the cloud profile:
fluvio profile switch <your-cloud-profile>
To know your cloud profile, run:
fluvio profile list
Then, consume with:
fluvio consume mirror-topic --mirror docker-remote -B
A
B
🎉 Congratulations! You have successfully tested mirroring on the cloud.