Easy Replication

[psytron]

Would it be possible to have a replication screen in Studio?
With option to select Local or Remote database, and then click replicate to have them magically synchronize like a mesh fungus.

For easy backups and moving datasets around, prod to local test etc.

[lvca]

@psytron I like this idea. We can absolutely create a new tab in Studio to host HA configuration, statistics, and manual operations. I have to wrap my head and come up with the stuff we already have via API to create a meaningful page for the user. I'll follow up on this discussion in a few days. Anybody can contribute with idea!

[lvca]

Following the settings regarding HA we could display/manage from Studio:

Setting	Description	Type	Default
ha.clusterName	Cluster name. By default is 'arcadedb'. Useful in case of multiple clusters in the same network	String	arcadedb
ha.enabled	True if HA is enabled for the current server	Boolean	false
ha.k8s	The server is running inside Kubernetes	Boolean	false
ha.k8sSuffix	When running inside Kubernetes use this suffix to reach the other servers. Example: arcadedb.default.svc.cluster.local	String
ha.quorum	Default quorum between 'none', 1, 2, 3, 'majority' and 'all' servers. Default is majority	String	MAJORITY
ha.quorumTimeout	Timeout waiting for the quorum	Long	10000
ha.replicationChunkMaxSize	Maximum channel chunk size for replicating messages between servers. Default is 16777216	Integer	16777216
ha.replicationFileMaxSize	Maximum file size for replicating messages between servers. Default is 1GB	Long	1073741824
ha.replicationIncomingHost	TCP/IP host name used for incoming replication connections. By default is 0.0.0.0 (listens to all the configured network interfaces)	String	0.0.0.0
ha.replicationIncomingPorts	TCP/IP port number used for incoming replication connections	String	2424-2433
ha.replicationQueueSize	Queue size for replicating messages between servers	Integer	512
ha.serverList	Servers in the cluster as a list of <hostname/ip-address:port> items separated by comma. Example: localhost:2424,192.168.0.1:2424	String

[lvca]

This is just a sketch

[lvca]

The current server where you're logged is at the center. You can see the server names, their IPs, the status and if online the latency, otherwise when they went offline. It should be easy to add/remove servers.

We could also show latest transaction committed, so you can see how far behind are servers.

[emdax]

It would be great if this system could also be used to migrate data, backup data and reflow specific datasets.

Say you're working with a dataset and your schema holds risk of somehow corrupting your data: 1. replicate backup copy. 2. run app 3. check corrupt 4. delete whole database and replicate from backup instance.

Would be great if this setup would also allow you to pull down a production dataset into local for experimenting , and keep it synced.

So many possibilities with this great feature. Arcade DB rules.

[lvca]

Without waiting too much for a nice GUI, I was thinking on providing the features first. I've just added a new "Servers" tab in Studio with some basic information.

The next step is to display the databases and implement some actions, like:

• force a full sync of a database
• add and remove servers

[lvca]

Created an issue to track the progress: #749

[emdax]

Maybe in your list of all servers there could be a drop down on each line to select target ?
🤖-Server1--[ address ][ user ][ pw ]--target:[ Server3 ][ db ]🔄Sync▶️
🤖-Server2--[ address ][ user ][ pw ]--target:[ select ][ db ]🔄Sync▶️
🤖-Server3--[ address ][ user ][ pw ]--target:[ select ][ db ]🔄Sync▶️
🤖-Server4--[ address ][ user ][ pw ]--target:[ Server3 ][ db ]🔄Sync▶️

And then drop down for targets is automatically populated with all servers in main list

[emdax]

Ok. Got it: #749 (comment)

[lvca]

On the main there is a nice update of the cluster page: it enlists the replicated databases with a button to force an alignment.

The alignment is a command that computes a checksum of the file locally, sends it over all the replicas, check if the checksum is the same, and otherwise asks for the portions that are different from the leader.

In normal conditions, databases are always aligned, but if it's not the case you can "sync", or align, the database from the leader to the replicas.

[psytron]

Just wanted to add to this discussion. From a PlayWithData frame of thinking, it would be ideal if users didn't have to struggle with "HA configuration" or "leader election". Basically enable the users to just Play With Data without getting stuck in a swamp of complex configuration. Ideal scenario would be pasting a URL and the system is just like a smart mesh, it figures itself out and synchronizes everything auto-magically.

[lvca]

Actually, it's like this: you just enable arcadedb.ha.enabled=true setting and pass one or a list of other servers. They see each other, one will be elected leader, etc.

It's all transparent, so the replication. You shouldn't need to synchronize manually things, unless there is an edge case we didn't cover or a bug in the replication.