Neo4j on Kubernetes

This is a first attempt at getting Neo4j 3.1 to run on Kubernetes.

You can either setup Kubernetes locally using minikube or on one of the cloud providers e.g. Google Compute Engine

Once you’ve got that setup you can run the following command to see where Kubernetes is running. e.g. on my machine

$ kubectl cluster-info
Kubernetes master is running at https://192.168.99.100:8443
KubeDNS is running at https://192.168.99.100:8443/api/v1/proxy/namespaces/kube-system/services/kube-dns
kubernetes-dashboard is running at https://192.168.99.100:8443/api/v1/proxy/namespaces/kube-system/services/kubernetes-dashboard

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Ok. Now we’re ready to install Neo4j.

Core Servers

First let’s install the Core servers. We’d typically have a small number of Core servers in a cluster and they can handle both read and write traffic.

Run the following commands to create the Core servers:

$ ./neo4j.sh
...
persistentvolume "pv0" created
persistentvolumeclaim "datadir-neo4j-core-0" created
persistentvolume "pv1" created
persistentvolumeclaim "datadir-neo4j-core-1" created
persistentvolume "pv2" created
persistentvolumeclaim "datadir-neo4j-core-2" created

$ kubectl create -f neo4j_svc.yaml
service "neo4j" created

$ kubectl create -f neo4j_core.yaml
statefulset "neo4j-core" created

We can check that those servers have spun up by running:

$ kubectl get pods
NAME           READY     STATUS    RESTARTS   AGE
neo4j-core-0   1/1       Running   0          23s
neo4j-core-1   1/1       Running   0          19s
neo4j-core-2   1/1       Running   0          16s

And we can check that Neo4j is up and running by running:

$ kubectl logs neo4j-core-2
Starting Neo4j.
2016-12-07 11:23:23.133+0000 INFO  Starting...
2016-12-07 11:23:24.551+0000 INFO  Bolt enabled on 0.0.0.0:7687.
2016-12-07 11:23:24.566+0000 INFO  Initiating metrics...
2016-12-07 11:23:24.820+0000 INFO  Waiting for other members to join cluster before continuing...
2016-12-07 11:23:45.546+0000 INFO  Started.
2016-12-07 11:23:45.700+0000 INFO  Mounted REST API at: /db/manage
2016-12-07 11:23:46.683+0000 INFO  Remote interface available at http://neo4j-core-2.neo4j.default.svc.cluster.local:7474/

Neo4j also exposes the cluster topology via a procedure:

$ kubectl exec neo4j-core-0 -- bin/cypher-shell "CALL dbms.cluster.overview()"
id, addresses, role
"484178c4-e7ae-46c9-a7d5-aaf6250efc7f", ["bolt://neo4j-core-0.neo4j.default.svc.cluster.local:7687", "http://neo4j-core-0.neo4j.default.svc.cluster.local:7474"], "FOLLOWER"
"0acdb8dd-3bb1-4c76-bbe7-213530af1d23", ["bolt://neo4j-core-1.neo4j.default.svc.cluster.local:7687", "http://neo4j-core-1.neo4j.default.svc.cluster.local:7474"], "LEADER"
"ca9b954c-c245-418f-9803-7790d36efdd9", ["bolt://neo4j-core-2.neo4j.default.svc.cluster.local:7687", "http://neo4j-core-2.neo4j.default.svc.cluster.local:7474"], "FOLLOWER"

Bye!

Read Replicas

Now that we’ve got the Core Servers up and running let’s add some Read Replicas. Read Replicas are used to scale out reads and they don’t accept any write operations.

Run the following command to create a single Read Replica:

$ kubectl create -f neo4j_rr.yaml
replicationcontroller "neo4j-replica" created

If we query for the list of pods now we’ll see that it’s been added to the list:

$ kubectl get pods
NAME                  READY     STATUS    RESTARTS   AGE
neo4j-core-0          1/1       Running   0          4m
neo4j-core-1          1/1       Running   0          4m
neo4j-core-2          1/1       Running   0          4m
neo4j-replica-rdwsh   1/1       Running   0          9s

Let’s check if the replica has joined the cluster and is ready to go:

$ kubectl logs neo4j-replica-rdwsh
Starting Neo4j.
2016-12-07 11:27:31.388+0000 INFO  Starting...
2016-12-07 11:27:33.072+0000 INFO  Bolt enabled on 0.0.0.0:7687.
2016-12-07 11:27:33.094+0000 INFO  Initiating metrics...
2016-12-07 11:27:39.693+0000 INFO  Started.
2016-12-07 11:27:39.878+0000 INFO  Mounted REST API at: /db/manage
2016-12-07 11:27:41.172+0000 INFO  Remote interface available at http://neo4j-replica-rdwsh:7474/

Yep, looks good!

Let’s check that Neo4j knows about our new server:

$ kubectl exec neo4j-core-0 -- bin/cypher-shell "CALL dbms.cluster.overview()"
id, addresses, role
"484178c4-e7ae-46c9-a7d5-aaf6250efc7f", ["bolt://neo4j-core-0.neo4j.default.svc.cluster.local:7687", "http://neo4j-core-0.neo4j.default.svc.cluster.local:7474"], "FOLLOWER"
"0acdb8dd-3bb1-4c76-bbe7-213530af1d23", ["bolt://neo4j-core-1.neo4j.default.svc.cluster.local:7687", "http://neo4j-core-1.neo4j.default.svc.cluster.local:7474"], "LEADER"
"ca9b954c-c245-418f-9803-7790d36efdd9", ["bolt://neo4j-core-2.neo4j.default.svc.cluster.local:7687", "http://neo4j-core-2.neo4j.default.svc.cluster.local:7474"], "FOLLOWER"
"00000000-0000-0000-0000-000000000000", ["bolt://neo4j-replica-rdwsh:7687", "http://neo4j-replica-rdwsh:7474"], "READ_REPLICA"

Bye!

It does indeed.

Now let’s scale up to 3 read replicas:

$ kubectl scale rc neo4j-replica --replicas=3
replicationcontroller "neo4j-replica" scaled

And give it a few seconds and Neo4j will know about those servers as well:

$ kubectl exec neo4j-core-0 -- bin/cypher-shell "CALL dbms.cluster.overview()"
id, addresses, role
"484178c4-e7ae-46c9-a7d5-aaf6250efc7f", ["bolt://neo4j-core-0.neo4j.default.svc.cluster.local:7687", "http://neo4j-core-0.neo4j.default.svc.cluster.local:7474"], "FOLLOWER"
"0acdb8dd-3bb1-4c76-bbe7-213530af1d23", ["bolt://neo4j-core-1.neo4j.default.svc.cluster.local:7687", "http://neo4j-core-1.neo4j.default.svc.cluster.local:7474"], "LEADER"
"ca9b954c-c245-418f-9803-7790d36efdd9", ["bolt://neo4j-core-2.neo4j.default.svc.cluster.local:7687", "http://neo4j-core-2.neo4j.default.svc.cluster.local:7474"], "FOLLOWER"
"00000000-0000-0000-0000-000000000000", ["bolt://neo4j-replica-48r7z:7687", "http://neo4j-replica-48r7z:7474"], "READ_REPLICA"
"00000000-0000-0000-0000-000000000000", ["bolt://neo4j-replica-j28g0:7687", "http://neo4j-replica-j28g0:7474"], "READ_REPLICA"
"00000000-0000-0000-0000-000000000000", ["bolt://neo4j-replica-rdwsh:7687", "http://neo4j-replica-rdwsh:7474"], "READ_REPLICA"

Bye!

Bit of explanation

For the core servers we’re using a Kubernetes beta feature introduced in 1.5.0 called StatefulSets. This is helpful because Neo4j Core Servers need to discover each other so that they can take participate in a consensus commit algorithm. We therefore need to know the hostnames of these servers so that we can specify it in the config file.

For the read replicas we don’t have this constraint so we can use the standard replication controller approach. Read Replicas list the hostnames of Core Servers in their config file.