Help us for better understand of the ConfiguredGraphFactory

[rayala-nag]

We preferred to use ConfiguredGraphFactory as we have to store data using multi tenant approach. We have around 650 customers data and want to store this data with JanusGraph. We created these many graphs to persist the data. We want to understand ConfiguredGraphFactory approach is good for our use case or not?. Below are our observations from this.

1. Hosted JanusGraph service is requiring more heap space. Right now it's utilising 5GB out of 60GB. Some times JanusGraph service giving OOM and creating heap dumps with size of 50GB's. We are unable to analyse that much size to understand the issues.
2. The customer provision is happening seamlessly and JanusGraph would require more heap in this case. How we can address this memory issue if we onboarding more clients.
3. Another issue we are facing with index creation. If we want to place any new index, we have to create with all the Graphs(650 in our case). We are unable to create new indexes if data ingestions are in progress. The indexes are holding with status of INSTALLED in this case.
4. Another challenge is we have a Graph schema with 200+ properties, 150+ edges, 100 Vertex labels and 200+ indexes. We have to create this schema during new customer provision. This schema creation is taking around 30mins and my service is kept waiting to central provision service until this.
5. We are using ApacheCassandra as backend store. Is this good choice to use with these many configured Graphs?
6. Observing JanusGraph is failing to establish new connections from gremlin console.
7. Unable to collect metrics as well. I tried to export by using jmx-promethus exporter. But JanusGraph service hosted instance CPU's utilisation touched high.

We needed JanusGraph expert's opinion on this issues. It will help us to scale Graph better.

[rngcntr]

Regarding (1) and (2): What is your cache configuration in JanusGraph and what are your JVM settings, especially Xms and Xmx? In the past, we observed high memory utilization by JanusGraph due to an oversized cache.

[FlorianHockmann]

(4): Did you try executing your schema creation script on a separate environment where JanusGraph and Cassandra are mostly idle? Would be interesting to know how long it takes there. 30 minutes definitely sound extremely long.

(5): Cassandra is definitely a solid choice and it has the advantage that you can easily scale it up if its load is too high. You could try whether Scylla performs better, especially if you have a high number of requests in parallel. (I would try this with a test environment first where you try to replicate a scenario that closely matches your production use case.)

(6) and (7): Do you maybe just have too few JanusGraph Server instances for the number of traversals you are trying to execute in parallel? Metrics would of course help to answer this question, but you could also just try to scale JanusGraph up and see whether it helps. You should definitely be able to export metrics.

[rayala-nag]

@FlorianHockmann Thanks for your response
Yes, I've tried schema creation where JanusGraph & Cassandra idle. I've executed the same script while launching the one of the new environment. Observed it took same amount of time.

As i mentioned we have 600+ graphs and traversal bindings are not happening during JanusGraph service startup when we using jmx-promethus exporter.

[FlorianHockmann]

Executing your schema creation script takes about 30 minutes with an idle backend and for just one graph?

Not sure why the Prometheus exporter should have anything to do with the ConfiguredGraphFactory. Can you please show how exactly you are configuring the exporter?

[rayala-nag]

@FlorianHockmann I followed the below steps to configure jmx-prometheus exporter

• Added java agent as blow
  JAVA_OPTIONS="-javaagent:/data/jars/jmx_prometheus_javaagent-0.19.0.jar=6767:$JANUSGRAPH_CONF/jmx-config.yaml"

• Started the janusgraph service.
  We have almost 600+ graphs and observing WARN logs as below during startup.
  938598 [pool-19-thread-1] WARN com.datastax.oss.driver.internal.core.session.DefaultSession - You have too many session instances: 605 active, expected less than 4 (see 'advanced.session-leak.threshold' in the configuration)

During this startup JanusGraph hosted instance CPU going high and these WARN logs are not appearing after 200 session instances (If you relate this to above log warning).

This is the behaviour i observed during startup .

[FlorianHockmann]

Sorry for the late reply, I simply didn't know an obvious answer to your problems and then probably forgot to answer completely.

Are you only getting these warning logs if you enable the exporter or are these two points independent of each other? I have to say that I don't know much about the Datastax driver, but if it only occurs with the exporter then that would at least narrow down the problem.

Do you have anything special in your jmx-config.yaml?

[rayala-nag]

@FlorianHockmann I'm not using any special configuration in jmx-config.xml. You can refer the content of my JMX config file as below.

rules:
- pattern: ".*"