g.V().count() returning very low count ?

[hexa00]

Hi,
Is it normal for Janus using a Cassandra backend to return very low vertex counts when using g.V().count() for example:

[Traversal Metrics
Step                                                               Count  Traversers       Time (ms)    % Dur
=============================================================================================================
JanusGraphStep([])                                                     9           9          13.656    96.04
  constructGraphCentricQuery                                                                   0.020
  GraphCentricQuery                                                                           13.461
    \_condition=()
    \_isFitted=false
    \_query=[]
    \_orders=[]
    \_isOrdered=true
    scan                                                                                      13.362
      \_condition=VERTEX
      \_query=[]
      \_fullscan=true
CountGlobalStep                                                        1           1           0.562     3.96
                                            >TOTAL                     -           -          14.218        -]

However there's 100k + vertices in that graph ...

[vtslab]

No, that is not normal. Please check other scenarios. E.g. if you use one JanusGraph instance to add 100k vertices, a separate JanusGraph instance cannot see them until they are committed to the backends.
If you would happen to encounter a bug, we would need a scenario to reproduce it.

[hexa00]

OK yeah the vertices are in the backend for sure, they can be returned by hitting the mixed index and they have been ingested for a while.

Maybe this has something to do with our use of keyspaces, I'm thinking we'll trace the CQL queries and see what's going on, hopefully that helps.

[hexa00]

Requests for g.V().count()

437208 [gremlin-server-exec-6] WARN  org.janusgraph.graphdb.transaction.StandardJanusGraphTx  - Query requires iterating over all vertices [()]. For better performance, use indexes
437215 [JanusGraph Session-io-2] INFO  com.datastax.oss.driver.internal.core.tracker.RequestLogger  - [JanusGraph Session|999836508][Node(endPoint=3.12.23.184/3.12.23.184:9142, hostId=ac9901f0-287c-3994-9da3-3eeec9af4705, hashCode=28184da4)] Success (6 ms) [2 values] SELECT key,column1,value FROM graph.edgestore WHERE column1>=:slicestart AND column1<=:sliceend ALLOW FILTERING [slicestart=0x02, sliceend=0x03]
437224 [JanusGraph Session-io-3] INFO  com.datastax.oss.driver.internal.core.tracker.RequestLogger  - [JanusGraph Session|2124075595][Node(endPoint=3.12.23.188/3.12.23.188:9142, hostId=bbc8d4cc-caa3-3b7e-ba41-c5c01169de46, hashCode=5a20bf89)] Success (8 ms) [2 values] SELECT key,column1,value FROM graph.edgestore WHERE column1>=:slicestart AND column1<=:sliceend ALLOW FILTERING [slicestart=0x02, sliceend=0x03]
437230 [JanusGraph Session-io-0] INFO  com.datastax.oss.driver.internal.core.tracker.RequestLogger  - [JanusGraph Session|1321987316][Node(endPoint=3.12.23.137/3.12.23.137:9142, hostId=0497d827-22ae-3776-9b9d-14a60347756f, hashCode=197a02ed)] Success (5 ms) [2 values] SELECT key,column1,value FROM graph.edgestore WHERE column1>=:slicestart AND column1<=:sliceend ALLOW FILTERING [slicestart=0x02, sliceend=0x03]
437234 [JanusGraph Session-io-1] INFO  com.datastax.oss.driver.internal.core.tracker.RequestLogger  - [JanusGraph Session|1328466181][Node(endPoint=REDACTED/3.12.23.142:9142, hostId=05deae2d-6405-494d-a965-c0e5836bcb3c, hashCode=30518d40)] Success (4 ms) [2 values] SELECT key,column1,value FROM graph.edgestore WHERE column1>=:slicestart AND column1<=:sliceend ALLOW FILTERING [slicestart=0x02, sliceend=0x03]
437243 [JanusGraph Session-io-2] INFO  com.datastax.oss.driver.internal.core.tracker.RequestLogger  - [JanusGraph Session|915777815][Node(endPoint=3.12.23.171/3.12.23.171:9142, hostId=b5bff396-4e0c-3939-a5c6-64d66d198d69, hashCode=22a8ae77)] Success (8 ms) [2 values] SELECT key,column1,value FROM graph.edgestore WHERE column1>=:slicestart AND column1<=:sliceend ALLOW FILTERING [slicestart=0x02, sliceend=0x03]

[li-boxuan]

I created a bug report on #3390 and also pointed out the direction to a fix. If you have some knowledge of Java and even better, Cassandra java driver, I would recommend you try fixing it and building a JanusGraph release locally (mvn clean install -Pjanusgraph-release -DskipTests=true --batch-mode --also-make -Dgpg.skip=true) and testing it out.

[hexa00]

Oh very interesting, nice find!!

Yeah I do plan to use indexes only but I was just not understanding why it returned so few results and wanted to make sure things were not broken in ways I did not understand.

I can surely compile and run it... for implementing a fix I'll have to wrap my head around, I did so java and Cassandra code but it's been years..

I can't imagine the java driver not supporting this seems even cqlsh is aware that there is more results coming even if it receives an empty page.

Could this be affecting other parts of janusgraph or these ALLOW FILTERING queries are only relevant for fullscans ?

[li-boxuan]

Could this be affecting other parts of janusgraph or these ALLOW FILTERING queries are only relevant for fullscans?

Good question. No. JanusGraph only uses ALLOW FILTERING for full scans.

[hexa00]

😌 thanks for your help!!!

[li-boxuan]

My pleasure and thanks for your patience. Glad we (partially) figured this out together.

[vtslab]

I am lost here, but probably others will read this. Can you elaborate on "with our use of keyspaces"?

[hexa00]

Oh sorry I can see how this can be confusing I mean we use AWS Keyspaces as our Cassandra backend

[li-boxuan]

Did you set a TTL by any chance?

[hexa00]

No I enabled however TTL like so:

aws keyspaces update-table --keyspace-name graph --table-name edgestore --ttl status=ENABLED;
aws keyspaces update-table --keyspace-name graph --table-name graphindex --ttl status=ENABLED;
aws keyspaces update-table --keyspace-name graph --table-name edgestore_lock_ --ttl status=ENABLED;
aws keyspaces update-table --keyspace-name graph --table-name graphindex_lock_ --ttl status=ENABLED;
aws keyspaces update-table --keyspace-name graph --table-name janusgraph_ids --ttl status=ENABLED;
aws keyspaces update-table --keyspace-name graph --table-name system_properties --ttl status=ENABLED;
aws keyspaces update-table --keyspace-name graph --table-name system_properties_lock_ --ttl status=ENABLED;
aws keyspaces update-table --keyspace-name graph --table-name systemlog --ttl status=ENABLED;
aws keyspaces update-table --keyspace-name graph --table-name txlog --ttl status=ENABLED;

And I can see in console:

Default Time to Live (TTL) period Info
Never expire (TTL = 0)

The fact that I can see all the data doing a query / valueMap suggests the data is there too...?