How to upload rdf data(turtle or RDF/XML) format to janus graph in bulk .

[Arpan03jain]

I have data in RDF(ttl) format. It is having around 6 million triplets. Currently, I have used rdf2gremlin python script for this conversion but it's taking to much time i.e. for 10k records it took around 1 hour. I am using Scylla DB as a Janus graph backend. Below is the python code I am using.

from rdf2g import setup_graph
DEFAULT_LOCAL_CONNECTION_STRING = "ws://localhost:8182/gremlin"
g = setup_graph(DEFAULT_LOCAL_CONNECTION_STRING)

import rdflib
import pathlib

OUTPUT_FILE_LAM_PROPERTIES = pathlib.Path("path/to/ttl/file/.ttl").resolve()

rdf_graph = rdflib.Graph()
rdf_graph.parse(str(OUTPUT_FILE_LAM_PROPERTIES), format="ttl")
Same RDF data in neo4j is taking around only 10 mins to load the whole data. But I want to use the Janus graph.

Kindly suggest to me the best way to upload bulk RDF data to Janus graph using python or java.

[farodin91]

Just to link a different discussion on the same topic. #2297

[mbrukman]

@Arpan03jain — have you taken a look at the docs on batch loading data into JanusGraph and followed those suggestions?

[Arpan03jain]

@mbrukman yes I have seen that documentation but the problem is if I set storage.batch-loading = true then I have to create my own schema and in my RDF data, I have vertices properties in the form of URLs so it is impossible to define my own schema.

I have rfd data which is in turtle or xml format. Now the data is having the vertex property name in the form of URL so when I upload that data to Janus graph it converts those URL values to ns1,ns2 and goes on for each unique url.

Now my first question is how the Janus graph is doing that and how I can get those ns values.

Because I need to define my own schema for Janus graph because I need to upload bulk data and for bulk upload, the manual schema creation is required for fast upload speed. How I can crate schema for those URLs. I mean during schema creation we have to declare the property of vertices and in my case properties are url so what is the best way to achieve this.