-
Notifications
You must be signed in to change notification settings - Fork 62
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Avoid deadlock on write transaction #942
Comments
What you see there is basically back-pressure from Neo4j not being able to handle as many concurrent relationship writes. One option would be to reduce concurrency, another would be to re-partition the relationship DF on source, target. However, depending on the degree distribution this could also lead to multiple partitions / threads that write relationships between the same nodes, but maybe less often than just random partitioning. |
I've noticed that writing relationship table (here) containing dozens of millions of rows with large batches (default value in the configuration, 100000) generates a lot of warnings like this
I wonder whether this can be minimized by re-partitioning of the Spark DF. Maybe it is not actually needed to write relationships in parallel because of a lot of retried transactions?
The text was updated successfully, but these errors were encountered: