Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replace unary SNARK / PP-SNARK with their batched variants. #331

Closed
wants to merge 3 commits into from
Closed

Replace unary SNARK / PP-SNARK with their batched variants. #331

wants to merge 3 commits into from

Conversation

huitseeker
Copy link
Member

Opening as a PoC, to start the conversation.

Follows #225, Fixes #197.

Benchmarks

Table of Contents

Benchmark Results

Comparison between CompressedSNARK and BatchedCompressedSNARK

Condition CompressedSNARK ref=2699725 BatchedCompressedSNARK ref=2699725
Prove-NumCons-0 274.85 ms 271.68 ms
Verify-NumCons-0 19.05 ms 20.28 ms
Prove-NumCons-6559 289.33 ms 290.99 ms
Verify-NumCons-6559 18.82 ms 20.28 ms
Prove-NumCons-22943 324.71 ms 325.74 ms
Verify-NumCons-22943 19.12 ms 22.44 ms
Prove-NumCons-55711 300.13 ms 289.36 ms
Verify-NumCons-55711 20.49 ms 23.47 ms
Prove-NumCons-121247 346.24 ms 328.16 ms
Verify-NumCons-121247 23.71 ms 25.44 ms
Prove-NumCons-252319 438.15 ms 414.80 ms
Verify-NumCons-252319 25.74 ms 32.22 ms
Prove-NumCons-514463 596.27 ms 591.86 ms
Verify-NumCons-514463 34.26 ms 50.07 ms
Prove-NumCons-1038751 982.27 ms 965.43 ms
Verify-NumCons-1038751 55.77 ms 92.50 ms

Comparison between CompressedSNARK-Commitments and BatchedCompressedSNARK-Commitments

Condition CompressedSNARK-Commitments ref=2699725 BatchedCompressedSNARK-Commitments ref=2699725
Prove-NumCons-0 2.04 s 2.06 s
Verify-NumCons-0 19.59 ms 19.53 ms
Prove-NumCons-6559 3.30 s 3.24 s
Verify-NumCons-6559 19.65 ms 19.62 ms
Prove-NumCons-22943 2.76 s 2.72 s
Verify-NumCons-22943 19.61 ms 19.66 ms
Prove-NumCons-55711 4.91 s 4.99 s
Verify-NumCons-55711 19.71 ms 19.76 ms
Prove-NumCons-121247 2.49 s 2.51 s
Verify-NumCons-121247 19.48 ms 19.48 ms
Prove-NumCons-252319 4.69 s 4.77 s
Verify-NumCons-252319 19.68 ms 19.67 ms

@huitseeker
Copy link
Member Author

!benchmark --bench compressed-snark --bench compressed-snark-supernova --features cuda

@github-actions GitHub Actions
Copy link
Contributor

Benchmark for 5bdc0ee

Click to view benchmark
Test Base PR %
CompressedSNARK-NIVC-1/Prove/StepCircuitSize-514444 1385.1±11.38ms 1412.0±29.55ms +1.94%
CompressedSNARK-NIVC-1/Prove/StepCircuitSize-6540 613.4±2.70ms 601.4±1.87ms -1.96%
CompressedSNARK-NIVC-1/Verify/StepCircuitSize-514444 139.1±1.26ms 141.2±1.24ms +1.51%
CompressedSNARK-NIVC-1/Verify/StepCircuitSize-6540 32.7±0.20ms 35.6±0.19ms +8.87%
CompressedSNARK-NIVC-2/Prove/StepCircuitSize-514444 1927.0±29.83ms 1850.3±17.68ms -3.98%
CompressedSNARK-NIVC-2/Prove/StepCircuitSize-6540 635.4±3.85ms 625.3±3.68ms -1.59%
CompressedSNARK-NIVC-2/Verify/StepCircuitSize-514444 250.4±1.38ms 248.0±1.93ms -0.96%
CompressedSNARK-NIVC-2/Verify/StepCircuitSize-6540 40.1±0.24ms 42.9±0.43ms +6.98%
CompressedSNARK-NIVC-Commitments-2/Prove/StepCircuitSize-514444 31.3±0.76s 32.2±0.74s +2.88%
CompressedSNARK-NIVC-Commitments-2/Prove/StepCircuitSize-6540 13.6±0.11s 13.9±0.17s +2.21%
CompressedSNARK-NIVC-Commitments-2/Verify/StepCircuitSize-514444 60.5±0.64ms 59.8±0.37ms -1.16%
CompressedSNARK-NIVC-Commitments-2/Verify/StepCircuitSize-6540 59.9±0.43ms 60.3±0.62ms +0.67%

@github-actions GitHub Actions
Copy link
Contributor

Benchmark for 5bdc0ee

Click to view benchmark
Test Base PR %
BatchedCompressedSNARK-Commitments/Prove/StepCircuitSize-0 4.3±0.07s N/A N/A
BatchedCompressedSNARK-Commitments/Prove/StepCircuitSize-121247 5.1±0.24s N/A N/A
BatchedCompressedSNARK-Commitments/Prove/StepCircuitSize-22943 5.3±0.14s N/A N/A
BatchedCompressedSNARK-Commitments/Prove/StepCircuitSize-252319 7.8±0.50s N/A N/A
BatchedCompressedSNARK-Commitments/Prove/StepCircuitSize-55711 8.2±0.55s N/A N/A
BatchedCompressedSNARK-Commitments/Prove/StepCircuitSize-6559 6.0±0.09s N/A N/A
BatchedCompressedSNARK-Commitments/Verify/StepCircuitSize-0 29.5±1.00ms N/A N/A
BatchedCompressedSNARK-Commitments/Verify/StepCircuitSize-121247 29.1±0.20ms N/A N/A
BatchedCompressedSNARK-Commitments/Verify/StepCircuitSize-22943 29.4±0.11ms N/A N/A
BatchedCompressedSNARK-Commitments/Verify/StepCircuitSize-252319 29.2±0.21ms N/A N/A
BatchedCompressedSNARK-Commitments/Verify/StepCircuitSize-55711 29.2±0.24ms N/A N/A
BatchedCompressedSNARK-Commitments/Verify/StepCircuitSize-6559 29.2±0.26ms N/A N/A
BatchedCompressedSNARK/Prove/StepCircuitSize-0 550.3±8.47ms N/A N/A
BatchedCompressedSNARK/Prove/StepCircuitSize-121247 651.2±13.71ms N/A N/A
BatchedCompressedSNARK/Prove/StepCircuitSize-22943 610.8±8.91ms N/A N/A
BatchedCompressedSNARK/Prove/StepCircuitSize-252319 809.7±21.47ms N/A N/A
BatchedCompressedSNARK/Prove/StepCircuitSize-55711 583.5±6.27ms N/A N/A
BatchedCompressedSNARK/Prove/StepCircuitSize-6559 580.5±7.29ms N/A N/A
BatchedCompressedSNARK/Verify/StepCircuitSize-0 24.3±0.24ms N/A N/A
BatchedCompressedSNARK/Verify/StepCircuitSize-121247 34.1±0.33ms N/A N/A
BatchedCompressedSNARK/Verify/StepCircuitSize-22943 25.1±0.30ms N/A N/A
BatchedCompressedSNARK/Verify/StepCircuitSize-252319 44.6±0.40ms N/A N/A
BatchedCompressedSNARK/Verify/StepCircuitSize-55711 28.0±0.31ms N/A N/A
BatchedCompressedSNARK/Verify/StepCircuitSize-6559 24.6±0.27ms N/A N/A
CompressedSNARK-Commitments/Prove/StepCircuitSize-0 4.4±0.04s 4.4±0.06s 0.00%
CompressedSNARK-Commitments/Prove/StepCircuitSize-121247 5.1±0.21s 5.0±0.17s -1.96%
CompressedSNARK-Commitments/Prove/StepCircuitSize-22943 5.5±0.15s 5.3±0.12s -3.64%
CompressedSNARK-Commitments/Prove/StepCircuitSize-252319 7.9±0.66s 8.0±0.68s +1.27%
CompressedSNARK-Commitments/Prove/StepCircuitSize-55711 8.3±0.74s 8.0±0.67s -3.61%
CompressedSNARK-Commitments/Prove/StepCircuitSize-6559 5.9±0.10s 5.9±0.45s 0.00%
CompressedSNARK-Commitments/Verify/StepCircuitSize-0 29.0±0.25ms 29.1±0.24ms +0.34%
CompressedSNARK-Commitments/Verify/StepCircuitSize-121247 29.0±0.19ms 29.5±0.32ms +1.72%
CompressedSNARK-Commitments/Verify/StepCircuitSize-22943 29.1±0.13ms 29.1±0.27ms 0.00%
CompressedSNARK-Commitments/Verify/StepCircuitSize-252319 29.2±0.53ms 29.0±0.21ms -0.68%
CompressedSNARK-Commitments/Verify/StepCircuitSize-55711 28.9±0.22ms 29.1±0.15ms +0.69%
CompressedSNARK-Commitments/Verify/StepCircuitSize-6559 29.1±0.18ms 29.0±0.16ms -0.34%
CompressedSNARK/Prove/StepCircuitSize-0 568.6±4.95ms 553.9±5.19ms -2.59%
CompressedSNARK/Prove/StepCircuitSize-1038751 2.1±0.04s 2.0±0.03s -4.76%
CompressedSNARK/Prove/StepCircuitSize-121247 674.9±7.34ms 676.1±10.03ms +0.18%
CompressedSNARK/Prove/StepCircuitSize-22943 617.9±7.44ms 614.1±6.29ms -0.61%
CompressedSNARK/Prove/StepCircuitSize-252319 876.0±20.33ms 811.7±19.85ms -7.34%
CompressedSNARK/Prove/StepCircuitSize-514463 1271.2±21.02ms 1247.8±41.57ms -1.84%
CompressedSNARK/Prove/StepCircuitSize-55711 601.8±14.52ms 591.5±10.29ms -1.71%
CompressedSNARK/Prove/StepCircuitSize-6559 584.2±7.36ms 578.9±4.70ms -0.91%
CompressedSNARK/Verify/StepCircuitSize-0 22.2±0.18ms 24.4±0.25ms +9.91%
CompressedSNARK/Verify/StepCircuitSize-1038751 128.9±0.69ms 221.4±1.23ms +71.76%
CompressedSNARK/Verify/StepCircuitSize-121247 26.6±0.55ms 34.3±0.50ms +28.95%
CompressedSNARK/Verify/StepCircuitSize-22943 22.5±0.19ms 25.5±0.28ms +13.33%
CompressedSNARK/Verify/StepCircuitSize-252319 39.6±0.93ms 48.8±2.58ms +23.23%
CompressedSNARK/Verify/StepCircuitSize-514463 72.5±1.00ms 118.8±0.72ms +63.86%
CompressedSNARK/Verify/StepCircuitSize-55711 23.8±0.41ms 28.5±0.36ms +19.75%
CompressedSNARK/Verify/StepCircuitSize-6559 22.2±0.13ms 24.9±0.24ms +12.16%

@adr1anh
Copy link
Contributor

adr1anh commented Feb 20, 2024

I'll have to take a look at the exact differences. My intuition here would be that the rlc operations we are performing manually with "powers" of challenges are not correctly skipping the first element of the list. Investigating...

@huitseeker
Copy link
Member Author

@adr1anh I'll rebase and re-run benches so you can see the results on the latest RLC (#330). My overall concerns are related to the huge variance of the performance of the batched_ppsnark, which is also related to #281.

@huitseeker
Copy link
Member Author

!benchmark --bench compressed-snark --bench compressed-snark-supernova --features cuda

@github-actions GitHub Actions
Copy link
Contributor

!gpu-benchmark action failed ❌

https://github.com/lurk-lab/arecibo/actions/runs/8026416422

@huitseeker
Copy link
Member Author

!gpu-benchmark --bench compressed-snark --bench compressed-snark-supernova

@samuelburnham
Copy link
Member

!benchmark --bench compressed-snark --bench compressed-snark-supernova --features cuda

@github-actions GitHub Actions
Copy link
Contributor

Benchmark for 74792f9

Click to view benchmark
Test Base PR %
CompressedSNARK-NIVC-1/Prove/StepCircuitSize-514444 1376.0±20.22ms 1370.5±32.65ms -0.40%
CompressedSNARK-NIVC-1/Prove/StepCircuitSize-6540 621.0±11.18ms 606.1±5.78ms -2.40%
CompressedSNARK-NIVC-1/Verify/StepCircuitSize-514444 140.5±2.08ms 142.2±1.19ms +1.21%
CompressedSNARK-NIVC-1/Verify/StepCircuitSize-6540 33.9±0.24ms 37.3±0.74ms +10.03%
CompressedSNARK-NIVC-2/Prove/StepCircuitSize-514444 1876.4±38.98ms 1855.0±41.90ms -1.14%
CompressedSNARK-NIVC-2/Prove/StepCircuitSize-6540 637.0±5.99ms 623.0±3.07ms -2.20%
CompressedSNARK-NIVC-2/Verify/StepCircuitSize-514444 248.5±1.84ms 256.8±3.83ms +3.34%
CompressedSNARK-NIVC-2/Verify/StepCircuitSize-6540 41.5±0.13ms 44.6±0.33ms +7.47%
CompressedSNARK-NIVC-Commitments-2/Prove/StepCircuitSize-514444 31.7±1.21s 31.4±1.19s -0.95%
CompressedSNARK-NIVC-Commitments-2/Prove/StepCircuitSize-6540 14.1±0.14s 13.9±0.25s -1.42%
CompressedSNARK-NIVC-Commitments-2/Verify/StepCircuitSize-514444 65.0±0.62ms 63.4±0.51ms -2.46%
CompressedSNARK-NIVC-Commitments-2/Verify/StepCircuitSize-6540 64.2±0.37ms 64.6±1.49ms +0.62%

@github-actions GitHub Actions
Copy link
Contributor

!gpu-benchmark action succeeded! 🚀

https://github.com/lurk-lab/arecibo/actions/runs/8028162664

@github-actions GitHub Actions
Copy link
Contributor

Benchmark for 74792f9

Click to view benchmark
Test Base PR %
BatchedCompressedSNARK-Commitments/Prove/StepCircuitSize-0 4.2±0.03s N/A N/A
BatchedCompressedSNARK-Commitments/Prove/StepCircuitSize-121247 4.9±0.25s N/A N/A
BatchedCompressedSNARK-Commitments/Prove/StepCircuitSize-22943 5.2±0.19s N/A N/A
BatchedCompressedSNARK-Commitments/Prove/StepCircuitSize-252319 8.1±0.56s N/A N/A
BatchedCompressedSNARK-Commitments/Prove/StepCircuitSize-55711 8.1±0.67s N/A N/A
BatchedCompressedSNARK-Commitments/Prove/StepCircuitSize-6559 5.7±0.40s N/A N/A
BatchedCompressedSNARK-Commitments/Verify/StepCircuitSize-0 29.6±0.61ms N/A N/A
BatchedCompressedSNARK-Commitments/Verify/StepCircuitSize-121247 29.6±0.19ms N/A N/A
BatchedCompressedSNARK-Commitments/Verify/StepCircuitSize-22943 29.7±0.24ms N/A N/A
BatchedCompressedSNARK-Commitments/Verify/StepCircuitSize-252319 29.7±0.66ms N/A N/A
BatchedCompressedSNARK-Commitments/Verify/StepCircuitSize-55711 29.5±0.28ms N/A N/A
BatchedCompressedSNARK-Commitments/Verify/StepCircuitSize-6559 29.9±1.11ms N/A N/A
BatchedCompressedSNARK/Prove/StepCircuitSize-0 539.2±4.34ms N/A N/A
BatchedCompressedSNARK/Prove/StepCircuitSize-121247 621.0±13.73ms N/A N/A
BatchedCompressedSNARK/Prove/StepCircuitSize-22943 593.6±5.14ms N/A N/A
BatchedCompressedSNARK/Prove/StepCircuitSize-252319 778.9±25.64ms N/A N/A
BatchedCompressedSNARK/Prove/StepCircuitSize-55711 563.6±4.06ms N/A N/A
BatchedCompressedSNARK/Prove/StepCircuitSize-6559 554.4±6.52ms N/A N/A
BatchedCompressedSNARK/Verify/StepCircuitSize-0 24.3±0.20ms N/A N/A
BatchedCompressedSNARK/Verify/StepCircuitSize-121247 34.4±0.41ms N/A N/A
BatchedCompressedSNARK/Verify/StepCircuitSize-22943 25.3±0.36ms N/A N/A
BatchedCompressedSNARK/Verify/StepCircuitSize-252319 43.6±0.85ms N/A N/A
BatchedCompressedSNARK/Verify/StepCircuitSize-55711 28.1±0.27ms N/A N/A
BatchedCompressedSNARK/Verify/StepCircuitSize-6559 24.9±0.23ms N/A N/A
CompressedSNARK-Commitments/Prove/StepCircuitSize-0 4.2±0.06s 4.2±0.08s 0.00%
CompressedSNARK-Commitments/Prove/StepCircuitSize-121247 4.9±0.23s 4.8±0.25s -2.04%
CompressedSNARK-Commitments/Prove/StepCircuitSize-22943 5.1±0.12s 5.2±0.18s +1.96%
CompressedSNARK-Commitments/Prove/StepCircuitSize-252319 7.5±0.48s 7.7±0.70s +2.67%
CompressedSNARK-Commitments/Prove/StepCircuitSize-55711 8.3±0.52s 7.9±0.66s -4.82%
CompressedSNARK-Commitments/Prove/StepCircuitSize-6559 6.0±0.30s 5.7±0.44s -5.00%
CompressedSNARK-Commitments/Verify/StepCircuitSize-0 29.6±0.27ms 29.8±0.33ms +0.68%
CompressedSNARK-Commitments/Verify/StepCircuitSize-121247 29.6±0.40ms 29.7±0.22ms +0.34%
CompressedSNARK-Commitments/Verify/StepCircuitSize-22943 29.9±0.44ms 29.9±0.37ms 0.00%
CompressedSNARK-Commitments/Verify/StepCircuitSize-252319 29.7±0.27ms 29.5±0.23ms -0.67%
CompressedSNARK-Commitments/Verify/StepCircuitSize-55711 29.6±0.29ms 29.7±0.19ms +0.34%
CompressedSNARK-Commitments/Verify/StepCircuitSize-6559 29.5±0.36ms 29.6±0.34ms +0.34%
CompressedSNARK/Prove/StepCircuitSize-0 540.6±5.50ms 538.2±3.46ms -0.44%
CompressedSNARK/Prove/StepCircuitSize-1038751 2.0±0.05s 1976.8±32.13ms -1.16%
CompressedSNARK/Prove/StepCircuitSize-121247 646.3±9.76ms 635.6±13.03ms -1.66%
CompressedSNARK/Prove/StepCircuitSize-22943 598.2±5.96ms 590.0±4.23ms -1.37%
CompressedSNARK/Prove/StepCircuitSize-252319 827.5±23.26ms 792.6±20.37ms -4.22%
CompressedSNARK/Prove/StepCircuitSize-514463 1199.8±27.35ms 1186.1±25.96ms -1.14%
CompressedSNARK/Prove/StepCircuitSize-55711 582.1±6.64ms 567.4±7.21ms -2.53%
CompressedSNARK/Prove/StepCircuitSize-6559 565.0±3.11ms 555.8±5.88ms -1.63%
CompressedSNARK/Verify/StepCircuitSize-0 22.1±0.13ms 24.3±0.24ms +9.95%
CompressedSNARK/Verify/StepCircuitSize-1038751 127.4±0.61ms 221.3±0.84ms +73.70%
CompressedSNARK/Verify/StepCircuitSize-121247 26.7±0.52ms 34.5±0.35ms +29.21%
CompressedSNARK/Verify/StepCircuitSize-22943 23.1±0.25ms 25.4±0.39ms +9.96%
CompressedSNARK/Verify/StepCircuitSize-252319 36.2±1.68ms 47.3±2.23ms +30.66%
CompressedSNARK/Verify/StepCircuitSize-514463 68.0±0.37ms 120.4±0.59ms +77.06%
CompressedSNARK/Verify/StepCircuitSize-55711 24.1±0.47ms 28.0±0.53ms +16.18%
CompressedSNARK/Verify/StepCircuitSize-6559 22.2±0.21ms 25.1±0.28ms +13.06%

@github-actions GitHub Actions
Copy link
Contributor

!gpu-benchmark action succeeded! 🚀

https://github.com/lurk-lab/arecibo/actions/runs/8028162664

@huitseeker
Copy link
Member Author

Arecibo is in maintenance mode.

@huitseeker huitseeker closed this Aug 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@adr1anh adr1anh Awaiting requested review from adr1anh

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

refactor: replace direct SNARK with batched SNARK
3 participants
@huitseeker @adr1anh @samuelburnham