Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf: Benchmark nodes-base #9355

Closed
wants to merge 6 commits into from
Closed

perf: Benchmark nodes-base #9355

wants to merge 6 commits into from

Conversation

ivov
Copy link
Contributor

@ivov ivov commented May 10, 2024

WIP

@ivov ivov mentioned this pull request May 10, 2024
Copy link

codspeed-hq bot commented May 10, 2024

CodSpeed Performance Report

Congrats! CodSpeed is installed 🎉

🆕 93 new benchmarks were detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

Detected benchmarks

  • nodes/Aws/S3/test/V1/AwsS3.file.upload.V1.workflow (39.9 ms)
  • nodes/Aws/S3/test/V2/AwsS3.file.upload.V2.workflow (39.9 ms)
  • nodes/Code/test/Code.workflow (245.5 ms)
  • nodes/Crypto/test/CryptoTest.workflow (31.5 ms)
  • nodes/Discord/test/v2/node/channel/create.workflow (43.7 ms)
  • nodes/Discord/test/v2/node/channel/deleteChannel.workflow (43.4 ms)
  • nodes/Discord/test/v2/node/channel/get.workflow (42.5 ms)
  • nodes/Discord/test/v2/node/channel/getAll.workflow (61.3 ms)
  • nodes/Discord/test/v2/node/channel/update.workflow (44.6 ms)
  • nodes/Discord/test/v2/node/member/getAll.workflow (44.3 ms)
  • nodes/Discord/test/v2/node/member/roleAdd.workflow (43.7 ms)
  • nodes/Discord/test/v2/node/member/roleRemove.workflow (72 ms)
  • nodes/Discord/test/v2/node/message/deleteMessage.workflow (43.7 ms)
  • nodes/Discord/test/v2/node/message/get.workflow (44.1 ms)
  • nodes/Discord/test/v2/node/message/getAll.workflow (44 ms)
  • nodes/Discord/test/v2/node/message/react.workflow (56.5 ms)
  • nodes/Discord/test/v2/node/message/send.workflow (46.2 ms)
  • nodes/Discord/test/v2/node/webhook/sendLegacy.workflow (15.8 ms)
  • nodes/Files/ConvertToFile/test/toText.workflow (42.4 ms)
  • nodes/Files/ReadWriteFile/test/ReadWriteFile.workflow (4 ms)
  • nodes/Files/test/convert_extract.workflow (30.7 ms)
  • nodes/Google/BigQuery/test/v2/node/executeQuery.workflow (5.7 ms)
  • nodes/Google/BigQuery/test/v2/node/insert.autoMapMode.workflow (42.7 ms)
  • nodes/Google/BigQuery/test/v2/node/insert.manualMode.workflow (5.8 ms)
  • nodes/Html/test/extractHtmlContent.workflow (282.9 ms)
  • nodes/Jwt/test/jwt.workflow (106.4 ms)
  • nodes/Microsoft/Excel/test/v2/node/table/addTable.workflow (10.3 ms)
  • nodes/Microsoft/Excel/test/v2/node/table/append.workflow (12.1 ms)
  • nodes/Microsoft/Excel/test/v2/node/table/convertToRange.workflow (10.4 ms)
  • nodes/Microsoft/Excel/test/v2/node/table/deleteTable.workflow (10.5 ms)
  • nodes/Microsoft/Excel/test/v2/node/table/getColumns.workflow (11.1 ms)
  • nodes/Microsoft/Excel/test/v2/node/table/getRows.workflow (11.7 ms)
  • nodes/Microsoft/Excel/test/v2/node/table/lookup.workflow (11.7 ms)
  • nodes/Microsoft/Excel/test/v2/node/workbook/addWorksheet.workflow (18.6 ms)
  • nodes/Microsoft/Excel/test/v2/node/workbook/deleteWorkbook.workflow (10.2 ms)
  • nodes/Microsoft/Excel/test/v2/node/workbook/getAll.workflow (9.7 ms)
  • nodes/Microsoft/Excel/test/v2/node/worksheet/append.workflow (18.9 ms)
  • nodes/Microsoft/Excel/test/v2/node/worksheet/clear.workflow (10.9 ms)
  • nodes/Microsoft/Excel/test/v2/node/worksheet/deleteWorksheet.workflow (10.8 ms)
  • nodes/Microsoft/Excel/test/v2/node/worksheet/getAll.workflow (10.9 ms)
  • nodes/Microsoft/Excel/test/v2/node/worksheet/readRows.workflow (15.2 ms)
  • nodes/Microsoft/Excel/test/v2/node/worksheet/update.workflow (19.7 ms)
  • nodes/Microsoft/Excel/test/v2/node/worksheet/upsert.workflow (20.1 ms)
  • nodes/Microsoft/Outlook/test/v2/node/calendar/create.workflow (120.7 ms)
  • nodes/Microsoft/Outlook/test/v2/node/calendar/delete.workflow (104.3 ms)
  • nodes/Microsoft/Outlook/test/v2/node/calendar/get.workflow (104.7 ms)
  • nodes/Microsoft/Outlook/test/v2/node/calendar/getAll.workflow (104.7 ms)
  • nodes/Microsoft/Outlook/test/v2/node/calendar/update.workflow (536.4 ms)
  • nodes/Microsoft/Outlook/test/v2/node/contact/create.workflow (104.4 ms)
  • nodes/Microsoft/Outlook/test/v2/node/contact/update.workflow (105.3 ms)
  • nodes/Microsoft/Outlook/test/v2/node/draft/create.workflow (104.7 ms)
  • nodes/Microsoft/Outlook/test/v2/node/draft/send.workflow (105 ms)
  • nodes/Microsoft/Outlook/test/v2/node/event/create.workflow (106.2 ms)
  • nodes/Microsoft/Outlook/test/v2/node/folder/create.workflow (105 ms)
  • nodes/Microsoft/Outlook/test/v2/node/folderMessage/getAll.workflow (106.9 ms)
  • nodes/Microsoft/Outlook/test/v2/node/message/move.workflow (118.2 ms)
  • nodes/Microsoft/Outlook/test/v2/node/message/reply.workflow (106.5 ms)
  • nodes/Microsoft/Outlook/test/v2/node/message/send.workflow (105.3 ms)
  • nodes/Microsoft/Teams/test/v2/node/channel/create.workflow (5.5 ms)
  • nodes/Microsoft/Teams/test/v2/node/channel/deleteChannel.workflow (5.5 ms)
  • nodes/Microsoft/Teams/test/v2/node/channel/get.workflow (5.5 ms)
  • nodes/Microsoft/Teams/test/v2/node/channel/getAll.workflow (5.6 ms)
  • nodes/Microsoft/Teams/test/v2/node/channel/update.workflow (5.5 ms)
  • nodes/Microsoft/Teams/test/v2/node/channelMessage/create.workflow (5.5 ms)
  • nodes/Microsoft/Teams/test/v2/node/channelMessage/getAll.workflow (5.5 ms)
  • nodes/Microsoft/Teams/test/v2/node/chatMessage/create.workflow (5.5 ms)
  • nodes/Microsoft/Teams/test/v2/node/chatMessage/get.workflow (5.5 ms)
  • nodes/Microsoft/Teams/test/v2/node/chatMessage/getAll.workflow (5.5 ms)
  • nodes/Microsoft/Teams/test/v2/node/task/create.workflow (5.7 ms)
  • nodes/Microsoft/Teams/test/v2/node/task/deleteTask.workflow (5.5 ms)
  • nodes/Microsoft/Teams/test/v2/node/task/get.workflow (5.5 ms)
  • nodes/Microsoft/Teams/test/v2/node/task/getAll.workflow (5.6 ms)
  • nodes/Microsoft/Teams/test/v2/node/task/update.workflow (5.5 ms)
  • nodes/MoveBinaryData/test/MoveBinaryData.workflow (7.1 ms)
  • nodes/MySql/test/v1/executeQuery.workflow (11.6 ms)
  • nodes/QuickChart/test/QuickChart.workflow (147.9 ms)
  • nodes/ReadBinaryFile/test/ReadBinaryFile.workflow (3.7 ms)
  • nodes/ReadBinaryFiles/test/ReadBinaryFiles.workflow (18 ms)
  • nodes/ReadPdf/test/ReadPDF-encrypted.workflow (3.8 ms)
  • nodes/ReadPdf/test/ReadPDF.workflow (3.9 ms)
  • nodes/RenameKeys/test/RenameKeys.workflow (18.9 ms)
  • nodes/Set/test/Set.v3.workflow (267.2 ms)
  • nodes/Set/test/Set.v3_3.workflow (237 ms)
  • nodes/Set/test/Set.workflow (47.2 ms)
  • nodes/Switch/V1/test/switch.expression.workflow (46.2 ms)
  • nodes/Switch/V1/test/switch.rules.workflow (23.9 ms)
  • nodes/Switch/V2/test/switch.expression.workflow (71.2 ms)
  • nodes/Switch/V2/test/switch.rules.workflow (56.3 ms)
  • nodes/Switch/V3/test/switch.expression.workflow (73 ms)
  • nodes/Switch/V3/test/switch.regex.workflow (33.1 ms)
  • nodes/Switch/V3/test/switch.rules.workflow (96.3 ms)
  • nodes/Wait/test/Wait.workflow (21.8 ms)
  • nodes/WriteBinaryFile/test/WriteBinaryFile.workflow (3.8 ms)

Copy link
Member

@netroy netroy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking pretty good. It's unfortunate that we have "only" 93 tests that can be used here. but, this is a great start 🎉 👏🏽

@n8n-assistant n8n-assistant bot added n8n team Authored by the n8n team node/improvement New feature or request labels May 10, 2024
.github/workflows/benchmark-nodes-base.yml Outdated Show resolved Hide resolved
@@ -14,6 +14,8 @@
"url": "git+https://github.com/n8n-io/n8n.git"
},
"scripts": {
"benchmark:build": "tsc -p tsconfig.benchmark.json > /dev/null; tsc-alias -p tsconfig.benchmark.json; pnpm build:metadata",
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why the > /dev/null ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To silence type errors when transpiling test helpers. Did not fix those here to avoid a larger PR.

@@ -373,6 +386,9 @@ export const workflowToTests = (workflowFiles: string[]) => {
);
}
});

if (process.env.NODE_ENV === 'benchmark') workflowData.pinData = {};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we use pinData for assertions. does this change mean that we disable those assertions when running nodes tests for benchmarking?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Benchmarks use executeWorkflow but do not use equalityTest so they sidestep all Jest assertions.

This line specifically I added because a handful of workflows (MoveBinaryData, QuickChart) were getting caught by the pindata check right after this line, because they have their own implementation of testWorkflows.

@@ -0,0 +1,9 @@
{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we do a on-the-fly compilation of TS code (like ts-jest does), instead of needing yet another tsconfig?

netroy
netroy previously approved these changes May 10, 2024
Copy link

cypress bot commented May 10, 2024

1 flaky test on run #4925 ↗︎

0 359 12 0 Flakiness 1

Details:

🌳 🖥️ browsers:node18.12.0-chrome107 🤖 ivov 🗃️ e2e/*
Project: n8n Commit: f581e40fe4
Status: Passed Duration: 04:23 💡
Started: May 10, 2024 2:24 PM Ended: May 10, 2024 2:29 PM
Flakiness  cypress/e2e/5-ndv.cy.ts • 1 flaky test

View Output Video

Test Artifacts
NDV > Stop listening for trigger event from NDV Screenshots Video

Review all test suite changes for PR #9355 ↗︎

Copy link
Contributor

✅ All Cypress E2E specs passed

@ivov ivov closed this Jun 12, 2024
@netroy netroy deleted the benchmark-nodes-base-v2 branch June 12, 2024 12:29
@netroy netroy restored the benchmark-nodes-base-v2 branch June 12, 2024 12:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
n8n team Authored by the n8n team node/improvement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants