Frontend changes for the user column migration work. #12158

samwho · 2023-10-23T16:58:05Z

This PR hooks up the frontend to the new POST /api/tables/:table_id/migrate endpoint. The video demonstrates how the functionality works.

CleanShot.2023-10-25.at.15.42.49.mp4

…n for the migrated column.

…di-7607-migrate-user-relationship-columns-to-the-new-user-column-4

codecov-commenter · 2023-10-25T14:12:11Z

Codecov Report

Merging #12158 (26100ff) into feature/budi-7607-migrate-user-relationship-columns-to-the-new-user-column (b3c08f0) will increase coverage by 28.70%.
The diff coverage is 74.57%.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

@@                                               Coverage Diff                                               @@
##           feature/budi-7607-migrate-user-relationship-columns-to-the-new-user-column   #12158       +/-   ##
===============================================================================================================
+ Coverage                                                                       46.82%   75.52%   +28.70%     
===============================================================================================================
  Files                                                                             164      338      +174     
  Lines                                                                            5811    14123     +8312     
  Branches                                                                         1164     2961     +1797     
===============================================================================================================
+ Hits                                                                             2721    10667     +7946     
- Misses                                                                           2824     3231      +407     
+ Partials                                                                          266      225       -41

Files	Coverage Δ
packages/server/src/api/controllers/table/index.ts	`71.84% <100.00%> (ø)`
packages/server/src/sdk/app/tables/getters.ts	`78.78% <100.00%> (ø)`
packages/server/src/integrations/utils.ts	`85.18% <0.00%> (ø)`
packages/server/src/sdk/app/tables/migration.ts	`80.26% <88.23%> (ø)`
packages/server/src/websockets/builder.ts	`10.95% <42.85%> (ø)`
packages/server/src/websockets/websocket.ts	`8.26% <0.00%> (ø)`

... and 496 files with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

packages/server/src/integrations/utils.ts

samwho · 2023-10-25T14:36:17Z

packages/server/src/sdk/app/tables/migration.ts

@@ -105,14 +113,17 @@ function getColumnMigrator(
 throw new BadRequestError(`Unknown migration type`)
 }

-class SingleUserColumnMigrator implements ColumnMigrator {
+abstract class UserColumnMigrator implements ColumnMigrator {


The vast majority of the 2 classes that now inherit from this was identical, so it felt natural to use an abstract class here.

samwho · 2023-10-25T14:37:24Z

packages/server/src/sdk/app/tables/migration.ts

+export interface MigrationResult {
+ tablesUpdated: Table[]
+}


Created and propagated this type back out of the migrator so that we can fire webhook messages back to the client and ensure the client state is correct. This migration endpoint is unusual in that it affects the current table and another table in the same operation, which the UI doesn't automatically account for.

samwho · 2023-10-25T14:38:46Z

packages/server/src/tests/utilities/api/table.ts

+ if (res.status !== expectStatus) {
+ throw new Error(
+ `Expected status ${expectStatus} but got ${
+ res.status
+ } with body ${JSON.stringify(res.body)}`
+ )
+ }


This made error messages from tests that call the migration endpoint significantly easier to debug. I would like functionality akin to this for all of our supertest expects, but I saw no easy way to do this generally.

hmm - we might be able to overload the supertest object that the config returns, e.g. add it to what this.request does - I'll take a look at this!

samwho · 2023-10-25T14:40:08Z

packages/server/src/websockets/builder.ts

+ emitTableUpdate(ctx: any, table: Table, options?: EmitOptions) {
+ // This was added to make sure that sourceId is always present when
+ // sending this message to clients. Without this, tables without a
+ // sourceId (e.g. ta_users) won't get correctly updated client-side.
+ if (isInternalTable(table._id!)) {
+ table = processInternalTable(table)
+ }


Two important changes here:

The addition of EmitOptions to indicate when we want a websocket update to also be processed by the event originator (necessary to update the state of the user table in the UI).

The enriching of the table with processInternalTable. This adds sourceId to the table, without which the UI won't use it to update its state.

Happy to leave this in here, but I think this maybe needs another look. With this enrichment, and a new app, the users table will not have a sourceId when fetched from the tables list endpoint (as it does not actually exist in the doc), but then will have one when a table change event is broadcast. That means that the original frontend issue of not being able to create a relationship to the users table will still exist, and will only be fixed if the client receives a table change event with this enriched property.

So I think the sourceId issue is a totally separate thing.

@mike12345567 could you confirm if removing the sourceId property from users table in new apps is intended? If so then I'm leaning towards removing this enrichment here, and instead updating the frontend to understand that it's optional.

The sourceId was really only designed for external datasources - but internal tables had one thrown on just for consistency - if the frontend expects it to be there for all tables then we can update the typing to make it so that it is non-optional on all tables - right now it is required for external but is optional for internal tables, however we can make that required I think.

@samwho - let me know if you'd like a hand with that!

I just tried creating a brand new app and inspecting the return values from some API endpoints and here's what I found:

GET /api/tables returns a sourceId (tacked on, the sourceId is not present in Couch) on each table

GET /api/tables/ta_users does not return a sourceId

GET /api/tables/ta_<id> does return a sourceId for another internal table (checking Fauxton, this appears to be because the table document has a sourceId)

@mike12345567 I'd appreciate a hand with this, thank you 🙏

packages/server/src/websockets/websocket.ts

…eded.

packages/frontend-core/src/components/grid/controls/MigrationModal.svelte

aptkingston

Incredibly impressive for one of your first features @samwho! Fantastic job. Left a few questions but overall this looks great.

packages/frontend-core/src/components/grid/controls/MigrationModal.svelte

packages/server/src/sdk/app/tables/getters.ts

packages/server/src/sdk/app/tables/migration.ts

mike12345567

Amazing work @samwho - fantastic implementation :)

…the-new-user-column' into feature/budi-7607-migrate-user-relationship-columns-to-the-new-user-column-4

samwho added 11 commits October 23, 2023 17:57

Frontend changes for the user column migration work.

a701933

Introduce modal to show warning to users, and toast to show success.

2f0a40e

Add an input to allow the user to choose the new column name.

d3670dd

Make sure the user cannot enter the same column name as the old colum…

f4c6ab7

…n for the migrated column.

Merge branch 'master' of github.com:Budibase/budibase into feature/bu…

a8ae168

…di-7607-migrate-user-relationship-columns-to-the-new-user-column-4

Handle a couple more edge cases more gracefully.

0f5c2cf

Emit table updates to the builder socket, and some minor refactoring.

78afba6

Attempt to emit events that include the originator.

52f97fb

Fix missing sourceId on Table sent to the websocket.

70f39b6

Spring cleaning before review.

007e919

Merge base branch.

fbf60ec

samwho added 3 commits October 25, 2023 15:29

Add column name input validation.

5779e97

User constant for user table ID.

402a53c

Remove stray console.log

0ac6b17

samwho commented Oct 25, 2023

View reviewed changes

packages/server/src/integrations/utils.ts Show resolved Hide resolved

samwho commented Oct 25, 2023

View reviewed changes

packages/server/src/websockets/websocket.ts Show resolved Hide resolved

samwho requested review from mike12345567 and aptkingston October 25, 2023 14:45

samwho marked this pull request as ready for review October 25, 2023 14:45

samwho added the firestorm Data/Infra/Revenue Team label Oct 25, 2023

samwho added 2 commits October 25, 2023 16:00

Remove the previous attempt at syncing grid state, it is no longer ne…

c9e3b7a

…eded.

Remove unused dispatch import.

7acce7b

aptkingston reviewed Oct 25, 2023

View reviewed changes

packages/frontend-core/src/components/grid/controls/MigrationModal.svelte Show resolved Hide resolved

aptkingston reviewed Oct 25, 2023

View reviewed changes

mike12345567 reviewed Oct 25, 2023

View reviewed changes

packages/frontend-core/src/components/grid/controls/MigrationModal.svelte Outdated Show resolved Hide resolved

Use FieldSubtype enum instead of raw strings.

2c5dd99

mike12345567 reviewed Oct 25, 2023

View reviewed changes

packages/server/src/sdk/app/tables/getters.ts Outdated Show resolved Hide resolved

samwho added 3 commits October 25, 2023 16:38

Use RelationshipType enum instead of raw string.

ef84e96

Use FieldType constant instead of raw string.

c509748

Make sure new column name cannot be the same as an existing column name.

e03b1be

mike12345567 reviewed Oct 25, 2023

View reviewed changes

packages/server/src/sdk/app/tables/migration.ts Outdated Show resolved Hide resolved

Simplify the function signature of processInternalTables

4a00649

mike12345567 approved these changes Oct 25, 2023

View reviewed changes

samwho added 2 commits October 25, 2023 16:49

Simplify try-catch in the migrate function.

6c3b535

Merge branch 'feature/budi-7607-migrate-user-relationship-columns-to-…

26100ff

…the-new-user-column' into feature/budi-7607-migrate-user-relationship-columns-to-the-new-user-column-4

samwho merged commit 9edaca0 into feature/budi-7607-migrate-user-relationship-columns-to-the-new-user-column Oct 26, 2023
8 checks passed

samwho deleted the feature/budi-7607-migrate-user-relationship-columns-to-the-new-user-column-4 branch October 26, 2023 09:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Frontend changes for the user column migration work. #12158

Frontend changes for the user column migration work. #12158

samwho commented Oct 23, 2023 •

edited

Loading

codecov-commenter commented Oct 25, 2023 •

edited

Loading

samwho Oct 25, 2023

samwho Oct 25, 2023

samwho Oct 25, 2023

mike12345567 Oct 25, 2023

samwho Oct 25, 2023

aptkingston Oct 25, 2023

mike12345567 Oct 25, 2023

samwho Oct 25, 2023

aptkingston left a comment

mike12345567 left a comment

Frontend changes for the user column migration work. #12158

Frontend changes for the user column migration work. #12158

Conversation

samwho commented Oct 23, 2023 • edited Loading

codecov-commenter commented Oct 25, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aptkingston left a comment

Choose a reason for hiding this comment

mike12345567 left a comment

Choose a reason for hiding this comment

samwho commented Oct 23, 2023 •

edited

Loading

codecov-commenter commented Oct 25, 2023 •

edited

Loading