Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge DS Defense patches into Slack's v14 release branch #90

Closed
wants to merge 24 commits into from

Commits on Apr 5, 2023

  1. slack-vitess-r14.0.5-defense: vttablet tx-throttler backports, pt 1 (

    …#67)
    
    * Emit per workload labels for existing per table vttablet metrics (vitessio#12394)
    
    * Emit per workload labels for existing per table vttablet metrics
    
    This adds the possibility to configure vttablet (via CLI flag) to also have a
    workload label for existing per table metrics (query counts, query times, query
    errors, query rows affected, query rows returned, query error counts). Workload
    can be any string that makes sense for the client application. For example, API
    endpoint name, controller, batch job name, application name or something else.
    
    This is usefult to be able to gain observability about how the query load is
    distributed across different workloads.
    
    This is achieved with two new CLI flags, namely:
    
    * `enable-per-workload-table-metrics`: whether to enable or disable per
      workload metric collection - disabled by default to preserve the current
      behavior, thus making the new feature opt-in only.
    * `workload-label`: a string to look for in query comments to identify the
      workload running the current query.
    
    The workload is obtained by parsing query comments of the form:
    
    /* ... <workload_label>=<workload_name>; ... */
    
    For example, if vttablet is started with
    
    `--enable-per-workload-table-metrics --workload-label app_name`
    
    anda query is issued with a comment like
    
    /* ... app_name=shop; ... */
    
    then metrics will look like
    
    ```
    vttablet_query_counts{plan="Select",table="dual", workload="shop"} 15479
    ```
    
    instead of
    
    ```
    vttablet_query_counts{plan="Select",table="dual"} 15479
    ```
    
    Query comment parsing only takes place if `--enable-per-workload-table-metrics`
    is used, as to not incur parsing performance impact if the user does not want
    per workload metrics.
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * make linter happy
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * fix flags e2e test
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Address PR comments:
    
    * Obtain workload information on the vtgate instead of the vttablet, avoiding
      double parsing.
    * Treat workload name as a query directive.
    * Send workload name from vtgate to vttablet as ExecuteOptions.
    
    Additionally, annotate tabletserver's execution span with the workload name
    to also enrich traces with workload name data, in addition to metrics.
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * A few fixes:
    
    1. Rebuild some files with `make proto`.
    2. Protect against nil ExecuteOptions on the tabletserver.
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Fix flags e2e test
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Address PR comments
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Fixes
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Fix a comment
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Fix e2e flag test
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Update JS code for protobuf changes.
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Fix QueryEngine unit test
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Fix e2e flag test
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Fix spurious tab in comment
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Address PR comment
    
    Don't use dual format flag for new flags - stick with - separated ones.
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    ---------
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Fix cherry-pick
    
    * Add basic metrics to `vttablet` transaction throttler (vitessio#12418)
    
    * Add basic stats to vttablet tx throttler
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * test new metrics
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * reorder
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * short names
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * Add max rate
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * Move NewGaugeFunc to under conditional
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * Use env
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * Remove env from TxThrottler struct
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * Fix tests
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * PR suggestion
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * Fix unit test
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * reorder test vars
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    ---------
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * Fix go/vt/vttablet/tabletserver/query_engine_test.go
    
    ---------
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    Co-authored-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    timvaillancourt and ejortegau authored Apr 5, 2023
    Configuration menu
    Copy the full SHA
    26a8bbe View commit details
    Browse the repository at this point in the history
  2. slack-vitess-r14.0.5-dsdefense: set CODEOWNERS (#68)

    * `slack-vitess-r14.0.5-dsdefense`: set `CODEOWNERS`
    
    * Use a sub-team
    timvaillancourt authored Apr 5, 2023
    Configuration menu
    Copy the full SHA
    5a3b3ed View commit details
    Browse the repository at this point in the history

Commits on Apr 6, 2023

  1. slack-vitess-r14.0.5-dsdefense: vttablet tx-throttler backports, pt…

    … 2 (#69)
    
    * Skip recalculating the rate in MaxReplicationLagModule when it can't be done
    
    This defends against lag records with nil stats which can lead to segfaults.
    See vitessio#12619
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Address PR comments.
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Make linter happy
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Add support for criticality query directive, and have TxThrottler respect that
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Remove unused variable
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Fix CI pipeline
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * Address PR comments.
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Make linter happy & add extra test cases.
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Address PR comments.
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Fix circular import
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Make linter happy
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Address PR comments:
    
    * Invalid criticality in query directive fails the query.
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Fix executor.go
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * Fix go/vt/vtgate/executor.go again
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * Fix TestNewMaxReplicationLagModule_recalculateRate
    
    * Fix go/vt/vtgate/executor_test.go
    
    * Regen protos from linux
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    ---------
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    Co-authored-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    timvaillancourt and ejortegau authored Apr 6, 2023
    Configuration menu
    Copy the full SHA
    9a9e058 View commit details
    Browse the repository at this point in the history

Commits on Apr 19, 2023

  1. Configuration menu
    Copy the full SHA
    cfdbd1e View commit details
    Browse the repository at this point in the history

Commits on May 10, 2023

  1. Throttled transactions return MySQL error code 1041 ER_OUT_OF_RESOURC…

    …ES (vitessio#12949) (#81)
    
    This error code seems better suited to represent the fact that transactions are
    being throttled by the server due to some form of resource contention than the
    current code 1203 ER_TOO_MANY_USER_CONNECTIONS.
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    Co-authored-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    timvaillancourt and ejortegau authored May 10, 2023
    Configuration menu
    Copy the full SHA
    3d306c8 View commit details
    Browse the repository at this point in the history

Commits on May 12, 2023

  1. slack-vitess-r14.0.5-dsdefense: backport txthrottler cleanup PR v…

    …itessio#12901 (#83)
    
    * Cleanup panics in `txthrottler`, reorder for readability (vitessio#12901)
    
    * Cleanup tx_throttler.go
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * Cleanup tx_throttler.go #2
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * Fix throttlerFactoryFunc
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * Undo if-cond consolidation
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * Undo struct shuffling
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * prove that disabled config returns nil error
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * Improve test
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    ---------
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * remove unused cell string
    
    ---------
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    timvaillancourt authored May 12, 2023
    Configuration menu
    Copy the full SHA
    9604c05 View commit details
    Browse the repository at this point in the history

Commits on Jun 5, 2023

  1. Configuration menu
    Copy the full SHA
    2533172 View commit details
    Browse the repository at this point in the history
  2. Fix linter issues & broken vtadmin autogenerated code from bad merge

    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    ejortegau committed Jun 5, 2023
    Configuration menu
    Copy the full SHA
    7ee19fd View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    b14c77e View commit details
    Browse the repository at this point in the history

Commits on Jun 6, 2023

  1. Fix transaction throttler ignoring the initial rate (vitessio#12618)

    * Fix transaction throttler ignoring the initial rate
    
    This addresses the issue reported in vitessio#12549
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Add missing override of max replication lag in `throttler.newThrottler()`
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Reorder functions to make diff easier to read
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Fix check for maxRate in `newThrottlerFromConfig()`
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Fix some CI pipeline issues
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Address PR comment.
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Fix typo
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    ---------
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    Signed-off-by: Eduardo J. Ortega U. <5791035+ejortegau@users.noreply.github.com>
    ejortegau committed Jun 6, 2023
    Configuration menu
    Copy the full SHA
    1630982 View commit details
    Browse the repository at this point in the history
  2. TxThrottler support for transactions outside BEGIN/COMMIT (vitessio#1…

    …3040)
    
    * TxThrottler support for transactions outside BEGIN/COMMIT
    
    This change allows the TxThrottler to throttle requests sent outside of
    explicit transactions (i.e. explicit BEGIN/COMMIT blocks) when configured to do
    so via a new config flag. Otherwise, it preserves the current/default behavior
    of only throttling transactions inside BEGIN/COMMIT.
    
    In addition, when this flag is passed, and because the call to throttle is done
    in a context in which the execution plan is already known, this change uses the
    plan type to make sure that throttling is triggered only when the query being
    executed is INSERT/UPDATE/DELETE/LOAD, so that SELECTs and others no longer get
    throttled unnecessarily, as they do not contribute to increasing replication
    lag, which is what the TxThrottler aims at controlling.
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Fix e2e flag tests & TxThrottler unit test
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Throttle auto-commit statements in QueryExecutor instead of TxPool
    
    This changes where we call the transaction throttler:
    
    1. Statements in `BEGIN/COMMIT` blocks keep being throttled in
       `TabletServer.begin()`.
    2. Additionally, throttling is added in QueryExecutor.execAutocommit() and
       `QueryExecutor.execAsTransaction()`.
    
    We also change things so that throttling in this new case is not opt-in via
    configuration flag but instead is the new and only behavior.
    
    Finally, we remove some previously added changes to example scripts that had
    been added with the intention of testing and are not part of the PR.
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Adds test cases for QueryExecutor.Execute() with TxThrottle throttling
    
    To make unit testing simple here, we separated the interface and implementation
    of the TxThrottle, and simply used a mock implementation of the interface in
    the tests.
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Add note on new TxThrottler behavior in v17 changelog
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Fix PR number in changelog entry for TxThrottler behavior change.
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Make linter happy
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * Address PR comments
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    ---------
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    ejortegau committed Jun 6, 2023
    Configuration menu
    Copy the full SHA
    c4f369b View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    1fc8b0e View commit details
    Browse the repository at this point in the history
  4. Remove spurious file

    ejortegau committed Jun 6, 2023
    Configuration menu
    Copy the full SHA
    bb7e6fc View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    c322ded View commit details
    Browse the repository at this point in the history

Commits on Jul 3, 2023

  1. Merge branch 'slack-vitess-r14.0.5' into slack-vitess-r14.0.5-dsdefense

    Signed-off-by: Eduardo J. Ortega U. <5791035+ejortegau@users.noreply.github.com>
    ejortegau committed Jul 3, 2023
    Configuration menu
    Copy the full SHA
    5885bdf View commit details
    Browse the repository at this point in the history
  2. txthrottler: further code cleanup (vitessio#12902) (#97)

    * txthrottler: further code cleanup
    
    
    
    * Fix bad merge resolution
    
    
    
    ---------
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    timvaillancourt authored Jul 3, 2023
    Configuration menu
    Copy the full SHA
    a67fd1c View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2023

  1. slack-vitess-r14.0.5-dsdefense: Add flag to select tx throttler tab…

    …let type (vitessio#12174) (#95)
    
    * Add flag to select tx throttler tablet type (vitessio#12174)
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * Remove mistaken git add
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    ---------
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    timvaillancourt authored Jul 5, 2023
    Configuration menu
    Copy the full SHA
    d7a37b3 View commit details
    Browse the repository at this point in the history

Commits on Jul 11, 2023

  1. Revert "fix parameter name (#93)" (#100)

    This reverts commit 76159fd.
    Roderick Yao authored and ejortegau committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    cecfcfd View commit details
    Browse the repository at this point in the history
  2. apply patch 12178 to v14 (#102)

    Signed-off-by: Priya Bibra <pbibra@slack-corp.com>
    Signed-off-by: 'Priya Bibra' <pbibra@slack-corp.com>
    pbibra authored and ejortegau committed Jul 11, 2023
    Configuration menu
    Copy the full SHA
    bd682ad View commit details
    Browse the repository at this point in the history

Commits on Aug 1, 2023

  1. Add per workload TxThrottler metrics (#110)

    This is a backport of upstreamed vitessio#13526
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    ejortegau authored Aug 1, 2023
    Configuration menu
    Copy the full SHA
    7203c35 View commit details
    Browse the repository at this point in the history

Commits on Aug 3, 2023

  1. DRE-8939 Backport TxThrottler dry run mode vitessio#13687 (#112)

    * BackportAdd dry-run mode to the TxThrottler
    
    This is a backport of upstreamed vitessio#13604
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    ejortegau authored Aug 3, 2023
    Configuration menu
    Copy the full SHA
    333c488 View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2023

  1. Upgrade go slack vitess r14.0.5 (#91)

    * Cleanup usage of go.rice in favor of go:embed (vitessio#10956)
    
    * Cleanup usage of go.rice in favor of go:embed
    
    The usage of go.rice predates the availability of go:embed, but we
    should switch to using go:embed instead to ship specific assets like
    config files that we need.
    
    go.rice is also incompatible with Go 1.19 and while it might see a fix
    in the future, it seems better to go with the recommended Go approach
    that is available these days.
    
    Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
    
    * Move vtctld to also use `go embed` instead of go.rice
    
    Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
    
    * Remove last rice-box related comments
    
    Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
    
    * Remove config moving
    
    This right now breaks building the actual tests since the tests might
    also end up loading the regular code which has a `go embed` and refers
    to the package with the config embeds.
    
    This doesn't mean that the config isn't properly included in the
    binaries. Also with using `go embed` we have a build time dependency on
    the files and we always know the latest is included, so we don't have
    the issue of potentially outdated files either.
    
    All in all, it seems simplest to remove this logic and trust that Go
    itself works as advertised.
    
    Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
    
    * fix vtrootbin
    
    Signed-off-by: 'Stanislav Maksimov' <smaksimov@slack-corp.com>
    
    * update the build to 1.19.10
    
    Signed-off-by: 'Stanislav Maksimov' <smaksimov@slack-corp.com>
    
    * update the workflows to 1.19.10
    
    Signed-off-by: 'Stanislav Maksimov' <smaksimov@slack-corp.com>
    
    * update the docker image to 1.19.10
    
    Signed-off-by: 'Stanislav Maksimov' <smaksimov@slack-corp.com>
    
    * update the bootstrap version to get docker to go 1.20.5
    
    Signed-off-by: 'Stanislav Maksimov' <smaksimov@slack-corp.com>
    
    * use 1.18.7 for static checks
    
    Signed-off-by: 'Stanislav Maksimov' <smaksimov@slack-corp.com>
    
    * lower the build version requirement to allow static checks to pass
    
    Signed-off-by: 'Stanislav Maksimov' <smaksimov@slack-corp.com>
    
    * experiment with using old and new Go for a single workflow
    
    Signed-off-by: 'Stanislav Maksimov' <smaksimov@slack-corp.com>
    
    * use old and new Go for a upgrade/downgrade workflows
    
    Signed-off-by: 'Stanislav Maksimov' <smaksimov@slack-corp.com>
    
    * set -buildvcs=false
    
    Signed-off-by: 'Stanislav Maksimov' <smaksimov@slack-corp.com>
    
    * address build errors
    
    Signed-off-by: 'Stanislav Maksimov' <smaksimov@slack-corp.com>
    
    * Revert "fix parameter name (#93)" (#100)
    
    This reverts commit 76159fd.
    
    * apply patch 12178 to v14 (#102)
    
    Signed-off-by: Priya Bibra <pbibra@slack-corp.com>
    Signed-off-by: 'Priya Bibra' <pbibra@slack-corp.com>
    
    * more workflows to use self-hosted runner
    
    * even more workflows to use self-hosted runner
    
    * partially switch upgrade-downgrade to self-hosted runner
    
    * Ejortegau/larger runners (#115)
    
    * Empty commit to trigger CI
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * `slack-vitess-r14.0.5-dsdefense`: use larger runner
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * Use runner group instead
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * Rename group
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * Move more jobs to runner group
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * use vitess-ubuntu20 runner group
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * Rever change of runner type for e2e ERS PRS new features heavy test
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    ---------
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    Co-authored-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * Revert change of runner type for shardedrecovery_stress_verticalsplit_heavy. (#116)
    
    We do this because the test is taking a lot longer & failing in some cases with
    the larger runner.
    
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    
    * `slack-vitess-r14.0.5`: use dedicated larger runner (#113)
    
    * `slack-vitess-r14.0.5`: use dedicated larger runner
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * fix fileNameFromPosition
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * empty commit
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * empty commit
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    ---------
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * `slack-vitess-r14.0.5`: allow conn overrides in consul topo (#111)
    
    * `slack-vitess-r14.0.5`: allow conn overrides in consul topo
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    * fix e2e test
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    ---------
    
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    
    ---------
    
    Signed-off-by: Dirkjan Bussink <d.bussink@gmail.com>
    Signed-off-by: 'Stanislav Maksimov' <smaksimov@slack-corp.com>
    Signed-off-by: Priya Bibra <pbibra@slack-corp.com>
    Signed-off-by: 'Priya Bibra' <pbibra@slack-corp.com>
    Signed-off-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    Co-authored-by: Dirkjan Bussink <d.bussink@gmail.com>
    Co-authored-by: Roderick Yao <roderick.yao@slack-corp.com>
    Co-authored-by: pbibra <pbibra@slack-corp.com>
    Co-authored-by: Eduardo J. Ortega U <5791035+ejortegau@users.noreply.github.com>
    Co-authored-by: Tim Vaillancourt <tim@timvaillancourt.com>
    6 people authored Aug 10, 2023
    Configuration menu
    Copy the full SHA
    356828b View commit details
    Browse the repository at this point in the history

Commits on Sep 4, 2023

  1. slack-vitess-r14.0.5: remove legacy sharding, add MySQL 8.x backup …

    …test + compressors (#119) (#124)
    
    * Delete all legacy sharding related code (vitessio#10278)
    
    * Delete all legacy sharding related code
    
    
    
    * Move used until functions from initialsharding to cluster
    
    
    
    * Remove vtctl commands
    
    
    
    * Kill vtworker and SetKeyspaceServedFrom cmd
    
    
    
    * WaitForDrain related stragglers
    
    
    
    * Legacy local straggler workflow
    
    
    
    * Get rid of SetKeyspaceShardingInfo & wait for drain stragglers
    
    
    
    * Remove vtworker stragglers
    
    
    
    * Update throttlerservice protobuf
    
    
    
    * Rename test 24, add hashicorp vault test to it (now mysql_server_vault)
    
    
    
    * Remove last mentiones of legacy sharding in vtctl
    
    
    
    * remove binlog_use_v3_resharding_mode
    
    
    
    * Address review comments
    
    
    
    * Address review comments
    
    
    
    * Correct vtgate help output
    
    
    
    * go fmt
    
    
    
    * Remove v2 resharding fields (vitessio#10409)
    
    * cleanup: remove sharding_column_name and sharding_column_type
    
    
    
    * cleanup: remove sharding_column_name and sharding_column_type
    
    
    
    * cleanup: remove sharding_column_name and sharding_column_type
    
    
    
    * cleanup: remove sharding_column_name and sharding_column_type
    
    
    
    * generate vtadmin files
    
    
    
    * cleanup: remove sharding_column_name and sharding_column_type from vtadmin
    
    
    
    * Merge from main
    
    
    
    * Fix bad merge conflict resolution
    
    
    
    * Fix missing 'sharding' import
    
    
    
    * Fix bad conflict resolution in go/test/endtoend/cellalias/cell_alias_test.go
    
    
    
    * Backup/Restore: add support for external compressors and decompressors (vitessio#10558)
    
    * change to support an external decompressor
    
    
    
    
    * add external compressor support + builtin additional compressors
    
    
    
    
    * wrap external compressor/decompressor
    
    
    
    
    * go mod tidy + comments
    
    
    
    
    * add copyright notices
    
    
    
    
    * add support for builtin engine
    
    
    
    
    * Adding test case for buckup compression
    
    
    
    * Fixing unit test and run mod tidy
    
    
    
    * Removing unwanted unit tests
    
    
    
    * Increase timeout of backup tests
    
    
    
    * fixing linter errors
    
    
    
    * Change test logic to accomodate running selective tests
    
    
    
    * removing lint warning
    
    
    
    * fixing test failure
    
    
    
    * Removing un-necessary test
    
    
    
    * Fixing code review feeback
    
    
    
    * Change builtinEngine to consider 'auto' decompressor
    
    
    
    * fixing Upgrade/Downgrade test
    
    
    
    * Fix type & add summary under release notes
    
    
    
    * Fixing typos in summary
    
    
    
    * Fixing flag name typos
    
    
    
    
    
    
    * Add MySQL 8 Support to Backup Tests (vitessio#10691)
    
    * Add support for MySQL 8.0 in backup tests
    
    
    
    * Add 8.0 workflow
    
    
    
    * whitespace
    
    
    
    * Use vtctldclient SetKeyspaceDurabilityPolicy to manage semi-sync
    
    This needed to be done after the shard was setup in order to satisfy
    the semantic assumptions related to semi-sync in the tests.
    
    
    
    * Remove extraneous changes
    
    
    
    * We need lz4 for TestXtrabackupStreamWithlz4Compression
    
    
    
    * Try using Percona Repo for MySQL 8 to align mysqld and xtrabackup versions
    
    
    
    * Specify stream type everywhere
    
    
    
    * Remove repeated server install
    
    
    
    * Moar...
    
    
    
    * Move vtctlbackup test to 8.0
    
    
    
    * Rename vtbackup test and move to MySQL 8
    
    
    
    * Split the xbstream tests so the workflow doesn't time out
    
    Otherwise it was going over the 10min limit and getting
    killed.
    
    
    
    * Use MySQL 8 compat method for setting passwords
    
    
    
    * Test increasing timeout at another level
    
    
    
    * Don't use the init passwords file with 8.0
    
    mysqlctl doesn't start...
    
    This is likely due to the change in initialization behavior with
    MySQL 8.0 as it goes though two phases and you can't simply
    start up mysqld and pass it data, it has to initialize and restart
    first.
    
    
    
    * Fix incorrect password update statement for vt_repl user
    
    
    
    * Bump timeouts for 8.0 backup tests
    
    
    
    * Bump it more :(
    
    
    
    * Increase backup/restore timeout in backup_utils
    
    
    
    * Apply new 8.0 template everywhere
    
    
    
    * Fix bugs around how the compression flags were getting passed
    
    
    
    * Use 45m timeout for the workflow, 30m for the run.
    
    
    
    * These changes were no longer needed so limiting diff.
    
    
    
    * Explicitly skip new linter check
    
    
    
    
    
    * Fix test file merge issues
    
    
    
    * Go 1.18.7 to fix error
    
    
    
    * Fix lint
    
    
    
    * Fix vttablet.txt
    
    
    
    * Remove sharding_column_name and sharding_column_type from vtctld2 (vitessio#10459)
    
    * Remove sharding_column_name and sharding_column_type
    
    
    
    * Run make web_build to compile production files
    
    
    
    * Remove vars from test
    
    
    
    * Remove sharding column type select box test from web test
    
    
    
    ---------
    
    Signed-off-by: Matt Lord <mattalord@gmail.com>
    Signed-off-by: Tim Vaillancourt <tim@timvaillancourt.com>
    Signed-off-by: Arvind Murty <arvind.murty@gmail.com>
    Signed-off-by: notfelineit <notfelineit@gmail.com>
    Co-authored-by: Matt Lord <mattalord@gmail.com>
    Co-authored-by: Arvind Murty <arvind.murty@gmail.com>
    Co-authored-by: Rameez Sajwani <rameezwazirali@hotmail.com>
    Co-authored-by: Renan Rangel <rrangel@slack-corp.com>
    Co-authored-by: Renan Rangel <renan@slack-corp.com>
    Co-authored-by: Rohit Nayak <rohit@planetscale.com>
    Co-authored-by: Frances Thai <31225471+notfelineit@users.noreply.github.com>
    8 people authored Sep 4, 2023
    Configuration menu
    Copy the full SHA
    76fcc86 View commit details
    Browse the repository at this point in the history
  2. [cmd/*] Switch to pflag for all CLI flag parsing (vitessio#10619) (#123)

    * Switch to `pflag` for all parsing
    
    This transparently swaps the cli parsing library used by `internal/flag`
    from the standard library `flag` package to `spf13/pflag`.
    
    It also introduces hook points for packages throughout the vitess codebase
    to register their flags for either all commands using `servenv` or a
    particular subset of commands. This allows these packages to continue to
    define their flag variables in a package-private way, but without
    polluting the global flagset.
    
    
    
    * Workaround exit code difference between stdlib `flag` and `pflag`
    
    tl;dr stdlib `flag` has [this][1] and `pflag` does not
    
    [1]: golang/go@dcf0929
    
    
    
    * adjust test data for difference in spacing between pflag/stdflag
    
    
    
    * update lingering legacy flag tests
    
    
    
    * Update vtgate/tabletgateway.go to use new interface to isolate flags, update test data
    
    
    
    * update flags in java test code
    
    Signed-off-by: Andrew Mason <andrew@planetscale.com>
    Co-authored-by: Andrew Mason <andrew@planetscale.com>
    timvaillancourt and Andrew Mason authored Sep 4, 2023
    Configuration menu
    Copy the full SHA
    c07d8ac View commit details
    Browse the repository at this point in the history