Skip to content
This repository has been archived by the owner on Jun 10, 2024. It is now read-only.

RelEng: ReleaseProcess

Jump to bottom
Andrew Klychkov edited this page Jun 10, 2021 · 95 revisions

Table of Contents (Last Generated August 27, 2018)

Permissions and Accounts Required

Access to post to the following Google groups:

Access to post to the following Ansible Slack channels:

  • #core_internal (private channel)
  • #core_network
  • #ship_it
  • #testing

Access to post to the following Red Hat distribution lists:

  • ansible-releases (are there even any restrictions on this?)

Access to change the channel topic on irc channels:

  • #ansible

Accounts on:

Credentials for:

GitHub team:

  • release manager team to push to stable branches.

During the Development Stage

  • Add links to all of the items on the roadmap for the release.
    • This lets others see the status of each feature
    • Can link to proposal, PR, issue, project board depending on the feature

Alpha 1

  • Check in with developers whether features are merged
  • Remove features that are going to be punted from the roadmap for the release.

Starting a new release branch

  • Usually around beta 1

  • Tag the branchpoint in devel so people can tell what commits were made to devel during the X.Y development cycle

    • git tag -a 'stable-X.Y-branchpoint' -m 'Branch stable-X.Y'

  • Create branch stable-X.Y from devel

    • git checkout -b stable-2.8

  • in new branch lib/ansible/release.py:

    • verify __codename__ is set to the selected codename for the release
    • __version__ should be right as it is updated by the release scripts. When we branch after alpha1, it should be 'X.Y.0a1.post0'. If we branch before alpha1 it should be 'X.Y.0.dev0'

  • Open a PR against devel with the following changes:

    • lib/ansible/release.py:
      • set __version__ to X.$(Y+1).0.dev0 (If we're releasing 2.6, __version__ in devel should be '2.7.0.dev0'
      • set __codename__ to the selected codename for the next release
      • Add the new codename to .github/RELEASE_NAMES.yml so that we know that name has already been used.
    • Delete everything under changelogs/fragments. This allows the next release to accumulate its changelog fragments without duplication, since the fragments are sorted into releases by tags that aren't applied to devel). Then create a stub changelog fragment for the new branch as changelogs/fragments/vX.Y+1.0-initial-commit.yaml with just an empty dict ({})
    • Delete any generated changelogs from any “in place” releases under changelogs/CHANGELOG-*.rst.
    • Delete changelogs/.changes.yaml which keeps state information for the changelog tool to know what releases a fragment was added for.
    • Create a new blank porting guide and move it into place: 
       hacking/build-ansible.py porting_guide --version X.Y+1
 mv porting-guide-X.Y+1.rst docs/docsite/rst/porting_guides/porting_guide_X.Y+1.rst
 git add docs/docsite/rst/porting_guides/porting_guide_X.Y+1.rst
    • The PR bumping version in devel will also cause some of our checks to fail. You'll need to create a PR to fix those as well:
      • Error E103 (example: lib/ansible/modules/cloud/amazon/_ec2_ami_find.py:0:0: E103 Did not find a call to removed_module()) means that the module needs to be replaced with a removed_module() stub (so ansible-doc and documentation can say the module has been removed).
        • Change the status in ANSIBLE_METADATA to removed.
        • Replace the remaining code with: 
              from ansible.module_utils.common.removed import removed_module


    if __name__ == '__main__':
        removed_module(removed_in='2.9')
        • Removed modules also need to be skipped for ansible-doc tests. Add them to test/sanity/ansible-doc/skip.txt
        • Add the removed modules to the porting-guide-X.Y.rst that you created above (Modules removed section)
        • You may also have to touch up previous porting guides which linked to the removed modules. The docs no longer exist so they can no longer be links
      • Error E322 (example: test/sanity/validate-modules/ignore.txt:1:1: A102 Remove since "lib/ansible/modules/cloud/amazon/_ec2_ami_find.py" passes "E322" test). If this refers to a module which is being turned into a removed_module() stub, simply remove it from the specified ignore.txt as it no longer has meaning.

  • in Launchpad under the "ansible" account Click "Create a new PPA" and add one for 'ansible-X.Y' (Consult your secret credentials for account credentials)

  • Alert @qe in #testing that the stable-X.Y branch needs to be added to jenkins for tower testing.

    • There is also a point person for tower<=>ansible core integration per release. These are listed in #testing's topic.

  • Fix changelog path in porting guide like this 2.7 example

  • Update the Jenkins Nightly RPM job to include the new stable branch.

Reporting Recurring Cleanups

This procedure is to discover cleanups that need to be performed every release and get them assigned to Core Team members for fixing every release. Currently reported cleanups are:

  • deprecated features in code
  • deprecated features in config
  • bundled libraries that need updating

Steps:

  • Remove ansible-deprecated-version from test/lib/ansible_test/_data/sanity/pylint/config/default.cfg
  • Remove deprecated-config.py from test/sanity/code-smell/skip.txt
  • Remove update-bundled.py from test/sanity/code-smell/skip.txt
  • Submit a pull request with this change
  • Once CI has completed, if there are any cleanups to be performed, CI should fail and give a list of those failures. You can wait for the bot to comment providing this list.
  • Create issues for all scheduled code deprecations. This can be done using the hacking/create_deprecated_issues.py script (This script does not work for deprecated-config currently).
    • Create a file with the ansible-deprecated-version issues discovered by pylint in the pylint standard output. (can copy-paste the output from shippable's pylint test)
    • Execute hacking/build-ansible.py file-deprecation-tickets deprecated.txt --template hacking/deprecated_issue_template.md --project-name 'ansible 2.10'
      • The project-name is the name of a github project for the next release to add the issues to. You may omit this if you don't want to track them.
  • Create issues for deprecated-config
  • Create issues for update-bundled
  • Close the meta-pull request enabling the tests once you've opened the new issues.

Protecting the stable branches

Our github admins (mattclay, jctanner, jimi-c) can modify branch permissions:

  • Protect this branch - Disables force-pushes to this branch and prevents it from being deleted.
  • Include administrators - Enforce all configured restrictions for administrators.
  • There's a github group, release manager team, that needs to be given access to all branches.

Any branch matching stable-2.[0-9]* is protected so only the release manager team and admins can push to them. If you have problems pushing or the branch naming changes, talk to a github admin to figure out what is wrong.

Ongoing lead up to release

  • Review and merge backport PRs against the stable-X.Y branch
  • If this is an rc-less release (minor releases as opposed to major releases), stop merging backport PRs several days before release (Monday is good last merge day for a release on a Thursday), so that Tower has a chance to test.
  • Ping the tower liaison in the #testing slack channel once you're done merging so they can be sure to test before the next rc/final release.

Release procedure

Release pre-check

  • Check status of Jenkins Tower Compatibility Test (VPN) for target branch for "green" (talk w/ tower QE point person if not)
    • This doesn't block the alpha and beta releases because we're working off devel which is a moving target but will block rcs. Check it at alpha and beta and let the Tower QE team know that something is failing so that it can be fixed before rc.
    • If the Test hasn't run since the last commit, you will need to run the Jenkins Stable Ansible Test (VPN) with GIT_TESTED_BRANCHES set to origin/stable-X.Y (and GIT_REFSPEC just left to its default value). This will kick off Ansible builds, Tower Installs, and the Tower compatibility tests. The test pipeline takes about 1.5 hours to complete. The final results will be available at Jenkins Tower Compatibility Test (VPN).
    • Tower is in the process of deprecating this test in favour of a different one. So this test will almost always be red. For now, you need to ping them every release to ask them if the branch is green. It's best to do this on Monday, after you've merged all of the backports for this release.
  • Check https://app.shippable.com/github/ansible/ansible/runs?branchName=stable-X.Y for "Success" (devel for the alpha release)
  • Ensure no open issues/PRs tagged tower_blocker, https://github.com/ansible/ansible/labels/tower_blocker , or coordinate with tower QE point person and others on Tower
  • Ensure no open issues/PRs tagged P2, https://github.com/ansible/ansible/labels/P2 , or coordinate with core team manager about those
  • Notify members of the documentation team who have merge privileges for backports that a release is pending and they should hold off on any backport merges until the release is completed.

Make release (2.6+)

  • From stable-X.Y (devel for alpha release) checkout root: 
    checkout stable-X.Y from ansible/ansible remote
  • Make sure you’re using the checked out version of ansible: 
    source hacking/env-setup
  • Change to the release directory 
    cd packaging/release
  • Make the release: 
    make release version={version}
    Where the version is:
    • Alpha releases: X.Y.0a1
    • Beta releases: X.Y.0b1
    • Release candidates: X.Y.0rc1
    • Final: X.Y.0
  • Push the release if you are satisfied with the changes from the previous step: 
    git push

Builds

  • Wait for the Shippable build of the stable branch to pass before continuing: https://app.shippable.com/github/ansible/ansible/dashboard

  • Run Ansible Release Jenkins job

    • Jenkins: Ansible Release
      • GIT_SHA == release SHA
      • PUBLISH release (or preview if it’s a preview)
      • MAJOR_MINOR X.Y portion of the release
      • LATEST checked if X.Y is the current major release, else not
    • Hit Build

  • If X.Y is current major release:

    • Do not do this for pre-releases!!
    • Once DEB build has finished, visit https://launchpad.net/~ansible/+archive/ubuntu/ansible-X.Y/+packages (logged in as jenkins@ansible.com), poll for completed build status
    • Once all Launchpad package builds have green checkboxes (can take anywhere from 15m to several hours), click "Copy packages" in upper right corner, select current X.Y.Z packages, select Destination PPA as ansible, Destination series as "The same series", Copy options as "Copy existing binaries", then hit "Copy Packages"

NOTE: You don’t need to wait for the docs to be published unless they will be referenced in the release announcements and are new or have changed since the last release. For a new X.Y.0 release you will likely want to wait, but probably not for a X.Y.1+ release.

Publish release (2.6+)

  • Once packaged releases are verified working, publish (tag) the release:

    make publish

  • Push the release if you are satisfied with the changes from the previous step:

    git push --follow-tags

  • If this is a rc or beyond, the red hat release manager now needs to make rpm builds

Job cleanup

  • If this is a new major release (X.Y.0) remove the now unsupported stable branch from the Jenkins Nightly RPM job.
  • Alert tower @qe in #testing that the branch is EOL and doesn't need to be tested anymore.

Notifications

Google Groups (mailing lists)

Internal email

  • Beta and beyond: ansible-releases@redhat.com
    • You can use the template from above and add a line to say that this is an upstream/community release.

Twitter

  • RC and beyond (mention @ansible for marketing ReTweet)
  • Final + servicing: Use the script in hacking/build-ansible.py release-announcement

NOTE: Copying a tweet such as the one above from the Twitter UI may result in additional space(s) being added after the URL.

IRC

Ansible Slack

  • All: #core_internal
  • rc and beyond: @elathram in #marketing - Include the link to the tweet
  • Final + servicing: #ship_it

This Wiki is used for quick notes, not for support or documentation.

(ARchived) Working groups

Working groups are now in the Ansible forum

Ansible project:
Community, Contributor Experience, Docs, News, Outreach, RelEng, Testing

Cloud:
AWS, Azure, CloudStack, Container, DigitalOcean, Docker, hcloud, Kubernetes, Linode, OpenStack, oVirt, Virt, VMware

Networking:
ACI, AVI, F5, Meraki, Network, NXOS

Ansible Developer Tools:
Ansible-developer-tools

Software:
Crypto, Foreman, GDrive, GitLab, Grafana, IPA, JBoss, MongoDB, MySQL, PostgreSQL, RabbitMQ, Zabbix

System:
AIX, BSD, HP-UX, macOS, Remote Management, Solaris, Windows

Security:
Security-Automation, Lockdown

Tooling:
AWX, Galaxy, Molecule

Communities

Modules:
unarchive, xml

Plugins:
httpapi

Wiki

Roles, Communication, Reviewing, Checklist, TODO

Clone this wiki locally