NatWest Group Hosted Roundtable - FINOS Open Source Readiness - How to Measure InnerSource, Open Source and Enterprise Projects - CHAOSS, Agur, 8KNOT, Backstage, GitLab and GitHub

[mcleo-d]

NatWest Group is running a How to Measure InnerSource, Open Source and Enterprise Projects “FINOS Members + Limited Guests, Chatham House Rule” roundtable, to celebrate OSFF London, on behalf of FINOS Open Source Readiness.

Tuesday 18th June at 2:00pm BST / 6:30pm IST / 9am EST - Fully Hybrid over Zoom
The roundtable is fully virtual, with a number of places available at NatWest, London for those wanting to join us in the room (registration in the comments below).

• Home - CHAOSS
• Info - osci.io
• Software - CHAOSS

This meeting follows on from Open Source Readiness #274 on 5th June 2024 by moving into the How and Next Steps for FINOS banks like NatWest Group.

We look forward to celebrating OSFF London with you, on behalf of NatWest Group!

James.

Meeting Minutes

Participants

• 20 FINOS Members and external open source guests attended the event.

Agenda

• Overview of the CHAOSS Project and its tools (Augur and GrimoireLab).
• Discussion on metrics for open source and InnerSource projects.
• Feedback from various participants on their experiences and needs related to metrics.
• Tactical implementations of CHAOSS metrics using GraphQL API.
• Challenges and considerations in implementing metrics.
• Next steps for integrating CHAOSS tools with internal banking systems.
• Closing remarks and future plans for similar discussions.

Key Presentation - CHAOSS Project

• Overview of the CHAOSS Project: Community health analytics for open source software.
• Emphasis on open source project health, not just code contributions.
• Main tools: Augur and GrimoireLab for gathering and analyzing metrics.
• Practitioner Guide Series for interpreting metrics.
• Importance of contribution and consumption metrics.
• Encouragement for a strategic goal-focused approach to metrics.

FINOS Member Bank Metrics System

• Measures include Dora metrics, flow metrics, and engineering capability metrics.
• Challenges: Consensus building around metrics takes time.
• Metrics reported to CIOs and Executives.

InnerSource vs. Open Source Metrics

• Discussion on using InnerSource and Open Source interchangeably.
• InnerSource metrics can include community engagement, reuse, and contribution.
• Potential overlap in metrics between engineering success and community engagement.

Feedback from Various Participants

• Importance of reusable components and developer happiness.
• Qualitative measures like storytelling and simple metrics for beginner teams.
• FINOS member bank uses extensive metrics to drive strategic decisions, including a patented internal system.

Tactical Implementations

• Team working on implementing CHAOSS metrics using GraphQL API.
• CHAOSS are complex tools and the approach should start with GraphQL API for simpler, more manageable metrics.
• Focus on community metrics and feasibility within the bank.
• Weekly sprints and retrospectives to refine the approach.

Challenges and Considerations

• Complexity of CHAOSS tools, importance of starting small with GraphQL API.
• Need to balance between qualitative and quantitative data.
• Importance of trust and clear communication in surveys and metrics.

Metric Specific Discussions

Specific metrics discussed in the meeting that a bank should measure for InnerSource projects included:

• Contribution Metrics:
  • Number of merge requests
  • Number of commits
  • Number of contributors
• Consumption Metrics:
  • Number of forks
  • Number of stars (if applicable)
  • Usage statistics
  • "Libyear" for measuring how far behind a project is from the mainline, which was highlighted as important for mitigating potential risks.
• Engagement Metrics:
  • Number of issues opened and closed
  • Response time to issues and pull requests
  • Number of comments on issues and pull requests
• Quality Metrics:
  • Code review times
  • Test coverage
  • Frequency of code refactoring
  • Number of bugs reported and fixed
• Reuse Metrics:
  • Number of teams using the InnerSource project
  • Number of reusable components/modules
  • Instances of code reuse across projects
• Community Health Metrics:
  • Developer satisfaction and feedback (via surveys)
  • Net promoter score (NPS) for the InnerSource initiative
  • Onboarding time for new contributors

The meeting emphasized the importance of balancing both qualitative and quantitative data and considering the trust and communication aspects when collecting these metrics.

It’s important to have trust and clear communication when collecting survey data and metrics, this makes sure the survey is acceptance and the data is interpreted correctly.

Next Steps

• Explore integration of CHAOSS tools with internal banking systems.
• Consider iterative approach to implementing and refining metrics.
• Engage with data lake and contribution model examples.
• Connect with CHAOSS Africa for broader community engagement.

Conclusion

• Positive feedback on the roundtable format and participation.
• Plan to continue similar discussions to complement FINOS special interest groups.
• Acknowledgment of the importance of community and collaboration in driving metrics success.

[psmulovics]

@mcleo-d , do you want to record a small video to advertise this to be posted on the FINOS linkedin feed?

[caradelia]

@mcleo-d will Dawn Foster be sharing 8Knot? Unfortunately, I can't attend as we have FINOS AI meetings at the Red Hat office.

[mcleo-d]

Hi @caradelia - It's great to hear from you. I can confirm that Dawn is involved in the discussion.

[mcleo-d]

Hi @psmulovics - The roundtable is Chatham House Rule, so I would prefer to limit the inclusion to OSR members that we know. Thanks for the great suggestion :)

[geekygirldawn]

Here are the slides that I used today (along with some bonus backup slides).
CHAOSS_Overview_NatWest.pdf

Some links:

• https://chaoss.community
• https://chaoss.community/kb-getting-started/
• https://chaoss.community/software/
• https://chaoss.community/about-chaoss-practitioner-guides/
• https://chaoss.community/kbtopic/all-metrics-models/