Get to a place where we can reliably create release tags

[sirkitree]

Problem:

Our open-source project is experiencing rapid growth with numerous contributions from our community members. While we have the infrastructure to create releases, we currently face challenges in ensuring that these releases are reliable and stable. The key issues are:

• Broken Test Suite: Our automated tests are failing, which means we cannot confidently verify the stability and functionality of new code changes.
• Rapid Code Changes: With many contributors and fast-moving code, it's difficult to track and integrate changes without introducing new bugs.
• Unreliable Releases: Without a consistent and dependable release process, we risk distributing versions of our project that may not meet quality standards.

---

Proposed Solution:

To overcome these challenges and establish a reliable release process, we propose the following steps:

1. Prioritize Fixing the Tests:

   • Assess and Repair Tests: Identify all failing tests and determine the causes. Fix broken tests to ensure they accurately reflect code functionality.
   • Community Involvement: Encourage contributors to help fix tests by creating specific issues and assigning tasks.
   • Testing issues can be found with the 'testing' label: https://github.com/ai16z/eliza/issues?q=is%3Aopen+is%3Aissue+label%3Atesting

2. Establish Continuous Integration (CI):

   • Automate Testing: Set up a CI system to automatically run the test suite on every commit and pull request. For the most part this is already in place but only runs a basic test currently.
   • Enforce Test Passing: Configure the CI to block merging of code that doesn't pass all tests, keeping the main branch stable. We currently have these running on every pull request, but it's not a blocking rule.

3. Implement a Branching Strategy:

   • Use Feature Branches: Have contributors work on separate branches for new features or fixes to isolate changes. For the most part this is happening already.
   • Adopt Release Branches: Create dedicated branches for preparing releases, allowing for final testing and stabilization.

4. Define a Release Workflow:

   • Release Types: Clearly define different release types (e.g., alpha, beta, stable) and their purposes.
   • Semantic Versioning: Use semantic versioning to communicate the nature of changes in each release.

5. Automate the Release Process:

   • Automated Builds and Tagging: Utilize tools to automate the building of releases and tagging of versions.
   • Generate Release Notes: Automatically create release notes from commit messages to document changes. This is already in place and is updated when a tag is created through CI: https://github.com/ai16z/eliza/blob/main/CHANGELOG.md

6. Establish Release Criteria:

   • Set Clear Requirements: Define what must be completed before a release (passing tests, code reviews, updated documentation).
   • Use a Release Checklist: Implement a checklist to ensure all criteria are met before releasing.

7. Enhance Collaboration and Communication:

   • Regular Updates: Hold regular meetings or send updates to keep everyone informed about release progress.
   • Transparency: Make the release roadmap and plans accessible to all contributors.

8. Implement Code Reviews and Quality Assurance:

   • Mandatory Code Reviews: Require that all code changes are reviewed by at least one other contributor. We've talked about this in discord, but it hasn't been implemented as a rule yet.
   • Use Quality Tools: Integrate static analysis and other quality assurance tools to catch issues early.

9. Plan for Different Release Channels:

   • Stable Releases: Provide thoroughly tested versions for general use.
   • Beta and Nightly Builds: Offer less stable versions for testing new features and gathering feedback.

10. Monitor and Iterate:

    • Collect Feedback: After each release, gather input from users and contributors to identify improvements.
    • Continuous Improvement: Regularly refine the release process based on feedback and performance metrics.

---

Next Steps:

• Immediate Actions:

  • Fix the Test Suite: Prioritize repairing all broken tests with the help of the community.
  • Set Up CI System: Implement a continuous integration pipeline to automate testing and enforce code quality.

• Community Engagement:

  • Communicate Plans: Share this proposed solution with the community to align efforts and expectations.
  • Encourage Participation: Invite contributors to take active roles in improving the testing and release processes.

• Documentation:

  • Create Guidelines: Document the release workflow, branching strategy, and contribution guidelines.
  • Provide Resources: Offer tutorials or guides on how to work with the new processes and tools.

---

By taking these steps, we aim to build a robust and reliable release process that can handle rapid development while maintaining high-quality standards. This will not only improve the stability of our releases but also enhance collaboration within our community, ultimately leading to a better project for all users and contributors.

[yodamaster726]

this is a great idea. we need to implement something that gets us to some continuous incremental stability.
create some basic unit tests that run via github actions that notes a certain level of stability and only promote code to main if it's good. also maybe have another dev-readme that maybe documents what all the functions are that are known to be stable, what is under development and what is in the pipeline.
I know, most of this is already covered above - but just had to express my thoughts. This project is awesome, let's keep making it better.