Environment:Apache Kafka Committer Tools Environment
| Knowledge Sources | |
|---|---|
| Domains | Infrastructure, Developer_Tools |
| Last Updated | 2026-02-09 12:00 GMT |
Overview
Python 3 environment with GitHub API, JIRA, and Git CLI for executing the PR merge, cherry-pick backport, and JIRA resolution workflow.
Description
The Committer Tools Environment supports the interactive `kafka-merge-pr.py` script used by Apache Kafka committers to merge pull requests, perform cherry-pick backports to maintenance branches, and resolve associated JIRA issues. It requires Python 3 with `beautifulsoup4` (for HTML parsing), `PyGithub` (for GitHub REST API interaction), and `ruamel.yaml` (for YAML configuration). Additionally, the `jira` Python library is needed for JIRA issue resolution. The script interacts with Git via subprocess calls for merge, cherry-pick, and push operations.
Usage
Use this environment when executing the PR Merge and Backport workflow. This includes fetching PR metadata from GitHub, standardizing JIRA references in commit messages, performing squash merges, cherry-picking to maintenance branches, pushing to the remote, and resolving JIRA issues.
System Requirements
| Category | Requirement | Notes |
|---|---|---|
| OS | Linux or macOS | Interactive terminal required for user prompts |
| Python | Python 3.x | Required for merge script |
| Git | Git CLI | Must have push access to apache/kafka remote |
| Network | Internet access | GitHub API and JIRA API access |
Dependencies
System Packages
- `python3`
- `git`
Python Packages
- `beautifulsoup4` == 4.12.3
- `PyGithub` == 2.4.0
- `ruamel.yaml` == 0.18.6
- `jira` == 3.8.0 (for JIRA resolution)
Credentials
The following credentials must be available:
- GitHub Token: Personal access token with repo read access (used by `get_json` for GitHub API calls).
- JIRA Credentials: Apache JIRA username and password for issue resolution via `resolve_jira_issues`.
- Git SSH Key: SSH key with push access to the `apache/kafka` repository (or HTTPS credentials).
Quick Install
# Install Python dependencies for committer tools
pip install beautifulsoup4==4.12.3 PyGithub==2.4.0 ruamel.yaml==0.18.6 jira==3.8.0
# Or use the requirements file
pip install -r committer-tools/requirements.txt
Code Evidence
Python dependencies from `committer-tools/requirements.txt:18-20`:
beautifulsoup4==4.12.3
PyGithub==2.4.0
ruamel.yaml==0.18.6
GitHub API usage from `committer-tools/kafka-merge-pr.py:77-90`:
def get_json(url):
try:
request = Request(url)
if GITHUB_OAUTH_KEY:
request.add_header('Authorization', 'token %s' % GITHUB_OAUTH_KEY)
return json.loads(urlopen(request).read())
except HTTPError as e:
if "rate limit" in str(e).lower():
print("Error: GitHub API rate limit exceeded.")
JIRA resolution from `committer-tools/kafka-merge-pr.py:264-320`:
def resolve_jira_issues(title, merge_branches, comment):
jira_ids = re.findall("KAFKA-[0-9]+", title)
for jira_id in jira_ids:
resolve_jira_issue(jira_id, merge_branches, comment)
Common Errors
| Error Message | Cause | Solution |
|---|---|---|
| `HTTPError: 403 rate limit exceeded` | GitHub API rate limit | Set `GITHUB_OAUTH_KEY` environment variable with a personal access token |
| `jira.exceptions.JIRAError: 401 Unauthorized` | Invalid JIRA credentials | Verify JIRA username and password |
| `fatal: Could not read from remote repository` | Git SSH key issue | Ensure SSH key is added to ssh-agent and has push access |
| `ModuleNotFoundError: No module named 'github'` | PyGithub not installed | Run `pip install PyGithub==2.4.0` |
Compatibility Notes
- GitHub Token: Without a token, API requests are rate-limited to 60/hour. With a token, 5000/hour.
- Interactive Mode: `kafka-merge-pr.py` requires an interactive terminal for user prompts (branch selection, author confirmation).
- JIRA Resolution: Optional step; can be skipped if JIRA credentials are not configured.