Skip to main content

72. GitHub API Integration for Worker

Status: Accepted Date: 2025-07-06

Context​

Some of our automated workflows require interaction with GitHub. For example, a background job might need to publish a performance report as a comment on a commit, upload a new data artifact to a release, or update a file in a repository. We need to decide where the logic and credentials for these GitHub API interactions will reside.

Decision​

The mercury-worker will be directly integrated with the GitHub API. We will create a dedicated GitHubService within the application that encapsulates the logic for making authenticated calls to the GitHub REST API.

The mercury-worker process will be configured with a GitHub Personal Access Token (PAT) or a GitHub App credential with the necessary permissions to perform its tasks. Specific consumers within the worker will then use the GitHubService to perform their required actions.

Consequences​

Positive:

  • Centralized Integration: All GitHub interaction logic is centralized in one service (GitHubService), making it reusable and easy to manage.
  • Enables Automation: Allows for powerful automation by connecting our internal trading system events directly to our source control and project management system.
  • Leverages Existing Infrastructure: The integration runs within our existing mercury-worker process, avoiding the need for a separate service just for GitHub interactions.

Negative:

  • Security Risk: Storing a GitHub credential with write access in the worker's configuration is a security risk. If the worker process is compromised, an attacker could gain access to our GitHub repositories.
  • Dependency on External Service: This makes parts of our background processing dependent on the availability and reliability of the GitHub API.
  • Rate Limiting: Our worker will be subject to the GitHub API's rate limits, which we need to handle gracefully.

Mitigation:

  • Least Privilege Principle: The GitHub credential used by the worker will be configured with the absolute minimum set of permissions required for its tasks. It will not be a full-access administrator token. For example, it might only have write access to one specific repository.
  • Secrets Management: The GitHub token will be stored securely in our secrets management system (e.g., HashiCorp Vault) and injected into the worker's environment at runtime. It will never be hardcoded in the source code.
  • Resilient API Client: The GitHubService will be built to be resilient to GitHub API failures. It will use a library like octokit which has built-in support for handling rate limits, network errors, and retries with exponential backoff.
  • Asynchronous by Design: Since these tasks run in background queues, an outage of the GitHub API will not block core trading operations. The jobs can simply be retried later when the API is available again.