Skip to main content

37. Question-Driven Metrics

Status: Accepted Date: 2025-07-06

Context​

In a data-rich environment like a trading system, it is easy to fall into the trap of collecting vast amounts of data without a clear purpose. This leads to a "data swamp" where it is difficult to find meaningful insights. The goal of the Mercury experimental framework is not just to gather statistics, but to answer specific business questions about the performance of different trading strategies.

Decision​

We will adopt a "Question-Driven Metrics" approach. For every experiment, we must first define the specific, measurable business question(s) we are trying to answer (e.g., "Does variant B generate a higher Sharpe ratio than variant A during high-volatility periods?"). Every metric that is collected, every analysis script that is written, and every dashboard that is built must be directly tied to answering one of these predefined questions. We will explicitly avoid collecting data "just in case" it might be useful later.

Consequences​

Positive:

  • Focused Engineering: Engineering and analysis efforts are focused on what really matters, preventing wasted time on collecting and storing data that provides no value.
  • Clearer Insights: The results of experiments are clear and unambiguous because they directly answer the questions that were asked.
  • Reduced Complexity: Avoids the complexity and cost of building and maintaining a large, unfocused data collection and analysis pipeline.
  • Actionable Results: The process is inherently geared towards producing actionable results that can inform business decisions.

Negative:

  • Requires Upfront Thinking: This approach requires more discipline and upfront thinking to clearly define the questions before starting an experiment.
  • Less Exploratory Data: We may miss out on unexpected insights that could have been found through more open-ended, exploratory data analysis. This is a deliberate trade-off.

Mitigation:

  • Structured Experiment Definition: The code-based configuration for each experiment will include a mandatory questions field, which is an array of strings describing the questions the experiment is designed to answer.
  • Regular Review: The questions themselves should be reviewed regularly to ensure they are still relevant to the business goals.
  • Balanced Approach: While the primary approach is question-driven, we can schedule periodic, time-boxed "exploratory analysis" sprints if we suspect there may be valuable insights hidden in the data that we are not currently capturing.