experiment - Details

---

  In Manage Goals, we spent time challenging our assumptions. Keep in mind  that there are assumptions related to pursuing the goal itself and assumptions related to the process of attaining the goal. When assumptions related to the process change, they often spark new approaches to working toward the goal. Small changes are known as Kaizen. Incremental small changes yield steady but controllable improvements to performance. However, Kaizens are not enough for optimal performance. Often we need a revolutionary change to obtain the next level improvements that bring about extraordinary results. These revolutionary changes, called Kaikaku, are the result of a complete revamping of the way of thinking. These new ways of thinking are paradigm shifts that come with a whole new set of assumptions.*Note - the idea of “stretch goals” are meant to inspire paradigm shifts as people will have to think differently in order to accomplish these difficult goals. However, research has proven that most people are demotivated by goals that they perceive to be impossible to reach. It generally causes more anxiety and mistakes for most.Assumptions are the foundations of an organization's thought process. When assumptions are incorrect, the decisions made from those assumptions may be incorrect, resulting in unnecessary time, energy, and money spent on undesirable outcomes.  The assumptions need to be validated via hypothesis tests.

The second step is determining an order for testing our assumptions. When a plan is created to reach a goal, we determine the riskiest assumptions made and test those first, systematically derisking each one in a way that allows for learning. We’ll prioritize our list of assumptions by conducting a Risk Magnitude assessment. For each assumption, to assess the Risk Magnitude, Impact Severity and Probability are scored on a 1-10 scale, with 1 equaling least severe/probable if incorrect and 10 equaling most severe/probable if our assumption is incorrect. The Risk Magnitude is then computed by multiplying the Impact Severity by the Probability of the risk occurring. Risk Magnitude = Impact Severity x Probability of occurrenceThe assumptions are prioritized in order from highest to lowest risk. By doing this as a team, members are allowed to weigh-in and decide together what assumptions are necessary to test to reduce the highest risk in attaining the future state. The output of this step is a documented and prioritized list of assumptions.The team creates the hypothesis by first stating the assumption they agree to test. The assumption makes up the GIVEN part of the hypothesis statement. Next, the team states the expected outcome in the THEN part of the statement. The outcome should be formed with adjectives describing a noun.  Concentration and attention should spent forming the outcome, ensuring it is specific, measurable, attainable, reasonable, and time-bound (SMART), before continuing. Once the outcome meets the SMART criteria, the team can then determine the evidence that must be produced in the given timebox in order for stakeholders to agree with the assumption. This evidence, the externally observable characteristics that would indicate the outcome has been successfully reached, is our Acceptance Criteria,. These characeristics can be derived by asking the question, “What would we observe if the expected outcome has been achieved?” A technique for doing this is to take each adjective in the outcome statement one-by-one, and describe what observable pieces of evidence would exist in order to know that the described qualities or state exists. At the end of this activity, the team agrees on the Acceptance Criteria, before moving on, and believe it is the evidence needed by stakeholders to accept the assumption as true.Lastly, the IF part of the hypothesis statement is formed by deciding on the one thing that must be done to achieve the desired results. The general rule is that the IF statement should be a singular statement to test so that a clear analysis of cause and effect can be performed. When there are no actions that can be taken to make the assumption true, then we should carefully consider whether the results are realistic or if the assumptions are really true. The outputs of this step are a documented and approved hypothesis statement and consensus on the Acceptance Criteria as the evidence necessary to agree with the assumption.

In this step, the test to be conducted is planned. Planning includes decomposing the work into smaller pieces that fit within the associated timebox, grooming those smaller elements, determining the resources for doing the work, and estimating the size of each element. The outputs of this step include an updated list of the decomposed and groomed work elements in the backlog with details for conducting the test. TME then provides their approval of the hypothesis test plan.

Defining the metrics and creating the analysis plan is the next step. To start, the team should agree that executing the hypothesis will result in a decision or learning about the hypothesis determined by measurable evidence. Once that is established, the team is ready to determine which metrics will be used to provide the evidence by referring back to the list of externally observable characteristics created earlier. At least one metric is needed to quantify each characteristic, to either prove or disprove it exists, and build the case for the hypothesis. Once the metrics are agreed upon, the team gets consensus as to how the results will be interpreted before testing starts and any results are measured. This analysis plan is created in advance to allow for an objective discussion after the results are obtained and avoids bias. This plan documents the predetermined values that each metric needs to reach, within specific timeframes, in order to form a conclusion about a learning or a decision. These values are the metric targets, the minimum numerical value that the measurement should meet or succeed in order to form a successful conclusion. In addition, the analysis plan may include thresholds for metrics, which are measured values that indicate when a specific action will be taken. In summary, the analysis plan tells us at what point we will conclude a cause and effect relationship does or does not exist as we hypothesized, or the results are inconclusive and additional testing is needed.The outputs of this step are a documented set of specific metrics that will be used to measure results of the test, as well as an analysis plan for how those metrics will be interpreted, In addition, we should now have decided how the execution of the hypothesis test will be conducted (EFT, POC, sprint review, etc.).

In this step, we execute our hypothesis test plan. Measurements are taken before the experiment begins in order to establish a baseline for the metrics. Then, as the decomposed work elements are executed and completed, measurements are taken. Each set of measurements are analyzed in order to assess results along the way to determine if a learning or decision point has been reached, and to forecast results going forward based on the trending of data. The test ends when either enough evidence has been made to conclude a decision or learning, or the timebox ends.  The outputs of this step are a set of published results, analyses and proposed decisions.

Next, the proposed decision and supporting results and analyses is shared with the stakeholders.  This communication informs the stakeholders of the actions that will occur as a result of the hypothesis testing and ensures everyone understands the new current state. A new published and communicated decision is the output from this step.

Our published decision should result in change.  In this step, we take all of the necessary actions to make the changes and updates as a result our of our new decision and learning.  This includes updating our assumption list, risk probability and magnitude, hypothesis, backlogs, prioritization of backlogs, communications plan, strategy in Aha, Business Epic in Aha, and the next agile commit. 

At this point, we decide if all of our riskiest assumptions have been validated. If so, we end the hypothesis planning process for this goal. If we need to continue testing additional assumptions, we repeat the process by forming a hypothesis for the next riskiest assumption, as shown in the flowchart.