A Note on Developing and Testing Hypotheses

This post is a continuation of the discussion I started in A Note On Startup Business Model Hypotheses. In this post I will describe how one might go about developing and testing a hypothesis about any aspect of a startup’s  business model.¹

I will use a fine-dining restaurant as a motivating example.

To ensure we are on the same page, first some definitions.

Definition #1: What is a startup? A startup is a temporary organization built to search for the solution to a problem, and in the process to find a repeatable, scalable and profitable business model that is designed for incredibly fast growth. The defining characteristic of a startup is that of experimentation – in order to have a chance of survival every startup has to be good at performing the experiments that are necessary for the discovery of a successful business model.²

Definition #2: What is a business model? A business model is the description of how a startup will create, deliver and capture value. Alex Osterwalder’s Business Model Canvas is one framework for describing and documenting the elements of a startup’s business model.

Definition #3 What is a hypothesis? A hypothesis is a statement, or a group of statements, that proposes an answer to a question, or a solution to a problem, in a manner that is testable through experimentation. The goal of experimentation and testing is to determine if the hypothesis is correct, and to inform the subsequent actions that the startup should take on the basis of that evidence.

A hypothesis is;

1. A guess about the process underlying a set of observations that have been made by the founder.
2. A testable guess, in the sense that it attempts to establish and predict the basic relationship between two or more variables that interact with one another to lead to the observed phenomena. This allows the researcher to test what happens when one of those variables is allowed to change, while others are held constant.
3. Not the same as a research question; in the sense that a research question is broad while a hypothesis is more narrow in scope.

Motivating example; A fine-dining restaurant is experiencing an ongoing slump in revenues. The restaurateur wishes to test a number of possible approaches to reversing that trend. From prior experience they believe that the following factors each has a positive impact on overall sales at the restaurant;

1. Wine Sales
2. Liquor Sales
3. Appetizers
4. Seasonal Menu Changes
5. Table Turns
6. PR, Advertising and Marketing

How should management determine where to make an adjustment in order to improve overall sales without incurring a large capital outlay?

Discussion

Let us assume that Sales can be modeled by the following relationship.³

Remember that the goal is to try to figure out a course of action without spending too much by way of capital until the restaurateur is fairly certain that any capital deployed to this end will yield a disproportionately positive result.

To keep the discussion brief, let’s focus on two of the factors that management believes play an important role in driving revenues from the preceding list. Let’s focus on Wine Sales and PR, Advertising, and Sales.

Assume the restaurant is laid out such that the front-of-the-house is organized as two nearly identical dining rooms, they are separated by an ornate vestibule. During the experiment one dining room is operated status quo, while the other is operated as part of the experiment. Management feels this makes sense because guests are typically evenly split between the two dining rooms. Let’s call them Dining Room #1 – the control dining room, and Dining Room #2 – the dining room in which the experimental change is implemented.

Possible Hypotheses for Wine Sales Program – Experiment #1

1. General Hypothesis: Implementing a wine-program will have an effect on sales.
2. Directional Hypothesis: Implementing a wine-program will have a positive effect on sales.
3. Testable Hypothesis: If we implement a wine-program we will increase revenue significantly because average check size increases substantially when wine sales increase.

Experiment #1: Management might hold all the factors constant, but implement an in-house training program to get wait-staff more comfortable speaking to guests of the restaurant about it’s current small collection of wine. Waiters in Dining Room # 2 are coached to discuss wines before guests order their meal, and afterwards when a guest might be considering a dessert. Let’s assume this program lasts a month. At the end of the month, management compares data from Dining Room #1 with data from Dining Room #2.

Possible Hypotheses for Inside-Sales Program – Experiment #2

1. General Hypothesis: Implementing an inside-sales program will have an effect on sales.
2. Directional Hypothesis: Implementing an inside-sales program will have a positive effect on sales.
3. Testable Hypothesis: If we implement an inside-sales program we will increase revenue significantly because average check size increases substantially when wait-staff are encouraged to engage with guests.

Experiment #2: In this experiment, which also lasts a month, management trains wait-staff in Dining Room #2 to engage guests in casual conversation and to speak more enthusiastically and informatively about each day’s specials. Wait-staff are trained to highlight upcoming events at the restaurant that regular guests might find interesting enough to return for – a special prixe-fixe Wine Dinner the following week, for example. On occasion, every 90 minutes or so, the restaurant’s chef de cuisine spends about 10 minutes walking through Dining Room #2 and socializes with guests. The restaurateur wishes to determine if making things like this a more consistent part of how the restaurant operates makes sense from the perspective of increasing revenue.

During each experiment management collects the following data for each day, each week, and also for the month:

1. Total Revenue
2. Average Check Size – Revenue per Table
3. Average Wine Sales – Wine Sales per Table
4. Positive Social Media Mentions
5. Negative Social Media Mentions
6. In-restaurant Reservations – future reservations made while the guest is at dinner as a result of learning about an upcoming special event. This only applies to Experiment #2.

Now that the restaurant has collected some data it is time to test the data to see what conclusions the restaurateur might be able to reach based on each of the experiments.

Stating Null and Alternative Hypotheses

To test the hypotheses our restaurateur must make two different statements that will form the basis of a test; The Null Hypothesis states that the effect our restaurateur thinks exist does not in fact exist. The Alternative Hypothesis makes a statement opposite to that made in the null hypothesis, and typically is the statement we want to prove.

Experiment #1: Null and Alternative Hypothesis

1. Null Hypothesis: Implementing a wine sales program has no effect on revenue.
2. Alternative Hypothesis: Implementing a wine sales program has a positive effect on revenue.

Experiment #2: Null and Alternative Hypothesis

1. Null Hypothesis: Implementing an inside-sales program has no effect on revenue.
2. Alternative Hypothesis: Implementing an inside-sales program has a positive effect on revenue.

Note that the null and alternative hypotheses stated above are merely examples. The restaurateur could formulate each of those statements in more general terms, or with more specificity than I have done. To some extent that choice depends on the granularity of the data that was collected from the experiments.

At this point our restaurateur can perform a test of statistical inference to reach a conclusion – the outcome would be a “rejection” or “a failure to reject” the null hypothesis. A rejection of the null hypothesis leads us to accept the alternative hypothesis. A failure to reject the null hypothesis leads us to fail to accept the alternative hypothesis.

Assuming that the restaurateur rejects the null hypothesis in both instances, then it makes sense to spend some capital trying to build out a more robust version of each of the experiments we described, with the intention of operationalizing what was done during the experiment and making those practices a permanent part of how the restaurant is managed and run on an ongoing basis.

Closing comments:

1. I have glossed over a significant amount of detail. That was deliberate. The goal of this post is not to discuss statistical theory, but to think about how statistical thinking can help startup founders who need to make important choices about how to utilize scarce resources. More detail can be found in any good introductory level business statistics text book.
2. While going through this process our restaurateur needs to ask more questions than I did in this post. For example, what does “significant” mean? Is a 5% increase in revenue significant? Why? Is a 20% increase in revenue significant? Why? Or, why not? Are the costs associated with implementing the changes necessary to make what was done during the two experiments a permanent operating practice of the restaurant justified by the restaurateur’s forecasts of the long term benefit of doing so? Why, or why not?
3. It is always important to think about sources of error whenever one is conducting an experiment that is supposed to yield data that decisions like the one I described in this post will be based upon. In this instance, one concern might be that the restaurateur is not collecting the appropriate data on which this decision should be based. Or, for example, that a single month is not a wide enough window of time to determine if the effect observed during this period of the experiment persists during the remaining 11 months of the year.

Notwithstanding those concerns, I believe that when it is possible, this kind of analysis should always complement management’s intuition.

1. Let me know if you feel I have failed to attribute something appropriately. Tell me how to fix the error, and I will do so. I regret any mistakes in quoting from my sources. ?
2. I am paraphrasing Steve Blank and Bob Dorf, and the definition they provide in their book The Startup Owner’s Manual: The Step-by-Step Guide for Building a Great Company. I have modified their definition with an element from a discussion in which Paul Graham, founder of Y Combinator discusses the startups that Y Combinator supports. ?
3. I am in no way suggesting this is the appropriate model for this problem. I am using this only for the purpose of this discussion. Assume the restaurateur has developed this model after years of experience in the industry, and based on trial and error. This model is supported by the last 5 years of sales data. ?