Business Experiments: Steps and Examples
Business Experiments: this article provides a practical explanation of various forms of Business Experiments. After reading, you’ll understand the basis of this supportive function for decision making, the various experiments, and the added value of experiments for both the organisation and the customer.
What are Business Experiments?
As a manager, but also as an entrepreneur, coming up with new ideas and executing them is a matter of course. Creating new, innovative ideas and ways to improve is essential for stimulating success within a company. Having or lacking success is not only beneficial or negative for the organisation itself, but could also have far-reaching positive or negative consequences for society.
For example, think of the development of the smartphone, cars, and aeroplanes. To some extent, these innovations were all the product of extensive testing and experimenting. It is therefore not enough to merely have ideas. The question is whether these ideas will yield results.
That’s why evidence is needed that these ideas and methods are actually effective. This evidence lies in the data obtained during Business Experiments. It should be clear that business experiments are necessary. First of all, to test and validate the ideas and methods, but also to determine the right method of implementation.
In fact, experiments are processes in which research methods and techniques are applied to assumptions or hypothesis. By testing these hypotheses and subjecting them to analysis, measurements, and validation, a conclusion can be formed based on facts. If the conclusion is positive, the decision is made to continue with the product or service.
Should the conclusion show that the results are still insufficient, the product or service is either discarded or improved and tested again.
Different Forms of Business Experiments
A business experiment, then, is a process in which different strategies, designs, and configurations are tested using a structured test-and-learn approach. Business experiments and testing revolutionary ideas are often linked to innovation. The exact goal of the experiment depends on the type of experiment.
In each type of experiment, however, data is obtained that is analysed and used in decision making. A distinction is made between, among others, the following types of experiments:
A/B Tests
A/B testing is a testing method that is also based on a hypothesis. This experiment often makes use of two configurations of a certain set-up, such as a website. The old situation is compared to the situation after the experiment. The terms “control group” and “treatment group” are also used.
It’s a popular test method among website developers, who use it to test certain content and configurations. Using such an experiment, certain statistics such as conversion ratios and click-through rates can be optimised.
Prototype Tests
Another form of business experiment is developing a prototype. In the case of a prototype, a certain hypothesis about a product or service is tested by actually developing it. Based on the results after analysis of the prototype, it is then decided whether or not to continue developing the end product.
Two kinds of prototypes are usually distinguished. An evolutionary prototype is a prototype that is constantly adjusted and improved in order to realise a change to a product or a future product. Evolutionary prototypes are often expensive and are used in sectors that require products to be tested extremely thoroughly before they are introduced to the market. Examples include the automotive and aerospace industries.
In addition, throwaway prototypes are also development and used. These are cheap and quickly designed to demonstrate a certain idea or feature. Throwaway prototypes are used in early phases of a product’s design.
Concept Testing
Concept testing is essentially a marketing experiment in which input from the potential customer is used in early phases of the design of a product or prototype. An important aspect in concept testing is gathering customer feedback through interviews, focus groups, or market research.
Mock-up
Mock-ups are used in production and design units within a company, and are often scale or full-size models of a design or device. The mock-up is used for promotion, design evaluation, knowledge transfer, and other ends. A mock-up can be seen as a prototype in which at least part of the original design is functional. Mock-ups are mainly used to gather feedback from users.
Thought Experiments
In a thought experiment, the consequences and effects of a certain hypothesis, procedure, or idea are evaluated. Thought experiments are developed in order to research ideas that don’t require physical experiments. Concrete thought experiments can be carried out using logic and arguments.
Abstract thought experiments often concern ideas about the future and go hand in hand with speculations.
With thought experiments, various problems and hypothesis can easily be approached from various perspectives, especially when the experiment is carried out by multiple people.
Pilots
A pilot, also known as feasibility study or experimental test, is a relatively small-scale experiment that tests whether a large project could also work well in practice. It provides organisations with a platform for testing certain projects, proving their worth, and uncovering shortcomings. It’s important that this be done in advance, before a considerable amount of money, energy, and time is spent on a large project.
Doing Business Experiments Yourself (Step-by-step Plan)
Step 1: Formulating a Hypothesis
The process of an experiment or test always starts with formulating a verifiable, measurable hypothesis. Verifiable means it is possible to make the test or the experiment succeed or fail based on the objectives specified in the hypothesis. The details of the test are then designed, and it is determined which of the above methods, or an alternative, is suitable for the test or experiment.
Step 2: Designing the Experiment
If the hypothesis is verifiable and has been approved, the next step in the process is designing the experiment itself. This step concerns, among other things, identifying the objects, locations, or units to be tested; selecting control groups and treatment groups (the A/B tests discussed above); and defining testing and control mechanisms.
Determining which experiment is suitable for which hypothesis may be difficult. An example. The customer service department at a large multinational would like to improve their customer service.
To this end, they decide to conduct an experiment in which part of the customer service staff is taught a new approach (the treatment group), while another part (the control group) continues to do what they were doing before. By investigating the results of both groups over time, it can be determined whether the method used in the experiment is sufficiently effective or not.
Step 3: Data Analysis
After the experiment finishes, the data and results are analysed. In the experiment with the customer service, that means analysing the difference between the performances of the control group and the treatment group. Does the difference match our hypothesis? Has the quality of customer service actually increased? Have profit margins gone up? Is the outcome surprising? Are any additional experiments needed?
Step 4: Completion
After the test has been performed within the determined period at the determined location, the data obtained is analysed in order to determine the results. These results are ideally stored in a company library, and may lead to a broader roll-out of the experiment, or testing a revised hypothesis.
Step 5: Re-testing
A difficult aspect when setting up a testing method or experiment is deciding when to re-test. It’s difficult to determine when a test is outdated. This requires an experienced analyst to judge whether enough environmental factors have changed to make previous results obsolete.
For example, Netflix decided in 2006 that it was time to perform customer tests again. After five years, their user base had evolved from nothing but internet pioneers to members from all layers of society. Retailers and producers may also want to re-test pricing, for example because the prices of resources go up fast.
Business experiments: what is a Test & Learn Approach?
Test and Learn is a series of practices used by traders, banks, and other consumer-focused companies to test ideas and predict their impact. These business experiments are often performed at a small number of branches, or on a small number of customers. The process was designed to answer three questions about the testing program:
- What is the impact of the pilot/test/experiment on important performance indicators?
- Does the pilot/test/experiment have a greater impact on some customers or stores than on others?
- Which components of the pilot/test/experiments work well? Which ones work less well?
Part of the Company Strategy
In its simplest form, the Test & Learn approach is a series of methods a company can use to experiment with new concepts and ideas. There are various reasons why Test & Learn should be added to any organisation’s business operations.
A great advantage of this method is that it allows for testing with real customers and real employees. They provide realistic data during the experiment, on the basis of which strategic decisions can eventually be made.
The Test & Learn method brings the initiative, the proposal, directly to the end user. This could be a new channel of communication, a reward structure, a new process, or something else. If it is received positively, the initiative will be taken on board for the future. If the initiative is not received positively, it may be further developed based on experience, or it can be decided to discard it.
Whatever the result, the decision is made on the basis of something that is proved by the data obtained, rather than presumptions or theories.
Practical Examples of Business Experiments
At eBay, applying changes, testing and experimenting is an important part of the company. Just like other online companies, eBay profits from the fact that it is relatively easy to carry out randomised tests of different website variations.
The online retailer’s administrators have carried out thousands of experiments. Because the site receives more than a billion visitors daily, they can carry out various experiments at the same time, while still having control groups and treatment groups in abundance. The simple A/B tests, in which two versions of a website can be compared, can be set up within a matter of days, and usually last at least a week.
Experimenting is so interesting to the company that it has built its own application called eBay Experimentation Platform. It is used by testers to keep track of what happens and which pages are tested during which times.
Besides the use of online experiments, the company also carries out offline tests, such as laboratory studies, home visits, focus groups, and website function analyses.
Business Experiments summary
Experiments are processes in which research methods are applied in order to come to a conclusion about a certain hypothesis or assumption. Business experiments come in several different forms. A well-known experiment, often used by website developers, is A/B Testing. This involves, for example, rendering a website in two configurations. Other methods are concept tests, prototypes, and pilots.
Before the experiment, a hypothesis is formulated. It’s important that the hypothesis be verifiable. Verifiable means it is possible to make the test or the experiment succeed or fail based on the objectives specified in the hypothesis.
The data gathered during the experiment is analysed afterwards. Based on this, a decision is made whether implementation should continue or the testers should go back to the drawing board.
In any case, the data obtained during the experiment is stored in the central company library. After some time, it may be necessary to test an experiment again because too many environmental factors have changed.
Companies that acknowledge the value of experimenting and testing make it into an important part of their business operations. In that case, striving for innovation through experimenting and testing is expressed in a so-called Test & Learn Approach. An advantage of this approach is that it allows for experiments to be applied to real customers and employees. It also helps testers to go through the testing process or experiment in a structured way.
Now it’s your turn
What do you think? Do you recognise the explanation about Business Experiments? Are any experiments carried out in your working environment? Do you know any examples of large and successful experiments performed in the past at large companies? Do you have any tips or additional comments?
Share your experience and knowledge in the comments box below.
More information
- Anderson, E. T., & Simester, D. (2011). A step-by-step guide to smart business experiments. Harvard Business Review, 89(3), 98-105.
- Bocken, N. M. P., Bom, C. A., & Lemstra, H. J. (2017). Business experiments as an approach to drive sustainable consumption: The case of homie. Delft University of Technology, 8, 10.
- Davenport, T. H. (2009). How to design smart business experiments. Strategic Direction.
- Ganguly, A., & Euchner, J. (2018). Conducting Business Experiments: Validating New Business Models Well-designed business experiments can help validate assumptions and reduce risk associated with new business models. Research-Technology Management, 61(2), 27-36.
- Website Vantazo. Retrieved 02/20/2024 from Vantazo.com
How to cite this article:
Janse, B. (2020). Business Experiments. Retrieved [insert date] from Toolshero: https://www.toolshero.com/decision-making/business-experiments/
Original publication date: 24/02/2020 | Last update: 02/20/2024
Add a link to this page on your website:
<a href=”https://www.toolshero.com/decision-making/business-experiments/”>Toolshero: Business Experiments</a>
Ben Janse
Ben Janse is a young professional working at ToolsHero as Content Manager. He is also an International Business student at Rotterdam Business School where he focusses on analyzing and developing management models. Thanks to his theoretical and practical knowledge, he knows how to distinguish main- and side issues and to make the essence of each article clearly visible.
Related ARTICLES
Product Life Cycle Stages Theory with Examples
Five Product Levels by Philip Kotler
Buy Sell Hierarchy (Miller and Heiman)
Raymond Vernon biography, theory and books
