On September 3, Bloomberg reported that Episerver had acquired Optimizely — the leading conversion rate optimization tool. During Optimizely’s last funding round in June 2020, the company was valued at $600 million. According to sources, Episerver paid LESS than that amount:
The exit is a muted outcome for Optimizely since startup founders and venture investors tend to hope for major exits at unicorn valuations. ... Like many startups, Optimizely had to cut staff in July due to the impact of Covid-19 on its business. It reduced its workforce by 60 people, a spokesperson said.
When the coronavirus hit, advertising and adtech spend was quickly impacted. Instead of cutting advertising costs across the board, though, many companies followed a similar playbook. Brand spend dropped dramatically, but “pay for performance” marketing held strong.
When times are tough, companies slash cost centers, but they do not cut revenue drivers. When a business is spending $1 on paid search to make (they think) $2 in profit, and the metrics are not changing, they would be irrational to abandon that strategy. But companies will postpone some future plans — perhaps their next TV campaign — until they understand the long-term impact of the pandemic on their finances.
Conversion rate optimization (CRO) should fall under the “performance marketing” category. Suppose you run a test that shows “Page A” has a 10% higher conversion rate than “Page B” — that should result in real dollars dropping to your bottom line. Plus, CRO delivers a more cost-efficient approach than paid search, because you don’t even need to spend the $1 to earn an additional $2 — you just need to come up with a new page design idea and run the [Optimizely] test. If the new page performs better than the old page, you will know (with 95% confidence) and you should capture additional revenue.
Even if each test only yields a 1% improvement on average, think about what happens if you run 100 tests a year: you have now doubled your business on zero incremental marketing or product spend.
So why did spend on Optimizely’s value DROP during the pandemic? Why were companies treating conversion rate optimization like television brand spend, instead of like paid search spend?
Menu design
On August 29, The Hustle published an interesting profile on Michele Benesch — a professional “menu designer” (a business she inherited from her grandfather, Walter Baker, who started Menu Makers back in 1968).
Benesch claims to use “design, psychology research, and general food knowledge to build a more scientific menu” that can significantly boost the amount of money spent by each customer. The article cites a specific example of her process for a Las Vegas restaurant:
She cut their 4-page layout down to a simple 2-page panel, upped the font size, did some dish repositioning, and cut loose some of the dishes that weren’t selling well. The new menu led to a spending bump equivalent to $9 more per customer.
Diners at top restaurants in Vegas spend around $100 per meal, so this menu revamp might represent an increase of roughly 10%.
Of course, The Hustle was not the first publication to run a story on menu optimization. Media outlets know that people love stories about the “tricks” that businesses use to manipulate customers into spending more money than they would otherwise “choose.”
A 2018 article from The Globe and Mail presented some highlights from a Cornell University study about menu optimization. Consider, for instance, three different ways to list the price of a menu item:
$20.00
20
Twenty dollars
According to the Cornell research, diners spent “significantly more” when the price was expressed as “20” — the option without any direct association to money.
How much is “significantly more” money? By digging into the research report, we can identify the specific impact of each pricing format.
The study concludes that using “20” instead of “$20.00” or “Twenty dollars” results in a lift of $5.55 (3.70 + 1.85). In the terms of their experiment, the boost of $5.55 is regardless of the party size or time spent at the table. But if we assume two people eating for an hour (and ignoring the tip), their model predicts an average spend of $29.33. An extra $5.55 would mean an 18% increase in check size. That’s huge!
Another Cornell study reviewed the language used on menus:
Regular labels, like “Sweet potato fries”
Descriptive labels, like “Hand-sliced Portuguese sweet potato planks tossed in virgin olive oil and Maldon sea salt”
The research determined that “sales of menu items with descriptive labels were 27% higher than items without descriptive labels” and that, “customers who ate items menued with descriptive labels consistently rated the items as being of higher quality and a better value than customers rating items with regular labels.”
The Hustle article provides a few other suggestions for restauranteurs (without any supporting data or evidence):
Offer a “magic number” of items per category (7 appetizers/desserts and 10 main courses).
Use large fonts and limit white space to no more than 20% of the page.
Include “decoy” items that drive sales to the rest of the menu (e.g., Ben & Jerry’s 20-scoop “Vermonster”).
Restaurants and reality
Running a restaurant is VERY difficult.
When I ran a restaurant marketing start-up, business owners told us about their tight margins. For an average restaurant with well-run operations and steady customers, their cash flow statements looked something like this:
30% — Cost of labor
30% — Cost of rent
30% — Cost of food (also known as COGS: cost of goods sold)
10% — Profit margin (less owner time and debt financing)
Let’s suppose two things:
The Cornell study about pricing formats is accurate.
Some of these successful restaurants do NOT use “non-dollar denominated prices.”
A restaurant could presumably make that one small change to their menu and see an 18% increase in customer spend — with no additional costs. In that case, their revised cash flow statement would look like this:
25% — Cost of labor
25% — Cost of rent
25% — Cost of food (aka COGS: cost of goods sold)
25% — Profit margin (less owner time and debt financing)
Making one tiny adjustment would increase the restaurant’s profit by 2.5x.
So why are ANY restaurants still including dollar signs on their menu? They are missing out on free money!
Here’s where the theory differs from the reality.
No one actually believes that revenue will jump by 18%, simply by removing dollar signs from menu prices. Even the original researchers admitted the limitations of their research: “...these findings may apply only to lunch at this particular restaurant.” Their “study” relied on ONE particular restaurant during ONE meal per day over ONE brief time frame. That said, the conclusions were so eye-popping that media outlets produced many stories about ways that “restaurants manipulate diners.”
In recent years, many of these types of studies have been largely discredited. (The “psychology replication crisis” has shown that several landmark studies don’t hold up when other researchers try to reproduce the original experiments).
The Globe & Mail article about menu optimization claims that, “Customers who chose from the more limited menu were more likely to say they were happy with their selection.” The article also cites Cornell’s suggestion that restaurants would be smart to reduce their number of menu items, based on some “paradox of choice” evidence.
You might be familiar with the famous jam example:
Back in 2000, a series of experiments in grocery stores found that people were more likely to purchase jams when presented with a limited array of six choices in comparison to a more extensive selection of 24 choices (30 per cent in the first group made a purchase compared with only 3 per cent in the second).
One problem: choice experiments like this one have faced replication issues. We shouldn’t take these ideas as universal truths.
There ARE good reasons for a restaurant to pare back its menu selections: limiting inventory complexity should reduce food wastage. In other words, restaurants could focus on decreasing COGS rather than increasing check size.
Still, restaurants thinking about simplifying their menu selections must also remember to provide enough variety for a range of potential guests. McDonald’s, for example, did not introduce salads to drive more sales from their SuperFans. The salads appear on the menu to satisfy the NonFans who are (reluctantly) dragged to the Golden Arches by the rest of their family and friends.
Here’s one more example of an idea that appears in every “how businesses trick you” article: grocery stores place milk in the very back corner to force customers to walk through various aisles, making impulse purchases along the way.
This notion sounds plausible, but it ignores the actual reasons that milk is stored near the back of the store: it is heavy, needs to be refrigerated, and turns over quickly. Grocery stores place milk in the back corner due to cost reduction/operations/logistics optimization, NOT for customer-marketing manipulation. As Carly Simon might say, “this song is not about you.”
Conversion Overrated Optimization
Improving conversion by only a little bit can make a HUGE difference for businesses.
Because conversion rate improvements do not require additional marketing spend, even small increases in conversion can appear to generate outsized impacts. If you are a $100 million break-even business selling a digital product with no marginal costs, then a 5% increase in conversion means you are now a $105 million business with $5 million in EBITDA (Earnings Before Interest, Taxes, Depreciation, and Amortization). With another 5% conversion improvement, you would grow your top line by 5% but DOUBLE your bottom line.
If you want approval for a new initiative, you’ll probably need to demonstrate its ROI potential. You COULD try to persuade decision-makers by describing ways to lower costs, improve lead generation per $ spend, etc., but there’s an easier way to secure approval: claim that your plan will increase conversion rate by 5% (or even 1%). It’s a “believable” outcome and it will show huge returns. Which, in turn, rationalizes significant spend (which is what you are trying to get approved).
Meanwhile, the Conversion Rate Optimization team will look back at their annual results and boast that their average successful test resulted in a 5% increase in conversion after it was fully implemented. And because they ran 20 successful tests this year, they will claim responsibility for doubling the business (5% x 20). At this point, someone might challenge their conclusions, saying something like: “Your analysis says you were responsible for a 2x increase in conversion this year and a doubling of the company’s revenue, but our actual conversion this year has dropped 3% and revenue is only up 20% on significantly more marketing spend and traffic.” Without missing a beat, here’s how the CRO team will respond: “that is because other things changed — different traffic, competitors, consumer spending patterns, etc. Without our team’s great CRO work, our CR would be half of what it is now and our business would be in real trouble!” You can’t really argue against counterfactual hypotheticals, so people just nod their heads and continue to invest in CRO, even though they don’t REALLY believe the magnitudes claimed by the reports.
This idea brings us back to the start of this essay, and the question why companies cut back on their Optimizely subscriptions after the pandemic struck.
In marketing today, there’s a clear dichotomy of strategies:
Many brand marketers want to roll out their new brand or page design or CMS platform without testing (“It’s so clearly better, we don’t need to test it”).
Many “performance marketers” want to test everything and believe that a +10% result on their A/B test will save the business.
I caution against both philosophies.
Conversion rate optimization has a place in marketing. I am the first to bang the table demanding that we A/B test any significant (or even insignificant) changes to the business. In most cases, though, I think that A/B tests help ensure that you do not destroy significant value with your changes, rather than the belief you can layer a bunch of tests about colors and formats to achieve a 50% revenue increase.
CRO for growth is most impactful when testing best practices. You want to ensure that widely used tactics will apply for your particular website. Additionally, CRO can help you identify ways to remove friction from your purchase funnel (and to make sure your friction removal isn’t causing unintended confusion somewhere else).
A caveat: when I work with small start-ups, they often want to test everything they do. The problem? Their traffic levels are so low, collecting “significant” results would take months. At small scale, you HAVE to make decisions based on judgement. Even at mid-sized companies, you can’t test everything — you need to prioritize the questions you think will yield the greatest impact. In theory, the bigger companies could test (almost) everything, but even these firms need to recognize the costs involved in widespread testing. We hear so much about “data-driven organizations” that do not believe anything UNLESS it has been tested. We should all recognize that such a philosophy forces these companies to not only optimize based on short-term feedback, but to also ignore long-run consequences.
The alternative is a balance. Study other places to learn what GENERALLY works. When starting something new, begin with the familiar best practices. When you reach scale, test your new ideas against your baseline. Test to make sure your new directions are not worse than what you are currently doing. Plus, look into anything that reduces friction and makes things easier for your customers. Not all changes are sexy: finding issues with your credit card entry form could be more impactful than testing new photos on your landing page.
And if you REALLY want to maximize your dining experience, I recommend Tyler Cowen’s “An Economist Gets Lunch.” Here are some of his recommendations:
Eat at ethnic restaurants in strip malls. (Low rent might mean lower prices. In addition, they need to offer high-quality food to attract ethnic diners to a location with limited foot traffic).
Avoid restaurants with attractive people and lively noise (you are paying for the atmosphere instead of the food).
When at a nice restaurant, order the most unusual item on the menu. (Steak and chicken dishes are there due to consumer demand. If sweetbread appears on the menu, it’s probably because the chef is proud of the dish and insists on its inclusion).
Wherever you decide to eat — don’t worry about how many digits the menu designer typed after the decimal.
Keep it simple,
Edward