How to Test Your Product Idea Using Concept Validation Surveys

Getting early market feedback is crucial for consumer product development. While this is widely known, the different available methods can be confusing for inventors and marketers.

Few things compare with concept surveys for speed, validity (statistical power), and depth of results. The main purpose of this early-stage survey is to benchmark the appeal of a new product idea against similar alternatives that are already in the market. If the product scores higher than the alternatives, it’s a good sign of potential product viability. If not, the results will show where the concept is weak, and may suggest improvement ideas.


An excerpt from a new product concept survey report. The tested product’s viability metrics are compared against those of benchmark products. The study type shown is for testing the viability of new beauty products.

Various study configurations are common, but they all typically employ 200 or more respondents, are conducted through online questionnaires (which significantly improves speed and cost).

Respondent recruiting is an important issue, and targeting criteria will depend on the purpose and stage of the survey. A “mass-market” sample (which is representative of the general adult population) can help identify the product’s target market by reporting which segments have the greatest interest in buying the product.


Example demographic response to a new invention idea, used to help identify the target market.

If the target market is already determined, the survey sample can be recruited from that specific population (for example, “US women age 18 to 24”, or “dog owners age 50+ in Florida”). From that population, different product variations (“concepts”) can tested to determine which has the best sales metrics.


The purchase intent for a new product is displayed by region, indicating where the best region to launch the product may be.

Some early-stage developers choose to conduct their own informal surveys, which can help determine consumer preference or critical product flaws. Professional research can be a worthwhile investment because do-it-yourself surveys may not pass the Investor Test. If you conduct your own surveys, be sure to address these important issues:

  • Questionnaire design: it’s easy to write questions that are confusing, leading (causing bias), or can be interpreted in several ways. The manner and order in which you ask questions can also significantly affect the responses.
  • Benchmarking: if you learn 50% of respondents “like” your product, is that good or bad? Without a database of other results, or running benchmarking surveys, the results are inconclusive.
  • Sampling: finding respondents who represent your target market is always a challenge.


An excerpt from a new product concept selection survey. The results for one concept variation are shown, which would be compared to other variations also tested.

Using the “Investor Test” to Prove New Product Ideas

Inventors face many challenges, and perhaps the most serious is the issue of marketability. After all their invested time and effort, will they be able to raise funding for manufacturing? Will they be able to get retail distribution? Will anyone actually buy it?

This issue is known as concept validation, and there are two key components for product development:

  1. Technical Validation: Proving the invention is technically possible (that it will work properly)
  2. Market Validation: Proving that people will buy the product (in sufficient quantity, at the required price)

A characteristic of inventors is that they tend to like inventing, leading them to focus on technical validation. At some points, perhaps after perfecting their prototype, or receiving their first sample batch, they must address the second and larger challenge: proving people will buy it.

An easy way to know if you’re validating your product idea sufficiently is with the “Investor Test”. Ask yourself:

If I showed the results to an investor, would it give them the confidence to invest in my product?

This is a very relevant question because many inventors eventually find themselves asking that exact question.  A good invention validation method will be:

  • Quantifiable: instead of vague observations, results should be measurable (numeric).
  • Reproducible: you could run the test again, and get similar results.
  • Trustworthy: conducted by competent people, in an unbiased manner.

While asking family and friends what they think of your new invention is a good idea, it clearly fails the “Inventor Test” of concept validation:

  • It’s not quantifiable because the results are typically qualitative responses like “that’s a good idea, can I get it in blue?” Even if you attempt to measure their responses, the results won’t be statistically significant unless you survey many hundreds of friends.
  • It’s not reproducible because you can only observe the initial reaction of your friends and family once.
  • It’s not trustworthy, because your friends and family have inherent biases not present in typical consumers.

It’s naturally a good idea to validate your invention idea in multiple ways. This will give investors more confidence to fund your product, and give you more assurance that you’re on the right track.

Increasing Beauty Product Sales Using Claims


A study was conducted in May 2016 by HIT Laboratories, revealing consumers may be up to 266% more likely to purchase a beauty product if subjective claims are added to its description.

A subjective claim is one that consumers can judge for themselves, such as if they think the product made their skin softer. This is contrasted with an objective claim, which usually requires scientific measurements of results (such as actual measurements of their skin’s softness using a device).

Beauty Claims Study Results

In the study, survey respondents were shown an “anti-wrinkle complex” cream and short description. Some respondents saw the description alone, while other respondents also saw subjective claim study results.

When asked about their likelihood to purchase the product, the group that saw the claims was 60% more interested in purchasing the product, and 266% more likely to pay the $29.95 product price.

The effect of claims on different types of beauty products is being tested by HIT Laboratories. New study results are available at

FTC Regulations and Claims

Beauty product claim regulations are generally set by the Federal Trade Commission, which advises that all claims must have competent and reliable scientific evidence to support them.

The required extent of testing depends on the strength of the claims and the impact they would have on consumers, among other factors. Furthermore, claims that suggest the product can prevent, diagnose, treat, mitigate, or cure any disease would effectively classify the product as a drug instead of a cosmetic according to FDA policy.

How to Substantiate Claims

Several U.S. companies including HIT Laboratories provide claim substantiation services. For subjective claims like the ones tested in the survey, the tested product is typically provided to between 20 and 50 subjects who use it for a specified period. Their results are measured or recorded, analyzed, and compiled into a final report. The claims that can be made in advertising are based on the final results.

This article is intended for informational purposes only and not for the purpose of providing legal advice.

How not to launch a product: A $4 Billion Marketing Lesson in 2 Minutes

When is a product launch a “sure thing”? How about when you’re the market leader, employ an army of marketers and designers, enchant the media, and invest four billion dollars?

Ford had every advantage, and the result was disaster. What went so wrong?

It was the mid-1950s, and Ford needed a new medium-priced model to round-out their line. They invested heavily in marketing research to determine what consumers wanted in a new car. They meticulously analyzed the competitive landscape. They designed genuine innovations that would later be copied by other manufacturers. In short, they did everything one would think sensible and prudent.

Yet the launch was a complete flop. Manufacturing was halted within three years — long before reaching break-even sales — and the “Edsel” brand became synonymous with monumental failure.

If they did such excellent planning, where did it go wrong? How can we avoid making similar mistakes with our own product launches?

1. Ford paid lip-service to their customers’ preferences, but disregarded them on key points. For example, they produced an incredible list of 6,000 name ideas ranked by consumer reactions and expert opinion, but top executives couldn’t pick a winner. The puzzling “Edsel” brand (a low-ranked name in their research) was chosen by the chairman of the board to break the debate. As John Brooks’s concluded in his 1969 essay on the topic,

“Although the Edsel was supposed to be advertised, and otherwise promoted, strictly on the basis of preferences expressed in polls, some old-fashioned snake-oil selling methods, intuitive rather than scientific, crept in.”

2. It’s hard to believe that in a $3 billion development budget ($250 million in mid-50’s), basic test marketing was never conducted. This can be done from a very early stage in the form of concept testing: showing a finished concept to the target market and getting feedback. Done quantitatively, several concept variations can be compared to determine the best one. In a later stage, consumers can try prototypes. Ford took the opposite approach: keeping their Edsels literally “under wraps” and hidden from public view until the grand unveiling.


3. Watch people shop for as much as a toaster, and it becomes obvious that pricing is crucial. The Edsel’s pricing was confusing, overlapping with other product lines so consumers didn’t know where it was supposed to stand. Furthermore, the positioning was unclear: highlighting innovative features that could only be considered “premium”, while lacking features common to other premium cars.

4. For over a year the Edsel was slowly revealed to the public in what some called and “automotive striptease”. It was hyped through stunts and ads like no other car had been, so expectations were naturally high. But after release, the only sensible media story remaining was “does it live up to the hype?” With such an intense spotlight, minor problems typical of any new model became national headlines, fueling a devastating negative news cycle.

Key takeaways:

  1. Apply your marketing research at each stage of development and marketing.
  2. Test market early and often to avoid launching an “Edsel”.
  3. The consumers’ purchasing decisions consider the competition, so you should too: position clearly.
  4. Don’t build unrealistic expectations or over-hype your product: it invites piercing scrutiny.


Survey Shows Tim Cook Probably Right About 2-in-1 Apple Hybrids

On November 15, Cook told the Irish Independent, “We feel strongly that customers are not really looking for a converged Mac and iPad….Because what that would wind up doing, or what we’re worried would happen, is that neither experience would be as good as the customer wants. So we want to make the best tablet in the world and the best Mac in the world. And putting those two together would not achieve either. You’d begin to compromise in different ways.”

His comment made ripples in tech news, and answered a lingering question: will Apple attempt to merge their MacBook and iPad lines into some kind of hybrid product? Such a move would follow computing trends: according to IDC research analyst Jitesh Ubrani, “2-in-1s are the only reason we expect the overall tablet market to experience positive growth from 2016 onward.”

In the week following Cook’s statement, research firm Hit Laboratories conducted an internet survey of 512 US consumers to investigate two questions that arise from his statement:

  1. Do Apple customers want to buy a hybrid device?
  2. Would a hybrid be the best experience for those that want one?


31% of Apple customers want to buy a 2-in-1 hybrid

Although Microsoft’s Surface and other Windows hybrids have demonstrated a market for 2-in-1 devices, is it possible that Apple customers have different needs and desires? It doesn’t seem so: among the respondents (of which 47% were Apple customers) there was no statistically significant difference between the groups when asked “what are you most interested in purchasing as your next primary computing device?”


Remainder of responses were “I don’t know” and “None of the above.” “Apple customers” were respondents who owned at least one device running the iOS or OS X operating systems. “Primary computing device” was defined to respondents as “your ‘main device’, with a screen 10 inches or larger, which you would use for anything more than simple tasks.”


Hybrids would not provide the best experience for everyone

The second question is whether a 2-in-1 hybrid would actually deliver a great experience for those who want one. As Cook implied, designing a hybrid device necessitates design compromises which cannot yet be resolved by engineering. For example, a device cannot have both a large screen (possibly a valued attribute in a laptop), and a small portable screen (maybe better for a tablet).

We examined ten different attributes and found that among people who wanted a hybrid as their next device, their preferences were potentially incompatible with what a hybrid would actually deliver.


Larger screen vs. smaller portable screen

On, the average laptop screen size is 14.5 inches, and the average tablet size is 7.3 inches. It stands to reason that a hybrid with an intermediate screen size (around 11 to 12 inches) would be too big when used as a tablet, and too small when used as a laptop. The below chart shows that among Apple customers who want to buy a hybrid, there’s a broad range of preferences which couldn’t be satisfied by any one device.

30% prefer a "smaller" and 59% prefer a "larger" screen size.

30% prefer a “smaller” and 59% prefer a “larger” screen size.

Touchscreen vs. Trackpad

When Apple customers were asked what they’d prefer in a hybrid, only 11 percent wanted a touchscreen without trackpad. Most respondents wanted some degree of trackpad support to complement the touch input, which could be interpreted as wanting a small trackpad. It’s interesting to note that the iPad Pro with Smart Keyboard, the closest thing Apple has to a 2-in-1 hybrid, has no trackpad at all. Fortunately, a trackpad should be easy for Apple to add to any future hybrid design: Microsoft’s Surface tablet has a small trackpad, and the Surface Book hybrid has a large trackpad.

30% tend toward a trackpad, and 55% toward a touchscreen

Performance vs. Battery life

While leading laptops and tablets have similar battery life when being actively used, laptops tend to have stronger processors, and tablets can be used for days in connected standby mode. A 2-in-1 would need both high performance and great battery life to satisfy demand, as seen below.

58% prefer faster a processor; 35% longer battery life.
58% prefer faster a processor; 35% longer battery life.

Tim Cook may be right: maybe Apple customers are not really looking for a converged Mac and iPad, but they are clearly interested in buying a hybrid device of some kind. The next question is if the iPad Pro and Smart Keyboard combination, with its relatively small screen and no trackpad, will fill that need.

Preventing a Crowdfunding Bubble

Last week, a group of customers were denied 1.14 million dollars of pre-ordered product. There are no clear consequences for the company responsible.

3,203 people gave an average of $356, each expecting to receive a flying radio controlled miniature marvel called the “Robot Dragonfly”. This week they were told there would be no dragonfly. To soften the blow, the company will give customers access to the blueprints. Any geniuses among them are welcome to build it themselves.

Should American consumers be shocked this can happen? Isn’t this a clear violation of basic consumer rights?

The problem is those 3,203 “customers” were shopping in murky waters. They ordered off the popular crowdfunding site Indiegogo, fully knowing the product wasn’t manufactured yet, and that their contributions would be used to develop and manufacture it.

This million-dollar loss could be the tip of the iceberg. Crowdfunding has been growing exponentially for the past few years, is already an estimated $34 billion market, and is expected to overtake all of US venture capital funding in 2016.

I love how crowdfunding democratizes product development, and I hope to see it expanded across other industries. However, there are two challenges that need to be addressed first.

The first issue is evaluating product feasibility. Campaigns are not vetted for legal, economic, or technical possibility. Stray patents, escalating manufacturing costs, and pesky physical laws can all lay to waste the best plans – after the money is taken.

Feasibility is an issue because crowdfunding inverts the normal product development process. Development traditionally happens first, so at least there is product available when marketing begins. With crowdfunding, the marketing effort happens first. In the case of non-trivial physical goods, a successful campaign requires only a good idea, a cool video, and a lot of Facebook friends. However, successful delivery requires knowledge of engineering, law, financial planning, project management, safety regulations, sourcing, manufacturing, and fulfilment.

The second issue is consumer perception of risk. As Seth Godin theorizes, the first backers are likely supportive friends, followed by wishful speculators, and once the funding goal is nearly reached and delivery is “assured”, backers transition to having an ecommerce shopping mentality. However just because many others are backing a project is no guarantee of successful delivery.

The legalization of equity crowdfunding may go a long way to addressing the second point: the upcoming wave of recreational “crowd investors” should at least recognize there are risks. It’s the degree of risk they may not be qualified to evaluate. With no vetting mechanism other the “wisdom of crowds”, it’s hard not to imagine a large crowd bubble forming.