Why broadcast questions aren't validation

The most common research mistake I see solo founders make is treating a broadcast question as customer research. You post on a forum or in a Slack community or on LinkedIn, you ask what people wish existed in the space, and you treat the replies as signal. They aren't signal. They're a different thing wearing signal's clothes, and the difference matters more than almost any other research decision you'll make this year.

When you ask a crowd what's missing from a tool, you're asking a room full of people to perform an opinion. A lot of them haven't thought about it before you asked. The ones who reply are the ones who can generate a plausible-sounding answer quickly, which selects for people who are good at sounding thoughtful, not for people who actually have the problem you're trying to solve. The replies look like data because they're written down and there are a lot of them, and looking like data is the entire trap.

What you get from broadcast questions is feature requests. People will tell you they'd like a darker dark mode, they'd like keyboard shortcuts, they'd like an integration with whatever tool they happen to be using right now. None of that tells you anything about whether a problem exists. It tells you what comes to mind when someone is asked to fill a text box with an answer about a product they don't use yet. Founders take this and build a roadmap, and the roadmap is built on nothing.

The version that actually works is smaller and much slower-feeling and infinitely more useful. You pick five people who you believe have the problem. You ask each of them, in a separate conversation, what happened the last time the thing actually went wrong. Not what they wish existed. Not what features they'd like. What happened. Walk me through the day. What did you try. What did you do instead. What did you say to the person next to you. Who else got pulled in. How long did it take. What did you decide afterwards.

That question gets you behavior. Behavior is the thing that survives contact with a product. When someone tells you they spent two hours on Tuesday afternoon fighting with a spreadsheet and ended up texting their cofounder a screenshot at midnight because they couldn't get the numbers to reconcile, you have a moment. You have a place in their week. You have a stakeholder and an emotional shape and a cost. You can build for that moment because the moment is concrete. You can't build for the abstract wish that a different tool would be nicer.

The other reason five conversations beat a hundred replies is that five conversations let you notice the second story. The first time someone tells you about the bad Tuesday afternoon, it's an anecdote. The third time someone unprompted describes the same shape of bad afternoon, it's a pattern. You can't find patterns in a forum thread because everyone is answering a different question in a different mood with a different level of investment. You can find patterns in five careful conversations because the conditions are controlled enough that the repetition means something.

Founders resist this because broadcast feels productive. You posted, you got fifty replies, you have something to show. The five-conversation version feels slow and underwhelming and you have nothing to screenshot for your investor update. That feeling is the cost of the approach that works. The approach that doesn't work is fast and feels great and scales beautifully into a roadmap that nobody asked for.

If you're trying to decide whether to keep building what you're building or change it, throw out the survey. Throw out the forum post. Pick five names, write five messages, ask each of them one careful question about a specific moment. The signal will be slower to gather and impossible to misread. That's the trade. It's the right trade every time.

This connects to a question I answer in depth: Why do people say they like my idea but don't buy and How do I validate a product idea before building it.