One of the recent YCombinator dinners that we attended featured Joe Kraus (who founded Excite and then later JotSpot, which sold to Google).
Like all YC guests, Joe had piles of startup wisdom… One of the things that stuck out to me (which I’d never heard much) was when he said, “when we launched JotSpot in beta, we launched it to too many people.” Huh? Too many users? That’s bad?
But as I look at my current workday, there are times when I wish that we had fewer users.
There’s some common wisdom about usability testing: Beyond 5-7 people, you really aren’t going to get much new/interesting data. There are diminishing returns.
Similarly, in a beta test where you are trying to understand your market, figure out your users, hone your funnel, hunt and slay bugs, and make your product better, there has got to me a point at which you have enough users to get the data that you need. For us, given that we have a web app as well as an installable app for both Mac and PC, our need for a diverse body of testers (in terms of the technologies they use) is probably higher than most. But I have no idea what the magic number is.
But we opened it up. To give you an idea of the consequences of this, here’s roughly the amount of communication that I do in a given day:
- 3-8 emails from our contact form – I respond to every single one of these. Sometimes that results in a follow up email that requires a second response.
- 20-30 feedback emails. We have a tiny form in our app asking for any thoughts a user might have. Even though we say below the form that we can’t respond to all of ‘em (and we point them towards the contact form / support forum if they want a response) I read them all and tend to respond to some.
- 3-5 posts on GetSatisfaction, which weird forum-sorta-thing. We try to respond to all of these, and many times they require some ongoing discussion.
When you add all of this up, it’s a pretty tremendous amount of communication. Say it requires an average of 3 minutes to digest and/or respond to each entity (this is 7 days a week, mind you)… That’s about 2 hours and 10 minutes PER DAY. Every day. Not counting the times that some emails require that I involve the whole team in a solution/discussion. That’s a lot of time for a company where all of the founders really ought to be spending almost all of their time working on development. And, as an effectively bootstrapped company, we don’t really have the budget for support staff.
On the Brighter side…
Still, despite the “costs” mentioned above, there are some pretty huge advantages to launching early and openly.
First off, you get over the biggest early hurdle that can slow most startups to a crawl. I think all founders are terrified that when they finally launch their business, no one will want what they have. So they’ll find any reason to delay it. Maybe they should focus on patents? Trademark research? Clever and innovative stock plans? Business cards? Fancy spreadsheets? Business plans? Marketing plans? Anything at all that will allow them to delay the possibility that people don’t like your app. When you’re out there and getting buried in feedback, all of that other stuff falls away… It’s incredible how much it focuses you on your app.
Second, you get to test your inherent marketability. Do people like talking about you? Do they tell their friends? Do they blog about you? Does your app have any virality? A closed beta really doesn’t allow this.
Third (and most important), in the sea of people who reach out to you is (hopefully) people who LOVE you. We’ve gotten feedback emails that simply say, “I love you” (3 so far!). We get long essays from users talking about their time management strategies and how RescueTime has helped. Literally 1-3 emails a day make me walk on air. Reduce that number to 1 a week and I’m not sure I could manage to make the sacrifices I’m making now to push the business ahead.
Fourth, there’s SEO. No need to get into the nitty-gritty, but starting the campaign of building incoming links and pagerank is something you should start as early as possible.
And, of course, there are nebulous concepts like “tipping points” and marketing momentum… If you hear about RescueTime enough, maybe eventually you’ll try it?
On the balance, I don’t know the right answer… And I suppose it’s different for each startup. I’d love to get other folks’ thoughts in the comments.
