Guide to Evaluating Startup Ideas

A great developer I once worked with was kvetching at lunch one day. He’d been working at a well-funded startup for about a year and had come to terms with the fact that the startup was really a pretty dumb idea. He’d wasted a year of his life and had a pile of stock options that weren’t very interesting. His last two jobs had been similar. He asked me a question that, at the time, I didn’t have a good answer for. “How can you possibly know when joining a startup if it’s going to be successful?” In other words, how can you spot a good startup idea?

Since I’ve announced that I’m moving on in the coming weeks/months, I’ve been bombarded with cool offers at existing startups, larger companies, and, of course, I’ve been pondering some of my own startup ideas. So his question which I didn’t really consider very carefully at the time is now one that I’m thinking a LOT about.

So without further ado, here is my “checklist for good startup ideas”. No startup will do great on every aspect of the checklist, but this allows me to put startups/products to a sniff test that I think is pretty darn useful. Note, this list is in rough order of importance.

  1. How deeply do you think the startup will effect people’s lives? Can you imagine them using it every day? Can you imagine them being royally pissed if they couldn’t use it? This can range from utility (gmail) to emotion (twitter), but if a product isn’t in the “I’d rather chew off my own arm than lose it” category for a meaningful percentage of it’s users, it should be a non-starter.
  2. Are the hypotheses that form the basis of the startup tractable? In other words, can test the idea(s) in a short period of time? I’ve talked about the importance of tractability before (hat tip, Ev Williams). Bottom line is that most initial hypotheses are wrong to varying degrees. Twitter was very tractable. Tesla is not. I’ll re-use the money quote from Fred Wilson: “…Of the 26 companies that I consider realized or effectively realized in my personal track record, 17 of them made complete transformations or partial transformations of their businesses between the time we invested and the time we sold. That means there a 2/3 chance you’ll have to significantly reinvent your business between the time you take a venture capital investment and when you exit your business.”
  3. How does the cost-of-acquisition, cost-of-goods-sold (COGS) and revenue-per-customer stack up? Most software startup have a pretty low COGS, so this question generally comes down to, “How much does it cost to buy a customer and how much revenue does that customer represent over their life?” This obviously requires a lot of guesswork early on, but experience is a helluva teacher here. If you haven’t been on the wrong side of this ratio a few times, find a mentor who has. Any way you slice it, you need to fine a “scalable, cost-effective way to get your customer’s attention”. I can’t count the number of startups that aimed squarely at small businesses or “prosumers” with sub-$100 price point and have no idea on how they’re going to buy a customer (other than word of mouth, SEM/SEO, and PR).

    I love extremes here.
    Zynga, Twitter, and Facebook has nailed one extreme– their cost of acquisition is free and nearly infinitely scalable. If you can build a service that grows virally (free and growing customer acquisition), you can focus most of your attention on value creation and revenue-per-user. With a little success there and a little time to let the virus spread, and you can almost not help but succeed. I think it’s hard to overestimate the power of free marketing/customer acquisition.

    There are certainly extremes on the other side. What do you think Oracle’s revenue per customer is? How much can they afford to “buy” a customer for? What about Groupon?

    Pro Tip: If you’re raising angel or Series-A money and you say you’ll be using the proceeds for things like magazine ads and wrappers on busses, you’ve probably already lost.

  4. How MANY lives could you imagine touching in 5 years? This is different than asking about total addressable market (TAM). Craigslist started as a classified ads mailing list for San Francisco. Amazon started selling books. Have some imagination and consider what your company could morph into. Is it interesting enough to justify the opportunity cost and the fact that you’re looking at a drastically reduced salary for 2-5 years?
  5. Is it an invention or re-invention? Hats off to you inventors out there, but I strongly prefer an existing market to creating one from scratch. The companies whose equity I covet didn’t build anything NEW, they just built something BETTER (Google, Facebook, Apple, Amazon, Craigslist, eBay, Zynga etc). In short, the first mover advantage is a crock of shit (most of the time).
  6. Is it worth talking about? Can you tell a story about the product that would make a blogger say, “Holy crap– I could write a story around that that would get tons of links, tweets, and comments?” One of my favorite products is Visual Website Optimizer (it’s a brilliant A/B testing tool). The founder (a great product designer who I’ve had a few conversations with) sent out a barrage of emails to major tech bloggers and heard nothing but crickets (he appealed to Hacker News readers for advice– I think the discussion is interesting). His fundamental problem is that he doesn’t have a story that will drive links/tweets/comments/pageviews– all of the metrics that pro-bloggers care about. Oftentimes, clever PR people can create a story out of something that has nothing to do with the product (see: 37Signals & Zappos), but it certainly helps a lot if your product is funny, controversial, unusually useful, or inherently exhibitionist.
  7. Are you passionate about the end-game? This one is hard to rank. All of the points above assume you are a “mercenary” founder (maximizing for opportunity) rather than a “missionary” founder (passionate about a vision that keeps you awake at night). Great video on that point here. Regardless of whether your end game is a vision realized or a big pile of cash (or some combination thereof), you need to be passionate about it… You need to have something that powers you through the bumps in the road where a rational person would cut and run. Both motivations are dangerous, by the way. If you’re motivated by cash, you might have a hard time sticking through tough times when you realize what you’ve built might only be a single or a double. If you’re motivated by vision, you might not like the pivots your startup needs to take to survive/succeed.
  8. Is the market moving in the right direction? Can you imagine there being a LOT of growth and consolidation in the next 5-10 years? I just saw my first RedBox the other day (it’s a cool box outside of supermarkets that allow you to rent DVDs). They are currently on the wrong side of a market shift away from physical media– can you imagine people renting DVDs in 10 years? I think this one is particularly hard to get right (which is why it’s low on the list).

That’s my list. Am I missing something that’s on yours?

  • http://www.GoMiles.com Michael Komarnitsky

    Interesting post. Commenting on #3, Customer acquisition cost:

    Building a credible Customer Acquisition Cost (+ Customer Lifetime Value) model is more straightforward in a business like mobile phones – with defined inputs ($/ad, people reached/ad, conversion rate %, etc) that can be optimized based on your business objective. The challenge in early-stage startups is that a CAC analysis is EXTREMELY uncertain, highly dependent on assumptions, and extraordinarily non-linear.

    In other words, the CAC and CLTV models are concepts to pitch, rather than to be modeled analytically (again, for early stage seed startups).

  • http://cyberfox.com/blog Morgan

    As a early-stage startup 'joiner' so far (as opposed to founder) one of the best sniff tests I've found is whether it's something you yourself use or need.

    The two companies I was with when they went wildly successfully public were both products that I used already before joining. Using a different product in the same space is probably okay, as is personally feeling a deep need for the product, if there isn't anything else like it.

    There's certainly a place for building products that address pains you don't personally feel, but you won't be as good at gauging the pain relief. That's why it's a high value item for me.

    But your #1 is definitely the most killer point.

  • http://www.rescuetime.com webwright

    NiallSmart points out on Hacker News that folks should read Ev Williams Post on the topic: http://evhead.com/2007/12/how-to-evaluate-new-p

    They should. My post (and a few of my points) were certainly inspired by that post!

  • http://www.rescuetime.com webwright

    NiallSmart points out on Hacker News that folks should read Ev Williams Post on the topic: http://evhead.com/2007/12/how-to-evaluate-new-p

    They should. My post (and a few of my points) were certainly inspired by that post!

  • http://www.rescuetime.com webwright

    Totally agree! If you're an enthusiastic user of your own product, I think you have some incredible advantages– not the least of which is having great instincts about what users want (which is a helluva lot cheaper than market research… not that you shouldn't talk to other users, too). It's also great because it hells you sell it… And sales (this is another post) is really the #1 job of a startup founder. IF you succeed (whether you're CEO or CTO) your job will quickly become selling to customers, reporters, bloggers, employees, potential employees, etc. It's easy to sell something you believe in!

  • http://www.rescuetime.com webwright

    Yeah, it's certainly speculative, at best. Certainly you have to make some assumptions that might not prove out. But, some products just scream a difference between CAC and CLTV.

    If I sat you down and we talked about a groupon clone and a farmville clone, you can smell that difference (and it smells good!). If we talk about a small-business CRM solution, something won't smell quite the same.

    I guess it comes down to: Do you have some credible secret sauce in terms of CAC? And if not, do you have a product that you can imagine having an unusually high CLTV? The vast majority of startup ideas that are pitched to me tend to fall into the “useful tool for small business” category, which oftentimes can be grim on both fronts.

  • http://artvankilmer.wordpress.com/ Arthur Klepchukov

    I love #1. My personal goal is to I work on something I'm deeply attached to that's also insanely useful. I want to wake up to the sound of my phone exploding with customers pleading me to get my service back up not only because demand is toppling our servers but because my service is something those customers absolutely need every day. Don't get me wrong, I'm not asking for scaling issues or to be Techcrunched. I should be so lucky to have scaling problems working on something I love.

    #4 is approaching the same thing from a different perspective. It's still a worthwhile question but I would put it under #1.

  • http://twitter.com/danieldirico Daniel DiRico

    Enjoyed the article. In the end, it seems as though most of the web startups in the last decade have begun with very simple goals; Duplicate a yearbook online. Allow people to post little 140 character messages and have people follow them. Create a site where people can post videos of their cat playing a piano. All ideas that you'd say to yourself, “seriously? you expect to live off this idea?” but then something happens. It succeeds. What is it that makes a simple idea succeed? Perhaps the 8 things in this article. Or maybe something more..

  • Julian

    You completely forgot people. “How good is the team?” should be the first question. A good idea is necessary, but what's really important is the implementation. And that is done by people. Ambitious people, working perfectly together, complementing one another and each having what it takes to succeed are the best factors for a successful startup.

  • NWGuy

    I'd have to agree with Julian; the team is very important, maybe more important than the product. Based on #2 there is a 67% probability that the product is wrong to some extent. You need a passionate team that can learn and adapt.

    The other aspects of this appear to be very consumer focused, with the objective of a hockey stick business plan. Not everybody has to be huge to be successful.

  • http://www.rescuetime.com webwright

    Hi Arthur! Thanks for commenting!

    I like #4 as a different animal. Insanely useful (or addicting– like Twitter or World of WarCraft) is always my #1. But you can imagine a product that was insanely useful, but ONLY for left-handed dentists. Or mothers with twins. A lot of insanely useful products have small markets that are hard to break out of. Some small markets can expand (like Craigslist did out of SF or like Amazon did away from books).

    But the #1 and #4 are definitely kissing cousins. Given that it's a big world you can always expand like Amazon did, I think #4 really is only a big consideration if you want to have a HUGE “change the world” impact. Maybe you only need 1000 true fans?

    See: http://www.kk.org/thetechnium/archives/2008/03/

  • Vikas Shukla

    I originally stared the Finance and Stocks forum at http://diggsamachar.com/adrforum to discuss the stocks and ADRs ( American Depository Receipts) – primarily because I had good interest in the stocks and also to enable interaction and discussion about the stock market.

    It went very well. People posted relevant and good messages. To award some of them I enabled the signature which allowed people to put links to their blogs or website. But this led to the decrease in the quality of the people. How can I ensure that the quality of the messages do not go down when I enable signature ?

  • http://twitter.com/wfjackson3 Willis F Jackson III

    Nice post. I think you are wrong about RedBox though. In the long term, they are on the wrong side of the market. In the short term, they are cannibalizing staffed store sales (think Blockbuster) pretty hard by resegmenting the market.

  • http://twitter.com/wfjackson3 Willis F Jackson III

    Nice post. I think you are wrong about RedBox though. In the long term, they are on the wrong side of the market. In the short term, they are cannibalizing staffed store sales (think Blockbuster) pretty hard by resegmenting the market.

  • http://www.rescuetime.com webwright

    I *almost *wrote a bit about people, but was trying to stick with evaluating
    IDEAS… But your point is dead on– people are damn important. Not only
    for the success of the business, but for just enjoying how you spend your
    day. I tend to be a bit contrarian on people-importance however — I tend
    to think the market is the most important thing by a wide margin. I'd take
    a C level team working in a grade A market over the inverse any day (you can
    always fix a team– fixing a market is harder). Of course, I'm just being a
    parrot here… See: http://pmarchive.com/guide_to_startups_part4

  • http://www.rescuetime.com webwright

    That's a fair point (regarding consumer/hockey stick). This is a startup
    focused… My personal definition of “startup” is a “newish company that's
    aiming for rapid growth and equity upside”. Lots of people have different
    definitions of a startup, which is fine. If your definition is radically
    different, then take this post with a grain o' salt.

    My examples are often consumerish (heck, I'm a consumer!), but I think these
    generalize to b2b businesses as well. Solving a “hair on fire” business
    problem is probably quite a bit more powerful than just about any consumer
    play.

  • http://www.rescuetime.com webwright

    Yeah, don't get me wrong, I think Redbox is killing it. They have a great
    short-term value prop and their growth is impressive! But if I was going to
    dedicate my life to a startup and had a potential co-founder who was jazzed
    about creating a RedBox competitor, I'd be leery. I think RedBox is on the
    same train that Blockbuster is on, which is going over a cliff. Their
    train-car is just farther back in line. Of course, who knows about timing?
    RedBox might have 10-15 years of legs for all I know.

    Markets are hard to guess about though. Everyone with a brain thought
    search engines were a non-opportunity when Google came around. When the
    first iPod hit the market, was anyone thinking that MP3 players was a market
    waiting for someone to come in and clean up?

  • http://twitter.com/startupdojang StartupDojang

    This is an interesting post. So if I were to choose a startup to join, I would put on my venture capitalist hat. Most VCs will tell you they do not invest in ideas, but rather the startup team. Sequoia invested in GOOGLE when there were already hundreds of search engines available, some of which had significant market share. At the time, GOOGLE could have easily been seen as a terrible idea. However, as we know now, GOOGLE revolutionized web search. Sequoia invested in Larry and Sergey as opposed to the idea. That is not to say the idea has no merit, but the team is highly weighted when compared to the idea.

    http://www.startupdojang.com

  • http://www.rescuetime.com webwright

    I've always been a fan of Pmarca's masterful startup post: “The only thing
    that matters”. He contends that the only thing is “the market”. I tend to
    agree. I've seen a LOT of mediocre teams see great success and a lot of
    great teams fall flat. Here's the post:

    *http://pmarchive.com/guide_to_startups_part4*

  • http://www.wavepointconsulting Darryl

    Point number 3 is indeed very interesting. Is anyone aware of any benchmarks for cost-of-acquisition for financial products?

Recent Tweets
  • @_gordee Portland next weekend! Meeting a friend from Alaska who is flying down to buy an RV and drive it north.
  • I just bought a 1 way train ticket, some digital calipers, and a pair of shoes on the internet in the last 5min. LET'S DO THIS.
  • @arogers907 SILAHOOT.
Categories