How Accurate is Think Secret?

Everybody loves a good rumor, at least, thats what the editors over at Think Secret hope. As the web’s premiere Apple-related rumor monger they have become the place to go for all of the best Mac gossip. They may not be always correct, but they are always close, right? Well, as it turns out, not really.

First, let me say that I am a long time reader of TS and that I always enjoy their gossip. As a user of Mac products and a writer for a Mac related blog, I always like getting the best information I can, as soon as I can. However, recently I have begun to have doubts as to just how good they are at predicting future trends and products.

So, in an effort clear the air and find out just how good a job they do, I had a look at their archives and compared them to Apple’s press releases dating back to the beginning of 2005. Here is what I found.

Initial problems, how do you define “accurate”?
Before I can make a judgement on how accurate Think Secret’s reporting is I have to define just what I mean by “accurate”. Unfortunately this is a very tough problem as demonstrated by this example:

Lets say that I predict that Apple will release a new iPod in the next year. And sure enough 8 months from now they update one of their iPods to give it more storage. Was my prediction accurate?

Well, it was true. After all, I said they would release a new iPod and they released a new iPod, albeit 8 months later. So does “true” equal “accurate”? Not always. But wait, I can already guess what you are going to say. You are going to complain that I said “new iPod” when in fact I should have said “upgraded iPod”. Ok, consider the prediction changed. Now am I accurate?

Well, yes, technically. But it wasn’t a very useful prediction was it? Since the term “upgraded iPod” could mean anything. If it got a new paint job, more storage, a brighter screen or better headphones I could claim it was “upgraded”. I made an obvious guess, based on readily available data about a strong product line that ultimately turned out to be true. Was my “prediction” true? Technically, yes. Am I a market visionary with inside information? Not really.

So, right off the bat we need to change the term “accurate” to “accurate and useful”. This means that if Think Secret predicts that Apple will marginally increase the size of an iPod, bump the speed of a processor or do a point upgrade to a piece of existing software, then I can’t give them credit for having inside knowledge. All of those predictions are mundane enough that they could be made by anyone looking at Apple’s historic road map. So, no credit for the easy ones!

However, this still leaves us with the problem of the “machine gun” approach to forcasting the future. A technique used all to often by sites like Think Secret. In this method you spew the same predictions as many times as you can and hope that you hit something eventually. How do you define “accurate” in such a situation? Here is an example of this problem:

I come to work one Monday and tell my co-worker that I predict it will rain that day. The day passes with no rain. So I come in Tuesday and once again predict that it will rain that day. And once again I am wrong. I repeat this every day of the week. On Friday, it rains. I then look at my co-worker and proudly proclaim “See! I told you it would rain today.”

Once again, my prediction was technically correct, but practically useless since I had made the same prediction over and over throughout the week. So, in the same vein, I can’t give Think Secret credit for predicting something if they make the same “prediction” with “insider knowledge” every month for half a year until the product is eventually released.

So where does this leave us? Well, we have eliminated from consideration the obvious and the redundant, so now, all I am counting are the important predictions. I have broken the remaining portion of this article into two different parts.

The first section lists predictions from Think Secret’s point of view. Here is how to read this information. On each date listed below, Apple released some product or service. I reviewed Think Secret’s predictions for the week leading up to that date. Then, I compared what they predicted would be released with what actually came out. If they predicted things close enough to be in the ball park then I marked that guess as “right”. If they predicted something that didn’t show up then that guess was marked “wrong”. And anything Apple released that they didn’t mention at all was marked as “missed”.

Think Secret’s Accuracy for Major Product Launches from January 2005 - September 2006

January 11, 2005
Right: iPod shuffle, Mac Mini
Wrong: FireWire Audio Breakout Box

June 6, 2005
Missed: Intel switch. Repeating what the WSJ said doesn’t count!

August 2, 2005
Missed: Mighty Mouse

September 7, 2005
Right: New iPods, ROKR phone
Wrong: ALL DETAILS - Nano (not the Mini), Colors wrong, size wrong, size of ROKR phone, Shuffle not upgraded

October 12, 2005
Right: iPod
Wrong: Wrong size for iPod
Missed: Video iPod, iMac/FrontRow/iSight, buying videos from the iTMS

January 10, 2006
Right: Nothing of substance
Wrong: MacBook Pro came out, NOT the MacBook. iMac when Intel, Shuffle did not get an upgrade

February 28, 2006
Right: HiFi speakers
Missed: Intel based Mac Mini

April 5, 2006
Missed: Boot Camp

April 24, 2006
Right: 17” MacBook Pro

May 16, 2006
Right: MacBook

May 23, 2006
Missed: Apple & Nike

August 7, 2006
Right: Mac Pro
Wrong: Mac Pro in same case (not different case as they reported), no new Nano
Missed: All of Leopards features

September 12, 2006
Right: Upgraded iPod and Nano, new colors, movies
Wrong: iMac mounting kit?
Missed: iTV, games

Results:
11 correct predictions
8 incorrect predictions
11 missed announcements

What do these numbers mean? Well, it means that for all major announcements from Apple, Think Secret only gets about half of them right (11/22). For every Mac Book Pro they correctly predict, they miss a Video iPod. For every HiFi speaker they get right they miss a switch-to-Intel story. Taken another way a little less than half (8/19) of their predictions are completely wrong.

Putting it all together it looks like this: If you take all of the major predictions from Think Secret, then about half of them will be crap. Of the half that comes true (whatever number that may be), you can assume that there will be an equal number of surprises from Apple.

The second section of this piece looks at things from Apple’s point of view. Here, only the products Apple actually released are considered. (as opposed to the previous section which included TS’s incorrect guesses) This chart shows how close TS got to the actual product specs. A score of 0 means they failed to predict the product at all. The lower the score the less detail was correct, the higher the score the closer they got to the truth. For instance, TS completely nailed the iPod Shuffle (earning a 5) but only got partial details correct (they claimed a complete redesign when in fact the case stayed the same. Though they did get the processor correct) on the Mac Pro (earning a 3).

Complete list, Things Apple Released:
[0 = missed, 1(wrong) . . . 5(right)]
iPod shuffle - 5
Mac Mini - 5
Intel switch - 0
Mighty Mouse - 0
iPod Nano - 2
ROKR - 3
Video iPod - 2
iMac/FrontRow/iSight - 0
TV shows - 0
MacBook - 5
Intel iMac - 1
HiFi Speakers - 0
Intel Mac Mini - 0
Boot Camp - 0
17” MacBook Pro - 5
MacBook - 5
Apple & Nike - 0
MacPro - 3
New iPod & Nano - 5
Movies - 4
Games - 0

Looking at these results we see that, for the most part, Think Secret gets things either completely right or completely wrong. So, when they have a real source they apparently get everything from this source. Product name, specs, description, pricing, availability, I mean everything! However they do miss many products entirely. And I believe that this can be mainly credited to Apple’s lockdown on internal leaks. Because in reading over Think Secret’s archived postings it is clear that they tend to get leaks only in certain departments within Apple. Mainly their computer hardware section and their iPod section. But with software, media and periphials they don’t seem to do so well. So, when Think Secret has genuine insider information they do pretty good. However, when they have woefully incomplete information, all too often they put it out as the real thing and just fill in the missing pieces with their best guesses. The problem with this less-than-winning strategy is that they don’t guess very well. They misread the Nano announcement pretty bad, the ROKR was only partially right and they weren’t even sure the Video iPod was going to exist until the day before the launch (all that week they predicted it wouldn’t happen for another 6 months).

So, to recap, when they actually have insider information they tend to be close to 100% correct. However, when they only have a few clues then their guesses aren’t much better than any other pundit’s predictions. The annoying part about all of this is that they don’t make much of a distinction between the “locks” and the “guesses”. Of course, its all very much obvious in hind sight, but how does that help us now?

Some of you might read this article and conclude that I have some agenda against Think Secret. That I massaged the data to paint them in a negative light. Or maybe you think I didn’t interpret my own data correctly. Well, you are of course welcome to think anything you like, however the truth is much less sensational. Many people base their purchasing plans on the information that sites like Think Secret provide. And this is a good thing, in moderation. After all, who doesn’t mind waiting a bit for a better piece of technology? Or a better deal or current products? I am not out to get Think Secret, AppleInsider or any of the rest of those sites. I am simply trying to inform my readers that these sites aren’t nearly as accurate as they would have you think. I am trying to remind everyone that you need to judge for yourselves what to believe. Remember, if it seems to good to be true then its probably been confirmed by a super-secret “inside source”.