Millions of Shoppers + Scanning Millions of Product Barcodes + Millions of Retail Store Locations + More than 40,000 Retailers Delivering Inventory and Price Information = BIG DATA.

ShopSavvy is now three years old and during that time has become the largest mobile shopping community in the world. Delivering real-time pricing, inventory, reviews and product information to millions of users, for millions of products from thousands of retailers is hard work – perhaps harder than it sounds. Late last year we secured a $7M investment lead by Facebook Co-founder Eduardo Saverin with the goal of making our platform more social. Layering a set of social graph data over our existing product data entailed the creation of an entirely new platform that could leverage the latest ‘Big Data’ tools like Cassandra, Hadoop and Mahout – ENTER ‘ProductCloud’.

ProductCloud is ShopSavvy’s Big Data solution for real-time social product data. Before the release of ProductCloud we used traditional LAMP tools for data – tools like MySQL. By Black Friday we had more than 1 billion rows in MySQL – the realistic limit of the technology. Our team has feverishly been building ProductCloud and I am pleased to announce that it is going live as I write this blog post.

Using Cassandra with Hadoop for data storage and processing, ProductCloud is able to iterate over a MASSIVE dataset on demand while concurrently serving data requests and meeting our various internal and external SLAs for companies like CNET, About.com, PriceGrabber and Walmart. Cassandra’s built-in replication suite allows ProductCloud to maintain load during peek times (such as Black Friday or Cyber Monday) as well as during hardware failures across multiple machines, racks or even data centers. ProductCloud maintains HUGE histories of prices, products, scans and locations that number in the hundreds of billions of items.

These Big Data tools make it possible to leverage the massive amount of data we store in order to constantly create new analytics to provide actionable insights into the retail shopping business. ProductCloud’s open architecture allows our team to layer tools like Mahout on top of our platform to enable new features like price prediction, user recommendations, product categorization and product resolution. What sort of data are we talking about? Here are a few stats that boggle the mind (well at least ours):

-  More than 240,000,000 product pictures and user action shots
-  More than 3,040,000,000 product attributes (color, size, features and so on)
-  More than 14,720,000,000 prices from retailers
-  More than 100 price requests from ShopSavvy users per second

With the release of ProductCloud our team has laid the groundwork for the ‘socialization’ of ShopSavvy. Adding a social layer to such a rich set of data – data that must be accessed on a real-time basis wasn’t possible before the creation of the Big Data tools found in ProductCloud. Look for social features later this quarter.

We had some great press today regarding one of our newest features, SavvyListings. The enthusiasm has been overwhelming and we are excited about this feature as well. With the announcement, we wanted to take a quick opportunity to add more information about this feature related to a user’s ability to remove a listing from the ShopSavvy marketplace:

You absolutely can remove the listing on the Android version of ShopSavvy.  This capability is live in ShopSavvy today.

On iPhone, this capability is currently pending review and approval in the iTunes marketplace.

As an early taste, below are screenshots of what to expect in this release:

We just finished our fourth Christmas here at ShopSavvy (still can’t believe ShopSavvy has been around that long) and I thought it was worth writing a post about the differences we see between our Android users and our iOS users. This afternoon I posted a few stats we have gathered from our Wallet feature. One stat that really seemed to bother folks were the user adoption rates for our Wallet feature between iOS and Android users – we are seeing a 10:1 ratio on creation and usage between the two platforms.

First, we need to put our cards on the table. ShopSavvy was one of the first Android apps available and Google awarded us $300K in the Android Developer Challenge back in 2008. Needless to say, we are pretty fond of Google and their team. More recently our friends at Apple have been VERY cool to us as well – featuring ShopSavvy in the iTunes store.

Here are some general observations about our Android Users:

• more willing to beta test new/complete features (good trait)
• more forgiving of bugs/crashes(good trait)
• more willing to give you a second chance, and update their market ratings(good trait)
• less willing to respond to questionnaires and polls (frustrating trait)
• far more likely to report you as a spammer if you send them emails (frustrating trait)
• far more concerned about giving access to various permissions (frustrating trait)
• more likely to request support via email or phone (good trait)

When we released our wallet feature on both Android and iOS we updated our permission request to include the user’s address book. The permission was required so that we could auto-fill the user’s name, email, address and phone number from the user’s own address book. iOS users appreciated the feature and mentioned it in their ratings as a time saver. On the other hand, Android users immediately began giving us 1-star ratings in the Android Market and accused us of ‘stealing’ their contacts. We tried to explain the reasoning behind the address permission, but in the end we just gave up and turned off the permission. Android users would have to manually fill in their name, email, address and phone number themselves. As a result of this ‘fear’ we believe our uptake on Android wallets has suffered (the numbers tend to agree).

Why do iOS users seem more trusting than Android users? Perhaps they aren’t really more trusting in general, but perhaps the ecosystem that Apple has created offers them a sense of security. Apple is very strict about which apps get in the market and updates are scrutinized. Google’s open system – anything goes – forces users to fend for themselves. Which is better? I am not sure, but I really do believe that Apple has made it easier for developers like ShopSavvy to gain user trust more rapidly. On the other hand, Android users are far more accepting of bugs and are willing to help you make your app better – this is very cool and something we do not see among iOS users. Which users are best? The ones that use ShopSavvy, of course.

With more than 20,000,000 downloads – ShopSavvy provides a statistically relevant window into the shopping behavior of Americans (and to a lesser extent Europeans). Each time a user scans a product we store ALL of the prices seen by the user and with more than 40,000 retailers we store a LOT of prices. Based on this data we have learned Amazon has the best price about 6% of the time. This means that 94% at least one retailer has a better price than Amazon. This holiday season we began to see a BIG change in pricing from Amazon.

First, remember that $5 discount Amazon offered users of its mobile phone app? Amazon offered 5% or up to $5 off as many as three items purchased through its price check app on December 10th. We looked to see how Amazon’s prices compared with other retailers on the 10th and found that Amazon was the best price only 3.3% of the time. Translation? Either Amazon raised their prices or other retailers lowered theirs on that day. This is almost a 100% decline in their price ranking.

Once we saw this we began looking at other days. On certain days like Black Friday and Cyber Monday we noticed that Amazon actually did better than normal – between 7-9% of the time they had the best price – a 50% increase in their price ranking. On other days, similar to the 10th, we saw a decline in their ranking.

The best data came from my wife. On two occasions she was buying toys for our daughter Erin from Amazon (she buys everything from Amazon) and she found that Amazon’s prices were 100% higher than local retailers like Target. The one I recall the most was some sort of Barbie Jet which was listed on Amazon for $80 – using ShopSavvy she found the same toy for $40 from Target – a huge savings. I think we won Michele over this year. (note the picture is for the Jet today, the prices have gone up a lot – Amazon is still highest)

So what did we learn? Amazon manipulates their pricing all of the time. It is smart to compare Amazon to BOTH local retailers AND other online retailers – all of the time. You never know who is going to have the best deal – but you can be sure than between 90-96% of the time someone besides Amazon will have the best deal. How do you cope? Use ShopSavvy of course – a neutral price comparison app is ALWAYS the best solution.

We were pulling data out of ShopSavvy during the holidays and several folks ran this infographic we put together. Did you see it? Did we get it right?

Periodically we poll our users about their experiences with our app. This week we asked a statistically relevant number of users how they were using ShopSavvy this week and one answer surprised us. Evidently some users (about 14%) are scanning the items they received for Christmas to a) determine how much their family and friends spent and b) determine which store has the highest price so that they can return and get the most bang for their buck. Of course, we do not condone this sort of behavior, but thought it was worth sharing. Our scan volume on Christmas Day was huge – almost 3X bigger than any normal shopping day – despite most stores being closed. We were wondering what was going on – now we know. Remember it isn’t the cost of the gift, but the thought that counts…

In the early 90′s I was a member of the United States Marine Corps. Since then I have been a big supporter of the USMC Toys for Tots program. Earlier this year I decided to use ALL of our remnant ad inventory to promote the program. Since the start of the holiday season millions of shoppers have seen our Toys for Tots promotion – around 140,000 shoppers each day. The program, combined with our integrated wallet, makes it VERY easy for shoppers to contribute. We recommended several toys available for immediate purchase with our wallet – once purchased by a ShopSavvy user we automatically ship the products to a Toys for Tots distribution facility. There isn’t much time left, toys ordered in the next two days will make it in time for Christmas. Here is what the screens look like:

Each year we have seen big increases in traffic on Black Friday, but as we have grown it has been harder and harder to keep up the pace of growth. You can only grow 100% for so long. We thought we hit that wall in 2010 when our Black Friday scan growth was only 317% compared to 415% in 2009. Somehow we broke through not only our 2010 number, but also our 2009 record – scan volume on Black Friday this year was 535% higher than our baseline (the baseline is our average scan volume during the day). Here is a nice graph:

We wondered what product categories would increase the most on Black Friday so we compared data from the week before Black Friday, Thanksgiving and Black Friday itself:

Consumer electronics and toys did very well, but we were surprised by this year’s HUGE increase in video game scans – 2000% increase in scans compared to normal traffic times. Wow.

We also looked at how users compared between Android and iPhone. Normally Android users scan a lot more (almost double) than iPhone users. But on Thanksgiving iPhone users reached parity with Android users and by Black Friday the iPhone users put Android users to shame – doing a LOT more scanning. Here is a relative graph:

More to come…

This holiday season EVERY scan you make gives you an additional chance to win an iPad 2 or Kindle Fire (your choice). Each week until December 25th we will randomly select one winner – the more scans you make the more chances you have to win. The user with the most scans between now and December 25th will receive BOTH an iPad 2 and a Kindle Fire. To win you must setup a user account on ShopSavvy.

Contest Start: Today
Contest End: December 25th
Award(s): Apple iPad 2, Kindle Fire or Both

Rules: Open to any registered user (you must set up a user account in the app for your scans to count). One entry per unique scan for each weekly drawing. Each weekly winner will be notified by email and select either an iPad 2 or a Kindle Fire. The Grand Prize will be awarded to the user with the most unique scans by midnight CST on December 25th. Winners will be contacted by email and must respond within 72 hours and provide their birth date, mailing address and a headshot to claim their prize.