Close this search box.

Collecting Big Data

Collecting Big Data

Collecting Big Data

The term ‘Big Data’ is one that has crept into common usage increasingly over the past twelve months. Nick Fletcher, Account Director at Tradedoubler look at what it means for retailers operating in the online space.

The prevalence of the term Big Data is a reflection of not only how companies are starting to make data a more integral part of their business, but also the amount of data available to analyse. By 2020, it is estimated that there will be more than 5,200 gigabytes of data for every person in the world, according to IDC.

However, as it stands, companies are generally rather poor at using this data. IDC estimates that although 23% of Big Data would be useful if tagged and analysed, only 0.5% is currently being analysed. Clearly there is some way to go before we are effectively using Big Data on a daily basis.


It’s easy to take a “more is better” approach to Big Data, but the data itself is virtually worthless when it is not used correctly; and it is easy for companies to invest significant time and resources into data warehouses, analytics tools and the personnel to run these for little actual return. To avoid Big Data becoming a Big Mistake for your business, it is important to scale the data you collect as a company in-line with your ability to actually use this data – and neither one should outgrow the other.

When approaching the issue of what data to collect, think about what aspect of your business is most crucial to your success. Is it the user experience? Is it the product? Is it the price? Is it your supply chain and logistics?

Of course, all these areas are important to any business, but if your customers are likely to take their business to a competitor if they can find the same product even slightly cheaper elsewhere, then one of the key benefits to your business from using Big Data will be the ability it gives you to react to your competitors’ changing their prices.

Equally, if your business’ key selling point is your range of products and the fact that you can always be relied upon to have the required item in stock, then consider how you can use data from both internal and external sources to anticipate demand and integrate this into your purchasing process.

Once your business is adept at analysing and using Big Data, the types and sources of data you can use are almost endless. Details directly from your customers can be combined with social media data, geolocation data and even weather data from the Met Office (yes – we’ve done some analysis on the effect of weather on sales for certain products, and the impact is significant and clear).


For retailers, the key driver behind using Big Data should be the significant positive impact it can have on the user experience when shopping online. Just as a good shop assistant in a bricks and mortar store can quickly ascertain what their customer requires and then help them find it quickly, Big Data gives internet retailers the power to make the online shopping experience as painless and easy as possible.

Used correctly, Big Data can be used to completely personalise the experience for the customer from the moment they arrive on a site, to the moment they complete the order – and beyond.

Amazon is a perfect example of this in action. Upon visiting the homepage, the user is greeted with a mixture of recommendations based on previous shopping and browsing habits, and data from other customers with similar interests or purchase history. Naturally, the effectiveness of these placements is carefully measured, with this data being used in turn to tune the recommendation engine and ultimately ensure that every pixel on Amazon’s homepage is optimised for getting the customer to buy there and then.

The use of Big Data for customisation doesn’t necessarily need to be limited to returning customers.

IBM believes it can give a fairly accurate picture of an individual’s personality from as few as 200 tweets.

Businesses that are able to successfully gather and act on social media information from their prospective customers are at a significant advantage.

The internet gives consumers the ability to check product prices against multiple competitors with very little time or effort. Indeed, price comparison sites or engines such as PriceRunner and Google Shopping , allow consumers to instantly identify which retailers sell a specific product and then filter the results based on the total cost of the product.

As a result, many of the larger retailers are extremely flexible when it comes to pricing their products and, in some instances; adjust prices dynamically based on a number of different factors, including the prices that competitors are offering. The automation of this process (scraping other retailers’ sites to compare prices), has occasionally misfired when the latest games consoles have mistakenly been listed at £10 but for retailers that compete on price, this use of data is a must.

Big Data can also be used to analyse the effect of price changes against a number of different variables such as different times of day, different geographical locations and different customer segments (perhaps new vs. existing customers). This can then be used to create a flexible pricing strategy that protects margins as well as maximising sales volumes.

It may sound like something out of a sciencefiction movie, but Big Data can be used to predict future purchasing trends by combining data from historical consumer behaviour, current purchasing habits and, significantly, external sources such as social media.

If retailers are able to use this information more effectively than their competitors, this can create a huge competitive edge, particularly with regards to products in low supply. Being able to order stock in anticipation of future increases in demand could easily result in a scenario where you are able to sell a product to your customers, while a competitor is forced to list the item as out of stock.


Data on geographical location is playing an increasingly important role in the overall use of Big Data by retailers. Although retailers have been able to use location data (by taking and storing their customers’ addresses) to influence their strategic decisions for some time now, the rise of mobile commerce has meant that extremely accurate location data is suddenly available on a real-time basis, making it infinitely more powerful.

Retailers with both online and offline presence are now able to use smartphone push notifications to incentivise potential customers to visit their stores, and even those retailers without an offline presence are now beginning to “hijack the high street” – delivering mobile marketing messages to smartphone users in a certain location that fit a certain demographic. Given that research suggests that once a customer uses their smartphone in a high-street store to research a product they were considering buying, there is only a 48% chance that they will actually go on to purchase there and then in-store, the opportunity is significant.

The prospect of working with Big Data may seem somewhat daunting, but its potential to transform a business is huge. For anyone venturing into this area for the first time, I would recommend the following: start off small and remember the “Three Vs” of Big Data:

Volume (store and analyse as much data as you can, but no more), Velocity (how fast you must be able to respond to the data to ensure its value) and Variety (be creative – think about exploring and using data from a multitude of internal and external sources). Stick to these principles and you and your business are guaranteed to reap the benefits.

Read More

Register for Newsletter

Group 4 Copy 3Created with Sketch.

Receive 3 newsletters per week

Group 3Created with Sketch.

Gain access to all Top500 research

Group 4Created with Sketch.

Personalise your experience on