While huge companies like Alibaba and Amazon are in a constant competition to dominate consumer spending, the real battle remains hidden from view. What users are exposed to is rivalry around pricing and marketing, but some of the most intense competition is actually over data, and this takes place behind the scenes. When it’s boiled down, who truly wins and loses depends on the amount of data collected on the prices and offerings of competitors, as this data can be leveraged in each company’s own campaigns.

The Types of Data Required****

Pricing is at the helm of any data intelligence operation. In order to secure a bigger market share, platforms must continuously monitor the prices of their competitors across millions of products and adjust their strategies according to the findings in real time. In the currently highly competitive and saturated market, even the slimmest price difference could lead to lost sales. As a result, having the best sales data is invaluable.

The data that’s collected is not limited to base pricing; it also factors in shipping costs, bundle deals and flash sales. Therefore, given the scale of monitoring required to get an accurate view of the market, it is hardly surprising that web scraping is unavoidable in this industry. Platforms that are able to successfully collect such data can then implement dynamic pricing algorithms that adjust based on present demand and inventory levels. Roughly25%-30% of ecommerce retailers in Europe now use dynamic pricing algorithms. Additionally, Amazon managed to boost profitsby an average of 25% by utilising dynamic pricing.

It is crucial today that competitive data in ecommerce is geographically specific. Companies have to gather information, all the way down to the details of individual zip codes. Only by using this method can platforms actually optimise their inventory distribution, filling any gaps in competitor offerings or utilising dynamic pricing to their customers’ benefit across the map. This granularity becomes essential during peak shopping seasons when ecommerce businesses often lose customers to competitors due to stockouts.

Retailers also collect and analyse data from product descriptions, specifications, reviews and product images. This data is essential to devise sophisticated search engine optimisation strategies. By utilising real-time intelligence, platforms can capture organic traffic by leveraging carefully calibrated product listings that precisely align with what consumers are actually searching for online.

The Ecommerce Data Collection Battle****

There’s an ongoing fight between competing ecommerce platforms as they seek to arm themselves with as much data as possible while denying the same to competitors. In practice, this means deploying scrapers and attempting to block any operated by competitors.

The fact that competition is getting fiercer is evident in the changing usability of proxy IPs. Proxy servers are needed to distribute requests when collecting public data on a large scale. Without geo-specific proxies, it is not possible to collect regionally specific content such as prices.

Previously, data center proxies were enough to collect such data without being blocked. These proxies are not affiliated with an ISP and they come from a secondary corporation to provide you with completely private IP authentication.

However, since ecommerce platforms have started to block competitor data collection, only a few tech-savvy companies can do without more capable residential proxies today. Residential proxies are harder to identify and as such, they are better for use in competitor monitoring. While superior in this sense, residential proxies are pricier and so are favoured by platforms with deeper pockets.

Privileged Data Access Versus User Experience****

Some ecommerce giants are so fixated on this data battle that they implement anti-bot measures so aggressively that they negatively impact everyday user experience. The most obvious example of this is requiring several layers of CAPTCHA verification, putting some customers off the lengthy buying process. Even more aggressive measures include:

1. Multi-layered detection protocols, combining behavioural analysis, device fingerprinting and machine learning algorithms.
2. Automated mouse movement monitoring for patterns, typing cadence, network characteristics, browser configurations and navigation sequences.
3. Frequent changes to website structure, API endpoints and data formats.

The smallest deviation from parameters considered to be ‘typical human behaviour’ triggers scrutiny or even outright blocking. Since this inevitably frustrates legitimate users, ecommerce platforms also employ another strategy. By refusing to play the blocking game altogether and prioritising user experience and pricing models that benefit the buyer, they achieve both short- and long-term wins. The former leads to more short-term sales, while the latter results in a growing base of satisfied loyal customers.

Putting Customers First****

Data collection and competitive efforts to block it are a paradoxical part of today’s digital commerce landscape. Due to the value of real-time public web data, this behind-the-scenes battle is rarely seen but strongly felt by users, and it might seem to have no end.

However, since all companies use, and will undoubtedly continue to use, public web data for competitive advantage, it might be time to consider another competitive avenue. Instead of creating negative buying experiences for customers with the friction caused by excessive access restrictions, ecommerce vendors might reach better results by investing more in using the data they can collect to offer the best prices, deals and overall online shopping experience, creating a loyal customer base.

Andrius Palionis is Head of Sales at Oxylabs. Since 2015, he has been supporting major companies around the world in their journey towards data-driven decision-making. His motto, “persistence is progress,” has driven him to transform global attitudes towards the importance of public web data in business operations, success and growth. Over the years at Oxylabs, Palionis obtained an in-depth understanding of the main challenges of web data acquisition. Day to day, he uses his problem-solving and team management skills to accelerate the performance of numerous companies by successfully bridging their data needs with the most effective solutions.