Comparison shopping is a proven and accepted practice within the retail industry. Pre-internet era versions meant shoppers would physically visit the retailers to get their “bottom line” price. Early online comparison shopping meant you could use search to find the desired product and compare vendors from the comfort of your own home, only physically venturing to the brick and mortar location as needed.
The evolution has continued with retail automation taking some of the comparison efforts away through “price smash or price match” features. The goal of these features is to ensure the vendor makes the sale, regardless of price, and given that the intent of the consumer is to get the best price, the result is a positive one.
At the same time, retailer pricing strategies and tools have evolved using these same techniques, albeit in a formalized, legitimate manner with the intent of ensuring their products are priced accurately. The opportunity for automation and add-on services has spawned a new generation of tools focusing on the topic of pricing intelligence, with vendors that are outwardly focused on helping retailers gain and maintain a competitive edge.
Recently, one of our customers discovered some of the same search and comparison techniques used in an automated, yet malicious manner. This raises the question of when does the age-old practice of comparison shopping become malicious? Here is what we found.
- Search Abuse: Using automation to find a retail item for purchase is common practice. A search for sneakerbot, NikeBot, or Ticketbot will not only allow you to find a bot to automate finding the high demand item you desire, but it may also help you purchase them. Going one step further, the automated search bots found in our retail customer environment exhibited the following characteristics:
- The search queries targeted every single web application URI across all of their locations.
- The search patterns were too perfect and too fast to be human.
- The queries were distributed across a wide range of locations that didn’t match the locations of the search queries themselves.
- Many of the queried items did not exist, placing a significant strain on their infrastructure.
Taken collectively, the findings described provided strong evidence that the intent of the search was malicious.
- Content Scraping: As with search, the practice of copying web content is an accepted one, as evidenced by content aggregators in the hospitality/travel and healthcare industries. The scraping activity observed during our investigation exhibited the following characteristics:
- The automation targeted URIs that did not exist.
- Multiple masking/evasive techniques were used to disguise the attack, including browser spoofing and forgery along with sophisticated user agent rotation.
- As with search abuse, some of the items scraped did not exist.
Viewed under a single pane of glass, the intent of these activities was deemed to be malicious.
Online interactions are well known for lacking in context. Emails are easily misinterpreted, instant messages and social media posts even more so. Even further removed and lacking in context are search and browsing activity. In retail environments, where margins are razor-thin, and the actual intent of the transaction is unknown, the decision to allow will be more common than deny. With the added context around automation and the techniques used to mask the activity, the decision to deny can be made more confidently.
