AI technologies have become crucial tools for retail and ecommerce businesses.
Experts predict the autonomous AI market will reach $28.5 billion by 2028, while the global conversational AI market is expected to hit $13.9 billion by 2025. To “skate to where the puck is heading”, a vast majority of ecommerce companies (84%) are making AI integration a top priority in their operations.
So, what's making all this AI magic happen? Behind the scenes, “AI Agents” gather info from their surroundings (like user prompts or web searches), process it, and help AI models to make decisions or take action.
For retailers and ecommerce businesses, AI agents can automate things like recommending products to customers, answering questions, or managing stock. This helps businesses run more smoothly, give customers a better experience, and grow without needing as much hands-on work.
Like any big tech shift, the use of AI Agents isn’t met without skepticism. A prime example of this complex landscape is Perplexity AI, a company that's been making waves in the AI search space. They're also stepping into ecommerce, developing shopping features that promise "one-click purchases" and free shipping for their subscribers.
Perplexity AI is caught up in a big debate about data indexing practices. Several publishers, including Forbes and Wired, have accused the company of scraping and using their content without permission, even circumventing paywalls and ignoring robots.txt files (more to come on this later in the article).
Perplexity has taken steps to address these concerns. They've started a 'Publishers Program', which is a revenue-sharing model that aims to compensate content creators when their material is referenced in AI responses.
This controversy illustrates how AI companies are adapting to create more equitable relationships with content providers, but the ongoing tension between AI's need for vast amounts of data and content creators' rights to control and monetize their work goes on.
On the flip side, some ecommerce sites are now explicitly blocking AI agents from accessing their sites. To do this, website developers modify the robots.txt file in your website's root directory to instruct web crawlers on which pages they can and can’t access.
Think of the robots.txt file as a "No Trespassing" sign for websites, indicating the owner's preferences for how their site should be crawled. But keep in mind, the robots.txt file is more of a suggestion or request to web crawlers rather than a hard rule. Ignoring it might not break any laws, but it is generally considered unethical and against web crawling best practices.
A robots.txt file might look like this:
User-agent: *
Disallow: /private/
User-agent: GPTBot
Disallow: /
In this example, OpenAI's agent, GPTBot, is explicitly blocked from accessing the entire site. It's important to understand that AI agents like GPTBot often serve dual purposes:
Retailers may block AI agents for several reasons:
For instance, some retailers like Goop have implemented such blocks, likely to prevent their site content from being used to train OpenAI's models without compensation - a stance shared by many retailers and publishers alike. However, this approach creates a dilemma: while it may protect content from being indexed for training purposes without royalties, it also potentially loses out on valuable revenue from shoppers using AI agents to browse on their behalf.
While setting boundaries is reasonable—like restricting access to sensitive areas—outright blocking AI agents like GPTBot can be counterproductive. Reducing visibility to the growing market of AI-assisted shoppers negatively impacts the user experience, and in the case of current AI assistants, they are not yet fast enough to pose a significant threat to website hosting costs.
Retailers need to consider both the pros and cons of blocking AI agents. The key is to strike a balance that protects proprietary information while still leveraging the advantages of AI-assisted shopping. This means understanding AI agents' functions, their impact on user experience, and the changing dynamics of online retail.
Let’s use continue with OpenAI’s GPTBot as an example. GPTBot serves two main functions:
In the shoppers’ case, GPTBot is fetching information from your website in order to give the user a personalized recommendation that matches their prompt criteria, which would have a high likelihood to result in a sale.
“Find me a trendy barn jacket for plus sized women.”
The AI will look through search results to find the best match for the shoppers criteria. Your brand might be the perfect fit, but if you’re blocking AI Agents from crawling your site, you’ll miss out on the sale.
In turn, blocking AI agents can lead to:
Blocking AI agents is like turning away potential customers at your digital storefront. The future of retail lies not in restriction, but about finding a balance between protecting proprietary information and leveraging AI's potential to drive growth and enhance customer experiences.
By allowing AI agents to crawl your site, retailers can position themselves for success in an AI-driven future of e-commerce, where personalization and technological innovation are key drivers of success.
FindMine believes in allowing personal AI agents to browse your site on behalf of users:
robots.txt file.Blocking AI agents diminishes the user experience and reduces your potential customer base, and is unlikely to be fully effective in protecting it from being used as AI training data. If a company allows any web crawler traffic, or even if it blocks all in their robots file, it's very unlikely that the data won't make its way into training data sets for AI because of data brokers. Unless a website has a unique situation, the benefit of the deterrent might not outweigh the benefit of allowing users to browse using agents.
As AI continues to reshape ecommerce, retailers must anticipate and prepare for the additional changes on the horizon:
By jumping on these trends early, e-commerce businesses can position themselves for the future AI-driven future of retail, setting new standards for customer engagement and improved operations.
Ecommerce sites should take a smart approach that balances accessibility with protection:
robots.txt file.By balancing protection and using AI to drive growth and enhance customer experiences, retailers can ensure they remain visible, relevant, and competitive.