Why Blocking AI Agents Could Hurt Your Ecommerce Store

AI technologies have become crucial tools for retail and ecommerce businesses.

Experts predict the autonomous AI market will reach $28.5 billion by 2028, while the global conversational AI market is expected to hit $13.9 billion by 2025. To “skate to where the puck is heading”, a vast majority of ecommerce companies (84%) are making AI integration a top priority in their operations.

So, what's making all this AI magic happen? Behind the scenes, “AI Agents” gather info from their surroundings (like user prompts or web searches), process it, and help AI models to make decisions or take action.

For retailers and ecommerce businesses, AI agents can automate things like recommending products to customers, answering questions, or managing stock. This helps businesses run more smoothly, give customers a better experience, and grow without needing as much hands-on work.

Ethics of Content Use: The Case of Perplexity AI

Like any big tech shift, the use of AI Agents isn’t met without skepticism. A prime example of this complex landscape is Perplexity AI, a company that's been making waves in the AI search space. They're also stepping into ecommerce, developing shopping features that promise "one-click purchases" and free shipping for their subscribers.

Perplexity AI is caught up in a big debate about data indexing practices. Several publishers, including Forbes and Wired, have accused the company of scraping and using their content without permission, even circumventing paywalls and ignoring robots.txt files (more to come on this later in the article).

Perplexity has taken steps to address these concerns. They've started a 'Publishers Program', which is a revenue-sharing model that aims to compensate content creators when their material is referenced in AI responses.

This controversy illustrates how AI companies are adapting to create more equitable relationships with content providers, but the ongoing tension between AI's need for vast amounts of data and content creators' rights to control and monetize their work goes on.

The Impact of Blocking AI Agents in Ecommerce

On the flip side, some ecommerce sites are now explicitly blocking AI agents from accessing their sites. To do this, website developers modify the robots.txt file in your website's root directory to instruct web crawlers on which pages they can and can’t access.

Think of the robots.txt file as a "No Trespassing" sign for websites, indicating the owner's preferences for how their site should be crawled. But keep in mind, the robots.txt file is more of a suggestion or request to web crawlers rather than a hard rule. Ignoring it might not break any laws, but it is generally considered unethical and against web crawling best practices.

A robots.txt file might look like this:

User-agent: *
Disallow: /private/

User-agent: GPTBot
Disallow: /

In this example, OpenAI's agent, GPTBot, is explicitly blocked from accessing the entire site. It's important to understand that AI agents like GPTBot often serve dual purposes:

1. 1. Browsing on behalf of users, enhancing the shopping experience.
   2. Potentially crawling the internet to gather data for training AI models.

Retailers may block AI agents for several reasons:

• Content Protection Concerns: Fear that proprietary content might be scraped and misused.
• Avoiding AI Model Training: Worries that data could be used to train AI models without compensation.
• Increased Website Hosting Costs: Because bots can browse more pages faster than humans, retailers fear their website hosting costs will become more expensive due to increased traffic.
• Misunderstanding AI Agent Functions: Lack of clarity about how AI agents operate and their benefits.

For instance, some retailers like Goop have implemented such blocks, likely to prevent their site content from being used to train OpenAI's models without compensation - a stance shared by many retailers and publishers alike. However, this approach creates a dilemma: while it may protect content from being indexed for training purposes without royalties, it also potentially loses out on valuable revenue from shoppers using AI agents to browse on their behalf.

While setting boundaries is reasonable—like restricting access to sensitive areas—outright blocking AI agents like GPTBot can be counterproductive. Reducing visibility to the growing market of AI-assisted shoppers negatively impacts the user experience, and in the case of current AI assistants, they are not yet fast enough to pose a significant threat to website hosting costs.

Retailers need to consider both the pros and cons of blocking AI agents. The key is to strike a balance that protects proprietary information while still leveraging the advantages of AI-assisted shopping. This means understanding AI agents' functions, their impact on user experience, and the changing dynamics of online retail.

Unintended Consequences of Blocking AI Agents

Let’s use continue with OpenAI’s GPTBot as an example. GPTBot serves two main functions:

1. Web Crawler for OpenAI: It collects data from the internet to train and improve AI models like ChatGPT.
2. Agent for ChatGPT User Browsing: It acts on behalf of users when they use ChatGPT's browsing capabilities to fetch real-time information from websites.

In the shoppers’ case, GPTBot is fetching information from your website in order to give the user a personalized recommendation that matches their prompt criteria, which would have a high likelihood to result in a sale.

Say a shopper asks ChatGPT or Perplexity:

“Find me a trendy barn jacket for plus sized women.”

The AI will look through search results to find the best match for the shoppers criteria. Your brand might be the perfect fit, but if you’re blocking AI Agents from crawling your site, you’ll miss out on the sale.

In turn, blocking AI agents can lead to:

• Reduced Visibility to AI-Assisted Shoppers: Products may not appear in AI-generated recommendations.
• Potential Loss of Sales and Customer Engagement: Less visibility can result in fewer visits and sales.
• Poor SEO Performance: Similar to not appearing in search engine results, blocking AI agents limits reach.

Blocking AI agents is like turning away potential customers at your digital storefront. The future of retail lies not in restriction, but about finding a balance between protecting proprietary information and leveraging AI's potential to drive growth and enhance customer experiences.

FindMine's Perspective

By allowing AI agents to crawl your site, retailers can position themselves for success in an AI-driven future of e-commerce, where personalization and technological innovation are key drivers of success.

FindMine believes in allowing personal AI agents to browse your site on behalf of users:

• Enhanced Customer Experience: AI agents help customers discover products more easily.
• Increased Engagement: Connect with customers who prefer AI-assisted shopping.
• Balanced Approach to Site Access: Instead of blocking, set reasonable boundaries in your robots.txt file.

Blocking AI agents diminishes the user experience and reduces your potential customer base, and is unlikely to be fully effective in protecting it from being used as AI training data. If a company allows any web crawler traffic, or even if it blocks all in their robots file, it's very unlikely that the data won't make its way into training data sets for AI because of data brokers. Unless a website has a unique situation, the benefit of the deterrent might not outweigh the benefit of allowing users to browse using agents.

Future Implications

As AI continues to reshape ecommerce, retailers must anticipate and prepare for the additional changes on the horizon:

• Increasing Adoption of AI Tools: Consumers will expect seamless interactions between their AI assistants and online retailers.
• Development of New Standards: Industry standards for AI agent interactions may emerge. The industry will need to mature in developing standards as this evolves.
• Opportunity for Innovation: Retailers embracing AI now can become leaders in customer experience.

By jumping on these trends early, e-commerce businesses can position themselves for the future AI-driven future of retail, setting new standards for customer engagement and improved operations.

Best Practices for Ecommerce Sites

Ecommerce sites should take a smart approach that balances accessibility with protection:

• Allow AI Agents with Boundaries: Permit access to personal AI agents while setting limits in your robots.txt file.
• Stay Visible to AI-Assisted Shoppers: Ensure your products are accessible to AI tools consumers use.
• Leverage AI for Improved Customer Experience: Use AI technologies like FindMine to offer personalized recommendations.

By balancing protection and using AI to drive growth and enhance customer experiences, retailers can ensure they remain visible, relevant, and competitive.