Sitemap: https://deiequipment.com/sitemap.xml Sitemap: https://deiequipment.com/news-sitemap.xml User-agent: * Disallow: /wp-content/uploads/wc-logs/ Disallow: /wp-content/uploads/woocommerce_transient_files/ Disallow: /wp-content/uploads/woocommerce_uploads/ Disallow: /*?add-to-cart= Disallow: /*?*add-to-cart= Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php # --- Crawl management (WCCOM 11621805) --- # General rules for all crawlers: skip admin/account pages and faceted/cart # URLs that can't be cached, and slow the pace a little. # (Bing & Yandex honour Crawl-delay; Google ignores it.) User-agent: * Crawl-delay: 10 Disallow: /wp-admin Disallow: /myaccount Disallow: /myaccount/edit-account Disallow: /*?orderby= Disallow: /*?filter_ Disallow: /*?min_price= Disallow: /*?max_price= Disallow: /*?query_type_ Disallow: /*?add-to-cart= # Unwanted SEO / scraper bots (no shopping value, real crawl cost) User-agent: AhrefsBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: ia_archiver Disallow: / User-agent: AlphaSeoBot-SA Disallow: / User-agent: AlphaSeoBot Disallow: / User-agent: rogerbot Disallow: / User-agent: dotbot Disallow: / User-agent: SemrushBot Disallow: / User-agent: SemrushBot-SA Disallow: / User-agent: The Knowledge AI Disallow: / User-agent: SeznamBot Disallow: / User-agent: trendictionbot Disallow: / User-agent: SMTBot Disallow: / User-agent: DataForSeoBot Disallow: / # AI training-only crawlers (no shopping/citation value, real crawl cost) User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: CCBot Disallow: / User-agent: Bytespider Disallow: / User-agent: Amazonbot Disallow: / # AI search/citation crawlers — explicitly allowed so buyers researching # equipment via ChatGPT, Claude, Perplexity, Alexa, or Rufus can find and # be pointed to you User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Amzn-SearchBot Allow: / User-agent: Amzn-User Allow: / Sitemap: https://deiequipment.com/sitemap.xml