# robots.txt – allow everything except well-known SEO audit crawlers # --------------------------------------------------------------- User-agent: AhrefsBot # Ahrefs Disallow: / User-agent: MJ12bot # Majestic Disallow: / User-agent: SemrushBot # Semrush Disallow: / User-agent: SemrushBot-SA # Semrush SiteAudit variant Disallow: / User-agent: DotBot # Moz (Dot-com Monitor/Moz Link Explorer) Disallow: / User-agent: BLEXBot # Blex/linked data crawler Disallow: / User-agent: SEOkicks-Robot # SEOkicks Disallow: / User-agent: spbot # OpenLinkProfiler / Seobility Disallow: / # Everything else is welcome User-agent: * Disallow: # (An empty Disallow means “allow all”.)