User-agent: Scrapy Allow: / User-Agent: * Disallow: /wp-admin/ Disallow: /portal/ Crawl-delay: 10 Sitemap: https://www.hightea.nl/sitemap.xml