by The website includes a <meta name="robots" content="noindex, nofollow"/> tag within its <head> section. This meta tag instructs web crawlers, including Screaming Frog, to neither index the page nor follow any of its links. To override this and allow full crawling of the website with Screaming Frog, you need to adjust the spider’s settings to ignore these directives. This is not related to the robots.txt file but to the specific meta tags in the page headers. I’ve configured Screaming Frog to disregard the noindex and nofollow values, which enables a complete crawl. You can replicate this approach by loading the provided configuration file into your instance of Screaming Frog. I’ve also included an export of my search results. FYI, I’ve disabled crawling all images, css, js to speed things up. I’m not sure what the goal of the crawl is, but this should be a start to get us get unstuck.
