Uncategorized

Full crawling of the live site using screaming frog settings.

The website includes a <meta name=”robots” content=”noindex, nofollow”/> tag within its <head> section. This meta tag instructs web crawlers, including Screaming Frog, to neither index the page nor follow any of its links. To override this and allow full crawling of the website with Screaming Frog, you need to adjust the spider’s settings to ignore […]