This seems to be a very useful tool for crawling sites and getting insights into broken links and other issues. Just tried it against the dev site in my VM and when it had created 1GB of data I decided to stop. I’ll try it again on a smaller site .
SharePoint 2010 SEO Analysis with the IIS SEO Toolkit
by Tristan Watkins
The IIS.NET Search Engine Optimization (SEO) Toolkit provides a powerful analysis tool that can generate reports for web editors and can automatically generate sitemaps and robots.txt files as well. These reports not only provide insight in to page rank improvements but also help content editors identify missing/duplicate content and find broken links. This post provides an overview of how the tools can be used by content editors or web managers who do not have access to the server infrastructure and what you can expect to see when running an SEO Analysis against an out of the box SharePoint 2010 Publishing site