There are a million and one useless SEO tools, a quick search will tend to turn up any number of Firefox plugins which will happily ruin your browser with keyword densities, PageRank and Alexa scores. Often these “tools” will slow down your browser and collect your private information, providing little actual SEO value to you. Here’s a list of actually useful tools which can be used for all sorts of SEO tasks.
There are plenty of crawlers of varying quality available. Here are three that we use most regularly in bigmouthmedia, they all have different strengths and weaknesses and often I’ll use more than one to do everything I want to do. Some other paid services/software that I have not written about here but are also worth checking include 80Legs, A1 and PowerMapper. All have free trials.
Xenu is fast, light and has strong export options. It’s also free. URL filtering and control over what is and isn’t crawled could be improved and it would be handy to have more information about each page. It’s still number one for now but Screaming Frog is updating and improving regularly.
Screaming Frogis now used widely by SEOs and has become a real challenger to Xenu. It is quick and provides lots of useful information (canonical link elements, headings, etc.) for each page. It also has a custom filter which allows you to highlight pages with specific markup – this allows you to check the site for tracking code, iframes etc. There are still a few areas where screaming frog is weaker – it doesn’t export in any form other than raw data and that, whilst it has been improved recently, can be limited. For example, it’s not possible to export a list of images with the pages where they are found. So far, many of the weaknesses SF started out with have been overcome and there are more updates in the pipeline – in particular, improved scraping sounds like a winner which will place Screaming Frog as my number one crawler.
G-Site Crawlerwas written by John Mueller in his pre-Google days. It’s an awesome piece of software although it does suffer from a certain level of clunky-ness. Its exporting, filtering and disallow options are all excellent but on the downside it can be very slow and is less stable than the other two.
Web Developer Toolbar
Flagfoxconstantly displays the hosting country flag of any site in the address bar. It is regularly updated for accuracy and has a number of useful links built in to allow you to get more information about a particular IP address.
User Agent Switcher
User Agent Switcherallows you to change your browser’s user agent settings to anything you want. View a site as it would be seen by a mobile browser or search engine bot, for example. You can easily add a list of user agents in XML format.
IMacros allows you to automate your browser to complete automated tasks. It could and no-doubt is used for evil but can be a white hat SEO’s best friend too. Got 4,000 “brand protection” domains you want to gather data on? Want to automate some reporting? Want to write a competitor report without the leg-work of gathering data manually? iMacros lets you write simple macros to do all this. The basic Firefox plugin version is free but other, more powerful optionsare also available.
HTTP Header Checkers
I was introduced to WebBug when I first started in bigmouthmedia. It’s still my favourite – I love the simplicity of it. There are no whistles or bells, it just does its job well.
Live HTTP Headers
More whistles and bells – Live HTTP Headersis the preferred option for header checking amongst many SEOs. I use it myself regularly but digging through all the responses to find what you’re looking for can get confusing.
Web Sniffer is easy and straightforward, it’s not unlike WebBug except online. It’s hosted in Germany so that has to be taken into account when using (you may see something different than a UK/US browser if conditional redirection is being used) although that can also be a benefit. I use all of the above tools on a daily basis. These are not affiliate links. If you have any tool you think should go in this list, feel free to comment below. Please, no spam – all comments are moderated and any suggested tools will be checked.