Using screaming frog SEO spider's 'Custom Extraction' feature
James Richardson
Co-Founder & Partnerships
Posted: 25 Mar 2020
Screaming Frog SEO Spider has been a tool we have been using for a few years now, which isn't as well known over here as it is in the UK and US markets.
Some of you may know why it's useful to have in your arsenal of SEO tools. For those of you who don't, Screaming Frog is basically a program which crawls URLs for a given website and gives back various on-site data such as page titles, meta descriptions, heading tags, and much more!
One of the latest features is the 'Custom Extraction' function, which allows you to put in your own CSS or Xpath in order to extract an HTML element from within a URL. This is useful for retrieving Google Analytics IDs, social media tags, product descriptions and more depending on what you want to do!
The video shows how to identify duplicate content in regards to product descriptions on an eCommerce website using this new feature on Screaming Frog.
Here is a quick summary of the steps from the video:
- Know what you want to extract - in our example it's the product descriptions.
- Right click on the webpage and 'Inspect Element' (or similar depending on your personal browser prefence).
- With the magnifying glass tool (top left), find what section you want to extract. This will navigate you to the area within the HTML which you've selected.
- Right click and copy 'CSS path' or 'Xpath'.
- Now go back to Screaming Frog and navigate onto Configuration > Custom > Extraction.
- Select a channel with your chosen method (CSS path, Xpath or Regex) and enter what you copied earlier from Inspect Element.
- Put in your chosen domain, URL or list depending on what you want to crawl.
- Make sure you navigate to the Custom > Filter (Extraction) and once you click start, it should reel in your chosen filter information if set up correctly.
Please note the element selected may not be present on every page (as a product description wouldn't be on the homepage for example).
Let us know how you find the video and if you are using the tool in any creative ways? We'd love to know!
James Richardson
Co-Founder & Partnerships
Working in the SEO industry for many years alongside some of Australia’s biggest brands, James started his online career running online Sports Fan sites, as well as cutting his teeth on several successful eCommerce brands and content sites.
Previously holding various senior roles across the Sales and Marketing teams for ASX listed companies, he went on to found Optimising with Daniel and is proud he has helped mould it into one of Australia's leading SEO agencies.
When he’s not in the office he’s at home having pretend tea parties, or building a cubby house in the lounge room with his three young girls.
We value purpose over profit and take action.
Our values and beliefs have always set the tone and approach to our business. It's not just enough to grow as a company and produce profits, we have a global responsibility to make our economy more inclusive and sustainable. As both a B Corp and a member of 1% for the planet, we have further cemented this purpose within our organisation.
However, our work isn't done quite yet. For Optimising, this is simple the start of our journey towards building a better business and world!
Partner with
the real deal
Chat with us today and we’ll get you the results you deserve.
We acknowledge the Wurundjeri Woi Wurrung people as the Traditional Owners of the land now known as Richmond. We pay our respects to Elders from all nations - and to their Elders past, present and future.
Optimising is committed to cultivating and preserving a culture of inclusion and connectedness. We are able to grow and learn better together with a diverse team of employees.
