Using screaming frog SEO spider's 'Custom Extraction' feature

James Richardson

Sales & Marketing Director | Co-Founder

Posted: 25 Mar 2020

Screaming Frog SEO Spider has been a tool we have been using for a few years now, which isn't as well known over here as it is in the UK and US markets.

Some of you may know why it's useful to have in your arsenal of SEO tools. For those of you who don't, Screaming Frog is basically a program which crawls URLs for a given website and gives back various on-site data such as page titles, meta descriptions, heading tags, and much more!

Screaming Frog

One of the latest features is the 'Custom Extraction' function, which allows you to put in your own CSS or Xpath in order to extract an HTML element from within a URL. This is useful for retrieving Google Analytics IDs, social media tags, product descriptions and more depending on what you want to do!

The video shows how to identify duplicate content in regards to product descriptions on an eCommerce website using this new feature on Screaming Frog.

Here is a quick summary of the steps from the video:

  1. Know what you want to extract - in our example it's the product descriptions.
  2. Right click on the webpage and 'Inspect Element' (or similar depending on your personal browser prefence).
  3. With the magnifying glass tool (top left), find what section you want to extract. This will navigate you to the area within the HTML which you've selected.
  4. Right click and copy 'CSS path' or 'Xpath'.
  5. Now go back to Screaming Frog and navigate onto Configuration > Custom > Extraction.
  6. Select a channel with your chosen method (CSS path, Xpath or Regex) and enter what you copied earlier from Inspect Element.
  7. Put in your chosen domain, URL or list depending on what you want to crawl.
  8. Make sure you navigate to the Custom > Filter (Extraction) and once you click start, it should reel in your chosen filter information if set up correctly.

Please note the element selected may not be present on every page (as a product description wouldn't be on the homepage for example).

Let us know how you find the video and if you are using the tool in any creative ways? We'd love to know!


James Richardson

Sales & Marketing Director | Co-Founder

Working in the SEO industry for many years alongside some of Australia’s biggest brands, James started his online career running online Sports Fan sites, as well as cutting his teeth on several successful eCommerce brands and content sites.

Previously holding various senior roles across the Sales and Marketing teams for ASX listed companies, he went on to found Optimising with Daniel and is proud he has helped mould it into one of Melbourne’s leading SEO agencies.

When he’s not in the office he’s at home having pretend tea parties, or building a cubby house in the lounge room with his three young girls.

Back to blog

Partner with
the real deal

Chat with us today and we’ll get you the results you deserve.