Get a FREE Quote - We'll contact you right back!

Using Screaming Frog SEO Spider's 'Custom Extraction' Feature

Posted on November 28, 2015

Screaming Frog SEO Spider has been a tool we have been using for a few years now, which isn't as well known over here as it is in the UK and US markets.

Some of you may know why it's useful to have in your arsenal of SEO tools. For those of you who don't, Screaming Frog is basically a program which crawls URLs for a given website and gives back various on-site data such as page titles, meta descriptions, heading tags, and much more!

A screaming frog..A screaming frog

One of the latest features is the 'Custom Extraction' function, which allows you to put in your own CSS or Xpath in order to extract an HTML element from within a URL. This is useful for retrieving Google Analytics IDs, social media tags, product descriptions and more depending on what you want to do!

The video shows how to identify duplicate content in regards to product descriptions on an eCommerce website using this new feature on Screaming Frog.

Here is a quick summary of the steps from the video:

  1. Know what you want to extract - in our example it's the product descriptions.
  2. Right click on the webpage and 'Inspect Element' (or similar depending on your personal browser prefence).
  3. With the magnifying glass tool (top left), find what section you want to extract. This will navigate you to the area within the HTML which you've selected.
  4. Right click and copy 'CSS path' or 'Xpath'.
  5. Now go back to Screaming Frog and navigate onto Configuration > Custom > Extraction.
  6. Select a channel with your chosen method (CSS path, Xpath or Regex) and enter what you copied earlier from Inspect Element.
  7. Put in your chosen domain, URL or list depending on what you want to crawl.
  8. Make sure you navigate to the Custom > Filter (Extraction) and once you click start, it should reel in your chosen filter information if set up correctly.

Please note the element selected may not be present on every page (as a product description wouldn't be on the homepage for example).

Let us know how you find the video and if you are using the tool in any creative ways? We'd love to know!

About the Author

Google Partner badges

We're real people! Meet the Optimising team!

Optimising team photo

Optimising Logo

Optimising are more than an Digital Agency. We're a team of brilliant digital minds who aim to transform the business of our clients through clever, outside of the box solutions.

The Optimising team are a highly experienced, dynamic group of professionals specialising in generating websites primed for success, ensuring a positive return from clients online presence.

+ Click here for more!


Get In Touch

Optimising Melbourne
1/345 Punt Road
Richmond VIC 3121

Optimising Gold Coast
Suite 511, 15 Albert Ave
Broadbeach QLD 4218

Optimising Sydney
Suite 64, 246 Pacific Highway
Crows Nest NSW 2065