A crawler is a program used to automatically scan webpages for things like content and meta data. As an advertiser, it’s important to understand how our crawler works because our system needs to be able to successfully crawl your landing page in order for you to launch and run a campaign.

How The Taboola Crawler Works

As an advertiser, any URL you add to your campaign will be crawled by our crawler in order to scan the content of the page. Our crawler will also grab the title and thumbnail from your site’s meta data.

If the crawler does not succeed crawling the page the first time, it will try again a few more times.

How to Avoid Crawling Errors

You may need to whitelist our crawler in order for it to crawl your webpage. Here is the information you’d need to do so: User agent: Mozilla/5.0 (compatible; Taboolabot/3.7; +http://www.taboola.com)

Also, make sure that you have title and thumbnail information in your site code for our crawler to find:

• Our crawler will look for a title use for your campaign item title. Here are two examples:
  <meta property="og:title">
  <meta name="title">

• Our crawler will look for a thumbnail image use for your campaign item thumbnail. Here are two examples:
  <meta property="og:image">  
  <meta name="item-image">
  <meta property="thumbnail-taboola-internal-key">
  <link rel =’image_src’>
  
  

If you are running into any issues, please reach out to your Account Manager or support@taboola.com.

How Our Crawler Is Used For Publishers

Our crawler will also crawl all Taboola publisher pages. In other words, it will crawl the pages where your content items may appear. The crawler will look at the content of the publisher page in order to make smart decisions about where to place your campaign items.