Content
Commerce
Devices
Customers
Pricing
Resources
Get Started
Sign in
Logout

Archives

How can I block your site crawler from scanning specific pages?

June 14, 2017

You can do that in 2 different ways:
(1) Put it under robots.txt as our crawler will respect it.
(2) Use our crawler blacklist feature.

Newer Posts

facebook
Linkedin
twitter
WordPress
Youtube

Home
Content
Commerce
Devices
Pricing
FAQ
Guide
Blog
About Us
Contact Us
Terms
Privacy
Privacy Statement CA Residents
Do Not Sell My Information

We use cookies at on this site so we can serve up content and advertising that's relevant to you. You can find out more and control how cookies are used by clicking on our Privacy Policy.

We use cookies at on this site so we can serve up content and advertising... view more

Privacy & Cookies Policy

Privacy Overview

This website uses cookies to improve your experience while you navigate through the website. Out of these, the cookies that are categorized as necessary are stored on your browser as they are essential for the working of basic functionalities of the website. We also use third-party cookies that help us analyze and understand how you use this website. These cookies will be stored in your browser only with your consent. You also have the option to opt-out of these cookies. But opting out of some of these cookies may affect your browsing experience.

Necessary

Always Enabled

Necessary cookies are absolutely essential for the website to function properly. This category only includes cookies that ensures basic functionalities and security features of the website. These cookies do not store any personal information.

Non-necessary

Any cookies that may not be particularly necessary for the website to function and is used specifically to collect user personal data via analytics, ads, other embedded contents are termed as non-necessary cookies. It is mandatory to procure user consent prior to running these cookies on your website.

SAVE & ACCEPT