You can do that in 2 different ways:
(1) Put it under robots.txt as our crawler will respect it.
(2) Use our crawler blacklist feature.
June 14, 2017
You can do that in 2 different ways:
(1) Put it under robots.txt as our crawler will respect it.
(2) Use our crawler blacklist feature.