This Working Droid Episode 72 Robots.txt

Episode Guide

  • Keep websites and individual pages hidden from search engine crawlers
  • Tells search engines how much or how little to crawl
  • Not required to follow by any crawlers
  • Wikipedia

Direct Download

E-Mail me or fill out our online contact form!

Support Geek I/O on Patreon!

HeadphonesNeil Amazon Author Profile

[Sponsor - ProXPN] Visit http://proxpn.com and use promo code 'GEEK20' to get 20% off your order today!