Random Site of the Week: OpenWebSearch (opens in new tab)
Your control over your online presence is paramount. If you prefer to keep our web crawler from accessing your site, you can do so by updating your website’s robots.txt file. Just add our user-agent identifier Owler, which represent our main and experimental crawler. To prevent any current or future versions from accessing your site, simply add Owler to the file.
Read the original article