+- Curl Global Community (https://communities.curl.com)
+-- Forum: Blogs (https://communities.curl.com/forumdisplay.php?fid=17)
+--- Forum: Tech blog (https://communities.curl.com/forumdisplay.php?fid=18)
+---- Forum: Robert blog (https://communities.curl.com/forumdisplay.php?fid=20)
+---- Thread: robots.txt (/showthread.php?tid=486)
robots.txt - RobertShiplett - 05-01-2012
Your Curl web pages may need you to add this in your robots.txt file to prevent Google search from displaying and listing your Curl code :
Disallow: /*.curl$
Disallow: /*.scurl$
Disallow: /*.mcurl$
Disallow: /*.xcurl$
Disallow: /*.pcurl$
the first 4 being the most important.