How to find Robots.txt with 02

We already discussed a script to find the crossdomain.xml file with 02.  Today we are going to talk about how to find the Robots.txt file.  Many websites have Robots.txt file but sometimes they contain sensitive information inside of these files.   Today we are going to write a script that searches Google for these files.

 

 

Below is a sample Robots.txt from a sample web application.

 

 

 

 

 

 

var ie = panel.clear().add_IE().silent(true);
ie.open("http://www.google.com");
ie.field("Search").value("inurl:robots.txt filetype:txt");


ie.button("Google Search").click();
var targetUrls = new List<string>();

foreach(var link in ie.links().urls())
 if (link.ends("robots.txt"))
 targetUrls.Add(link);

return targetUrls;
return targetUrls;
return targetUrls;
return ie.buttons();

//O2File:WatiN_IE_ExtensionMethods.cs
//using  O2.XRules.Database.Utils.O2
//O2Ref:WatiN.Core.1x.dll

 

 

 

 

I don’t plan on showing you how to exploit robots.txt.  But the 02 script is a simple one to find robots.txt  out in the wild.

 

Parsons Software Security Consulting, LLC

Securing the Internet one Application at a time.

 

mparsons [at] gmail.com