How to find Robots.txt with 02

We already discussed a script to find the crossdomain.xml file with 02.  Today we are going to talk about how to find the Robots.txt file.  Many websites have Robots.txt file but sometimes they contain sensitive information inside of these files.   Today we are going to write a script that searches Google for these files.

 

 

Below is a sample Robots.txt from a sample web application.

 

 

 

 

 

 

var ie = panel.clear().add_IE().silent(true);
ie.open("http://www.google.com");
ie.field("Search").value("inurl:robots.txt filetype:txt");


ie.button("Google Search").click();
var targetUrls = new List<string>();

foreach(var link in ie.links().urls())
 if (link.ends("robots.txt"))
 targetUrls.Add(link);

return targetUrls;
return targetUrls;
return targetUrls;
return ie.buttons();

//O2File:WatiN_IE_ExtensionMethods.cs
//using  O2.XRules.Database.Utils.O2
//O2Ref:WatiN.Core.1x.dll

 

 

 

 

I don’t plan on showing you how to exploit robots.txt.  But the 02 script is a simple one to find robots.txt  out in the wild.

 

Parsons Software Security Consulting, LLC

Securing the Internet one Application at a time.

 

mparsons [at] gmail.com

 

Etherpad 02 scripting to search and click on a link in Google

I was on an Etherpad session with Dinis and Sarah from OWASP.   Below is the link to Etherpad.   Etherpad is a great tool for programmers and for us to help write scripts together and trouble shoot.

http://ietherpad.com/

We used the free version.

Below are our Etherpad sessions from December, 3, 2010.

http://ietherpad.com/xkckfhJGAY

http://ietherpad.com/UYmog5ljkj

Dinis blogs about it on his blog.   http://o2platform.wordpress.com/2010/12/04/o2-script-to-perform-a-google-search/

Below is today’s 02 script.  Dinis is the original author but I tweaked it to do some shameless self promoting.

panel.clear();
var ie = panel.add_IE().silent(true);
ie.disableFlashing(); // use this when developing the script to make it faster
Action<string> searchGoogle =
 (searchText)=> {
 ie.open("http://www.google.com");
 searchText = searchText.line();       // here......  <----
 ie.field("Search").value(searchText).flash();
 ie.button("Google Search").Click();
 };
Action<string> clickOnLink =
 (linkToClick)=> {
 if (ie.hasLink(linkToClick))
 ie.link(linkToClick).flash().click();
 else
 "Error: could not find link: {0}".error(linkToClick);
 };

searchGoogle("Parsons Software Security Consulting, LLC");
clickOnLink("Parsons Software Security Consulting, LLC - Home");


return ie.link("Parsons Software Security Consulting, LLC - Home").click();




//O2File:WatiN_IE_ExtensionMethods.cs
//using O2.XRules.Database.Utils.O2
//O2Ref:WatiN.Core.1x.dll

This code opens Google.  Disables flashing to make the search faster.  Then searches for my company, Parsons Software Security Consulting, LLC and clicks on the first link then opening my company’s website.

There is the script. Feel free to email me at mparsons1980@gmail.com for comments.

Matt Parsons, CISSP, MSM,
Parsons Software Security Consulting, LLC
Securing the Internet one Application at a time.

How to find Crossdomain.xml Cross Site Request Forgery with 02

Lately it seems that a lot of people are talking about the potential security vulnerabilities of having an unrestricted crossdomain.xml.   It’s public knowledge that this can be abused by an attacker setting up Cross Site Request Forgery.

Below is the sample code in the crossdomain.xml.   This is a simple one.  Some of the big websites that have these crossdomain.xml’s unrestricted have alot more data in the xml file.   From the blogs that I have read in the community and from HP Web Inspect Remediation Guide “Exploiting a Vulnerability Involves crafting a custom Flash Application”.

<cross-domain-policy>

<site-control permitted-cross-domain-policies="all"/>

<allow-access-from domain="*"/>

</cross-domain-policy>

The fix is “not to design and deploy Flash APIS meant to be accessible to arbitrary third parties.   It is also recommended “to host these on a sub domain”. We are not going to discuss how to exploit this vulnerability but rather to find it in the wild with 02.

So lets open up 02.   I like using Dinis Cruz’s version because it is more powerful.

Then lets open IE Automation.

Below is the default script in IE Automation that Dinis Created.  The default website is Google.

panel.clear();
var ie = panel.add_IE().silent(true);

ie.open("http://www.google.com");

//O2File:WatiN_IE_ExtensionMethods.cs
//using O2.XRules.Database.Utils.O2
//O2Ref:WatiN.Core.1x.dll

var topPanel = panel.<strong>clear</strong>().<strong>add_Panel</strong>();
var ie = topPanel.<strong>add_IE</strong>().<strong>silent</strong>(<strong>true</strong>);
ie.<strong>open</strong>("http://www.google.com");
ie.<strong>field</strong>("Search").<strong>value</strong>("inurl:crossdomain.xml filetype:xml");
ie.<strong>button</strong>("Google Search").<strong>click</strong>();
var targetUrls = <strong>new </strong>List<string>();
<strong>foreach</strong>(var link <strong>in </strong>ie.<strong>links</strong>().<strong>urls</strong>())
<strong>if </strong>(link.<strong>ends</strong>("crossdomain.xml"))
targetUrls.<strong>Add</strong>(link);
var listOfUrls = topPanel.insert_Left<Panel>(250).<strong>add_TextArea</strong>().<strong>wordWrap</strong>(<strong>false</strong>);
listOfUrls.<strong>set_Text</strong>(targetUrls.<strong>str</strong>());
return targetUrls;

//O2File:WatiN_IE_ExtensionMethods.cs
//using  O2.XRules.Database.Utils.O2
//O2Ref:WatiN.Core.1x.dll

var topPanel = panel.clear().add_Panel();
var ie = topPanel.add_IE().silent(true);
ie.open("http://www.google.com");
ie.field("Search").value("inurl:crossdomain.xml filetype:xml");
ie.button("Google Search").click();

var listOfUrls = topPanel.insert_Left<Panel>(350).add_TreeView();
var fileContents =listOfUrls.insert_Below<Panel>(200).add_SourceCodeViewer();

listOfUrls.afterSelect<string>(
 (selectedUrl)=> {
 listOfUrls.backColor(Color.LightPink);
 Application.DoEvents();
 var html = selectedUrl.uri().getHtml();
 fileContents.set_Text(html);
 listOfUrls.backColor(Color.White);
 });

foreach(var link in ie.links().urls())
 if (link.ends("crossdomain.xml"))
 listOfUrls.add_Node(link,link);

listOfUrls.selectFirst();

//O2File:WatiN_IE_ExtensionMethods.cs
//using  O2.XRules.Database.Utils.O2
//O2Ref:WatiN.Core.1x.dll

If the script executes properly it displays all the potential websites that are vulnerable to this vulnerability and then puts them inside of 02.  In order for them to be vulnerable allow-access-from-domain has to be set to *.    I don’t want to expose which sites are vulnerable due to legal reasons.   What is missing from the script is  is the rule that checks for the * value.

Many thanks to Dinis, all the security researchers that have been blogging about this vulnerability, HP, IBM and the Netsparker crew.

http://erlend.oftedal.no/blog/?blogid=107

http://shiflett.org/blog/2006/sep/the-dangers-of-cross-domain-ajax-with-flash

http://www.hp.com/go/appsec

www.ibm.com/software/awdtools/appscan/

http://www.o2platform.com/wiki/Main_Page

http://diniscruz.blogspot.com/

http://www.mavitunasecurity.com/

Matt Parsons, CISSP, MSM

mparsons [at]  gmail.com

Parsons Software Security Consulting, LLC Securing the Internet one Application at a time.