WebVulnCrawl searching excluded directories for hackable web servers
A misguided person is using the robots.txt exclusion file to search for
vulnerable web applications. What he plans on doing with this list of
vulnerable web applications is up to debate.
What he is doing is a violation of the RFC's (governing robots.txt..
Yes, hackers do that also)
The robots.txt file is NOT AN ACCESS CONTROL LIST, and SHOULD NOT BE
USED TO 'HIDE' DIRECTORIES. AALL DIRECTORIES SHOULD BE PROTECTED AGAINST
Directory listing.
The only files and directories in the robots.txt file should be files
and directories that are already exposed on the web, and not include any
hidden or private directories.
Take this opportunity to review your web logs, and robots.txt file to
see what normally unpublished information and hints it may expose.
If you see WebVulnCrawl.blogspot.com in your logs, you may wish to lodge
complaints (after tightening up your security)
Further, dshield shows them portscanning the net also, looking for
unpublished information on unpublished servers.
Spammers harvesting email addresses? (illegal under federal can-spam
law) Hackers looking for credit card info? Hackers looking for
vulnerable web servers? (WebVuln Crawl?)
Either case, illegal under FEDERAL 1990 computer abuse and fraud act,
'attempted access beyond authorization'
Several other people also think this is illegal:
http://webvulncrawl.blogspot.com/2005/12/what-am-i-doing.html
216.179.125.69 - - [11/Mar/2006:06:03:15 -0500] "GET /js/ HTTP/1.1" 403
213 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69
- - [11/Mar/2006:06:03:18 -0500] "GET /includes/ HTTP/1.1" 403 219 "-"
"WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - -
[11/Mar/2006:06:03:20 -0500] "GET /mailman/ HTTP/1.1" 404 214 "-"
"WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - -
[11/Mar/2006:06:03:22 -0500] "GET /cgi-bin/ HTTP/1.1" 403 218 "-"
"WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - -
[11/Mar/2006:06:03:25 -0500] "GET /icons/ HTTP/1.1" 403 216 "-"
"WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - -
[11/Mar/2006:06:03:27 -0500] "GET /ClientList/ HTTP/1.1" 404 217 "-"
"WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - -
[11/Mar/2006:06:03:29 -0500] "GET /OtherMedia/ HTTP/1.1" 404 217 "-"
"WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - -
[11/Mar/2006:06:03:34 -0500] "GET /testlist.html HTTP/1.1" 404 219 "-"
"WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - -
[11/Mar/2006:06:03:36 -0500] "GET /nessus/msiis_testlist.html HTTP/1.1"
200 31073 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803"
216.179.125.69 - - [11/Mar/2006:06:03:41 -0500] "GET /pipermail
HTTP/1.1" 404 215 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803"
216.179.125.69 - - [11/Mar/2006:06:03:43 -0500] "GET /errors/ HTTP/1.1"
404 213 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803"
216.179.125.69 - - [14/Mar/2006:23:48:31 -0500] "GET /js/ HTTP/1.1" 403
213 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69
- - [14/Mar/2006:23:48:34 -0500] "GET /includes/ HTTP/1.1" 403 219 "-"
"WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - -
[14/Mar/2006:23:48:37 -0500] "GET /mailman/ HTTP/1.1" 404 214 "-"
"WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - -
[14/Mar/2006:23:48:39 -0500] "GET /cgi-bin/ HTTP/1.1" 403 218 "-"
"WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - -
[14/Mar/2006:23:48:42 -0500] "GET /icons/ HTTP/1.1" 403 216 "-"
"WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - -
[14/Mar/2006:23:48:45 -0500] "GET /ClientList/ HTTP/1.1" 404 217 "-"
"WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - -
[14/Mar/2006:23:48:47 -0500] "GET /OtherMedia/ HTTP/1.1" 404 217 "-"
"WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - -
[14/Mar/2006:23:48:52 -0500] "GET /testlist.html HTTP/1.1" 404 219 "-"
"WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803" 216.179.125.69 - -
[14/Mar/2006:23:48:54 -0500] "GET /nessus/msiis_testlist.html HTTP/1.1"
200 31073 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803"
216.179.125.69 - - [14/Mar/2006:23:48:58 -0500] "GET /pipermail
HTTP/1.1" 404 215 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803"
216.179.125.69 - - [14/Mar/2006:23:49:01 -0500] "GET /errors/ HTTP/1.1"
404 213 "-" "WebVulnCrawl.blogspot.com/1.0 libwww-perl/5.803"
--
Michael Scheidell, CTO
561-999-5000, ext 1131
SECNAP Network Security Corporation
Keep up to date with latest information on IT security: Real time
security alerts: http://www.secnap.com/news