Programming Tools
 
Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
 
User Name:
Password:
Remember me
 
Go Back   Dev Articles Community ForumsProgrammingProgramming Tools

Reply
Add This Thread To:
  Del.icio.us   Digg   Google   Spurl   Blink   Furl   Simpy   Y! MyWeb 
Thread Tools Search this Thread Display Modes
 
Unread Dev Articles Community Forums Sponsor:
Get inside! Sample the range of functionality easily built with JMSL Library for Time Series Data Analysis, Heat Maps, Portfolio Optimization, Monte Carlo Simulation, Stock Price Charting and more. Download Now!
  #1  
Old February 24th, 2003, 09:15 AM
tomandlis tomandlis is offline
Junior Member
Dev Articles Newbie (0 - 499 posts)
 
Join Date: Feb 2003
Posts: 1 tomandlis User rank is Just a Lowly Private (1 - 20 Reputation Level) 
Time spent in forums: < 1 sec
Reputation Power: 0
Send a message via AIM to tomandlis
ServerXMLHTTP detected

I am trying to automate a process where I go and screen scrape a site that has my fantasy football information. I am using ServerXMLHTTP, but it isn't working. The site recognizes that I am using a robot and doesn't send me the information I seek. here is the code.

I have been working on this for weeks and I don't see how I am being detected. I am logining into the site and I post my user name and pwd. That's it, but somehow they detect me.

The function below is called 'getCookies' because it is designed to collect cookies that the remote site should send me if I login undetected. Unfortunately, they detect my bot and don't send my cookies.

<%
function getCookies(byRef sRef, sUrl, sMethod, sPostVars, byRef arrExistingCookie, sTheDomain)
'sUrl is the location you are spidering
'sPostVars - set to "" if using "GET" as method or no post vars
'arrCookies - contains cookies already received (must by a dynamic array)
'sRef will be set to the value of the last page (including redirects) sent by server

'*****************
'param testing
'*****************
' response.write "<p>sUrl: " & sUrl
' response.write "<p>sRef: " & sRef
set xmlhttp = server.Createobject("MSXML2.ServerXMLHTTP.4.0")
xmlhttp.setOption(0) = true
'check if array needs to be initialized
if isarray(arrExistingCookie) then
'check for empty array
bInit = false
On Error Resume Next
iUbound = uBound(arrExistingCookie)
If Err.Number > 0 Then
bInit = true
end if
else
arrExistingCookie = array()
bInit = true
end if
if bInit then
iUbound = 0
redim preserve arrExistingCookie(iUbound)
arrExistingCookie(0) = "placeholder"
end if
if instr(1,sMethod,"GET",1)>0 then
sPostVars = ""
end if
'*****************
'end param testing
'*****************
with xmlhttp
.open sMethod, sUrl, false
.setRequestHeader "Content-Type", "application/x-www-form-urlencoded"
.setRequestHeader "User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)"
.setRequestHeader "Referer", sRef
.setRequestHeader "ACCEPT-LANGUAGE", "en-us"
.setRequestHeader "Accept", "image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/vnd.ms-powerpoint, application/vnd.ms-excel, application/msword, */*"
if iUbound > 0 then
for j = 1 to iUbound
'set the outgoing request header with current cookies
.setRequestHeader "Cookie", arrExistingCookie(j)
' RESPONSE.WRITE "<br/> COOKIE" & j & ": " & arrExistingCookie(j) & "<br/>"
next
end if
.send sPostVars
' response.write "<p>getOption:" & xmlhttp.getOption(SXH_OPTION_URL) & "<p>cookie: " & bCookie & "<br/>"
While .readyState <> 4
.waitForResponse 1000
Wend
strHeaders = .getAllResponseHeaders()
sReturn = .responseText
sRef = xmlhttp.getOption(SXH_OPTION_URL)
' response.write "<p>sRef: " & sRef
end with
set xmlhttp = nothing
' response.write "<p>headers: " & strHeaders & "<br/>"
' response.write "<p>body: " & sReturn
' response.end
'********************************
'get cookie information from returned headers (strHeaders)
'********************************
' sTemp = findCookies(strHeaders,arrExistingCookie, sTheDomain)
sTemp = sParseHead(sHTML, arrExistingCookie, sTheDomain)
'********************************
'return body text for any processing
'********************************
getCookies = sReturn
end function
%>

Reply With Quote
Reply

Viewing: Dev Articles Community ForumsProgrammingProgramming Tools > ServerXMLHTTP detected


Thread Tools  Search this Thread 
Search this Thread:

Advanced Search
Display Modes  Rate This Thread 
Rate This Thread:


Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
View Your Warnings | New Posts | Latest News | Latest Threads | Shoutbox
Forum Jump


Forums: » Register « |  User CP |  Games |  Calendar |  Members |  FAQs |  Sitemap |  Support | 
  
 





© 2003-2008 by Developer Shed. All rights reserved. DS Cluster 4 hosted by Hostway