|
|
|||||||||
|
|||||||||
|
|||||||||
| |
|||
| |||||||||
![]() |
|
|
«
Previous Thread
|
Next Thread
»
|
Thread Tools | Search this Thread | Display Modes |
|
|
|
Get inside! Sample the range of functionality easily built with JMSL Library for Time Series Data Analysis, Heat Maps, Portfolio Optimization, Monte Carlo Simulation, Stock Price Charting and more. Download Now! |
|
#1
|
|||
|
|||
|
ServerXMLHTTP detected
I am trying to automate a process where I go and screen scrape a site that has my fantasy football information. I am using ServerXMLHTTP, but it isn't working. The site recognizes that I am using a robot and doesn't send me the information I seek. here is the code.
I have been working on this for weeks and I don't see how I am being detected. I am logining into the site and I post my user name and pwd. That's it, but somehow they detect me. The function below is called 'getCookies' because it is designed to collect cookies that the remote site should send me if I login undetected. Unfortunately, they detect my bot and don't send my cookies. <% function getCookies(byRef sRef, sUrl, sMethod, sPostVars, byRef arrExistingCookie, sTheDomain) 'sUrl is the location you are spidering 'sPostVars - set to "" if using "GET" as method or no post vars 'arrCookies - contains cookies already received (must by a dynamic array) 'sRef will be set to the value of the last page (including redirects) sent by server '***************** 'param testing '***************** ' response.write "<p>sUrl: " & sUrl ' response.write "<p>sRef: " & sRef set xmlhttp = server.Createobject("MSXML2.ServerXMLHTTP.4.0") xmlhttp.setOption(0) = true 'check if array needs to be initialized if isarray(arrExistingCookie) then 'check for empty array bInit = false On Error Resume Next iUbound = uBound(arrExistingCookie) If Err.Number > 0 Then bInit = true end if else arrExistingCookie = array() bInit = true end if if bInit then iUbound = 0 redim preserve arrExistingCookie(iUbound) arrExistingCookie(0) = "placeholder" end if if instr(1,sMethod,"GET",1)>0 then sPostVars = "" end if '***************** 'end param testing '***************** with xmlhttp .open sMethod, sUrl, false .setRequestHeader "Content-Type", "application/x-www-form-urlencoded" .setRequestHeader "User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)" .setRequestHeader "Referer", sRef .setRequestHeader "ACCEPT-LANGUAGE", "en-us" .setRequestHeader "Accept", "image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/vnd.ms-powerpoint, application/vnd.ms-excel, application/msword, */*" if iUbound > 0 then for j = 1 to iUbound 'set the outgoing request header with current cookies .setRequestHeader "Cookie", arrExistingCookie(j) ' RESPONSE.WRITE "<br/> COOKIE" & j & ": " & arrExistingCookie(j) & "<br/>" next end if .send sPostVars ' response.write "<p>getOption:" & xmlhttp.getOption(SXH_OPTION_URL) & "<p>cookie: " & bCookie & "<br/>" While .readyState <> 4 .waitForResponse 1000 Wend strHeaders = .getAllResponseHeaders() sReturn = .responseText sRef = xmlhttp.getOption(SXH_OPTION_URL) ' response.write "<p>sRef: " & sRef end with set xmlhttp = nothing ' response.write "<p>headers: " & strHeaders & "<br/>" ' response.write "<p>body: " & sReturn ' response.end '******************************** 'get cookie information from returned headers (strHeaders) '******************************** ' sTemp = findCookies(strHeaders,arrExistingCookie, sTheDomain) sTemp = sParseHead(sHTML, arrExistingCookie, sTheDomain) '******************************** 'return body text for any processing '******************************** getCookies = sReturn end function %> |
![]() |
| Viewing: Dev Articles Community Forums > Programming > Programming Tools > ServerXMLHTTP detected |
| Thread Tools | Search this Thread |
| Display Modes | Rate This Thread |
|
|
|
|