瀏覽器能打開的網(wǎng)址，用python卻采集不了

predatorymh

如題：
    http://www.bioonjob.com/hospital ... 0-8135-12A2C52548E3
用瀏覽器能正常打開，但是用python卻無法讀取其源代碼

我的代碼：
    req=urllib2.Request(url)
      req.add_header('User-Agent', "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)")
      response=urllib2.urlopen(req)

ubuntu_mark

首先，你這個url轉(zhuǎn)向別的的頁面了，再一個請求發(fā)過去，好像報500，需不需要登錄，你再試試

poper168

1.     res = opener.open(req)
   
2.   File "C:\software\Python25\lib\urllib2.py", line 387, in open
   
3.     response = meth(req, response)
   
4.   File "C:\software\Python25\lib\urllib2.py", line 498, in http_response
   
5.     'http', request, response, code, msg, hdrs)
   
6.   File "C:\software\Python25\lib\urllib2.py", line 425, in error
   
7.     return self._call_chain(*args)
   
8.   File "C:\software\Python25\lib\urllib2.py", line 360, in _call_chain
   
9.     result = func(*args)
   
10.   File "C:\software\Python25\lib\urllib2.py", line 506, in http_error_default
    
11.     raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
    
12. urllib2.HTTPError: HTTP Error 500: Internal Server Error
    

復(fù)制代碼

挺奇怪，我抓過來也是提示500，瀏覽器打開正常

renxiao2003

有沒有代理。

hipeace86

1. 
   
2.     import pycurl
   
3.     c = pycurl.Curl()
   
4.     c.setopt(pycurl.URL,"http://www.bioonjob.com/hospital/show.asp?id=5544112F-6542-4DD0-8135-12A2C52548E3")
   
5.     import StringIO
   
6.     b = StringIO.StringIO()
   
7.     c.setopt(pycurl.WRITEFUNCTION, b.write)
   
8.     c.setopt(pycurl.FOLLOWLOCATION, 1)
   
9.     c.setopt(pycurl.MAXREDIRS, 5)
   
10.     c.perform()
    
11.     print b.getvalue().decode('gb2312')
    

復(fù)制代碼

表示能采集

predatorymh

樓上的大俠，為什么用pycurl就能采集呢？

newcnad

其實是因為這個頁面本身返回的就是   500 Internal Server Error ，在瀏覽器中訪問返回的狀態(tài)也是500（雖然有部分內(nèi)容）

predatorymh

沒想到urllib2怎么都采不到的網(wǎng)址用pycurl能夠輕松采到啊，我平時用的都是urllib，看來以后要多研究研究pycurl了

descusr

www.samsung.com
這個網(wǎng)站用pycurl也采集不了，
    html = StringIO.StringIO()
    c = pycurl.Curl()
    c.setopt(c.URL, url.encode('utf-8') if type(url) is unicode else url )
    c.setopt(pycurl.URL,str(url))
    c.setopt(pycurl.WRITEFUNCTION, html.write)
    c.setopt(pycurl.NOBODY,0)
    c.setopt(pycurl.FOLLOWLOCATION, 0)
    c.setopt(pycurl.MAXREDIRS, 5)
    c.setopt(pycurl.CONNECTTIMEOUT, 60)
    c.setopt(pycurl.TIMEOUT, 300)
    #c.setopt(pycurl.USERAGENT, "Mozilla/5.0 (Windows; U;compatible; MSIE 8.0; Windows NT 6.1; SV1; .NET CLR 1.1.4322)")
    USER_AGENT = 'Mozilla/5.0 (X11; U; Linux i686; en-GB; rv:1.9.0.5) Gecko/2008121622 Ubuntu/8.10 (intrepid) Firefox/3.0.5'
    c.setopt(pycurl.USERAGENT, USER_AGENT)
    c.perform()
    ret = html.getvalue()
得到的ret為None!!!好郁悶～～