web scraping – Get Bing search results in Python
web scraping – Get Bing search results in Python
Actually, code youve written working properly, problem is in HTTP request headers. By default urllib use Python-urllib/{version} as User-Agent header value, which makes easy for website to recognize your request as automatically generated. To avoid this, you should use custom value which can be achieved passing Request object as first parameter of urlopen():
from urllib.parse import urlencode, urlunparse
from urllib.request import urlopen, Request
from bs4 import BeautifulSoup
query = programming
url = urlunparse((https, www.bing.com, /search, , urlencode({q: query}), ))
custom_user_agent = Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:47.0) Gecko/20100101 Firefox/47.0
req = Request(url, headers={User-Agent: custom_user_agent})
page = urlopen(req)
# Further code Ive left unmodified
soup = BeautifulSoup(page.read())
links = soup.findAll(a)
for link in links:
print(link[href])
P.S. Take a look on comment left by @edd under your question.