Getting service unavailable error in scrapy crawling

I am trying to crawl a forum website with scrapy. The crawler works fine if I have

CONCURRENT_REQUESTS = 1

But if I increase that number then I get this error

2012-12-21 05:04:36+0800 [working] DEBUG: Retrying (failed 1 times): 503 Service Unavailable

I want to know if the forum is blocking the request or there is some settings problem.

2 Answers

HTTP status code 503, "Service Unavailable", means that (for some reason) the server wasn't able to process your request. It's usually a transient error. I you want to know if you have been blocked, just try again in a little while and see what happens.

It could also mean that you're fetching pages too quickly. The fix is not to do this by keeping concurrent requests at 1 (and possibly adding a delay). Be polite.

And you will encounter various errors if you are scraping a enough. Just make sure that your crawler can handle them.

This answer maybe a bit late, but what worked for me is this. I added a header where I specified Mozilla/5.0 user agent. I then stopped getting "HTTP status code 503" error.

Code is below, just ran trough Amazon without issues. This code basically collects all links from Amazon's home page. Code is a Python programming language code.

 import urllib2 from bs4 import BeautifulSoup, SoupStrainer url = "" opener = urllib2.build_opener() opener.addheaders = [('User-Agent', 'Mozilla/5.0')] website = opener.open(url) html = website.read() soup = BeautifulSoup(html, "html.parser") for element in soup.find_all(['a','link']): link = element.get('href') print link