I need to extract the meta keywords from a web page using Python. I was thinking that this could be done using urllib or urllib2, but I'm not sure. Anyone have any ideas?
I am using Python 2.6 on Windows XP
13 Answers
lxml is faster than BeautifulSoup (I think) and has much better functionality, while remaining relatively easy to use. Example:
52> from urllib import urlopen 53> from lxml import etree 54> f = urlopen( "" ).read() 55> tree = etree.HTML( f ) 61> m = tree.xpath( "//meta" ) 62> for i in m: ..> print etree.tostring( i ) ..> <meta http-equiv="content-type" content="text/html; charset=ISO-8859-2"/> Edit: another example.
75> f = urlopen( "" ).read() 76> tree = etree.HTML( f ) 85> tree.xpath( "//meta[@name='Keywords']" )[0].get("content") 85> "xml,tutorial,html,dhtml,css,xsl,xhtml,javascript,asp,ado,vbscript,dom,sql,colors,soap,php,authoring,programming,training,learning,b eginner's guide,primer,lessons,school,howto,reference,examples,samples,source code,tags,demos,tips,links,FAQ,tag list,forms,frames,color table,w3c,cascading style sheets,active server pages,dynamic html,internet,database,development,Web building,Webmaster,html guide" BTW: XPath is worth knowing.
Another edit:
Alternatively, you can just use regexp:
87> f = urlopen( "" ).read() 88> import re 101> re.search( "<meta name=\"Keywords\".*?content=\"([^\"]*)\"", f ).group( 1 ) 101>"xml,tutorial,html,dhtml,css,xsl,xhtml,javascript,asp,ado,vbscript,dom,sql, ...etc... ...but I find it less readable and more error prone (but involves only standard module and still fits on one line).
3BeautifulSoup is a great way to parse HTML with Python.
Particularly, check out the findAll method:
Why not use a regular expression
keywordregex = re.compile('<meta\sname= ["\']keywords["\']\scontent=["\'](.*?)["\']\s/>') keywordlist = keywordregex.findall(html) if len(keywordlist) > 0: keywordlist = keywordlist[0] keywordlist = keywordlist.split(", ") 
