I'm trying to parse a website and get some info with the find_all() method, but it doesn't find them all.
This is the code:
#!/usr/bin/python3 from bs4 import BeautifulSoup from urllib.request import urlopen page = urlopen ("") # print (page.read ()) soup = BeautifulSoup (page.read ()) manga_img = soup.findAll ('a', {'class' : 'manga_img'}, limit=None) for manga in manga_img: print (manga['href']) It only prints half of them...
2 Answers
Different HTML parsers deal differently with broken HTML. That page serves broken HTML, and the lxml parser is not dealing very well with it:
>>> import requests >>> from bs4 import BeautifulSoup >>> r = requests.get(') >>> soup = BeautifulSoup(r.content, 'lxml') >>> len(soup.find_all('a', class_='manga_img')) 18 The standard library html.parser has less trouble with this specific page:
>>> soup = BeautifulSoup(r.content, 'html.parser') >>> len(soup.find_all('a', class_='manga_img')) 44 Translating that to your specific code sample using urllib, you would specify the parser thus:
soup = BeautifulSoup(page, 'html.parser') # BeatifulSoup can do the reading The quick way to grab all href elements is to use CSS Selector which will select all a tags with an href element that contains /manga at the beginning link.
Output will contain all links that starts with /manga/"title"(check this in dev tools using inspector):
import requests from bs4 import BeautifulSoup import lxml html = requests.get(').text soup = BeautifulSoup(html, 'lxml') for a_tag in soup.select('a[href*="/manga"]'): link = a_tag['href'] link = link[1:] print(f') Alternative method:
Change requests.get to a different URL (directory/2.html)
Here's the working code(works 2-3-4-5-6.. pages as well) and replit.com to play around:
import requests from bs4 import BeautifulSoup import lxml html = requests.get(').text soup = BeautifulSoup(html, 'lxml') for manga in soup.select('.line'): title = manga.select('.manga-list-1-item-title a') for t in title: print(t.text) for i in manga.findAll('img', class_='manga-list-1-cover'): img = i['src'] print(img) for l in manga.findAll('p', class_='manga-list-1-item-title'): link = l.a['href'] link = link[1:] print(f') Output(could be prettier), all in order:
A Story About Treating a Female Knig... Tales of Demons and Gods Martial Peak Onepunch-Man One Piece Star Martial God Technique Solo Leveling The Last Human Kimetsu no Yaiba Versatile Mage Boku no Hero Academia Apotheosis Black Clover Tensei Shitara Slime Datta Ken Kingdom Tate no Yuusha no Nariagari Tomo-chan wa Onna no ko! Goblin Slayer Yakusoku no Neverland God of Martial Arts Kaifuku Jutsushi no Yarinaoshi Re:Monster Mushoku Tensei - Isekai Ittara Honki... Nanatsu no Taizai Battle Through the Heavens Shingeki no Kyojin Iron Ladies Monster Musume no Iru Nichijou World’s End Harem Bleach Parallel Paradise Shokugeki no Soma Spirit Sword Sovereign Horimiya Dungeon ni Deai o Motomeru no wa Mac... Dr. Stone Berserk The New Gate Akatsuki no Yona Naruto Overlord Death March kara Hajimaru Isekai Kyo... Tsuki ga Michibiku Isekai Douchuu Eternal Reverence Minamoto-kun Monogatari Beastars Jujutsu Kaisen Hajime no Ippo Kaguya-sama wa Kokurasetai - Tensai-... Domestic na Kanojo The Legendary Moonlight Sculptor The Gamer Kumo desu ga, nani ka? Bokutachi wa Benkyou ga Dekinai Enen no Shouboutai Tsuyokute New Saga Fairy Tail Komi-san wa Komyushou Desu. Kenja no Mago Soul Land Boruto: Naruto Next Generations Hunter X Hunter History’s Strongest Disciple Kenichi Phoenix against the World LV999 no Murabito Gate - Jietai Kare no Chi nite, Kaku... Kengan Asura Konjiki no Moji Tsukai - Yuusha Yoni... Please don’t bully me, Nagatoro Isekai Maou to Shoukan Shoujo Dorei ... I found the best way (for me) with .find_all()/.findAll() methods is just to use for loop, same goes with .select() method.
And in some cases .select() giving better results. Check out SelectorGadget to quickly find css selector.
