Web Scraping with Scrapy | code comparison using items

Using Item Loaders to populate items when web scraping with Scrapy : The 'Items.py' and ItemLoader class are demonstrated here.

With this bare minimum spider I show 2 ways of populating the items containers in Scrapy ready to export.

The beauty of using items (items.py) is that you can use the same scrapy fields/items.py file with multiple spiders.

I use css selectors here and show how to use add_value to add literal values as well.
I also discuss 'SelectorGadget', 'neofetch'



To read more about Feed Exports : https://docs.scrapy.org/en/latest/topics/feed-exports.html
Feed Exports allow you to export to one of the following formats:

JSON
JSON lines
CSV
XML


⦿ The challenge is to rewrite the selectors using xpath! ➤ Scrapy reference : https://docs.scrapy.org/en/latest/topics/loaders.html?highlight=itemloader


A follow up video will show these, how to handle the pagination and collect the geo coordinates that I identified during this video.



Useful Links:



➤ SelectorGadget: point and click CSS selectors : https://selectorgadget.com/

➤ neofetch : https://github.com/dylanaraps/neofetch
(Neofetch supports almost 150 different operating systems inc. Linux and Windows)

Python scrapy.loader.ItemLoader() Examples :
➤ https://www.programcreek.com/python/example/89768/scrapy.loader.ItemLoader

Credit : https://www.pythongasm.com/introduction-to-scrapy/ for the original idea.

My Craiglsist Real Estate Python Scrapy code : https://github.com/RGGH/Scrapy4

Apologies if the cam was slightly out of synch with the sound - I'll fine tune it for next time. I need a clapperboard.

Feel free to buy me a coffee or tea!:
https://www.buymeacoffee.com/DrPi ☕️☕️☕️
else: ?


See you around yeah?