Web Scraping with Python | Real Estate

How to use Python and Scrapy with regular expressions and 'response.meta' - 2 important aspects of getting the desired CSV / output file for a real world project. Regular expressions can be complex so if you want to make money web scraping you'll need to be able to understand and create your own as quickly as possible without spending a lot of effort searching Stackoverflow etc.

I've shown an overview of the logic of how the scraper works and demonstrated how to run a scrapy script standalone without it being part of a scrapy project - i.e using CrawlerProcess.

The structure of this code could be copied, modified , reused for other real estate scraping projects so feel free to download it from my github repo : https://github.com/RGGH/Scrapy16

=== Chapter timings ===
0:00 intro
5:24 how to use reponse.meta
11:37 cloning the code from GitHub
15:20 how to get correct headers
20:20 running the code
32:10 checking the output

Visit redandgreen the notes to this video
=========================================
? https://redandgreen.co.uk/scrapy-response-meta/

Subscribe to the YouTube Channel
=================================
? https://www.youtube.com/c/DrPiCode

Follow on Twitter - to get notified of new videos
=================================================
? https://twitter.com/RngWeb

Buy Dr Pi a coffee (or Tea)
☕ https://www.buymeacoffee.com/DrPi

Proxies
=================================================
If you need a good, easy to use proxy, I was recommended this one, and having used ScraperAPI for a while I can vouch for them. If you were going to sign up anyway, then maybe you would be kind enough to use the link and the coupon code below?

You can also do a full working trial first as well, (unlike some other companies). The trial doesn't ask for any payment details either so all good! ?

? 10% off ScraperAPI : https://www.scraperapi.com?fpr=ken49
◼️ Coupon Code: DRPI10
(You can also get started with 1000 free API calls. No credit card required.)

Thumbs up yeah? (cos Algos..)

#webscraping #python #realestate