It was early in 2019, COVID had just hit Australia, and my lease was expiring on my over-priced and quite cockroach-y apartment in Bondi Beach. I needed somewhere to live, but didn't want to commit to another six month lease as I was hoping to return to Europe for a few months once things got back to normal (that's taking a while longer than expected!).
As travel restrictions and border closures being imposed across the country, the news was full of stories about how the tourism industry was in dire straights and then the idea struck me:
We should take advantage of this opportunity to travel around Australia and get some really cheap holiday accommodation.
We wanted to spend some time in Noosa, an upmarket resort town in Queensland. A quick Google search revealed literally hundreds of holiday rental properties, but how could we contact all of them to find the ones that were interested in offering cheap rates?
The Answer? We scraped the emails from the websites appearing in the search results, and emailed them all at once.
My girlfriend Mercedes is a machine learning engineer so is pretty handy with a Python script. I'm an SEO so know a thing or two about how Google works. Together we built an awesome script that allowed us to:
- Enter a search query and get the Google search results (in our case, 'holiday apartments Noosa')
- Scrape the top 100 URLs for the pages in the search results
- Look through the HTML code of those pages to find and extract a contact email address, and if there was none, look for a 'contact us' page and check that too
- Export the email addresses into a CSV file that we could download
After scraping the emails, I crafted an email which we sent out to all the addresses that we had found. The email I wrote basically explained our situation, what we were looking for, and our weekly budget (which was about a quarter of what you might usually expect to pay!).
It worked! After some emails back-and-forth emails with a number of resorts, we found a beautiful place called Bali Hai. The managers, Michael and Suzanne, were a lovely couple who were happy to have us stay there for a few months. We ended up paying $600 per week (less than half the usual weekly rate, and much cheaper than were I had been staying back in Bondi!).
How can you scrape email addresses from a Google search?
It's really easy!
Click here to go to our Google scraping tool.
It's built using Python script in a Google Colab file, but don't be put off if that all sounds a bit technical!
- Clone the notebook so you are able to make changes and save the output your own Google Drive
- Enter your search query in the space at the top of the notebook where it says 'Enter your search query between the speech marks here'
- Press CTRL + F9 (cmd +F9 on mac) to run all the cells.
- Give the tool a minute or two to run and voila! You will be prompted to download a CSV file containing all the email addresses found.
Any questions? Let me know in the comments section below.