Introduction
Welcome to the first instalment of a four part series exploring how a salary prediction model was constructed, using job descriptions from LinkedIn.
Part One: Scraping Job Descriptions
In web data extraction projects, the choice of tools can significantly impact the efficiency and success of a project. LinkedIn has very strict rules on scraping data, there are several measures set up to prohibit bots, and so, to scrape job listings from their public job search webpage it is necessary to use a tool that can manage the measures put in place.
Body
For this project, Scrapy, a Python-based web scraping framework renowned for its robust capabilities was used.
Why Scrapy? Scrapy excels at handling large-scale data extraction tasks with remarkable ease. Unlike its counterparts such as BeautifulSoup and Selenium, Scrapy stands out for its minimalistic approach, requiring fewer dependencies and delivering results swiftly.
It is excellent at navigating complex web structures with features like middleware, proxies, and user agents, ensuring smooth extraction.
The scraping process was divided into 2 parts
- Capturing initial job details from individual job cards.
- Acquiring job links to scrape additional job details.
LinkedIn’s public job search portal lists jobs roles like in the image. New jobs are returned when a user scrolls down the page.Scrapy uses spiders to crawl and scrape data from websites. The LinkedIn spider was built using the code code snippet below:
# scrapes initial details on job card on public linked in jobs portal
import scrapy
import time
class JobsSpider(scrapy.Spider):
name = 'jobs'
api_url = ["https://uk.linkedin.com/jobs-guest/jobs/api/seeMoreJobPostings/business-data-analyst-jobs-london?start="] # Replace with the actual URL pattern
def start_requests(self):
base_url = self.api_url[0]
print(base_url)
first_job_on_page = 0
first_url = base_url + str(first_job_on_page)
print(first_url)
yield scrapy.Request(url=first_url, callback=self.parse_job, meta={'first_job_on_page': first_job_on_page})
def parse_job(self, response):
first_job_on_page = response.meta['first_job_on_page']
job_item = {}
# Replace the CSS selectors and extraction logic to match the structure of the job listings on the website you are scraping.
jobs = response.css("li")
num_jobs_returned = len(jobs)
print("******* Num Jobs Returned *******")
print(num_jobs_returned)
print('*****')
time.sleep(10)
for job in jobs:
job_item['job_title'] = job.css("h3.base-search-card__title::text").get(default='not-found').strip()
job_item['job_detail_url'] = job.css(".base-card__full-link::attr(href)").get(default='not-found').strip()
job_item['job_listed'] = job.css('time::text').get(default='not-found').strip()
job_item['company_name'] = job.css('h4 a::text').get(default='not-found').strip()
job_item['company_link'] = job.css('h4 a::attr(href)').get(default='not-found')
job_item['company_location'] = job.css('.job-search-card__location::text').get(default='not-found').strip()
print(3)
yield job_item
# Wait for 10 seconds before making the next request
time.sleep(10)
if num_jobs_returned > 0:
first_job_on_page = int(first_job_on_page) + 25
base_url = self.api_url[0]
next_url = base_url + str(first_job_on_page)
print(next_url)
yield scrapy.Request(url=next_url, callback=self.parse_job, meta={'first_job_on_page': first_job_on_page})
The code block kicks off by collecting the initial URL from the API URL list. From there it fires off requests to the webpage using the start_requests function.
Navigating LinkedIn’s defences against scraping can be challenging. From IP blocking to rate limiting, they’ve got a few tricks up their sleeve. To outsmart these barriers, a proxy aggregation service was used.
What the proxy aggregation service did was grant the spider access to various ip addresses to use so it didn’t have to use the same ip to make requests over and over. It also rotated user agents.
The LinkedIn jobs page has an endless scroll feature, with each scroll a request is made to the jobs api to provide more job posts. Each request returns 25 jobs.
Based on this feature, the spider was designed to collect data using the api request url.
Here’s the breakdown: When the spider interacts with the LinkedIn API, it does so in increments of 25 with each call.
In the start_requests method, the URL string is augmented by adding 25 to it. This modified URL is then routed through the proxy aggregator, which handles the request.
The parse jobs function reads through the response returned and uses css selectors to locate and collect the necessary
- Job title
- Job detail url
- Date listed
- Company name
- Company’s LinkedIn url
- Company location
A counter is set for the number of jobs returned, to check if any jobs are returned. If the count is greater than 0, the initial counter value appended to the api url is increased by 25, this tells the api to provide the next 25 jobs.
This process is repeated until there are no more jobs left.
The data scraped, is stored in variables, outputted to a csv file.
After the initial details are collected, additional job details are scraped using the job detail url.
# add in individual job posts url to scrap further details of each job
import scrapy
import pandas as pd
class JobsSpider(scrapy.Spider):
name = 'jobs'
def start_requests(self):
file = pd.read_csv('links to scraped job details file')
urls = file['job_detail_url'].tolist()
# Generate requests for each URL
for url in urls:
yield scrapy.Request(url=url, callback=self.parse_job)
def parse_job(self, response):
job_item = {}
# Extract job details
job_details = response.css('.show-more-less-html__markup').getall()
# Since job_details is a list, you should join them first into a single string
job_details_string = ' '.join(job_details)
# Now you can use split() on the string and then join to clean up whitespace
job_item['job_details'] = ' '.join(job_details_string.split()).strip() if job_details else 'not-found'
# Extract other details using XPath
job_item['seniority_level'] = response.xpath("//h3[contains(text(), 'Seniority level')]/following-sibling::span/text()").get(default='not-found').strip()
job_item['employment_type'] = response.xpath("//h3[contains(text(), 'Employment type')]/following-sibling::span/text()").get(default='not-found').strip()
job_item['job_function'] = response.xpath("//h3[contains(text(), 'Job function')]/following-sibling::span/text()").get(default='not-found').strip()
job_item['industry'] = response.xpath("//h3[contains(text(), 'Industries')]/following-sibling::span/text()").get(default='not-found').strip()
# Add the current URL to the job_item
job_item['source_url'] = response.url
yield job_item
Conclusion
Using pandas and a for loop, a list of the job detail urls is passed to the spider. It makes a request to each url in the list and uses css selectors to identify and scrape the key details specified.
- Seniority level
- Employment type
- Job function
- Industry
These details are stored in variables, at the end of the process all the output is stored in a final csv format.
The next step is to clean the data gathered, read all about this here
Try the model here: (model)
