Introduction

Welcome to the third instalment of a four part series exploring how a salary prediction model was constructed, using job descriptions from LinkedIn.

In this part of the series, lets delve into how the data was prepared for regression modelling.

Regression is a statistical method used to investigate the relationship between one or more independent variables and a dependent variable. It is commonly used for predicting the value of a dependent variable based on the values of the independent variables.

To build the model the following packages were used:

import warnings
import itertools
import numpy as np
import pandas as pd
import scipy.stats as stats
from scipy.stats import chi2_contingency
from sklearn.model_selection import train_test_split

NumPy and Pandas were employed for various data processing tasks.

Warnings were suppressed using the warning module.

The scipy.stats module, imported as stats was used to compute F-statistic values.

The chi2_contingency function from the scipy.stats module was imported to compute chi-square values.

To begin, the dataset was imported into a pandas dataframe, which included the following columns:

• job_title: The title of the job.
• job_detail_url: Link to the job details.
• job_listed: Date when the job was listed.
• company_name: Name of the company offering the job.
• company_link: Link to the company’s LinkedIn page.
• company_location: Location of the company.
• job_details: Additional details about the job.
• seniority_level: Level of seniority of the job.
• employment_type: Type of employment (e.g., full-time, part-time, contract).
• job_function: Function or role of the job.
• industry: Industry of the company.
• source_url: Link of the job listing.
• company_city: City where the company is located.
• company_region: Region where the company is located.
• salaries_cleaned: Cleaned salary information.
• salary_lower_bound: Lower bound of the salary range.
• salary_upper_bound: Upper bound of the salary range.
• tech_stack_list: List of technologies or tools required for the job.
• job_category: Category of the job.

Job_title_two

Within the dataset, Job titles served as the title of the role being hired for, because there were several unique job titles, there was a need to recategorize them into broader groups to reduce the possibility of high cardinality within the model. Job titles were categorized into the following data careers:

• Data Engineering
• Insights Analyst
• Business Intelligence Analyst
• Data Scientist
• Data Management
• Data Governance
• Business Data Analyst
• Business Analyst
• Data Analyst
• Other Data Professional

In_out _london

Next Company Location was categorized into two groups.

• In Greater London
• Outside Greater London

Job functions values were categorized in a similar manner.

Tech_stack_group_count

The dataset included details specifying the technical tools and programming languages necessary for each job listing. Each job post required potential employees to know a different number of programming languages or tools. To categorize technical tools, the number of tools and languages needed for each role was noted, and based on the count, one of the following range values was assigned:

• 0–3
• 4–6
• 7–9
• 10+

Industry_category

Initially there were 553 unique industry values, the sheer number of these would contribute to high cardinality within the model, which could lead to overfitting, because of this industry values were recategorized into 35 broad industries.

Annual_salary

The goal of the model was to predict an annual salary value for the listed job roles. The dataset contained 2979 entries for the lower bound of salary ranges and 1768 entries for the upper bound of salary ranges. Considering that the number of salary values was less than 50% of the total dataset, annual salary values were imputed based on some parameters.

First, for job posts that had salary range values present, either higher bound or lower bound, the mean salary range value was calculated and used as the annual salary value.

condition = (jobs_main['salary_upper_bound'].isnull()) & (jobs_main['salary_lower_bound'] is not None)

jobs_main.loc[condition, 'salary_upper_bound'] = jobs_main.loc[condition, 'salary_lower_bound']

jobs_main['annual_salary'] = jobs_main[['salary_lower_bound', 'salary_upper_bound']].mean(axis = 1)


jobs_main['annual_salary'].count()

Following that, the count of salary values for each seniority level was examined. Some levels of seniority had no salary values present ,these were dropped.

Moving on to the remaining data, using seniority_level, job_title_two, in_out _london, and industry_category mean salary values were calculated, and imputed into annual salary based on the specified columns.

jobs_main['mean_salary'] = jobs_main.groupby(['seniority_level', 'job_title_two', 'in_out_london','industry_category'])['annual_salary'].transform('mean')

jobs_main['annual_salary'] = jobs_main['annual_salary'].fillna(jobs_main['mean_salary'])

Finally, null salary values and outliers were removed.

jobs_main = jobs_main[~jobs_main["annual_salary"].isna()]

jobs_main = jobs_main[jobs_main['annual_salary'] > 10000]
jobs_main = jobs_main[jobs_main['annual_salary'] < 80000]

Exploratory Data Analysis

Exploratory data analysis was performed on some columns, assessing category counts and changes in annual salary by category.

For instance, with level of seniority, the count for each seniority level was investigated.

jobs_main['seniority_level'].value_counts()

seniority_level
Mid-Senior level    3307
Entry level         2628
Associate            794
Name: count, dtype: int64

To perform regression, there should ideally be a relationship between the independent and dependent variables. Regression analysis seeks to model and understand the relationship between a dependent variable and one or more independent variables .

If there is no relationship between the independent and dependent variables, the regression analysis may not yield meaningful results. The strength and nature of the relationship between the variables are crucial for interpreting the coefficients and making predictions based on the regression model.

Consequently, the potential presence of a relationship between annual salary (dependent variable) and the independent variables was investigated.

Using a bar plot we visualized how mean annual salary differed per seniority level.

jobs_main.groupby('seniority_level')['annual_salary'].mean().sort_values().plot.barh();

The bar plot displayed differences in the mean annual salary values where seniority level was concerned. To further investigate a potential relationship, the F statistic value and an associated p-value were calculated.

The F statistic describes how well an independent variable explains variation in a dependent variable.

The p-value in statistics indicates how likely it is to see the observed data if there’s really no relationship or difference. A low p-value (typically less than 0.05) suggests that the results are unlikely under the null hypothesis, leading us to reject it and consider the observed effect as statistically significant.

If the F-statistic is high and the associated p-value is low (typically less than 0.05), it means the independent variables explain the dependent variable. This suggests a likely relationship between the dependent and independent variables.

Before calculating the F statistic we stated our hypothesis.

# null hypothesis: seniority level has no influence on annual salary
# alternate hypothesis: seniority level has influence on annualsalary

The F statistic and associated p-value were calculated as follows

f_statistic, p_value = stats.f_oneway(*[group[1]['annual_salary'] for group in jobs_main.groupby('seniority_level')])

and yielded these results:

f_statistic
1358.8740529657719


P_value
0.0

At approximately 1359 the F statistic value was considered sufficiently large and the p-value was less than the level of significance of 0.05, showing that the data provided significant evidence against the null hypothesis, suggesting a relationship between the dependent and independent variable.

The F statistic was also calculated for the following variables:

• Job title (job_title_two)
• Seniority_level
• Tech_stack_list ( tech_stack_count_group)
• Company_location (in_out_london)
• Industry_category

Up until this point, all the independent variables were categorical. Through external research, key information on average annual salary values for professionals in the data industry was discovered based on their job title and location. This information was added to the dataframe under the column name

• Average_uk_annual_salary

Information on the median salary values by location, in the United Kingdom from the Office for National statistics was added to the dataframe under the column name

• Median_salary_uk

Using correlation analysis, the possibility of relationship between Average_uk_annual_salary, Median_salary_uk and annual salary, was explored.

Correlation analysis is a statistical technique used to measure the strength and direction of the relationship between two numeric variables. It helps determine whether and to what extent changes in one variable are associated with changes in another variable. The primary measure in correlation analysis is the correlation coefficient which ranges from -1 to +1.

• The absolute value of the correlation coefficient indicates strength, with values closer to +1 indicating stronger relationships.
• The sign (positive or negative) of the correlation coefficient shows the direction:
• Positive correlation (r > 0): Both variables tend to increase together.
• Negative correlation (r < 0): One variable tends to decrease as the other increases.

continuous_vars = jobs_main[['annual_salary', 'average_uk_annual_salary']]
correlation_matrix = continuous_vars.corr()


                          annual_salary    average_uk_annual_salary
annual_salary               1.000000              0.543211
average_uk_annual_salary     0.543211             1.000000

After calculating the correlation between average_uk_annual_salary and annual salary (dependent variable), the coefficient obtained was 0.543211 indicating a positive relationship between these variables.

The correlation coefficient was calculated for

• Median_salary_uk and Annual salary
• Average_uk_annual_salary and Median_salary_uk

The correlation coefficient for Median_salary_uk and Annual salary was -0.02132, which pointed to a negligible negative relationship between these variables.

Multicollinearity occurs when two or more independent variables in a regression model are highly correlated with each other. In simpler terms, it means that some of the independent variables “move together” or are redundant in explaining the variation in the dependent variable.

To address multicollinearity one or more of the correlated variables from the model can be removed.

To check for multicollinearity between Median_salary_uk and Average_uk_annual_salary which were both independent variables, the correlation coefficient between the variables were calculated. The variables had a very weak negative relationship with a correlation coefficient of -0.076131.

As Median_salary_uk had a weak relationship with the annual_salary (dependent variable), it was not used in the final model.

Additional Features

Python, PowerBI, Microsoft Excel, SQL and Tableau are popular tools and programming languages needed to perform tasks in some data roles. Due to their popularity, they were evaluated to identify if they contributed to annual salary. New columns were created for each tool and language. For each job if any of these tools or languages were present, a value of 1 was assigned else 0.

Chi square values were calculated as a way to assess the presence of multicollinearity between categorical variables.

The chi-square test of independence helps us understand if there’s a strong relationship between two or more independent variables in a regression model. Chi-square results have a chi-square statistic and a p-value.

A low p-value (typically less than 0.05) means there’s likely a significant relationship between the variables. The chi-square statistic measures the extent of differences between observed and expected frequencies, with larger values indicating a stronger relationship.

If p-values are statistically significant and chi-square statistic values are large, it would point to a relationship between the variables, meaning there is a likelihood of multicollinearity.

After running a chi square test of independence for the independent variables in the dataset, many of the variables displayed relationships with other variables, this influenced the choice of regression used in building the model.

Test Train Split

After ensuring the columns were of the correct data type the dataset was split into X and y values.

X values consisted of the independent variables while the y value contained the dependent variable

X values were:

• average_uk_annual_salary: float64
• python: object
• excel: object
• sql: object
• tableau: object
• powerbi: object
• seniority_level: object
• Industry_category: object
• in_out_london: object
• tech_stack_count_group: category
• job_title_two: object

The y value, or target value was:

• annual_salary: float64

Finally the data was then split into test and training sets:

• X_train: Subset of features used for training the model.
• X_test: Subset of features used for testing the trained model.
• y_train: Corresponding target values for training.
• y_test: Corresponding target values for testing.

X_train, X_test, y_train, y_test = train_test_split(X,y ,
                                   random_state=42,  
                                   test_size=0.25,  
                                   shuffle=True)

A random seed value of 42 was set to ensure replicability of the dataset and 25% of data was used for testing, while 75% was used in training the model.

Conclusion

In the final part of this series we will explore how we built the prediction model.

Akosua Kissiedu