Introduction

Welcome to the final instalment of a four part series exploring how a salary prediction model was constructed, using job descriptions from LinkedIn.

In the final part of this series we will explore how we built a regression model to predict annual salary.

Regression is a statistical method used to investigate the relationship between one or more independent variables and a dependent variable. It is commonly used for predicting the value of a dependent variable based on the values of the independent variables.

To get to the final working model, separate models were built using Lasso Regression and XGBoost Regression.

The Least Absolute Shrinkage and Selection Operator; also Lasso was used to construct the initial model. Some essential components imported from the sklearn library included:

• r2_score: Imported for evaluating model performance.
• Pipeline: Employed for constructing the modelling pipeline.
• Lasso: Utilized as the regression model.
• ColumnTransformer: Used for feature transformation.
• GridSearchCV and KFold: Utilized for hyperparameter tuning and cross-validation.
• StandardScaler: Used for feature scaling.
• OneHotEncoder: Employed for categorical encoding.
• train_test_split: Employed for splitting the data into training and testing sets.
• mean_absolute_error: Utilized for performance evaluation.

The final model was created using XGBoost, a gradient boosting algorithm.

Gradient boosting is a step-by-step process of building many simple models, each one correcting the mistakes of the combined model from the previous step, leaving a powerful model that can make accurate predictions.

Gradient boosting is used for both regression and classification tasks.

The packages imported for regression using XGBoost were the same as those imported for Lasso regression, however, the model type Lasso, was replaced with XGBoost.

After the dataframe was split into testing and training data sets the first model, which was the baseline model was constructed.

Baseline Model

The baseline model serves as a basis for comparison. It is a simple model with few features which provides a single prediction. It provides predictions without any additional features.

For instance, a good marker of an annual salary value for a data analyst in a given industry would be the average salary for data analysts in the specified industry.

Based of this, we used the average of our annual salary values as the prediction for our baseline model.

mean_y_train = y_train.mean()
mean_y_train
45414.63305559859

With a baseline prediction of approximately £45415 before applying any complex predictive techniques, it was estimated that, on average, a data professionals average salary should be around £45415

Mean Absolute Error (MAE) is a metric used to measure how close the predictions are to the actual values on average, without considering the direction of the errors i.e. if the error term is negative or positive.

Calculating the Mean absolute error of the baseline model, the error term returned was:

y_baseline_prediction = [mean_y_train] * len(y_train)
mae_of_baseline_model = mean_absolute_error(y_train, y_baseline_prediction)
print("Baseline MAE:", (mae_of_baseline_model))


Baseline MAE: 12569.66404146221

This meant that the predicted value of £45415 was off by approximately £12570

Using advanced regression techniques (Lasso) an improved model was created with a lower mean absolute error value and the ability to generate better predictions.

Lasso Model — With arbitrary alpha value

Lasso regression is a type of regression that adds a penalty term to the ordinary least squares (OLS) objective function. It penalizes large coefficients using a regularization parameter. This penalty applied makes coefficients very small or even zero ensuring that only important features are selected which prevent overfitting.

Overfitting happens when a machine learning model learns too much from the training data, capturing noise or random fluctuations that aren’t really representative of the underlying pattern. As a result, the model performs well on the training data but doesn’t generalize well to new, unseen data. In simpler terms, it’s like memorizing answers instead of understanding concepts.

The regularization parameter helps control the balance between two objectives: fitting the training data well and keeping the model simple. It imposes a penalty on the ordinary least squares (OLS) objective function significantly reducing or setting coefficients to 0.

A higher regularization parameter leads to a simpler model which might not capture all the nuances in the data, however the model is less likely to overfit. On the other hand, a lower regularization parameter results in a model which can capture more complex relationships, but the model is more likely to overfit.

To create the model, the following steps were taken:

One-Hot Encoding

One-hot encoded the categorical independent variables ( One-hot encoding is a method used in machine learning to represent categorical variables as binary vectors (0,1) ).

categorical_transformer = Pipeline(steps=[
    ('onehot', OneHotEncoder(handle_unknown='ignore'))
])

Standard Scaling

Standardized numeric independent variables with the standard scaler ( Standard scaling adjusts numerical features in a dataset so they all have similar scales. This matters because some machine learning methods are sensitive to feature scales. If features have different scales, it can cause problems or errors in the results).

preprocessor = ColumnTransformer(
    transformers=[
        ('onehot', OneHotEncoder(handle_unknown='ignore'), categorical_column),  # Apply one-hot encoding to categorical columns
        ('scaler', StandardScaler(), numeric_column)  # Scale the numeric column
    ],
    remainder='passthrough'  # Pass through the remaining columns unchanged
)

The lasso model was initialised with a regularization parameter (alpha) value of 1

lasso_basemodel = Lasso(alpha=1.0, fit_intercept=True, random_state=42)

Processing steps and the model were put into a pipeline

model_A = Pipeline(steps=[('preprocessor', preprocessor), ('model', lasso_basemodel)])

And the model was fit to the data

model_A.fit(X_train, y_train)

The newly trained model was used on test data for predictions and the mean absolute error value of the predictions was calculated

y_pred_basemodel=model_A.predict(X_test)
mae_of_lasso_basemodel = mean_absolute_error(y_test, y_pred_basemodel)
Mae_of_lasso_basemodel


7668.764500097229

Comparing the mean absolute error value of the baseline model versus this model, the value reduced from an initial £12570 to approximately £7669. Assessing the mean absolute error value, this model already performed better however, there was no process behind picking the regularization parameter and there was no way to know if this regularization parameter was the best value or if it could change to create a better model.

From this standpoint, the next goal was to find a better regularization parameter or alpha value, which could deliver an even better model.

Lasso Model — Selecting alpha with hyperparameter tuning

Rather than randomly using, 1 as the alpha value, an optimal alpha value can be found using cross validation.

Cross validation iteratively uses data points in a given dataset for training and testing a model, such that each data point is used for both training and testing at least once.

Cross validation can be achieved using multiple packages from scikit-learn including GridSearchCV and KFold cross validation.

To train a model with the best alpha value GridSearchCV was used.

To perform GridSearchCV, first a range for likely alpha values between 10 exponent -4 and 10 exponent 10 was set. The range contained 50 numbers in total.

alphas = np.logspace(-4, 10, num=50)

This range was passed to a dictionary called param_grid (parameter grid).

GridSearchCV finds all combinations of all values in a parameter grid, and trains a model with the various combinations. It evaluates the model’s performance for each set of parameters used against a specified metric, in this case, the mean absolute error. GridSearchCV, returns the hyperparameters of the iteration with the highest metric score as the optimal hyperparameter.

In this case the only parameter was alpha, so after a number of iterations as specified by the cv value, GridSearchCV returned the alpha of the model with the best neg mean absolute error value. The neg mean absolute error value is a negation of the mean absolute error value.

With a neg mean absolute error value of approximately -7561 this model performed better than the initial model with the arbitrary alpha value.

Finding optimal alpha

Finding optimal alpha value

The final Lasso model was trained and employed for predictions using the obtained alpha value. The mean absolute error value for this model was approximately £7661.

In comparison to the model without the tuned alpha value with a mean absolute error value of £7669, the performance of this model showed a very negligible difference.

Given that predictions deviated by more than £7000, there was a need to enhance the model.

XGBoost Regression

Aside from Lasso, there are various other models which are used for regression analysis. XGBoost is one of these models.

XGBoost stands for Extreme Gradient Boosting. It operates on the concept of boosting, a technique where multiple weak decision trees are combined to create a strong learner. In XGBoost, each subsequent tree learns from the mistakes of the previous one, focusing on the data points that were poorly predicted. This iterative process gradually improves the model’s accuracy.

Imagine you’re trying to solve a puzzle, but you’re not very good at it. So, you ask for help from a bunch of friends, each of whom is good at solving a different part of the puzzle. You all work together to solve the puzzle, and each time you make a mistake, you learn from it and try to do better next time.

XGBoost works like this. It’s like having a team of friends (called decision trees) who help you make predictions. But instead of solving a puzzle, you’re trying to predict something, like the price of a house or whether an email is spam.

XGBoost starts with a simple model, just like you starting with your initial attempt at solving the puzzle. Then, it looks at where that model made mistakes and tries to fix them with the next model. It keeps doing this over and over, each time getting a little better at making predictions.

The “gradient descent” part is like figuring out which direction to adjust your strategy to make the fewest mistakes. It’s like taking small steps in the right direction, guided by how wrong your predictions were before.

An XGBoost regression model was fit to the data, without any additional parameters.

XGBoost Regression

XGBoost Regression

Similar to the Lasso model, the categorical variables were one-hot encoded.

Immediately there was a significant reduction in the mean absolute error value dropping from approximately £7661 in the final Lasso model to approximately £1724 in this model.

R-squared helps us understand how well our model captures the relationship between the variables. The closer it is to 1, the better the model fits the data. The model further had a score above 0.9 for its r2.

A smaller mean absolute error value and a high r2 showed that this model’s predictions were quite close to the true value.

In an attempt to further improve upon the result, parameters were introduced. Like the Lasso model’s alpha, XGBoost parameters are settings or configurations that can be adjusted to change how a model learns from data and makes predictions.

GridSearchCV was applied for cross-validation to find the optimal values, which were then utilized in the final XGBoost model.

The parameters used were as follows:

• subsample: Trains each tree on 90% of the data.
• colsample_bytree: Trains each tree on 50% of features.
• learning_rate: Controls the step size (set to 0.5).
• max_depth: Limits tree depth to 5.
• gamma: No minimum requirement for split.
• reg_lambda: Penalizes large coefficients to prevent overfitting. A higher reg_lambda increases regularization strength, while a value of 1.0 means large coefficients were penalised aggressively.

XGBoost regression with parameters

XGBoost regression with parameters

The mean absolute error value for this model was approximately £1840, slightly higher than the model without explicitly tuned parameters. Its r2 was also above 90% at 0.92.

The difference in performance between the XGBoost model with tuned parameters and the one without was negligible.

With these results we constructed the final interactive model where data professionals in the specified fields can generate an estimated annual salary value, based on the selections they make.

This brings us to the end of this series.

Akosua Kissiedu