Introduction
Linear regression is a powerful statistical method that helps us understand and quantify the relationship between two or more variables.It is a method that helps us see and measure how different things are related to each other. For example, it can show us how changes in one thing, like hours spent studying, might affect another, like test scores.
With linear regression, one can analyze past data points, and draw a line of best fit through them, offering a clear visual representation of trends and patterns.
This line isn’t just a graphical representation; it's the key to making informed predictions. Whether it's forecasting sales based on advertising spend or estimating the impact of temperature on crop yield, linear regression turns historical data into a guide for future decisions.
Let's start exploring how, using Python, we can use linear regression to make predictions and understand data better.
Example
Our dataset is a Spotify dataset which contains information on the most streamed songs on Spotify. It provides insights into each song's attributes, popularity, and presence on various music platforms.
Our goal is to analyze if the number of Spotify playlists (in_spotify_playlists) a song appears on has any relationship with the number of times it is streamed (streams).
The first step is to clean our dataset. Find a walkthrough on how this dataset was cleaned (here) (insert link)
Data Analysis
Scatter Plot
After we have cleaned our data we first need to identify if there is a linear relationship between our selected variables in_spotify_playlists and streams. A scatter plot and correlation coefficient value can be used to identify the presence of a linear relationship between variables.
A scatter plot is a type of graph that shows how two variables are related to each other. It does this by plotting data points on a two-dimensional chart, where each axis represents one of the variables. For example, if you were looking at the relationship between the number of hours studied and test scores, the horizontal axis might represent study hours and the vertical axis the test scores. Each point on the graph represents a pair of values from your data set. This visual representation helps us see if there's a pattern or relationship between the two variables.
On the other hand, a correlation coefficient is a numerical value that quantifies the strength and direction of the relationship between two variables. It ranges from -1 to 1. A value close to 1 indicates a strong positive relationship, meaning as one variable increases, the other tends to increase as well. A value close to -1 indicates a strong negative relationship, which means as one variable increases, the other tends to decrease. A value around 0 suggests that there is no linear relationship between the variables. This coefficient gives us a concise and quantitative way to assess the relationship we might have already visualized in the scatter plot.
Plotting both variables on the scatter plot, we observe a positive linear relationship
plt.scatter(data['in_spotify_playlists'], data['streams'], color = "b")
plt.xlabel('in_spotify_playlists')
plt.ylabel('streams')
plt.title("Playlists vs Streams")
which is further supported by the correlation coefficient value
playlist_streams = data['in_spotify_playlists'].corr(data['streams'])
playlist_streams
0.7804038503763728
In our project, we aim to build a model that predicts how many times a song will be streamed on Spotify, based on how many Spotify playlists it appears in. To achieve accurate predictions, it's essential to start with clean and reliable data. One key aspect of ensuring our data is dependable involves checking for outliers in our selected variables.
Outliers are data points that are significantly different from most other points. They can skew our analysis and affect the accuracy of our predictions.
To begin, we'll focus on the 'streams' column, which represents the number of times songs have been streamed. To understand this data better, we will use two main visualizations: the histogram and a boxplot.
plt.hist(data['streams'])
plt.xlabel('streams')
plt.ylabel('frequency')
plt.title("Streams")
plt.boxplot(data['streams'], vert=False)
plt.xlabel('streams')
From the histogram and boxplot we can see our data has some outliers. The outliers on the boxplot are denoted by the small circles, and by the skew on the histogram. To ensure accurate results, we need to remove these outliers by dropping some of the values in our dataset.
We use quantiles to achieve this. Think of quantiles like checkpoints in a race that divide the track into equal parts. In data, these checkpoints help us see how values are spread out. For example, if you line up all the data points from the smallest to the largest, the 'middle' point is a quantile called the median. Now, imagine dividing the data into four equal parts - these are called quartiles. The middle two quartiles cover the 'typical' range of the data.
Sometimes, we have data points that are very different from the rest, like a runner far ahead or behind in a race. These are called outliers. To find and remove these outliers, we use a method involving the quartiles. We look at the range of the middle two quartiles and set up 'fences' or limits around them. Anything lying outside these limits is an outlier. By removing these outliers, we make sure they don't mislead us when we analyze the data or make predictions
low, high = data['streams'].quantile([0.1, 0.8])
mask_aa = data[data['streams'].between(low, high)]
After removing outliers:
Comparing this to the first boxplot we see no circles and a reduced skew on the histogram. We will do the same for our ‘in_spotify_playlists’ column
Before removing outliers:
plt.hist(data['in_spotify_playlists'])
plt.xlabel('in_spotify_playlists')
plt.ylabel('frequency')
plt.boxplot(data['in_spotify_playlists'], vert=False)
plt.xlabel('in_spotify_playlists')
low, high = data['in_spotify_playlists'].quantile([0.2, 0.8])
mask_ab = data[data['in_spotify_playlists'].between(low, high)]
After removing outliers:
Scikit-learn
In our journey in data analytics with python, we'll be using several libraries and packages, each with its unique role in data analysis and model building. One of the most important packages we'll be using is Scikit-learn, often referred to as sklearn.
For our specific task, we'll be utilizing Scikit-learn's linear regression functionality. This will help us create a model that can predict the number of streams a song might receive based on various input features, like the number of playlists it's in. Linear regression in Scikit-learn is designed to be straightforward yet powerful, allowing us to find the relationship between our input and output variables with just a few lines of code.
Additionally, we'll also be using Scikit-learn's Mean Absolute Error (MAE) method.
Dataset Preparation
When building a predictive model, it's crucial to have two separate sets of data: a training dataset and a test dataset. The training dataset is used to build and 'train' the model, teaching it to understand patterns and relationships in the data. The test dataset, on the other hand, is used to evaluate how well the model performs on data it hasn't seen before.
In our project, we're taking a unique approach by using data from a single artist as our training dataset. This strategy allows us to build a model based on a focused subset of data. For this purpose, we've chosen songs by Taylor Swift as our training dataset. The idea is to train our model exclusively on her songs, examining the relationship between the number of playlists her songs are featured in and the number of streams they receive.
After the model is trained using Taylor Swift's songs, we'll then test its effectiveness on a broader dataset that includes songs from various other artists. This step is crucial to evaluate whether the insights and patterns learned from Taylor Swift's music can be generalized to predict streaming numbers for a wider range of artists. If our model, trained on this specific artist's data, shows accurate predictions on the diverse test dataset, it indicates that our model has strong generalizability and can be reliably used for broader music streaming predictions.
swift_df = regression_df[regression_df['artist(s)_name']=='Taylor Swift']In predictive modeling, for our training data, we split our data into two main parts. The first part is the 'target vector'. This is what we want to predict, like how much a house might cost or which song someone will like. We often call this target vector y_train. The second part is the 'feature matrix', which is all the information we think might help us make that prediction. In machine learning, we often call this feature matrix X_train. For example, if we're trying to figure out what kind of music someone likes, we might just use their Spotify playlist as our feature matrix. This division of data into target vector and feature matrix is really important because it helps us organize our model and make better predictions.
Feature Matrix:
X_train = pd.DataFrame(swift_df['in_spotify_playlists'])Target Vector
y_train = pd.DataFrame(swift_df['streams'])
Baseline Model
A baseline model in predictive modeling serves as a starting point for comparison. It's a simple model with not many features, and it typically predicts just one outcome, regardless of the different values in the feature matrix. Think of it as a basic guesser. Its performance is measured and summarized in a single value. This value is very important because it helps us understand how well our more complex models are doing. We compare this value with the performance of advanced models, like linear models, to see if they are doing any better. If our complex model performs better than this simple baseline, it means our predictions are likely more accurate and that the added complexity is worthwhile. The baseline model sets the standard for what we need to beat with our more sophisticated methods.
In linear regression, a baseline model could be as simple as predicting the mean or median of the target variable for all observations, regardless of the input features. For instance, if you're trying to predict house prices and the average house price in your dataset is $300,000, your baseline model might predict $300,000 for every house, irrespective of its size, location, or other characteristics.
Our baseline prediction will be the average number of streams.
y_mean_train = y_train.mean()
y_mean_trainstreams 3.148633e+08
dtype: float64
Our baseline prediction value is 314,863,300. This means that, initially, before applying more complex predictive techniques, we are estimating that, on average, a song, would have around 314.9 million streams.
Exploring this visually,
plt.plot(X_train, y_baseline_prediction, color="orange", label="Baseline_Prediction")
plt.scatter(X_train, y_train)
plt.xlabel("in_spotify_playlists")
plt.ylabel("streams")
plt.title("Streams vs. Playlists")
plt.legend();
In our situation, the baseline model, which predicts an average of around 314.9 million streams, doesn't perform very well because it doesn't consider the different trends or changes in the variables we're studying. But that's okay – a baseline model isn't expected to be highly accurate. It's just a starting point. To understand exactly how our baseline model is performing, we're going to use a mathematical method called the 'mean absolute error' as a performance metric. This will give us a clear number that shows how far off our baseline predictions are from the actual values. This number will help us see how much improvement we need from our more advanced models.
Mean absolute Error (MAE)
Mean Absolute Error (MAE) is a straightforward way to measure the accuracy of a model in predicting quantitative outcomes.
It's a metric used in statistics and machine learning to evaluate how close predictions are to
the actual outcomes.
Absolute Error: For each prediction, you calculate the "absolute error,"
which is the absolute difference between the predicted value and the actual value.
For example, if you predict that a house will sell for $200,000 and it sells for $180,000,
the absolute error is $20,000.
Mean (Average): You then take the average of these absolute errors across all predictions.
This average is what we call the Mean Absolute Error.
Interpretation: The MAE gives you a number that represents how off your predictions typically are.
A smaller MAE suggests that your model is more accurate, while a larger MAE indicates less accuracy.
Calculating the MAE of our baseline model we have:
y_baseline_prediction = [y_mean_train] *len(y_train)
mae_of_baseline_model = mean_absolute_error(y_train, y_baseline_prediction)print("Mean number of streams", round(y_mean_train, 2))
print("Baseline MAE:", round(mae_of_baseline_model, 2))Mean number of streams streams 3.148633e+08
dtype: float64
Baseline MAE: 128212452.69
If we use our baseline prediction of 314,863,300 streams for a song, we find that we're wrong by about 128,212,453 streams. Our aim now is to create a model that makes smaller errors in prediction.
Model Building
Using sklearn we we build our model:
model = LinearRegression()
In building the model what we are doing is fitting the model to the data.
Fitting the model to data
Fitting a model means adjusting the parameters of the model so that it best captures the relationship between the input and output of the training data. The model "learns" from this data by finding patterns or trends.
Predicting on training data
Now, we're going to make predictions using our training dataset and see how they compare to our baseline model.
y_predictions_on_training_data = model.predict(X_train)mae_training = mean_absolute_error(y_train, y_predictions_on_training_data)
print("Training MAE:", round(mae_training, 2))Training MAE: 29134603.71
Our model's mean absolute error (MAE) for predicting Taylor Swift's song streams on the training data is about 29,134,604 streams. This shows an improvement from being off by over 100 million streams to just 29 million, indicating that our model is performing better than the baseline, specifically for Taylor Swift's songs
Predicting on test data
Next, we're going to test our model with songs from different artists, not just Taylor Swift. This will help us see if our model is good at predicting stream counts for all kinds of music, or if it only works well for Taylor Swift's songs. This step is important to check if our model is really effective and can be used for a wide range of artists and music styles.
test_df = regression_df[regression_df['in_spotify_playlists'] != 'Taylor Swift']X_test = pd.DataFrame(test_df['in_spotify_playlists'])y_test = pd.DataFrame(test_df['streams'])y_predictions_on_test_data = model.predict(X_test)mae_training = mean_absolute_error(y_test, y_predictions_on_test_data)
print("Training MAE:", round(mae_training, 2))Training MAE: 118268407.82
The mean absolute error (MAE) from the test dataset was 118,268,407.82 streams, which is an improvement compared to the baseline MAE of 128,212,452.69 streams.
However, this test MAE is still much higher than the 29,134,604 streams MAE we observed in our training data for Taylor Swift's songs.
This difference in performance suggests that while our model was quite effective with Taylor Swift's music, its accuracy decreases when applied to a broader range of artists. Despite performing better than the baseline, the significant error in the test dataset indicates a lack of generalizability.
Our model, as currently configured, seems to be more suited to the patterns in Taylor Swift's songs and doesn't adapt as well to different musical styles or other artists' data. To make our model more universally applicable, it will require adjustments to better capture and learn from the diversity in music across various artists and genres.
Conclusion
In summary, our experience in creating a model to predict music streaming numbers shows how important it is to keep improving and making changes. Our model worked well for Taylor Swift's songs, but it wasn't as good with other artists, reminding us that building a model is a process that doesn't stop. We need to keep tweaking and testing our model, especially with different kinds of music. This step-by-step improvement is key to making our model better and more useful for a wider range of songs and artists. So, continuous improvement is a big part of making a successful model in data science.
