Introduction to exploratory data analysis in python

What is Exploratory Data Analysis

Think of Exploratory Data Analysis EDA as the thrilling detective phase in the data journey, bridging the gap between raw data collection and the mysteries of predictive modeling and deep analysis. It's where we roll up our sleeves, sift through the data, and unearth hidden patterns and stories waiting to be told. 

It involves delving deep into the data to uncover its underlying structure, extract important variables, identify anomalies and outliers, and test underlying assumptions.

Through visualizations, statistics, and intuitive insights, EDA helps analysts gain a comprehensive understanding of the dataset's patterns, relationships, and nuances. By fostering a thorough understanding of data, EDA ensures that subsequent analyses are grounded in reality and are more accurate and meaningful.

EDA processes

Here's an overview of the EDA process:

1. Gathering Your Data: Think of starting EDA like preparing for a big meal. First, you need to get all your ingredients, which means pulling together data from different places. Once you've got everything, you pop it into a software 'pot' to start cooking.
2. Quick Peek at Your Data: Now, take a quick look at what you've got. How big is your dataset? What kind of info is in there? Tools in pandas like head(), tail(), and info() are like little sneak peeks into your cooking pot.
3. Tidying Up Your Data: Just like prepping your veggies, you might need to clean up your data a bit. This means filling in missing bits, straightening out any wonky bits, and making sure everything is in the right format.
4. Understanding Your Data: Next, get a feel for your data's main features. Think of it as tasting the sauce and checking if it needs more salt or spices. Tools help you check things like the average, middle value, and how spread out the data is.
5. Visualizing Your Data: Time to make it look good! Using charts and graphs is like plating up your meal. You can see patterns and get a clear picture of what's going on.
6. Simplifying Complex Data: If you have a ton of ingredients and it's getting overwhelming, techniques like Principal Component Analysis are your best friend. They help you boil down the data to the essentials.
7. Principal Component Analysis (PCA): Imagine you're at a banquet with a plethora of dishes. If you had to summarize the meal with just a few key dishes, which would you choose? PCA operates on a similar principle with data. When faced with numerous variables, it's challenging to discern patterns. PCA simplifies this by creating 'principal components', which are like the key dishes that encapsulate the essence of the data. These components combine the original variables in a way that highlights their core differences and patterns, offering a streamlined yet comprehensive view of the data. Think of it as capturing the banquet's spirit with just a few dishes.
8. Checking Relationships: Imagine asking, "Do peas go well with carrots?" This step looks at how different parts of your data relate to each other.
9. Dealing with Categories: Got categories? No problem! Just like deciding whether to dice or slice, you'll decide how to handle and convert these for easy use.
10. Making New Features: Sometimes, you realize you need a pinch of something extra. Based on what you've observed, you might create new data features to capture things better.
11. Remember, EDA is Flexible: Lastly, remember that EDA is a bit like freestyle cooking. You might go back and forth, adding a bit here, tweaking a bit there, until it's just right!

EDA Example

Lets dive into a practical EDA example using the Spotify 2023 dataset. The dataset is a detailed compilation of 2023's top hits on Spotify, it sheds light on song attributes, cross-platform presence, and more.

We first load the necessary libraries and upload our dataset

import matplotlib.pyplot as plt
import pandas as pd

data = pd.read_csv("spotify2023.csv", encoding = "Latin-1")

To understand the key characteristics of the dataset, we use the command

data.info()

This allows us to view the number of rows and columns, column names, column count, and data types

<class 'pandas.core.frame.DataFrame'>
Int64Index: 817 entries, 0 to 952
Data columns (total 24 columns):
#   Column                Non-Null Count  Dtype
---  ------                --------------  -----
0   track_name            817 non-null    object
1   artist(s)_name        817 non-null    object
2   artist_count          817 non-null    int64
3   released_year         817 non-null    int64
4   released_month        817 non-null    int64
5   released_day          817 non-null    int64
6   in_spotify_playlists  817 non-null    int64
7   in_spotify_charts     817 non-null    int64
8   streams               817 non-null    object
9   in_apple_playlists    817 non-null    int64
10  in_apple_charts       817 non-null    int64
11  in_deezer_playlists   817 non-null    object
12  in_deezer_charts      817 non-null    int64
13  in_shazam_charts      817 non-null    object
14  bpm                   817 non-null    int64
15  key                   817 non-null    object
16  mode                  817 non-null    object
17  danceability_%        817 non-null    int64
18  valence_%             817 non-null    int64
19  energy_%              817 non-null    int64
20  acousticness_%        817 non-null    int64
21  instrumentalness_%    817 non-null    int64
22  liveness_%            817 non-null    int64
23  speechiness_%         817 non-null    int64
dtypes: int64(17), object(7)
memory usage: 159.6+ KB

data.head()

+-------------------------------------------+-----------------+--------------+---------------+----------------+--------------+---------------------+------------------+----------+-------------------+-----+-----+-----+-------+---------------+----------+--------+--------------+-------------------+-----------+------------+

|               track_name                  | artist(s)_name  | artist_count | released_year | released_month | released_day | in_spotify_playlists | in_spotify_charts| streams  | in_apple_playlists| ... | bpm | key | mode  | danceability_% | valence_%| energy_%| acousticness_%| instrumentalness_%| liveness_%|speechiness_%|

+-------------------------------------------+-----------------+--------------+---------------+----------------+--------------+---------------------+------------------+----------+-------------------+-----+-----+-----+-------+---------------+----------+--------+--------------+-------------------+-----------+------------+

| Seven (feat. Latto) (Explicit Ver.)       | Latto, Jung Kook|      2       |      2023     |       7        |      14     |         553          |       147        | 141381703|        43         | ... | 125 |  B  | Major |       80       |    89    |   83    |      31       |         0          |     8     |      4      |

| LALA                                      | Myke Towers     |      1       |      2023     |       3        |      23     |        1474          |        48        | 133716286|        48         | ... |  92 | C#  | Major |       71       |    61    |   74    |       7       |         0          |    10     |      4      |

| vampire                                   | Olivia Rodrigo  |      1       |      2023     |       6        |      30     |        1397          |       113        | 140003974|        94         | ... | 138 |  F  | Major |       51       |    32    |   53    |      17       |         0          |    31     |      6      |

| Cruel Summer                              | Taylor Swift    |      1       |      2019     |       8        |      23     |        7858          |       100        | 800840817|       116         | ... | 170 |  A  | Major |       55       |    58    |   72    |      11       |         0          |    11     |     15      |

| WHERE SHE GOES                            | Bad Bunny       |      1       |      2023     |       5        |      18     |        3133          |        50        | 303236322|        84         | ... | 144 |  A  | Minor |       65       |    23    |   80    |      14       |        63          |    11     |      6      |

+-------------------------------------------+-----------------+--------------+---------------+----------------+--------------+---------------------+------------------+----------+-------------------+-----+-----+-----+-------+---------------+----------+--------+--------------+-------------------+-----------+------------+

Let’s explore the categorical variables in our dataset, our data set has many songs and some artists may have multiple songs, let us count the unique artists within the data set: 

unique_artists = data["artist(s)_name"].unique()
len(unique_artists)

571

We have 571 unique artists in the data set.

As some artists may have multiple songs, we can find out the artists with the highest number of songs 

data["artist(s)_name"].value_counts().head(10)

Taylor Swift        29
SZA                 17
Bad Bunny           16
The Weeknd          15
Kendrick Lamar      12
Harry Styles        12
Morgan Wallen        9
Feid                 8
Drake, 21 Savage     7
BTS                  7
Name: artist(s)_name, dtype: int64

The query shows us the top 10 artists with the number of songs they have. Taylor Swift has the highest number of songs.

Our dataset has numeric variables too, summary statistics can give us a clearer understanding of our numeric data. Summary statistics provide a quick overview of the main characteristics of a dataset. They include measures like mean, median, mode, standard deviation, and range, offering insights into the distribution, central tendency, and variability of the data. These statistics help in understanding and interpreting data patterns efficiently.

The number of streams for each song on the Spotify app can be analyzed using the df.describe() method. This will provide insights into various statistics, including the average and median number of song streams.

num_of_streams.describe()

count    8.160000e+02
mean     4.689858e+08
std      5.231267e+08
min      2.762000e+03
25%      1.342848e+08
50%      2.638368e+08
75%      6.011986e+08
max      3.562544e+09
dtype: float64

While descriptive statistics offer insights into our data, visual representations give a more comprehensive understanding of its context. 

A box plot provides a visual summary of data, displaying the median, quartiles, and any outliers. This graphical representation offers insights into the data's central tendency and variability. Let's display the total stream values using a boxplot to gain insights.

On the boxplot, the orange line signifies the median. Numerous outliers are evident in the dataset.

We can visualize our data with matplotlib. Matplotlib is a leading Python library for creating static, animated, and interactive visualizations. Let us use matplotlib to visualize the number of streams Taylor Swift's songs have.

First, we need to filter our dataset to isolate all songs performed by Taylor Swift.

Taylor_swift_songs = data[data["artist(s)_name"] == "Taylor Swift"]

Next, we'll generate a bar chart displaying the songs and their corresponding stream counts.

plt.bar(Taylor_swift_songs['track_name'], Taylor_swift_songs['streams']);
plt.xticks(rotation=90, ha='center')
plt.xlabel('Taylor Swift Songs')
plt.ylabel('Number of Streams')
plt.title('Taylor Swift Song Streams')
y_formatter = ScalarFormatter(useOffset=False, useMathText=False)
y_formatter.set_scientific(False)
plt.gca().yaxis.set_major_formatter(y_formatter)

From the graph Blank Space has the highest number of streams and Better Than Revenge.

Another method to analyze our dataset is by determining if there's a connection between variables. This can be gauged by calculating the correlation between these variables. The correlation coefficient measures the strength and direction of this relationship. The coefficient ranges from -1 to 1: a value of 0 indicates no relationship, values between 0 and -1 represent a negative relationship, and values between 0 and 1 signify a positive relationship.

In our dataset, there's a column labeled bpm that represents the beats per minute of each song. It would be interesting  to determine whether the bpm affects the song's stream count.

First we visualize the relationship using a scatter plot 

plt.xlabel('Beats per Minute')
plt.ylabel('Streams')
plt.title("BPM vs Price")
y_formatter = ScalarFormatter(useOffset=False, useMathText=False)
y_formatter.set_scientific(False)
plt.gca().yaxis.set_major_formatter(y_formatter)

The scatter on the plot suggests a random distribution of data points, potentially indicating no clear correlation between the variables.

Let's further explore this using the pandas dataframe correlation function

bpm_streams = data['bpm'].corr(data['streams'])
bpm_streams

-0.025693846505562057

From the output, the results suggest a negligible negative correlation between beats per minute and the number of streams an artist receives.

Conclusion

Exploratory Data Analysis (EDA) is much like reading a captivating novel. Each dataset tells a unique story, and with EDA, we get to uncover its secrets. As we delve deeper, we might refine our approach, revisit certain chapters, and gain even richer insights. It's a continuous journey of discovery and understanding

Concluding our introduction to Exploratory Data Analysis (EDA) in Python, we've seen that EDA is an essential first step in the data analysis journey. With Python's powerful tools at our disposal, we can uncover insights, identify patterns, and better understand our data. It's a foundation upon which further analysis and decision-making are built. As you move forward, keep in mind that each dataset holds its unique story waiting to be discovered. Happy analyzing!

Akosua Kissiedu