Data Analysis with Pandas: Manipulating and Visualizing Data

Written by Yasin TANIŞ

Originally published on Medium

Illustration of tables and charts for data analysis with pandas

Introduction

Data analysis has become an integral part of decision-making. Python stands out in this field thanks to its powerful libraries for analysing and manipulating data. One of them, pandas, is indispensable for data scientists because it is flexible and easy to use. In this article we look at the core features of pandas and how it is used in data analysis, with practical examples on real or synthetic data.

DataFrame and Series

Pandas works with two main data structures. A DataFrame is a two-dimensional structure of rows and columns. A Series is one-dimensional and represents a single column of data.

import pandas as pd

# Creating a simple DataFrame
data = {
    'name': ['Alice', 'Bob', 'Charlie'],
    'age': [25, 30, 35],
    'city': ['New York', 'Los Angeles', 'Chicago']
}
df = pd.DataFrame(data)
print(df)
Code output: a DataFrame and a Series printed in the console

Creating DataFrames

There are many ways to create a DataFrame. You can load data from sources such as CSV, Excel and SQL and turn it into a DataFrame.

# Loading data from a CSV file
df = pd.read_csv('data.csv')

# Loading data from an Excel file
df = pd.read_excel('data.xlsx')

# Loading data from an SQL database
import sqlite3
conn = sqlite3.connect('database.db')
df = pd.read_sql_query('SELECT * FROM table', conn)

Loading and saving data

Pandas makes loading and saving data simple. You can read and process files in many different formats.

# Saving data
df.to_csv('new_data.csv', index=False)
df.to_excel('new_data.xlsx', index=False)

Cleaning and preparing data

Cleaning and preparation are critical steps in data analysis. Pandas makes it easy to handle missing values, convert data types and remove data you do not need.

# Checking for missing values
print(df.isnull().sum())

# Filling missing values
df.fillna(0, inplace=True)

# Converting data types
df['date'] = pd.to_datetime(df['date'])
Code output: missing-value counts for each column, all zero

Here we check whether the DataFrame has missing values. The output shows that none of the columns has any.

Statistical summaries

Pandas offers many tools for summarizing data quickly. Basic statistics and summaries are a single call away.

# Basic statistical summaries
print(df.describe())

# Summarizing specific columns
print(df['age'].mean())
print(df['age'].std())
Code output: statistical summary of the age column

In the summary the mean age is 30 with a standard deviation of 5; the minimum and maximum ages are 25 and 35. This gives a quick picture of the age distribution in the data.

Grouping and summarizing

Grouping and summarizing is one of pandas' strengths. The groupby and pivot_table functions make it straightforward.

# Grouping and summarizing
group = df.groupby('city')['age'].mean()
print(group)

# Creating a pivot table
pivot = df.pivot_table(values='age', index='city', columns='gender', aggfunc='mean')
print(pivot)
Code output: mean age grouped by city
Code output: pivot table of mean age by city

Grouping by city gives the average age for each one: Chicago (35), Los Angeles (30) and New York (25). The pivot table gives a similar summary broken down by gender; our example has no gender data, so those cells show NaN.

Visualizing data with pandas

Pandas has simple but effective plotting tools. Functions such as plot, hist and boxplot produce charts quickly.

# Basic plots
df['age'].plot(kind='hist')
df.plot(kind='scatter', x='age', y='income')

Working with Seaborn and Matplotlib

You can also visualize pandas data with Seaborn and Matplotlib, which allow more customized and detailed charts.

import seaborn as sns
import matplotlib.pyplot as plt

# Visualizing with Seaborn
sns.boxplot(x='city', y='age', data=df)
plt.show()

Time series analysis

Pandas handles time series data very well, which makes trends and patterns easy to find.

# Handling time series data
df['date'] = pd.to_datetime(df['date'])
df.set_index('date', inplace=True)
print(df.resample('M').mean())
Code output: monthly mean age after resampling

Here we convert the date column to datetime and set it as the index. Resampling by month and taking the mean gives an average age of 30 for January 2023.

Merging and combining data

Pandas has many functions for merging and combining data sets, such as merge, join and concat.

# Merging two DataFrames
df1 = pd.DataFrame({'name': ['Alice', 'Bob'], 'age': [25, 30]})
df2 = pd.DataFrame({'name': ['Alice', 'Bob'], 'city': ['New York', 'Los Angeles']})
merged_df = pd.merge(df1, df2, on='name')
print(merged_df)
Code output: two DataFrames merged on the name column

In this example we merge two DataFrames on the name column, combining each person's age and city.

Practical example: analysis with synthetic data

Creating the data

For a practical example we create synthetic data to show what pandas can do.

import pandas as pd 
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt

# Creating synthetic data
np.random.seed(0)
dates = pd.date_range('20230101', periods=100)
df = pd.DataFrame(np.random.randn(100, 4), index=dates, columns=list('ABCD'))
df['Category'] = np.random.choice(['Category1', 'Category2', 'Category3'], size=100)
print(df.head())
Code output: the first rows of a synthetic DataFrame

This creates a DataFrame of 100 rows with columns A, B, C and D and a categorical column Category. The values are random, and Category takes one of three values.

Analysis and visualization

With the synthetic data we run some basic analysis and draw charts.

# Statistical summary
print(df.describe())
Code output: statistical summary of the synthetic data

The summary gives a full overview of the data. The means of columns A, B, C and D are close to zero, as expected from a random normal distribution, and the standard deviations are close to 1, showing how widely the values spread around the mean.

# Grouping and summarizing
grouped = df.groupby('Category').mean()
print(grouped)
Code output: column means grouped by category

Grouping by Category shows the mean of each column for each category, which reveals how the categories differ.

# Visualizing data
df['A'].plot(kind='hist', title='Histogram of A')
plt.show()

sns.boxplot(x='Category', y='B', data=df)
plt.show()
Histogram of column A
Box plot of column B by category

The histogram shows how the values in column A are distributed; the box plot shows column B across the categories. Charts like these help you understand the distribution and spot outliers or patterns.

Conclusion

Pandas is a versatile and powerful tool for data analysis; its flexibility and ease of use make it indispensable for data scientists. In this article we covered its core features and how they fit into an analysis, and we showed practical use on synthetic data. With further practice and more advanced techniques, you can go deeper into data science and draw more insight from your data with pandas.

References

  • McKinney, W. (2010). Data Structures for Statistical Computing in Python. Proceedings of the 9th Python in Science Conference, 51–56.
  • McKinney, W. (2017). Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython (2nd ed.). O’Reilly Media.
  • The pandas development team. pandas documentation. pandas.pydata.org
  • NumPy documentation. numpy.org
  • Matplotlib documentation. matplotlib.org
  • Seaborn documentation. seaborn.pydata.org