Coder Legion - Recent posts tagged pandas

Building Decision Trees with Pandas and Scikit-learn

Fri, 14 Aug 2026 14:10:38 +0000

The Theory

So what's a Decision Tree Regressor and how does it work?
It's a regression model so it returns continuous numerical values. It gets a prediction by asking a series of yes/no questions about the chosen features, building a binary tree structure where each question splits the data further until it reaches a prediction at the leaf node.

It learns by choosing which questions to ask

So for a given feature, a "split" is a threshold value that divides the data into two groups, those above and those below.

So say we wanted to make predictions on house price and we have features like the number of rooms, the size of the land, etc. Then at each node the algorithm tries every possible split across every feature and picks the one that minimises the mean squared error (MSE) of the resulting groups i.e. the split that produces two groups where house prices within each group are as similar to each other as possible ( it sums the MSE from each set of two groups and picks the set with the lowest combined MSE).

Where,

\[\text{MSE} = \frac{1}{n} \sum_{i=1}^{n} (x_i - \bar{x})^2\]

example:

What value does the leaf node actually hold?

This is the key bit. During training, multiple houses end up at each leaf node. The predicted value stored at that leaf is simply the mean house price of all training houses that landed there.

How do we know if our model is any good?

A reasonable way to validate the model is to compare, on average, how close a prediction is to the actual value. We will use Mean Absolute Error (MAE).

\[MAE = \frac{1}{n} \sum_{i=1}^{n} |y_i - \hat{y}_i|\]

At this point you may be wondering why are we using MSE for training and MAE for validation. Can't we just pick one metric and use it for splitting the tree and validating it?

We use MSE for training because it's faster than MAE. During training, the tree has to calculate errors for millions of potential splits. Because MSE uses squared numbers, it unlocks a clever mathematical shortcut (variance reduction) that lets the computer calculate splits using a simple running tally.

\[\sum (y_i - \bar{y})^2 = \sum y_i^2 - \frac{(\sum y_i)^2}{n}\]

Look closely at the right side of the identity. There are no subtractions between individual data points and the mean. Instead, the entire formula is built out of only three components, The sum of all squared target values, the sum of all target values (which is then squared) and the count n.

The computer can keep a running tally of the sums, the squared sums, and the counts as it slides through split points. It never has to look back at individual data points to calculate the error for a group.

We validate with MAE because humans understand it: MSE leaves you with an error in "squared units" (like squared euros), which would probably raise a few eyebrows at your local bank.

Limitations to consider.

Decision Tree tends to overfit because it memorises the training data rather than learning general patterns. This may result in highly specific "rules" that only apply to historical data, causing it to struggle to generalise.

When you train a decision tree, it uses a greedy algorithm. This means at every single split, it only cares about making the absolute perfect decision for the data right in front of it at that exact moment. It doesn't look ahead, and it doesn't plan for the future.

If you don't stop it (by setting limits like max_depth), the tree will keep asking questions until it has separated every single unique case in your training set into its own tiny leaf node.

This is exactly why Ensemble Methods like XGBoost exist - it builds hundreds of shallow trees and combines them, which is far more robust. Think of a Decision Tree as the foundation concept that XGBoost is built on top of.

The Code

Learning path:

Explore the data and identify prediction target
Transform - Clean the data
Select features and Model type
Train the model - find patterns from the data
Predict and Evaluate the model
Optimise the model

Step 1: Load and Explore the data

Using Pandas to read from a csv file and return a pandas DataFrame.

import pandas as pd

housing_df = pd.read_csv('./datasets/housing_data.csv')

print(f"Statistical Summary \n {housing_df.describe()}")

print(f"Dataset Features \n {housing_df.columns}")

Output

Statistical Summary

Rooms Price ... Longtitude

Propertycount

count 13580.000000 1.358000e+04 ... 13580.000000 13580.000000

mean 2.937997 1.075684e+06 ... 144.995216 7454.417378

std 0.955748 6.393107e+05 ... 0.103916 4378.581772

min 1.000000 8.500000e+04 ... 144.431810 249.000000

25% 2.000000 6.500000e+05 ... 144.929600 4380.000000

50% 3.000000 9.030000e+05 ... 145.000100 6555.000000

75% 3.000000 1.330000e+06 ... 145.058305 10331.000000

max 10.000000 9.000000e+06 ... 145.526350 21650.000000

Output

Dataset Features

Index(['Suburb', 'Address', 'Rooms', 'Type', 'Price', 'Method', 'SellerG', 'Date', 'Distance', 'Postcode', 'Bedroom2', 'Bathroom', 'Car', 'Landsize', 'BuildingArea', 'YearBuilt', 'CouncilArea', 'Lattitude', 'Longtitude', 'Regionname', 'Propertycount'], dtype='object')

Step 2: Transform the data

(i) Remove any rows that have NaN (missing) value ( axis: 0 -> rows, 1 -> columns )

clean_housing_df = housing_df.dropna(axis=0)

(ii) Select the prediction target 'y' and the features 'X'.

y = clean_housing_df.Price
features = ['Rooms', 'Bathroom', 'Landsize', 'Longtitude', 'Lattitude']
X = clean_housing_df[features]

(iii) Split the data into a training set and validation set

from sklearn.model_selection import train_test_split

train_X, val_X, train_y, val_y = train_test_split(X, y, random_state=1)

Step 3: Select Model type and train.

(i) Scikit-learn Decision Tree Regressor ( random state = 1. If the split results in a tie then breaking is deterministic )

from sklearn.tree import DecisionTreeRegressor

housing_regressor_model = DecisionTreeRegressor(random_state=1)

(ii) Fit the model

housing_regressor_model.fit(train_X, train_y)

Step 4: Predict

predicted_home_prices = housing_regressor_model.predict(val_X)

print(predicted_home_prices)

Output

[ 503000. 1857000. 760000. ... 4200000. 800000. 1785000.]

Step 5: Validate

Using the Mean Absolute Error metric as discussed above.

from sklearn.metrics import mean_absolute_error

mae = mean_absolute_error(val_y, predicted_home_prices)
print(mae)

Output

251698.44673983214

Step 6: Optimise

How can we minimise the MAE?

By using a process known as Hyperparameter Optimisation .
We can tune our decision tree using the max_leaf_nodes parameter, which is passed to the model before training. This setting allows us to specify the upper limit of leaf nodes the model is allowed to create, helping us control the balance between underfitting (too few leaves) and overfitting (too many leaves).

Let see if we can find a value for max_leaf_nodes that minimise the MAE.

mae_scores = []
candidate_levels = [10, 100, 200, 400, 500, 1000, 5000]

for max_leafs in candidate_levels:
    # Initialize and fit the model with the current candidate size
    model = DecisionTreeRegressor(max_leaf_nodes=max_leafs, random_state=0)
    model.fit(train_X, train_y)
    
    # Predict and evaluate
    preds_val = model.predict(val_X)
    mae = mean_absolute_error(val_y, preds_val)
    
    # Store the result as a sublist: [error, node_count]
    mae_scores.append([mae, max_leafs])
    print(f"Max leaf nodes: {max_leafs} | MAE: {mae:.2f}")

# Find the sublist with the lowest MAE score
smallest_mae = min(mae_scores, key=lambda x: x[0])
print(f"\nThe optimal choice for max_leaf_nodes is: {smallest_mae[1]} (MAE: {smallest_mae[0]:.2f})")

Output

Max leaf nodes: 10 | MAE: 320570.21

Max leaf nodes: 100 | MAE: 256533.25

Max leaf nodes: 200 | MAE: 244453.42

Max leaf nodes: 400 | MAE: 243238.90

Max leaf nodes: 500 | MAE: 241920.26

Max leaf nodes: 1000 | MAE: 243230.84

Max leaf nodes: 5000 | MAE: 256050.02

The optimal choice for max_leaf_nodes is: 500 (MAE: 241920.26)

Trial by Fire: From Garbage Excel to Relational Graph with Python and Pandas

Mon, 27 Apr 2026 05:59:10 +0000

1. The Hook: Industrial Data Entropy

In standard academic theory, data sets are inherently clean. In the active reality of the industrial supply chain, obsolete ERPs continually export garbage arrays.

Receiving a flat Bill of Materials systematically exported from a legacy database immediately binds you to processing massive structural entropy: entirely void parameter cells, anomalous blank spacing hidden inside critical part numbers (e.g., " SN74LS00N "), unstandardized component manufacturer nomenclatures (inconsistently shifting between capitals and disparate acronyms like "ti"), and severe mixed data typing where strict numerals conflict natively with raw text variables.

Pushing this raw flat algorithmic entropy directly forward into a live relational matrix triggers a complete referential integrity failure. The process inherently requires deploying a clean staging environment.

2. The ETL Pipeline (Extract, Transform, Load)

Securing the database perimeter requires a rigorous intermediate layer. We employ Pandas, effectively operating as the strictly objective gatekeeper to our foundational PostgreSQL database.

Before a solitary piece of hardware telemetry reaches the backend schemas, the dedicated Python layer strictly enforces dataset standardization. We routinely execute hard operations securely using .str.strip(), .str.title(), and .fillna() to properly homogenize external manufacturer part numbers (MPNs), completely delete volatile residual spaces, and explicitly enforce explicit valid ENUM classifications replacing null lifecycle constraints natively prior to API linkage.

def load_and_clean_data(filepath: str) -> pd.DataFrame:
    df = pd.read_csv(filepath)
    
    # 1. Radical space elimination and uppercase standardization (Avoids hardcoding)
    df['Manufacturer_PN'] = df['Manufacturer_PN'].str.strip().str.upper()
    df['Manufacturer'] = df['Manufacturer'].str.strip().str.upper()
    
    # 2. Enforced data typing (Guaranteed numerical quantities)
    df['Quantity_per_Assembly'] = pd.to_numeric(
        df['Quantity_per_Assembly'].astype(str).str.strip(), 
        errors='coerce'
    ).fillna(1).astype(int)
    
    # 3. Containment of missing values
    df['Lifecycle'] = df['Lifecycle'].replace(r'^\s*$', np.nan, regex=True).fillna('Active')
    
    return df

3. The Graph Challenge (Shattering the Flat Table)

A Bill of Materials mathematically operates internally as a hierarchical tree structure natively, not a flat dimension stack.

In order to systematically populate the highly restrictive bom_lines array cleanly avoiding the database exclusive CHECK integrity blocker explicitly coded earlier, the python automation segregates and systematically separates the initial two-dimensional DataFrame sequentially into two independent logical grouping vectors.

Applying this distinct data segmentation strictly ensures the entire tree graphs properly:

Loop 1: Explicitly tying Final Parent Products (End_Product_SKU) entirely to specific internal Subassembly containers (Assembly_PN).
Loop 2: Structurally tying Internal Subassemblies (Assembly_PN) strictly natively downward to explicit Base Component items (Component_PN).

This absolute directional differentiation operates cleanly resolving integration faults natively satisfying the mutually exclusive table constraints enforced inside Supabase, correctly mimicking highly accurate factory layout trees algorithmically.

4. The Golden Rule: Ingestion Idempotency

Before a standard data module successfully reaches enterprise standard operation it strictly requires satisfying system Idempotency. This establishes directly that any localized operations engineer can forcefully execute the extraction module recursively infinite sequential times utilizing identical hardware array data completely strictly avoiding destroying structural records nor accidentally generating unconstrained graph hierarchy duplication.

Exploiting primitive INSERT API calls natively corrupts the database integrity by duplicating the graph edges. Instead, the implementation forcefully deploys the upsert() functions logically integrated seamlessly through the Supabase SDK backend architecture utilizing strictly declared Postgres UNIQUE keys firmly established systematically across the specific domains (sku directly crossing global product ranges, internal_pn binding isolated component IDs, and mpn natively locking foreign market parts respectively).

Executing automated telemetry pushes immediately executes cleanly verifying redundant array vectors explicitly reporting live structural compliance seamlessly across loop iterations:

--- Ingestando Products (Idempotente) ---
Upserted 3 products.
--- Ingestando Internal Parts (Idempotente) ---
Upserted 18 internal parts (assemblies & components).
--- Ingestando Manufacturer Parts Telemetry (Idempotente) ---
Upserted 16 manufacturer external parts.
--- Ingestando AML (Mapeo Interno -> Externo) ---
No new AML links to insert (Idempotent success).
--- Ingestando BOM Lines (Grafo Bidireccional Segregado) ---
Mapeando relaciones: Producto Final -> Subensamblaje...
Mapeando relaciones: Subensamblaje -> Componente...
El Grafo BOM está completamente sincronizado (Idempotent success).

[OK] Migración a entorno relacional completada al 100%.

The Sandbox: Execute it Yourself

In Operations Engineering, code is worth more than theory. I have prepared an interactive and secure environment (Zero Friction) for you to test this ETL pipeline.

You don't need to install anything or configure databases. The script will spin up an SQL engine in your browser's memory, ingest a corrupt CSV file, cleanse it using Pandas, and build the relational tree in milliseconds, demonstrating idempotency live.

Access the Interactive Google Colab Here

5. Closing & CTA (Towards the AI Brain)

The system structural data logic completely establishes pure stability. We strictly transported fundamentally corrupt, stagnant standard tabular Excel files completely across explicit programming loops directly natively injecting refined node values natively filling a vastly superior PostgreSQL relational database firmly.

The core foundations remain entirely structural and secure. We possess a natively stable IEC 62402 compliant relational architecture structurally fused directly with a robust Python processing ETL node securely operating live. Crossing into the incoming operational module (Block 3), we firmly awaken complex systematic intelligence cleanly processing logic directly on hardware trees. We deploy strictly scalable Agentic AI parameters (LangChain / CrewAI) comprehensively actively connected inside the unified database constantly autonomously identifying internal financial stress resulting dynamically processing external global EOL silicon alerts. Subscribe consistently witnessing the Agentic Radar launch sequence.

Mastering Pandas — Part 4: Data Visualization with Matplotlib & Seaborn

Thu, 02 Apr 2026 19:32:22 +0000

From Clean Data to Clear Insight

In Part 3, you learned how to clean messy data and combine multiple sources into one unified DataFrame. Now that your data is ready, the next step is to communicate what's inside it — and nothing communicates data faster or more clearly than a chart.

This article covers the two most important visualization libraries in Python: Matplotlib, the core engine that powers all plotting, and Seaborn, a higher-level library built on top of it that produces beautiful statistical charts with minimal code. By the end, you'll know which tool to reach for depending on what you need to show.

We'll use this sample dataset throughout:

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

data = {
    'Country':         ['China', 'India', 'USA', 'Brazil', 'UK'],
    'Continent':       ['Asia', 'Asia', 'North America', 'South America', 'Europe'],
    '2022 Population': [1412000000, 1380000000, 331000000, 214000000, 67000000],
    'Area (km2)':      [9597000, 3287000, 9834000, 8516000, 243000]
}
df = pd.DataFrame(data)

Part 1 — Matplotlib: The Engine

What is Matplotlib?

Matplotlib is the foundational plotting library in Python. Every chart you create in Python — whether through Pandas, Seaborn, or directly — eventually goes through Matplotlib to render the final output. Think of it as the engine: it handles the low-level work of drawing lines, shapes, colors, axes, and text.

The submodule you'll use in practice is matplotlib.pyplot, which gives you a clean, high-level interface without managing every visual detail manually:

import matplotlib.pyplot as plt

Matplotlib works across three styles, each suited to a different situation:

Style	When to use
`df.plot(kind="bar", y="col")`	Your data is already in a DataFrame — quickest way
`plt.plot(xs, ys)`	Working with raw lists, arrays, or math functions
`fig, ax = plt.subplots()`	Multiple subplots or full layout control

All three ultimately use Matplotlib to render the chart. plt.show() is always called at the end to display it.

Global Settings with `plt.rcParams`

plt.rcParams is a global settings dictionary. Any value you set here applies to every chart created afterward in the session — so you set it once at the top of your notebook and never repeat it.

plt.rcParams['figure.figsize'] = (12, 5)   # default width and height
plt.rcParams['font.size']      = 13         # default font size
plt.rcParams['figure.dpi']     = 100        # image sharpness
plt.rcParams['lines.linewidth'] = 2         # default line thickness
plt.rcParams['axes.grid']      = True       # show grid on all charts

Setting a Visual Theme with `plt.style.use()`

One line changes the entire look of your charts — colors, background, grid style, and fonts. Call it before your plot, not after.

plt.style.use("ggplot")
df.plot(kind="bar", y="2022 Population")
plt.show()

Style	Look
`"ggplot"`	Gray background, colored lines — popular R-like style
`"seaborn-v0_8-whitegrid"`	Clean, modern, white with grid
`"fivethirtyeight"`	Bold, thick lines — news article style
`"dark_background"`	Black background
`"bmh"`	Soft colors, Bayesian style
`"grayscale"`	Shades of gray only
`"tableau-colorblind10"`	Colorblind-friendly palette

Run print(plt.style.available) to see every available option.

Plotting from a DataFrame with `.plot()`

Since Pandas is built on Matplotlib, every DataFrame and Series has a .plot() method that creates a chart in one line.

df.plot(kind="bar", x="Country", y="2022 Population", figsize=(10, 5), title="Population by Country")
plt.show()

The kind= parameter selects the chart type. All common types are supported:

`kind=`	Chart
`"bar"`	Vertical bars
`"barh"`	Horizontal bars
`"line"`	Line chart (default)
`"pie"`	Pie chart
`"hist"`	Histogram
`"scatter"`	Scatter plot
`"box"`	Box plot

You can also call the chart type directly as a method — both styles produce identical output:

# These two are exactly the same:
df.plot(kind="bar", x="Country", y="2022 Population")
df.plot.bar(x="Country", y="2022 Population")

The direct method style (df.plot.bar()) is shorter and more common in practice.

Key parameters you'll use on almost every chart:

Parameter	What it does	Example
`kind`	Chart type	`"bar"`, `"line"`, `"scatter"`
`x` / `y`	Columns for axes	`x="Country"`, `y="2022 Population"`
`figsize`	Width and height in inches	`figsize=(10, 5)`
`title`	Chart title	`title="Population by Country"`
`color`	Bar/line color	`color="steelblue"`
`legend`	Show or hide legend	`legend=True`
`xlabel` / `ylabel`	Axis labels	`xlabel="Country"`
`rot`	Rotation of tick labels	`rot=45`
`grid`	Show background grid	`grid=True`
`alpha`	Transparency	`alpha=0.8`
`bins`	Bin count (histogram only)	`bins=20`

df.plot(
    kind="bar",
    x="Country",
    y="2022 Population",
    figsize=(10, 5),
    title="2022 World Population",
    color="steelblue",
    xlabel="Country",
    ylabel="Population",
    rot=0,
    grid=True,
    alpha=0.85
)
plt.show()

Plotting Directly with `plt.plot()`

When your data is not in a DataFrame — raw lists, NumPy arrays, or a mathematical function — use plt.plot() directly.

import numpy as np

xs = np.arange(-5, 5, 0.25)
ys = xs ** 2

plt.figure(figsize=(8, 5))
plt.plot(xs, ys, color="steelblue", linewidth=2, linestyle="--")
plt.title("Quadratic Function")
plt.xlabel("X")
plt.ylabel("Y")
plt.grid(True)
plt.show()

Matplotlib doesn't know your data is a curve — it connects the dots you give it with straight lines. When points are closely spaced (as with np.arange step 0.25), hundreds of short segments visually blend into a smooth curve.

Customizing plt.plot():

# Color
plt.plot(xs, ys, color='red')

# Line style
plt.plot(xs, ys, linestyle='--')    # dashed
plt.plot(xs, ys, linestyle=':')     # dotted
plt.plot(xs, ys, linestyle='-.')    # dash-dot

# Markers at each point
plt.plot(xs, ys, marker='o')        # circles
plt.plot(xs, ys, marker='s')        # squares
plt.plot(xs, ys, marker='^')        # triangles

# Everything combined
plt.plot(xs, ys, color='green', linestyle='--', linewidth=2, marker='o')
plt.show()

If you want dots without a line, use plt.scatter():

plt.scatter(xs, ys)
plt.show()

Decorating Any Chart

These functions work on any chart — whether drawn through Pandas or directly:

plt.title("Chart Title")
plt.xlabel("X Axis Label")
plt.ylabel("Y Axis Label")
plt.grid(True)
plt.legend()      # shows labels defined with label= on each line
plt.show()

When plotting multiple lines, use label= with each plt.plot() call, then plt.legend() to display them:

ys1 = xs ** 2
ys2 = xs ** 3

plt.plot(xs, ys1, label="f(x) = x²")
plt.plot(xs, ys2, label="f(x) = x³", linestyle="--")
plt.legend()
plt.show()

Multiple Subplots with Object-Oriented Style

When you need more than one chart in the same figure, use the object-oriented style. It gives you full control over each subplot independently.

# 1 row, 2 columns — side by side
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5))

ax1.bar(df["Country"], df["2022 Population"], color="steelblue")
ax1.set_title("Population by Country")
ax1.set_xlabel("Country")
ax1.set_ylabel("Population")

ax2.scatter(df["Area (km2)"], df["2022 Population"], color="tomato")
ax2.set_title("Area vs Population")
ax2.set_xlabel("Area (km²)")
ax2.set_ylabel("Population")

plt.tight_layout()
plt.show()

fig is the whole figure container. ax1 and ax2 are the individual plotting areas. The pattern is the same as plt.* calls, just moved to ax.set_*() methods.

Part 2 — Seaborn: Statistical Visualization

What is Seaborn?

Seaborn is a library built on top of Matplotlib, designed for statistical data visualization. It produces polished, publication-quality charts with far less code than raw Matplotlib, and works natively with Pandas DataFrames.

import seaborn as sns

Every Seaborn chart follows the same basic structure:

sns.chart_type(data=df, x='column_name', y='column_name')
plt.show()

The hue= parameter is Seaborn's most powerful feature — it automatically splits and colors your data by a category column, adding a legend with no extra work:

sns.scatterplot(data=df, x='Area (km2)', y='2022 Population', hue='Continent')
plt.show()

Global Styling with `set_theme()`

Call this once at the top. Every chart afterward follows these settings automatically.

sns.set_theme(style='whitegrid', palette='deep', font_scale=1.2)

style controls the background:

sns.set_theme(style='whitegrid')   # white background with grid — most common
sns.set_theme(style='darkgrid')    # dark background with grid
sns.set_theme(style='white')       # white background, no grid
sns.set_theme(style='ticks')       # minimal, axis ticks only

palette controls the colors of chart elements:

sns.set_theme(palette='deep')        # default, rich colors
sns.set_theme(palette='muted')       # softer, less saturated
sns.set_theme(palette='pastel')      # light pastel colors
sns.set_theme(palette='colorblind')  # accessible for colorblind readers

font_scale scales all text up or down:

sns.set_theme(font_scale=1.5)    # larger text
sns.set_theme(font_scale=0.8)    # smaller text

To apply a style to one chart only without changing global settings:

with sns.axes_style('whitegrid'):
    sns.barplot(data=df, x='Continent', y='2022 Population')
    plt.show()

To reset everything back to Matplotlib defaults: sns.reset_defaults()

Relational Charts — Showing Relationships Between Numbers

Use these when you want to see how two numeric columns relate to each other.

scatterplot() — each row becomes one dot on the chart:

# Basic
sns.scatterplot(data=df, x='Area (km2)', y='2022 Population')

# Colored by continent
sns.scatterplot(data=df, x='Area (km2)', y='2022 Population', hue='Continent')
plt.title("Area vs Population")
plt.show()

lineplot() — connects points with a line. Best for time series or ordered data:

sns.lineplot(data=df, x='Country', y='2022 Population', hue='Continent')
plt.show()

relplot() — a single figure-level function for both scatter and line. Use kind= to switch:

sns.relplot(data=df, x='Area (km2)', y='2022 Population', kind='scatter', hue='Continent')
sns.relplot(data=df, x='Country', y='2022 Population', kind='line')
plt.show()

Distribution Charts — Showing How Values Are Spread

Use these to understand the shape and spread of a numeric column.

histplot() — bars showing how many values fall within each range:

# Basic histogram
sns.histplot(data=df, x='2022 Population')

# With smooth density curve on top
sns.histplot(data=df, x='2022 Population', kde=True)

# Split by category
sns.histplot(data=df, x='2022 Population', hue='Continent')
plt.show()

kdeplot() — a smooth density curve, more refined than a histogram:

sns.kdeplot(data=df, x='2022 Population', hue='Continent')
plt.show()

For a 2D density plot showing where two numeric columns overlap:

sns.kdeplot(data=df, x='Area (km2)', y='2022 Population')
plt.show()

rugplot() — adds small tick marks along an axis showing where individual data points are. Typically layered on top of a kdeplot:

sns.kdeplot(data=df, x='2022 Population')
sns.rugplot(data=df, x='2022 Population')
plt.show()

ecdfplot() — shows the cumulative distribution: what percentage of values fall below each point:

sns.ecdfplot(data=df, x='2022 Population')
plt.show()

displot() — one function to rule all distribution charts. Use kind= to switch:

sns.displot(data=df, x='2022 Population', kind='hist', hue='Continent')
sns.displot(data=df, x='2022 Population', kind='kde')
sns.displot(data=df, x='2022 Population', kind='ecdf')
plt.show()

Categorical Charts — Comparing Groups

Use these when one axis is a category and the other is numeric.

barplot() — shows the mean value per category, with confidence interval bars:

sns.barplot(data=df, x='Continent', y='2022 Population', hue='Continent')
plt.title("Average Population by Continent")
plt.show()

Note: barplot() shows the mean, not the raw values. If you want raw counts by category, use countplot().

countplot() — counts how many rows belong to each category:

sns.countplot(data=df, x='Continent')
plt.show()

boxplot() — shows median, quartiles, and outliers. A fast way to spot unusual values:

sns.boxplot(data=df, x='Continent', y='2022 Population')
plt.show()

violinplot() — like a boxplot, but also shows the full distribution shape on both sides. More informative for larger datasets:

sns.violinplot(data=df, x='Continent', y='2022 Population')
plt.show()

stripplot() — shows every individual data point as a dot per category. Useful when you want to see actual values rather than a summary:

sns.stripplot(data=df, x='Continent', y='2022 Population', hue='Continent')
plt.show()

swarmplot() — same as stripplot but repositions overlapping points so each one is visible:

sns.swarmplot(data=df, x='Continent', y='2022 Population')
plt.show()

pointplot() — shows the mean per category as a dot, connected by lines across categories. Good for spotting trends:

sns.pointplot(data=df, x='Continent', y='2022 Population', hue='Continent')
plt.show()

catplot() — one function for all categorical charts. Use kind= to switch between them:

sns.catplot(data=df, x='Continent', y='2022 Population', kind='bar')
sns.catplot(data=df, x='Continent', y='2022 Population', kind='box')
sns.catplot(data=df, x='Continent', y='2022 Population', kind='violin')
sns.catplot(data=df, x='Continent', y='2022 Population', kind='strip')
sns.catplot(data=df, x='Continent', y='2022 Population', kind='swarm')
sns.catplot(data=df, x='Continent', y='2022 Population', kind='point')
plt.show()

Matrix & Regression Charts — Correlations and Trends

corr() + heatmap() — the most common combination for exploring how numeric columns relate to each other:

corr = df.select_dtypes(include='number').corr()
sns.heatmap(corr, annot=True, cmap='coolwarm')
plt.title("Correlation Matrix")
plt.show()

The values range from -1.0 (perfect negative relationship) to 1.0 (perfect positive relationship). annot=True displays the number inside each cell. cmap='coolwarm' colors high values red and low values blue.

clustermap() — same as heatmap but automatically reorders rows and columns by similarity using clustering:

sns.clustermap(corr, annot=True, cmap='coolwarm')
plt.show()

pairplot() — creates scatter plots for every combination of numeric columns at once, with distribution plots along the diagonal. The fastest way to get a complete overview of your data:

# All numeric columns
sns.pairplot(df.select_dtypes(include='number'))

# Colored by a category
sns.pairplot(df, hue='Continent')
plt.show()

jointplot() — shows a scatter plot of two columns, plus the distribution of each column on the margins:

sns.jointplot(data=df, x='Area (km2)', y='2022 Population')

# With a regression line
sns.jointplot(data=df, x='Area (km2)', y='2022 Population', kind='reg')
plt.show()

lmplot() — scatter plot with a regression line fitted through the data. Supports separate lines per group:

sns.lmplot(data=df, x='Area (km2)', y='2022 Population')

# Separate regression line per continent
sns.lmplot(data=df, x='Area (km2)', y='2022 Population', hue='Continent')
plt.show()

regplot() — the axes-level version of lmplot(). Does the same thing but doesn't support grouping. Use when you want to embed a regression plot inside a larger figure with subplots:

sns.regplot(data=df, x='Area (km2)', y='2022 Population')
plt.show()

residplot() — shows the residuals of a regression: how far each actual point is from the predicted line. A flat, random scatter around zero means the regression is a good fit:

sns.residplot(data=df, x='Area (km2)', y='2022 Population')
plt.show()

Combining Seaborn with Matplotlib

Seaborn creates the chart. Matplotlib customizes it. Both work together in the same block — you never have to choose one or the other:

sns.boxplot(data=df, x='Continent', y='2022 Population', palette='pastel')

plt.title('Population Distribution by Continent')
plt.xlabel('Continent')
plt.ylabel('Population')
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()

This is the standard workflow: Seaborn for the chart, Matplotlib for the title, labels, rotation, and layout.

Choosing the Right Tool

Use Matplotlib when:

Your data is in raw lists or NumPy arrays, not a DataFrame
You need multiple subplots in a specific layout
You want pixel-level control over every visual element

Use Seaborn when:

Your data is in a DataFrame and ready to visualize
You want a beautiful chart quickly with minimal code
You're doing statistical analysis — distributions, correlations, regressions, group comparisons

In most real projects, you'll use both: Seaborn to draw the chart, Matplotlib to label and polish it.

References

GitHub Repo: https://github.com/Hu8MA/Mastering-Pandas-Reference

Matplotlib Documentation: https://matplotlib.org/

Seaborn Documentation: https://seaborn.pydata.org/

Pandas Visualization Guide: https://pandas.pydata.org/docs/user_guide/visualization.html

S&OP: Why Your Excel Is Lying to You (and How to Interrogate It with Python)

Wed, 25 Mar 2026 07:22:37 +0000

In S&OP (Sales & Operations Planning) meetings, opinions are often discussed instead of facts. "I think we'll sell more", "Last month was weird".

The root problem is not the lack of business vision, it's the lack of signal integrity.

Most supply chains are managed on spreadsheets that accept anything: dates as text, blank spaces, and typos that turn a 100-unit order into 100,000. When you feed your prediction algorithm with that "garbage," you get amplified garbage (the financial Bullwhip effect).

Today we kick off the S&OP Engineering series. We're not going to talk theory; we're going to build a data architecture that audits your business automatically.

The Problem: Signal-to-Noise Ratio

In telecommunications (my original background), noise is any interference that corrupts the signal. In Supply Chain, "noise" is dirty data.

If you don't filter the noise before planning demand, you're immobilizing capital. An undetected outlier is money on fire. If your algorithm sees a false spike of 100,000 units, it will order raw materials you don't need, burning cash and taking up warehouse space. Data hygiene is not 'cleaning', it's protecting your operating margin.

The Visual Evidence

Before seeing a single line of code, look at the difference between what your ERP exports (top) and the statistical reality of your demand (bottom).

Top: Raw data with human errors. Bottom: The clean signal ready for AI algorithms.

The Solution: "Quality Valve" Architecture

To solve this, we apply First Principles Thinking. We don't need to "be more careful" with Excel. We need a system that mathematically prohibits the entry of dirty data into our "Single Source of Truth."

We've designed an automated pipeline with the following stack:

Brain: Python (Pandas + Scipy) for statistical logic.
Warehouse: Supabase (PostgreSQL) as the "Single Source of Truth."
Agent: A script that runs automatically when new files arrive.

The Code: Statistics > Intuition

We don't use fixed rules ("if greater than 1000, delete"). We use statistics. We implement the Z-Score, which measures how many standard deviations a data point is from the mean.

If a sale has a Z-Score > 3 (it's more than 3 sigmas from normality), it's mathematically unlikely to be standard behavior. The system doesn't delete it (it could be a real sale), but it flags it for audit and excludes it from automatic prediction.

Note: We use Z-Score assuming normality to simplify this example. In production scenarios with intermittent demand, we use methods like IQR (Interquartile Range) or MAD (Median Absolute Deviation) which are more robust for non-Gaussian distributions.

[Image of box plot showing Interquartile Range (IQR) and outliers]

Here's the core logic from our SupplyChainSanitizer class:

def detect_outliers_zscore(self, threshold=3):
    """
    Detects statistical anomalies using Z-Score.
    We don't delete the row (info loss), we label it.
    """
    # Calculate the standard deviation of the signal
    z_scores = np.abs(stats.zscore(self.df['qty']))
    
    # Flag what is mathematically suspicious
    self.df['is_outlier'] = z_scores > threshold
    return self

Open Kitchen: Try It Yourself

As an engineer, I distrust what I can't execute. That's why I've isolated the cleaning logic in an interactive Colab Notebook.

You don't need to install Python or configure databases. I've prepared an ephemeral environment where you can:

Generate a corrupt sales dataset (simulated).
Run the SupplyChainSanitizer cleaning engine.
See how the algorithm detects and separates the noise.

Click the button, hit "Play" on the cells, and observe data engineering in action.

Production Architecture (Behind the Scenes)

For technical profiles interested in how this scales in a real company (Datalaria Core):

Ingestion: CSVs are uploaded to a private Bucket in Supabase Storage or a local database.
Trigger: A Python worker detects the file.
Processing: Executes the cleaning in memory (Docker Container).
Persistence: Clean data is injected into PostgreSQL using Row Level Security (RLS) to ensure nobody can manually alter the financial history.

Security Note: In production, we never connect scripts with superuser permissions. We use specific Service Roles and strict RLS policies to ensure supply chain integrity.

Data Flow Visualization

The following diagram shows how "dirty" data passes through our Quality Valve before reaching the Single Source of Truth:

Legend:

Red: Raw data with noise (the problem)
Green: Clean signal ready for prediction
Orange: Anomalies labeled for human review
Blue: Centralized warehouse (Supabase)

Next Step: Scientific Prediction

Now that we have a clean database (a pure signal), we're ready to look into the future.

In the next chapter of the series, we'll connect this clean table with Facebook Prophet to generate probabilistic demand forecasts, forever abandoning Excel's simple moving averages.

Subscribe to receive Chapter 2: "Demand Planning: From Guessing to Probability".

Mastering Pandas — Part 3: Data Cleaning, Merging & Joining

Wed, 18 Mar 2026 23:24:55 +0000

Real Data Is Never Clean

In Part 2, you learned how to group and slice a DataFrame with precision. But in real projects, the data you receive rarely arrives ready to analyze. Column names have spaces. Ages are negative. Phone numbers mix letters with digits. Duplicate rows sneak in. And your data often lives across multiple files that need to be combined before you can do anything useful.

This article covers the two skills that bridge raw data and real analysis:

Data Cleaning — detecting and fixing everything wrong with your data before it corrupts your results
Merging & Joining — combining separate DataFrames into a single, unified dataset

These topics flow naturally into each other. You clean each source first, then you combine them. That's the order every data analyst follows in practice — and it's the order we'll follow here.

We'll use this sample dataset throughout the cleaning section:

import pandas as pd
import numpy as np

data = {
    "Name":             ["Alice", "bob", "Carol", "Alice", "  dave  ", "Eve"],
    "Age":              [25, -5, 200, 25, 30, None],
    "Phone":            ["055-123-4567", None, "55123x", "055-123-4567", "055-987-0000", "055abc"],
    "Email":            ["*Emails are not allowed*", "*Emails are not allowed*", None, "*Emails are not allowed*", "*Emails are not allowed*", "not-an-email"],
    "Paying_Customer":  ["Y", "N", "Yes", "Y", "N", "y"],
    "Score":            [85, 110, 72, 85, -10, 90],
}
df = pd.DataFrame(data)

Step 1 — Inspect Before You Touch Anything

The most important rule in data cleaning: look before you act. Jumping straight into fixes without understanding the full picture leads to mistakes you won't catch until much later.

These are the inspection tools you should run on every new dataset:

df.shape          # (rows, columns) — how big is this?
df.info()         # column names, data types, non-null counts
df.head()         # first 5 rows
df.tail()         # last 5 rows
df.describe()     # min, max, mean, std — spot impossible values here

For missing values specifically:

df.isna().sum()           # count NaN per column
df.isnull().sum()         # same as isna() — both work
df[df["Phone"].notna()]   # rows that actually have a phone number

For duplicates and inconsistent categories:

df.duplicated()                         # True/False per row
df[df.duplicated()]                     # show only the duplicate rows
df["Paying_Customer"].value_counts()    # reveals: Y, N, Yes, y — all meaning the same thing

Always run describe() on numeric columns — the min and max alone will tell you immediately if something is wrong. A minimum age of -5 or a score of 110 out of 100 is a red flag you can't miss.

Step 2 — Fix Column Names First

Messy column names make everything harder. Clean them before anything else, so every subsequent operation uses predictable, consistent names.

Rename specific columns:

df.rename(columns={"Old Name": "New_Name", "phone number": "Phone"})

Clean all column names at once — the standard one-liner:

df.columns = df.columns.str.strip().str.lower().str.replace(" ", "_")

This strips surrounding spaces, converts to lowercase, and replaces spaces with underscores in a single chain. After this, "First Name" becomes "first_name", " Age " becomes "age", and so on.

Before:              After:
"First Name"    →    "first_name"
"  Age "        →    "age"
"Phone Number"  →    "phone_number"
"ZIP Code"      →    "zip_code"

Step 3 — Handle Missing Values

Missing values appear as NaN in Pandas. Before deciding what to do with them, you need to understand why they're missing — because the right fix depends on the cause.

Fill missing values with a fixed value:

df["Phone"].fillna("000-000-0000")    # placeholder for missing phone
df["Age"].fillna(0)                   # fill numeric with 0
df["Name"].fillna("Unknown")          # fill text with label
df["Score"].fillna(df["Score"].mean()) # fill with column average

Fill missing numeric values by estimating from neighbors:

df["Sales"] = df["Sales"].interpolate()

interpolate() fills NaN by calculating the midpoint between the values above and below it. This is ideal for time series or sensor data where values change gradually — it would be wrong to use for random datasets.

Before:     After interpolate():
100         100
NaN    →    150   ← estimated midpoint
200         200
NaN    →    225   ← estimated midpoint
250         250

Remove rows with missing values:

df.dropna()                            # drop any row with at least one NaN
df.dropna(subset=["Phone"])            # drop only if Phone is NaN
df.dropna(thresh=3)                    # drop rows that have fewer than 3 non-NaN values

inplace rule: use either inplace=True or variable assignment — never both. They do the same thing, and combining them produces a bug where df becomes None.
df.dropna(subset=["Phone"], inplace=True)   # ✅ modifies df directly
df = df.dropna(subset=["Phone"])            # ✅ returns new df
df = df.dropna(subset=["Phone"], inplace=True)  # ❌ df becomes None

Step 4 — Remove Duplicates

Duplicate rows silently inflate counts, averages, and totals. Always check for them.

df.drop_duplicates()                         # remove fully identical rows
df.drop_duplicates(subset=["Name"])          # remove rows with duplicate Name only
df.drop_duplicates(subset=["Name", "Email"]) # remove rows where both columns match
df.drop_duplicates(keep="last")              # keep the last occurrence instead of the first

Before:                        After drop_duplicates():
Name   Email                   Name   Email
Alice  *Emails are not allowed*          Alice  *Emails are not allowed*
Bob    *Emails are not allowed*      →     Bob    *Emails are not allowed*
Alice  *Emails are not allowed* ← dup    Carol  *Emails are not allowed*
Carol  *Emails are not allowed*

Step 5 — Fix Data Types

Pandas sometimes reads numeric columns as strings, or dates as plain text. Operations on wrong types either crash or silently produce wrong results.

Change column type directly:

df["Zip_Code"] = df["Zip_Code"].astype(str)     # treat as text, not a number
df["Age"] = df["Age"].astype(int)               # convert to integer
df["Price"] = df["Price"].astype(float)         # convert to decimal

Handle messy numbers that contain symbols or text:

df["Price"] = pd.to_numeric(df["Price"], errors="coerce")

errors="coerce" converts anything that can't be parsed into NaN instead of crashing. This is the safe way to clean columns like "$1,200" or "N/A" that can't be directly cast with astype().

Before:           After pd.to_numeric(..., errors="coerce"):
"1200"    →       1200.0
"$850"    →       NaN     ← couldn't parse
"700.5"   →       700.5
"N/A"     →       NaN     ← couldn't parse

Convert to datetime:

df["Date"] = pd.to_datetime(df["Date"])
df["Date"] = pd.to_datetime(df["Date"], format="%d/%m/%Y")  # specify format if needed

Once a column is proper datetime, you unlock sorting by date, filtering by year/month, calculating time differences, and much more.

Step 6 — Clean Strings

String columns are almost always the messiest part of a real dataset. Pandas provides a full .str accessor with everything you need.

Strip unwanted characters from edges:

df["Name"] = df["Name"].str.strip()           # remove spaces from both sides
df["Name"] = df["Name"].str.strip(" ./-")     # remove spaces, dots, slashes, dashes
df["Name"] = df["Name"].str.lstrip()          # left side only
df["Name"] = df["Name"].str.rstrip()          # right side only

Replace characters anywhere in the string:

df["Phone"] = df["Phone"].str.replace(r"[a-zA-Z]", "", regex=True)   # remove all letters
df["Phone"] = df["Phone"].str.replace(r"\D", "", regex=True)          # keep digits only
df["Name"]  = df["Name"].str.replace(r"[./]", "", regex=True)         # remove dots and slashes

Standardize text case:

df["Name"] = df["Name"].str.lower()     # all lowercase → "alice"
df["Name"] = df["Name"].str.upper()     # all uppercase → "ALICE"
df["Name"] = df["Name"].str.title()     # title case    → "Alice"

Check text content:

df[df["Email"].str.contains("@")]           # filter valid-looking emails
df[df["Phone"].str.len() != 12]             # find rows where phone length is wrong
df[df["Name"].str.startswith("A")]          # names starting with A
df[df["Name"].str.endswith("e")]            # names ending with e

Extract a pattern using regex:

df["Zip"] = df["Address"].str.extract(r"(\d{5})")        # pull 5-digit zip code
df["Area_Code"] = df["Phone"].str.extract(r"^(\d{3})")   # pull first 3 digits

Split one column into multiple columns:

df[["Street", "State", "Zip"]] = (
    df["Address"]
    .str.split(",", n=2, expand=True)
)

n=2 limits the split to 2 times, producing exactly 3 parts regardless of how many commas the string contains. expand=True spreads those parts into separate columns.

Before:                              After:
"123 Main St, Texas, 75001"   →   Street="123 Main St"  State="Texas"  Zip="75001"

Step 7 — Replace and Standardize Values

Raw data often has the same value written multiple ways: "Y", "Yes", "yes", "y" — all meaning the same thing. Standardize them.

Replace whole cell values using a dictionary:

df["Status"] = df["Status"].replace({"Y": "Yes", "N": "No", "y": "Yes"})

# Apply to multiple columns at once
df[["Paying_Customer", "Do_Not_Contact"]] = (
    df[["Paying_Customer", "Do_Not_Contact"]]
    .replace({"Y": "Yes", "N": "No", "y": "Yes", "n": "No"})
)

Overwrite invalid values using loc[]:

df.loc[df["Age"] < 0,   "Age"] = None     # negative age → invalid
df.loc[df["Age"] > 120, "Age"] = None     # age over 120 → invalid
df.loc[df["Price"] < 0, "Price"] = None   # negative price → invalid

Keep valid values, replace everything else with NaN:

df["Salary"] = df["Salary"].where(df["Salary"] > 0)
# Rows where Salary <= 0 become NaN; the rest are untouched

Now Combine — Merging & Joining DataFrames

Once each source is clean, the next challenge is combining them. In the real world, data almost never lives in a single file. You might have orders in one table, customers in another, and products in a third. Merging is how you bring those pieces together.

Pandas provides three main tools for this:

Tool	What it does
`merge()`	Joins two DataFrames by matching on a shared column
`concat()`	Stacks DataFrames vertically or side by side
`join()`	Joins on the index instead of a column

We'll use these two DataFrames for all the examples:

customers = pd.DataFrame({
    "ID":        [1, 2, 3, 4],
    "Name":      ["Alice", "Bob", "Carol", "Dave"],
    "Country":   ["USA", "UK", "India", "USA"],
})

orders = pd.DataFrame({
    "ID":        [1, 2, 3, 5],
    "Product":   ["Laptop", "Phone", "Tablet", "Monitor"],
    "Price":     [1200, 800, 600, 400],
})

Notice that ID=4 (Dave) exists in customers but not in orders, and ID=5 exists in orders but not in customers. This is exactly the kind of mismatch you encounter in practice — and why the how= parameter matters so much.

1️⃣ `merge()` — The Main Tool

merge() works like a SQL JOIN. It aligns two DataFrames by finding rows where a shared column has the same value.

Basic syntax:

customers.merge(orders, how="inner", on="ID")

The `how=` Parameter — All Four Types

# INNER → only rows that match in both DataFrames (default)
customers.merge(orders, how="inner", on="ID")
# Result: IDs 1, 2, 3 only — Dave (4) and Monitor (5) are excluded

# LEFT → all rows from the left + matched rows from the right
customers.merge(orders, how="left", on="ID")
# Result: all 4 customers — Dave gets NaN for Product and Price

# RIGHT → all rows from the right + matched rows from the left
customers.merge(orders, how="right", on="ID")
# Result: all 4 orders — Monitor (ID=5) gets NaN for Name and Country

# OUTER → all rows from both, NaN where no match
customers.merge(orders, how="outer", on="ID")
# Result: all 5 unique IDs — gaps filled with NaN on both sides

`how=`	Keeps from Left	Keeps from Right	NaN Possible
`inner`	Matched only	Matched only	No
`left`	All rows	Matched only	Right side
`right`	Matched only	All rows	Left side
`outer`	All rows	All rows	Both sides

CROSS merge — every row with every row:

customers.merge(orders, how="cross")
# Produces 4 × 4 = 16 rows — every customer paired with every order
# No on= parameter — it will raise an error if you include it

The `on=` Parameter — What to Match On

# Match on a single column
customers.merge(orders, how="inner", on="ID")

# Match on multiple columns — both must match simultaneously
customers.merge(orders, how="inner", on=["ID", "Country"])

# Auto-detect — Pandas finds all shared column names automatically
customers.merge(orders)

Be careful with auto-detect. If both DataFrames happen to share a column that you don't intend to join on (like a generic "Name" column), Pandas will include it in the match condition and produce unexpected results. Explicit is safer.

Handling Shared Column Names — `_x` and `_y`

If both DataFrames have a column with the same name, but you only join on ID, Pandas keeps both copies and renames them automatically:

# Both DataFrames have a "Name" column, but you only merge on "ID"
customers.merge(orders, how="inner", on="ID")
# Result: Name_x (from customers), Name_y (from orders)

Three ways to fix this:

# Option 1 — merge on all shared columns so no duplicates appear
customers.merge(orders, how="inner", on=["ID", "Name"])

# Option 2 — use custom suffixes instead of _x and _y
customers.merge(orders, how="inner", on="ID", suffixes=("_customer", "_order"))

# Option 3 — drop the shared column from one DataFrame before merging
orders_clean = orders.drop(columns=["Name"])
customers.merge(orders_clean, how="inner", on="ID")

Other Useful Parameters

# left_on / right_on — when the key column has different names in each DataFrame
customers.merge(orders, how="inner", left_on="CustomerID", right_on="OrderID")

# indicator — adds a column showing where each row came from
customers.merge(orders, how="outer", on="ID", indicator=True)
# _merge column values: "left_only", "right_only", "both"

The indicator=True parameter is especially useful for auditing: after a merge, you can immediately see which rows matched, which came only from the left, and which came only from the right.

2️⃣ `concat()` — Stacking DataFrames

While merge() combines DataFrames horizontally by matching values, concat() simply stacks them — either rows on top of rows, or columns side by side.

pd.concat([df1, df2])        # stack vertically (default)
pd.concat([df1, df2], axis=1)  # stack horizontally

`axis=` Parameter

# axis=0 → stack rows vertically (append one DataFrame below another)
monthly_data = pd.concat([january, february, march], axis=0)

# axis=1 → stack columns side by side
combined = pd.concat([names_df, scores_df], axis=1)

`join=` Parameter

# outer (default) → keep all columns, NaN where a column doesn't exist in one source
pd.concat([df1, df2], join="outer")

# inner → keep only columns that exist in all DataFrames
pd.concat([df1, df2], join="inner")

`ignore_index=` Parameter

# Without ignore_index — original indices are preserved, which can repeat
pd.concat([df1, df2])
# Index: 0, 1, 2, 0, 1, 2  ← 0, 1, 2 repeats

# With ignore_index=True — fresh sequential index
pd.concat([df1, df2], ignore_index=True)
# Index: 0, 1, 2, 3, 4, 5  ← clean

Always use ignore_index=True after stacking unless you specifically need the original indices.

`keys=` Parameter

# Label each source so you can identify where each row came from
combined = pd.concat([df1, df2], keys=["source_a", "source_b"])

This creates a MultiIndex where the first level is your label and the second is the original row index — useful when you need to trace data back to its source after combining.

`verify_integrity=` Parameter

# Raises an error if any duplicate index values are found after concatenation
pd.concat([df1, df2], verify_integrity=True)

`merge()` vs `concat()` — When to Use Which

	`merge()`	`concat()`
Direction	Horizontal (by matching)	Vertical or horizontal (by stacking)
Needs a shared key column	Yes	No
Best for	Joining related tables	Combining same-structure data
Equivalent to	SQL JOIN	SQL UNION

A simple rule: if you're combining columns from different tables, use merge(). If you're stacking more rows of the same structure, use concat().

3️⃣ `join()` — Index-Based Joining

join() works like merge(), but it joins on the index instead of a regular column. You need to set the index first.

customers_indexed = customers.set_index("ID")
orders_indexed    = orders.set_index("ID")

customers_indexed.join(orders_indexed, how="inner")

The how= options are the same as merge(): inner, left, right, outer.

Why `merge()` Is Almost Always Better

	`merge()`	`join()`
Join on any column	✅ Yes	❌ No — index only
Join on index	✅ Yes	✅ Yes
Requires `set_index()` first	❌ No	✅ Yes
Simpler syntax	✅ Yes	⚠️ Extra steps

> Use `join()` only when your data is already indexed — for example, after a `groupby()` that produced a meaningful index. Otherwise, `merge()` is cleaner and more flexible.

This is Part 3 of the Pandas for Data Science series. Next up: Part 4 — Data Visualization with Matplotlib & Seaborn

To access the full article, please visit the link below or directly access the repository via the links below.

Refrences

GitHub Repo : https://github.com/Hu8MA/Mastering-Pandas-Reference
Website : https://pandas.pydata.org/
GitHub Library : https://github.com/pandas-dev/pandas
Course : https://youtu.be/Mdq1WWSdUtw
Full article : https://hussein16mahdi.medium.com/mastering-pandas-part-3-data-cleaning-merging-joining-817a070147ca

Mastering Pandas — Part 1: Reading, Sorting & Displaying Data

Thu, 26 Feb 2026 22:28:41 +0000

Pandas for Data Science Series — Article #1

What is Pandas and Why Does It Matter?

Pandas is an open-source Python library built specifically for data manipulation and analysis. Released in 2008 and named after "Panel Data" — a term from econometrics — it has since become one of the most essential tools in the entire Python ecosystem.

At its core, Pandas gives you two powerful data structures:

Series — a single column of data
DataFrame — a full table with rows and columns (think of it as a turbocharged spreadsheet you control with code)

Whether you're working in data science, machine learning, financial analysis, or business intelligence, chances are you'll be loading, exploring, and transforming data with Pandas before you do anything else. It integrates seamlessly with libraries like NumPy, Matplotlib, Seaborn, Scikit-learn, and TensorFlow, making it the essential starting point in nearly every data workflow.

In this first article of the series, you'll cover two essential skill sets: reading data into Pandas from virtually any file format, and sorting and displaying that data in ways that let you understand it quickly.

Let's get started

Part 1 — Reading Files into Pandas

Before you can analyze anything, you need to load your data. Pandas makes this simple with a family of read_*() functions — one for almost every file format you'll encounter in the real world.

CSV File — Most Common

CSV (Comma-Separated Values) files are the bread and butter of data work. read_csv() is almost certainly the first Pandas function you'll use in any project.

import pandas as pd

df = pd.read_csv("file.csv")                          # basic read
df = pd.read_csv("file.csv", header=0)                # first row as header
df = pd.read_csv("file.csv", index_col="order_id")    # set column as index
df = pd.read_csv("file.csv", nrows=100)               # read only first 100 rows
df = pd.read_csv("file.csv", skiprows=2)              # skip first 2 rows
df = pd.read_csv("file.csv", usecols=["price","quantity"])  # specific columns only
df = pd.read_csv("file.csv", na_values=["NA","?"])    # define missing values
df = pd.read_csv("file.csv", sep=";")                 # semicolons instead of commas

Excel File

Excel files are ubiquitous in business environments. read_excel() handles them cleanly, and you can target specific sheets by name or position.

df = pd.read_excel("file.xlsx")                       # basic read
df = pd.read_excel("file.xlsx", sheet_name="Sheet1")  # specific sheet by name
df = pd.read_excel("file.xlsx", sheet_name=0)         # first sheet by index
df = pd.read_excel("file.xlsx", skiprows=2)           # skip first 2 rows
df = pd.read_excel("file.xlsx", usecols="A:D")        # read columns A to D

JSON File

JSON is the standard format for web APIs. read_json() can parse various JSON structures, including a flat list of records.

df = pd.read_json("file.json")                        # basic read
df = pd.read_json("file.json", orient="records")      # list of records format

HTML File — Web Scraping Made Easy

One of Pandas' more surprising capabilities: it can extract tables directly from HTML pages, including live websites. Simple web scraping with zero extra libraries.

tables = pd.read_html("file.html")                          # returns list of all tables
df     = pd.read_html("https://website.com/table")[0]       # first table from a URL

SQL Database

When your data lives in a database, Pandas can query it directly. You bring a connection object and a SQL query, and it returns a DataFrame.

import sqlite3

conn = sqlite3.connect("database.db")

df = pd.read_sql("SELECT * FROM table", conn)    # full SQL query
df = pd.read_sql_table("table_name", conn)       # read entire table directly

Text File

read_table() is the text-file counterpart to read_csv(). The only real difference is its default separator — a tab instead of a comma — but it accepts all the same parameters.

df = pd.read_table("file.txt")                      # tab separated (default)
df = pd.read_table("file.txt", sep=",")             # comma separated
df = pd.read_table("file.txt", sep=";")             # semicolon separated
df = pd.read_table("file.txt", sep="|")             # pipe separated
df = pd.read_table("file.txt", header=None)         # file has no header row
df = pd.read_table("file.txt", names=["col1","col2"])  # add column names manually
df = pd.read_table("file.txt", skiprows=2)          # skip first 2 rows
df = pd.read_table("file.txt", nrows=100)           # read only 100 rows
df = pd.read_fwf("file.txt")                        # fixed-width text file

read_csv() vs read_table() — The only practical difference is the default separator: read_csv() uses a comma (,) while read_table() uses a tab (\t). Every other parameter works identically in both.

	`read_csv()`	`read_table()`
Default separator	`,` comma	`\t` tab
File type	CSV files	TXT files
Speed	Same	Same
Parameters	Same	Same

Other Formats

# Clipboard — copy any table, then run this
df = pd.read_clipboard()

# Parquet — the preferred format for big data (very fast)
df = pd.read_parquet("file.parquet")

# XML
df = pd.read_xml("file.xml")

Summary — All Read Functions at a Glance

Function	File Type	Common Use
`read_csv()`	CSV	Most common — daily use
`read_excel()`	XLSX	Excel files from business
`read_json()`	JSON	Web APIs and REST data
`read_html()`	HTML	Web scraping tables
`read_sql()`	Database	SQL queries
`read_table()`	TXT	Tab-separated text files
`read_clipboard()`	Clipboard	Quick copy-paste workflow
`read_parquet()`	Parquet	Large-scale / big data
`read_xml()`	XML	Structured XML data

Most Important Parameters — Work With ALL Functions

nrows      →  how many rows to read
skiprows   →  how many rows to skip at the top
usecols    →  which columns to read
index_col  →  which column to use as index
na_values  →  what to treat as missing value (NaN)
header     →  which row to use as column header
sep        →  what separator/delimiter to use

Part 2 — Sorting and Displaying Data

Once your data is loaded, the first thing you want to do is understand it. The functions in this section are your primary tools for exploring structure, distributions, relationships, and ordering — before you write a single line of analysis.

We'll use this sample DataFrame throughout all the examples:

import pandas as pd

data = {
    'Country':          ['China', 'India', 'USA', 'Brazil', 'UK'],
    'Continent':        ['Asia', 'Asia', 'North America', 'South America', 'Europe'],
    '2022 Population':  [1412000000, 1380000000, 331000000, 214000000, 67000000],
    'Area (km2)':       [9597000, 3287000, 9834000, 8516000, 243000]
}
df = pd.DataFrame(data)

Sorting Functions

1. sort_values()

The most commonly used sorting function. Reorders the DataFrame by the values in one or more columns. By default it sorts ascending; pass ascending=False to flip it.

# Ascending order (default)
df.sort_values('2022 Population')

# Descending order
df.sort_values('2022 Population', ascending=False)

# Sort by multiple columns: Continent A→Z, then Population largest first
df.sort_values(['Continent', '2022 Population'], ascending=[True, False])

2. sort_index()

Sorts by the DataFrame's index rather than column values. This becomes important after filtering, merging, or shuffling rows, which can leave the index disordered.

df.sort_index()                # sort index ascending
df.sort_index(ascending=False) # sort index descending

3. nlargest()

Returns the top N rows with the largest values in a specified column. More efficient and more readable than combining sort_values() with head().

# Top 3 most populated countries
df.nlargest(3, '2022 Population')

4. nsmallest()

Returns the top N rows with the smallest values in a specified column.

# 3 smallest countries by area
df.nsmallest(3, 'Area (km2)')

5. rank()

Assigns a rank to each row based on a column's values, without changing the row order. Useful for adding a ranking column to your existing DataFrame.

# Add a population rank column (1 = largest)
df['Population Rank'] = df['2022 Population'].rank(ascending=False)

Displaying Functions

1. head() & tail()

The simplest inspection tools. head() shows the first N rows (default 5) and tail() shows the last N rows. These are usually the first things you run on any new dataset.

df.head()    # first 5 rows
df.head(10)  # first 10 rows
df.tail()    # last 5 rows
df.tail(3)   # last 3 rows

2. sample()

Returns N randomly selected rows. Ideal for getting a representative look at a large dataset where the first or last rows might not be representative.

df.sample(3)  # 3 random rows

3. info()

Displays the DataFrame's structure: column names, data types, number of non-null values, and memory usage. This should be one of the very first functions you call on any new dataset — it immediately reveals missing data and wrong data types.

df.info()

4. describe()

Returns a statistical summary for all numeric columns: count, mean, standard deviation, min, max, and quartiles. A quick way to understand the scale and distribution of your data.

df.describe()

5. value_counts()

Returns the frequency of each unique value in a column. Invaluable for understanding the distribution of categorical data at a glance.

# How many countries exist per continent?
df['Continent'].value_counts()

6. corr()

Calculates the correlation between all numeric columns. Returns values from -1.0 (perfect negative relationship) to 1.0 (perfect positive relationship), with 0.0 meaning no relationship.

Important: corr() requires numeric columns only. Always use select_dtypes() before calling it to avoid a ValueError.

# Safe pattern — works on any DataFrame
df.select_dtypes(include='number').corr()

# Or select specific columns manually
df[['2022 Population', 'Area (km2)']].corr()

7. select_dtypes()

Filters and returns only the columns that match a specified data type. Useful when you need to work on a subset of columns without knowing their names in advance.

df.select_dtypes(include='number')              # numeric columns only (int and float)
df.select_dtypes(include='object')              # string columns only
df.select_dtypes(include='bool')                # boolean columns only
df.select_dtypes(include=['int64', 'float64'])  # specific numeric types
df.select_dtypes(exclude='object')              # drop all string columns
df.select_dtypes(exclude='number')              # drop all numeric columns
df.select_dtypes(include='number', exclude='bool')  # numeric but not boolean

8. Select Specific Columns

Returns a new DataFrame containing only the columns you need. Helps reduce output to what is relevant.

df[['Country', '2022 Population']]

Matplotlib Global Settings — plt.rcParams

While not a Pandas function, plt.rcParams is something you'll configure right alongside your Pandas setup. It controls the default appearance of all plots in your session — set it once at the top of your notebook and every chart inherits those settings automatically.

import matplotlib.pyplot as plt

plt.rcParams['figure.figsize'] = (20, 8)   # 20 wide, 8 tall (in inches)
plt.rcParams['font.size']       = 14       # default font size for all text
plt.rcParams['figure.dpi']      = 100      # resolution (higher = sharper)
plt.rcParams['lines.linewidth'] = 2        # default line width
plt.rcParams['axes.grid']       = True     # show grid on all plots by default

Common Patterns

Filter, then Sort

The most frequent real-world pattern: narrow the dataset to a subset, then rank within that subset.

# Top 3 Asian countries by population
df[df['Continent'] == 'Asia'].nlargest(3, '2022 Population')

Sort and Preview

Both lines produce the same result — the second is preferred for its clarity.

df.sort_values('2022 Population', ascending=False).head(10)
df.nlargest(10, '2022 Population')  # ✅ preferred

Clean Display of Numbers

Control how many decimal places are shown across the entire session.

pd.set_option('display.float_format', '{:.2f}'.format)

Complete Summary Table

Function	Category	Purpose
`sort_values()`	Sorting	Sort by column values
`sort_index()`	Sorting	Sort by index
`nlargest()`	Sorting	Get top N largest values
`nsmallest()`	Sorting	Get top N smallest values
`rank()`	Sorting	Assign rank to each row
`head()`	Displaying	Show first N rows
`tail()`	Displaying	Show last N rows
`sample()`	Displaying	Show random N rows
`info()`	Displaying	Show structure and data types
`describe()`	Displaying	Statistical summary
`value_counts()`	Displaying	Count frequency of unique values
`corr()`	Displaying	Correlation between numeric columns
`select_dtypes()`	Displaying	Filter columns by data type
`plt.rcParams`	Matplotlib	Set global defaults for all plots

This is Part 1 of the Pandas for Data Science series. Next up: Part 2—Filtering, Selecting & Cleaning Data.

Refrences

GitHub Repo : https://github.com/Hu8MA/Mastering-Pandas-Reference
Website : https://pandas.pydata.org/
GitHub Library : https://github.com/pandas-dev/pandas
Course : https://youtu.be/Mdq1WWSdUtw

Understanding the "TypeError: cannot convert the series to " in Pandas

Wed, 14 Jan 2026 11:58:14 +0000

Introduction

In Python, Pandas is a powerful and effective tool for data analysis and manipulation. It performs exceptionally well with data types and structures. However, as users, when you start working with the problems of real-world datasets, you may encounter many errors. One such error that often occurs is "TypeError: cannot convert the series to <class 'float'>." Generally, this error in Pandas occurs when we try to perform operations that expect numeric values on a Pandas series. In this article, we will dive deep into the causes behind this error, understand various scenarios where we can commonly find this error, and provide various debugging solutions to troubleshoot and resolve the issue.

Understanding the Error

The "TypeError: cannot convert the series to <class 'float'>" is a Pandas error that occurs when we try to perform operations that expect numeric values on a Pandas series. The error is an indication that the data type of the series is interrupting its conversion to a float, preventing the expected operation from being executed successfully.Let's imagine you're using Pandas to analyze customer data. You have a Series of customer ages, however some entries may be missing or contain information such as "unknown". When attempting to compute average customer age, Pandas will return this error because it is unable to translate "unknown" to a number. Before computing the average, you must first clean the data by deleting "unknown" items or replacing them with a numerical value (e.g., -1).

Common Scenarios causing this error

Mixing Data Types :
Consider the following example:

import pandas as pd

# creating df which is a data frame with mixed data types
data = {'column1': [1, 'two', 3, 4.5]}
df = pd.DataFrame(data)

# Attempting to perform a numeric operation on the mixed data
result = float(df['column1'])

In this example, the presence of a string ('two') in the 'column1' series hinders the conversion to a float which eventually leads to "TypeError: cannot convert the series to ". The output of this looks as follows:

Null or Missing Values :
Now, let's explore another example involving missing values:

import pandas as pd

# creating df which is a data frame with a missing value
data = {'column1': [1, 2, None, 4.5]}
df = pd.DataFrame(data)

# Attempting to perform a numeric operation on a Series with missing values
result = float(df['column1'])

In this example, the presence of a None in the 'column1' series hinders the conversion to a float, which eventually leads to "TypeError: cannot convert the series to ". The output of this looks as follows:

Solutions to the "TypeError"

The first step in fixing the "TypeError" is figuring out the cause of the error. Here are two common situations where it occurs and their respective solutions:

Handling Mixed Data Types

# Convert the 'column1' Series to numeric, coercing errors to NaN
df['column1'] = pd.to_numeric(df['column1'], errors='coerce')

# Now perform the numeric operation
result = df['column1'].apply(float)

In this solution, the 'column1' series is converted to a numeric format using the pd.to_numeric() function. The substitution of NaN for non-convertible values is ensured by the errors='coerce' parameter. The numeric operation can be carried out without causing a "TypeError" once the conversion is finished.

Handling Missing Values

# Filling missing values with a default value (e.g., 0)
df['column1'].fillna(0, inplace=True)

# Now perform the numeric operation
result = df['column1'].apply(float)

In this solution, the fillna() function is used to fill in the missing values in the 'column1' series with a default value, such as 0. This guarantees that every value in the series is a number, enabling the numeric operation to be carried out successfully.

Exploring Best Practices

In addition to fixing the current issue, using best practices when working with Pandas dataframes will help reduce the possibility of similar errors occurring in the future.

Data Type Consistency

A dataframe's ability to function properly depends on the data types inside it being consistent. Make sure the series only has numerical values in it before beginning any numerical actions. To determine a series' data type, utilize the dtype attribute.

# Checking the datatype of the 'column1' Series
print(df['column1'].dtype)

To standardise the data type, if it is not what you expected, think about using conversion functions like pd.to_numeric().

Error Handling

Adopting strong error-handling procedures becomes essential in situations where data quality fluctuates. The graceful handling of non-convertible values is made possible by the use of functions like pd.to_numeric() with error handling options like errors='coerce,' which prevent errors from stopping the process altogether.

Missing Value Strategies

In data analysis, handling missing values is a frequent problem. To make sure the data frame is still appropriate for numeric operations, use an appropriate approach, such as using fillna() to fill in missing values with a default or calculated value.

Data Cleaning and Preprocessing

Take the time to thoroughly clean and preprocess your data before executing any complicated operations. The chance of running into errors later on can be greatly decreased by spotting and fixing outliers, inconsistent data, and missing values early in the process.

Conclusion

Finally, the Pandas "TypeError: cannot convert the series to<class 'float'>" message is a useful reminder of how critical it is to preserve consistent, high-quality data in the field of data science. For data manipulation processes to be error-free, it is essential to comprehend the root causes of these errors and put workable fixes in place. Users can strengthen their data analysis workflows and derive valuable insights from a variety of datasets by implementing strong error handling, encouraging data type consistency, and developing strategies for handling missing values. Python is still the most popular language for data science and analysis, which emphasises how important it is to understand the subtleties of error correction and follow best practices in order to successfully negotiate the complexities of data manipulation. Gaining expertise in error correction and best practices implementation is not only a necessary skill in this ever-changing environment, but also a vital means of gaining access to insightful knowledge from the wide and diverse world of datasets.

References

1 pandas.to_numeric — pandas documentation. (n.d.). Retrieved from https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.to_numeric.html

Which method can be used to add a group of elements to a set?

Wed, 14 Jan 2026 11:54:37 +0000

Introduction

Adding a group of elements to a set in Python is a common task encountered in programming when dealing with collections of unique elements. This problem arises when there's a need to incorporate multiple items into an existing set efficiently. Sets in Python ensure uniqueness, meaning duplicate elements are automatically filtered out. However, adding a group of elements to a set requires special attention to maintain this uniqueness while achieving the desired functionality. Fortunately, Python offers various methods to address this challenge, providing flexibility and efficiency in managing sets. By understanding these methods, programmers can effectively manipulate sets to accommodate new elements while preserving their distinctiveness.

What is a Set?

In Python, sets are a fundamental data structure used for storing collections of unique elements. Unlike lists or tuples, sets do not allow duplicate values, ensuring that each element within the set is unique. Sets are unordered, meaning that the elements are not stored in any particular order, and they are mutable, allowing for dynamic changes to the set's contents.
Sets in Python are like containers that holds a collection of unique items.
Imagine you have an empty bag where you can only keep unique items; that's essentially what a set is.
Each item in a set is distinct, and the order doesn't matter. If you try to put a duplicate item, it simply won't go in.
Sets in Python are denoted by curly braces {} enclosing a comma-separated list of elements, or by using the set() constructor.

Let's look at a small example to understand what a set is,

EXAMPLE

CODE

  # Unique Elements
     fruits_1 = {'mango', 'apple', 'orange', 'kiwi'}
     print("Set of Fruits-1 : ", fruits_1)

     # Even if I repeat elements, it will consider that element to be unique only
     fruits_2 = {'mango', 'apple', 'orange', 'kiwi','orange'}
     print("Set of Fruits-2: ", fruits_2)

OUTPUT

Adding a group of elements to a set

Using update() method
Using | Operator
Using Set Union Method - union()
Using Set Comprehension
Using Set Constructor

These methods offer flexibility in adding a single element or a group of elements to a set in Python-3.

CODE

# Define a set
my_set = {1, 2, 3}

# Define a list of elements to add
to_add_elements = [4, 5, 6]

Method - 1

Using `'update()'` method

This method directly adds multiple elements to a set in Python-3.
It takes an iterable (such as a list, tuple, or set) as input and modifies the set by adding all elements from the iterable.
Example: my_set.update(elements_to_add).

CODE

# Method 1: Using update() method
my_set.update(to_add_elements)
print("my_set", my_set)

OUTPUT

Tip: Use `update()` method when you want to directly add multiple elements to a set.

Method - 2

Using `'|'` Operator

The set union operator (|) performs a union operation between two sets, adding elements from both sets to create a new set.
It combines the existing set with another set created from an iterable using the set() constructor.
Example: my_set |= set(elements_to_add).

CODE

# Method 2: Using '|' operator
my_set |= set(to_add_elements)
print("my_set : ", my_set)

OUTPUT

Method - 3

Using Set Union Method - `'union()'`

The union() method returns a new set containing all the elements from the existing set and an iterable.
It doesn't modify the original set but creates a new set with the
combined elements.
Example: my_set = my_set.union(elements_to_add)

CODE

# Method 3: Using union() method
my_set = my_set.union(to_add_elements)
print("my_set : ", my_set)

OUTPUT

Method - 4

Using Set Comprehension

Set comprehension is a concise way to create sets in Python by iterating over an iterable.
It allows for adding elements from the iterable to the set while creating it.
Example: my_set = {element for element in my_set} | set(elements_to_add).

CODE

# Method 4: Using set comprehension
my_set = {element for element in my_set} | set(to_add_elements)
print("my_set : ", my_set)

OUTPUT

Tip: Employ set comprehension for a concise way to create sets by iterating over an iterable.

Method - 5

Using Set Constructor

The set() constructor creates a new set from an iterable, automatically removing any duplicate elements.
It can be combined with set operations like union to add elements from the iterable to the existing set.
Example: my_set = my_set.union(set(elements_to_add)).

CODE

# Method 5: Using set constructor
my_set = my_set.union(set(to_add_elements))
print("my_set : ", my_set)

OUTPUT

All the 5 Methods in One Go!

# Define a set
my_set = {1, 2, 3}

# Define a list of elements to add
elements_to_add = [4, 5, 6]

# Method 1: Using update() method
my_set.update(elements_to_add)

# Method 2: Using set union (| operator)
my_set |= set(elements_to_add)

# Method 3: Using union() method
my_set = my_set.union(elements_to_add)

# Method 4: Using set comprehension
my_set = {element for element in my_set} | set(elements_to_add)

# Method 5: Using set constructor
my_set = my_set.union(set(elements_to_add))

Note: Pay attention to the performance implications of each method, especially when dealing with large sets.

Best Practices

Here are some best practices to follow when adding elements to a set:

Choose the most appropriate method:

Depending on your specific use case and preference, choose the method that best suits your needs. Each method has its advantages, but some might be more efficient or readable depending on the context.
Use update() for simplicity:

When you want to add multiple elements to an existing set and modify it in place, update() is the simplest and most straightforward method. It directly modifies the original set without creating a new one.
Document your code:

Clearly document your code, especially if you're using less common methods or if the purpose of the code might not be immediately obvious. Documenting your code helps other developers understand your intentions and use your code effectively.
Test thoroughly:

Test your code with various inputs and edge cases to ensure it behaves as expected. Consider writing unit tests to automate testing and catch regressions.

Note: Ensure that the elements you're adding to the set are unique to maintain set's uniqueness property.

Caution: Be cautious when using set comprehension with large iterables, as it may lead to increased memory usage.

Summary

Sets in Python are collections of unique elements, akin to a bag where each item is distinct.
They are useful for tasks requiring uniqueness, like removing duplicates or finding unique elements.
Various methods exist to add a group of elements to a set, including update(), set union (| operator), union() method, set comprehension, and set constructor.

Conclusion

In simpler terms, when working with sets in Python, you have different methods for adding multiple items. You can choose the one that suits your needs best:

If you want a straightforward approach, you can use update().
For a more elegant solution, you can use | (the pipe symbol).
If you need more flexibility, union() is a good option.
If you prefer concise code, you can use set comprehension.
And if you need to build a set from other iterable types, you can use the set constructor.

Understanding these methods helps you manage sets effectively, ensuring that each item you add remains unique.

References

Python documentation:https://www.python.org/doc/

"Only valid with datetimeindex, timedeltaindex or periodindex, instance of 'index'" Solved

Mon, 24 Nov 2025 00:10:04 +0000

Introduction

In Python, Pandas is a powerful and effective tool for data analysis and manipulation. It performs exceptionally well with data types and structures. However, when it comes to time-series data, users occasionally encounter the following errors: "Only valid with DatetimeIndex, TimedeltaIndex, or PeriodIndex, but got an instance of 'index'". The error message usually appears when attempting time-related operations on a DataFrame or Series that does not have a proper time-based index. To resolve this error, users should use pd.to_datetime() to convert the existing index to a DatetimeIndex or set a datetime column as the index with set_index(). In this article, we will try to explore and understand the significance of using these time-based indexes in Pandas and take a look into solutions for addressing this common issue.

Understanding The Types of Index in Pandas

In Pandas, there are various types of index errors, such as Int64Index, RangeIndex, and DatetimeIndex, among others. An index is important for effective data manipulation and retrieval. Using a time-based index for time-series data enhances the performance of temporal operations seamlessly.
A DatetimeIndex is the best option for time-series data analysis. It uses timestamps as the index to provide sophisticated time-based operations. You can easily calculate time differences, resample data to different frequencies, and filter by date ranges. This systematic technique dramatically increases your performance while working with temporal data.

Significance of Time-based Indexing

Understanding change relies heavily on time-series data. It's a set of measurements taken like snapshots at precise times, allowing us to see how things change over time. Stock prices ticking every minute, or daily temperature readings taken over a year, are typical examples. The key, however, is in the time-based index, an invisible timeline that assures each data piece is in the correct location.
This time index is the key to accessing powerful analytical tools. Want to compare average monthly sales data over the last year? Using a time index makes this simple. You may easily segment the data by month, generate monthly averages, and identify trends. Need to see the broad picture? You can aggregate data into larger periods, such as quarters. Are you analyzing at a different pace?. Resample the data from hourly to daily. However, without a time index, these procedures become error-prone. The system simply cannot grasp the order of your data points, resulting in annoying error messages. Remember that a time-based index is the cornerstone for unlocking the true potential of your time series data exploration.

Common Mistakes: 'Only valid with DatetimeIndex, TimedeltaIndex, or PeriodIndex'

Ever get annoyed when you try to examine data over time and just get error messages? It's as if the computer does not grasp what you're asking for. The root cause is a missing time-based index. This index functions as a calendar for your data, ensuring that each point is properly arranged.

The issue often appears when you do time-sensitive operations on data that does not include this calendar. This can occur when you import data without providing the time component, or when the data lacks temporal information. It's like attempting to examine a movie with every scene jumbled together; you can't follow the tale! So, a time-based index is critical for making sense of your data and identifying trends across time.

Debugging the Issue: Converting to Time-based Index

To debug the error, You are required to convert the existing index to a time-based index. There are several methods for that purpose, which are:

Using pd.to_datetime()

The pd.to_datetime() function is a flexible tool used for converting various date-like objects to Pandas datetime objects. You can transform your existing dataframe into a DatetimeIndex by implementing this function.

import pandas as pd

# df is a data frame with a generic index
df.index = pd.to_datetime(df.index)

Creating a DatetimeIndex

You can create a new DatetimeIndex based on that column with time-related information in your dataset.

import pandas as pd
# Here, 'time_column' is containing time related information such as date
df['time_column'] = pd.to_datetime(df['time_column'])
df.set_index('time_column', inplace=True)

Generating a TimedeltaIndex

You can convert your index to TimedeltaIndex if the index represents the duration.

import pandas as pd
# here df is a data frame with index representing durations
df.index = pd.to_timedelta(df.index)

Utilising PeriodIndex

You can create a PeriodIndex when you are dealing with data that is relevant for a specific period (e.g., monthly or quarterly).

import pandas as pd

# here df is data frame with an index representing periods with freq='M' seting it frequency to monthly.
df.index = pd.PeriodIndex(df.index, freq='M')

Understand the Debugging Process with a real-world example

Let's now understand the debugging process with a practical example. Suppose you have a data frame with a non time-based index and humidity readings.

import pandas as pd
import numpy as np

# Creating a sample data frame
data = {'Humidity': np.random.rand(5)}
df = pd.DataFrame(data)

# Trying a time-based operation without a suitable index
try:
 df['Date'] = ['2022-01-01', '2022-01-02', '2022-01-03', '2022-01-04', '2022-01-05']
 df.index = df['Date']
 daily_mean = df.resample('D').mean()
except ValueError as e:
 print(f"Error: {e}')

In the above example, we tried to assign a date column to the index, which is recognised as a string by Pandas. If you try any datetime operation on the index that results in errors, which require a proper time-based index to debug.
Now, let's convert the index to a DatetimeIndex for debugging.

# Convert the index to DatetimeIndex
df.index = pd.to_datetime(df.index)

This above change will allow you to use time-based operations on the DataFrame seamlessly.

Conclusion

In summary Pandas time series data handling requires the use of time-based indexes. They guarantee that your data is accurately arranged and give you the ability to carry out effective time-based analysis. It's a good idea to confirm that your data has the correct time index before beginning any time series analysis, as indicated by the error notice Only valid with DatetimeIndex, TimedeltaIndex, or PeriodIndex".

Fortunately, Pandas has ways to change your data into a temporally organized format, such as using pd.to_datetime(), DatetimeIndex, TimedeltaIndex, or PeriodIndex. By using these techniques, you may avoid mistakes and fully utilize Pandas' time series capabilities, which will enable you to gain insightful knowledge from your data.

References

Pandas Documentation: https://pandas.pydata.org/docs/
Python Documentation ( Datetime module ):
https://docs.python.org/3/library/datetime.html

Performance Comparison: Pandas vs. NumPy for Data Analysis

Sat, 04 Oct 2025 12:29:19 +0000

NumPy excels at fast, memory-efficient numerical computations on homogeneous data arrays and is significantly faster for basic operations like arithmetic, slicing, and mean calculations, especially on smaller datasets (less than 50,000 rows).

Pandas provides user-friendly tools for tabular, heterogeneous data, supports complex operations (like groupby, merging, handling missing data), and allows more flexible data manipulation, but has more overhead resulting in slower performance for basic calculations compared to NumPy.

For large datasets (500,000 rows or more), Pandas can outperform NumPy for certain data operations due to internal optimizations; between 50,000 and 500,000 rows, performance differences are operation-dependent.

NumPy uses less memory and is optimal for machine learning model inputs, while Pandas is preferred for rich data analysis, data cleaning, and working directly with external sources like CSV or Excel files.

Indexing is faster in NumPy, while Pandas offers more advanced indexing and labeling but at the cost of speed.

Mastering Data Analysis and Manipulation with Pandas: A Comprehensive Guide

Mon, 31 Mar 2025 21:40:00 +0000

This article covers Pandas, an essential Python data analysis and manipulation library. Data handling is made easier using Pandas, from pre-processing to more complex tasks like time series analysis. It addresses the generation of pivot tables, method chaining for effective processing, and handling of both text and category data, with an emphasis on advanced techniques. Users can get more robust insights and improve their data analysis operations by becoming proficient with these functionalities.

Table of Contents:

Data Cleaning and Preprocessing
Data Visualization with Pandas
Time Series Analysis with Pandas
Advanced Pandas Techniques
Conclusion

Data Cleaning and Preprocessing

1. Handling missing data

Pandas manages missing data in DataFrames effectively: fillna() and dropna() replace or drop missing values, respectively, and isnull() or notnull() detects them.

import pandas as pd 

data = {'A': [1, 2, None, 4],

        'B': [None, 5, 6, 7],

        'C': [8, 9, 10, 11]}

df = pd.DataFrame(data)

# Detecting missing values

print("Missing Values:")

print(df.isnull())

# Filling missing values with 0

filled_df = df.fillna(value=0)

print("\nDataFrame after filling missing values:")

print(filled_df)

clean_df = df.dropna()

print("\nDataFrame after dropping missing values:")

print(clean_df)

2. Data Type Conversion

Pandas frequently performs data type conversion to guarantee consistency and interoperability across various data types. You can convert the data types of columns in a Data Frame using the astype() method.

import pandas as pd

data = {'Name': ['Alice', 'Bob', 'Charlie'], 'Age': ['25.5, '30.3', '22.7'],# Age as string 'Date': ['2022-01-01', '2022-02-01', '2022-03-01']} # Date as strings

df = pd.DataFrame(data)

# Convert 'Age' column to float

df['Age'] = df['Age'].astype(float)

# Convert 'Date' column to datetime

df['Date'] = pd.to_datetime(df['Date'])

print(df)

Caution: Use caution when processing huge datasets in pieces, as this can still result in high memory usage, particularly when complicated calculations or data manipulation are involved.

3. Removing Duplicates

During data cleaning and pre processing, removing duplicates in Pandas Data Frame is a common procedure. Duplicates can be removed using the 'drop_duplicates()' method.

import pandas as pd

data = {'Name': ['John', 'Anna', 'John', 'Linda', 'Anna'],

        'Age': [28, 35,  28, 45, 35]}

df = pd.DataFrame(data)

# Remove duplicate rows

unique_df = df.drop_duplicates()

print("DataFrame after removing duplicate rows:")

print(unique_df)

4. Data normalization and scaling

Preprocessing data with scaling and normalisation helps to standardise feature ranges for better machine learning. Standardisation sets features to mean 0 and standard deviation 1, whereas normalisation scales characteristics to 0-1.

import pandas as pd 

from sklearn.preprocessing import MinMaxScaler, StandardScaler

# Sample DataFrame

data = {'Feature1': [10, 20, 30, 40, 50],

        'Feature2': [1, 2, 3, 4, 5]}

df = pd.DataFrame(data)

# Min-Max scaling

min_max_scaler = MinMaxScaler()

normalized_data = min_max_scaler.fit_transform(df[['Feature1', 'Feature2']])

# Standardization

standard_scaler = StandardScaler()

standardized_data = standard_scaler.fit_transform(df[['Feature1', 'Feature2']])

print("Normalized Data:")

print(normalized_data)

print("\nStandardized Data:")

print(standardized_data)

Data Visualization with Pandas

With Pandas, you can quickly gain understanding of your data by creating simple plots straight from Data Frames for data visualization. Pandas uses the robust charting tool Matplotlib to facilitate visualization.

1. Basic plotting

Pandas offers easy-to-use techniques for basic plotting, including scatter plots, histograms, bar graphs, and line plots. On a Data Frame or Series, you can directly call these graphing functions.

import pandas as pd

import matplotlib.pyplot as plt

# Creating a DataFrame

data = {'Date': ['2022-01-01', '2022-01-02', '2022-01-03', '2022-01-04'],

        'Value': [10, 15, 20, 25],

        'Category': ['A', 'B', 'C', 'D'],

        'Count': [5, 3, 4, 2],

        'Feature1': [1, 2, 3, 4],

        'Feature2': [5, 6, 7, 8]}

df = pd.DataFrame(data)

# Line plot

df.plot(x='Date', y='Value', kind='line')

plt.title('Line Plot')

# Bar plot

df.plot(x='Category', y='Count', kind='bar')

plt.title('Bar Plot')

# Show plots

plt.show()

2. Customizing plots

To alter the look of plots, pandas plotting functions support a wide range of choices, such as colours, labels, titles, grid lines, and more. You can customize the visualization to meet your own requirements by adjusting these settings.

import pandas as pd

# Creating a DataFrame

data = {'Date': ['2022-01-01', '2022-01-02','2022-01-03', '2022-01-04'],

        'Value': [10, 20, 15, 25]}

df = pd.DataFrame(data)

# Changing colors

df.plot(color=['blue'])

# Adding labels and title

df.plot(x='Date', y='Value', xlabel='Date',

        ylabel='Value', title='Value over Time')

3. Subplots and multiple axes

Pandas enables the setting up of numerous axes and subplots inside a single figure. You can compare several datasets or different features of your data by visualizing them side by side with this functionality.

import pandas as pd

# Creating a DataFrame

data = {'Date': ['2022-01-01', '2022-01-02', '2022-01-03', '2022-01-04'],

        'Value': [10, 20, 15, 25],

        'Count': [5, 3, 4, 2]}

df = pd.DataFrame(data)

# Convert 'Date' column to datetime format

df['Date'] = pd.to_datetime(df['Date'])  

# Subplots

df.plot(subplots=True, figsize=(10, 8))

# Multiple axes

ax1 = df.plot(x='Date', y='Value', color='blue')

ax2 = df.plot(x='Date', y='Count', color='red', secondary_y=True, ax=ax1)

Time Series Analysis with Pandas

Using Pandas for time series analysis involves handling and examining data that is indexed by time.

1. Handling time series data

Pandas can be used to work with time series data by creating datetime objects from strings or numeric representations of dates. Date-like objects can be converted into datetime objects using the pd.to_datetime() function.

import pandas as pd

data = {'Date': ['2024-03-13', '2024-03-14', '2024-03-15'],

        'Value': [10, 20, 30]}

df = pd.DataFrame(data)

# Parsing dates

df['Date'] = pd.to_datetime(df['Date'])

# Setting datetime index

df.set_index('Date', inplace=True)

print(df)

2. Resampling and frequency conversion

The resample() function in Pandas allows you to aggregate data at multiple intervals of time, either up-sampling or down-sampling (data to a higher or lower frequency).

import pandasa as pd

data = {'Date': pd.date_range(start='2024-01-01', end='2024-03-15'),

        'Value': range(1, 76)}

df = pd.DataFrame(data)

# Setting 'Date' column as datetime index

df.set_index('Date', inplace=True)

# Downsampling from daily to monthly

monthly_data = df.resample('M').mean()

# Upsampling from monthly to daily

daily_data = monthly_data.resample('D').ffill()  # Forward fill missing values

print("Monthly Data:")

print(monthly_data.head())

print("\nDaily Data:")

print(daily_data.head())

Tip: To handle data in smaller chunks, reduce memory errors, and increase efficiency for huge datasets, use chunksize when using pandas' read_csv() or read_sql() functions.

3. Time shifting and lagging

Pandas has methods like shift() and diff() that let you shift or lag time series data. Data shifting can be helpful in calculating variations or changes over time.

import pandas as pd

data = {'Date': pd.date_range(start='2024-01-01', end='2024-01-05'),

        'Value': [10, 15, 20, 25, 30]}

df = pd.DataFrame(data)

# Shifting data by one period

df['Shifted'] = df['Value'].shift(1)

# Calculating differences between consecutive elements

df['Diff'] = df['Value'].diff()

print(df)

Advanced Pandas Techniques

In the scope of fundamental functions, efficient and effective data manipulation and analysis are made possible by advanced Pandas approaches.

1. Handling categorical data

Categorical data is effectively managed by Pandas using pd. To create dummy variables, use pd.get_dummies() and categorical() for fixed distinct values.

import pandas as pd 

data = {'Category': ['A', 'B', 'C', 'A', 'B', 'C']}

df = pd.DataFrame(data)

# Convert 'Category' column to categorical

df['Category'] = pd.Categorical(df['Category'],categories=['A', 'B', 'C'])

# Convert categorical variable into dummy/indicator variables

dummy_df = pd.get_dummies(df['Category'], prefix='Category')

print("Original DataFrame:")

print(df)

print("\nDummy DataFrame:")

print(dummy_df)

2. Working with text data

Pandas has several string methods (such str.lower() and str.contains()) that can be used to work with text data in Data Frame columns. Advanced text processing activities can also be carried out with regular expressions.

import pandas as pd 

data = {'Text': ['This is a Text with Keyword', 'Another text without the keyword']}

df = pd.DataFrame(data)

# Convert 'Text' column to lowercase

df['Text'] = df['Text'].str.lower()

# Check if text contains a specific substring

contains_substring = df['Text'].str.contains('keyword')

# Extract text using regular expressions

extracted_text = df['Text'].str.extract(r'(\d+)')

print(df)

print(contains_substring)

print(extracted_text)

3. Pivot tables and cross-tabulations

To aggregate and sum data based on variables, pivot tables can be created using Pandas' pivot_table() function. In a similar manner, cross-tabulations are used by pd.crosstab() to compute frequency tables.

import pandas as pd 

import numpy as np

data = {'Date': pd.date_range(start='2024-01-01', periods=6),

        'Category': ['A', 'B', 'A', 'C', 'B', 'C'],

        'Value': [10, 20, 15, 25, 30, 35]}

df = pd.DataFrame(data)

# Create a pivot table

pivot_table = pd.pivot_table(df, values='Value', index='Date', columns='Category', aggfunc=np.sum)

print("Pivot Table:")

print(pivot_table)

# Sample DataFrame for cross-tabulation

data_cross_tab = {'Category1': ['A', 'B', 'A', 'C', 'B', 'C'], 'Category2': ['X', 'Y', 'X', 'Y', 'X', 'Y']}

df_cross_tab = pd.DataFrame(data_cross_tab)

# Create a cross-tabulation

cross_tab = pd.crosstab(df_cross_tab['Category1'], df_cross_tab['Category2'])

print("\nCross-Tabulation:")

print(cross_tab)

4. Method chaining

Code readability and workflow efficiency are increased by the Pandas method chaining, which reduces several processes into a single line.

import pandas as pd 

data = {'Category': ['A', 'B', 'A', 'C', 'B', 'C'],

        'Value': [10, 20, 15, 25, 30, 35]}

df = pd.DataFrame(data)

# Group by 'Category', calculate mean of 'Value', reset index, and rename column

result = df.groupby('Category').agg({'Value': 'mean'}).reset_index().rename(columns={'Value': 'Mean_Value'})

print(result)

FAQ Q: What is the significance of handling missing data?
A: Handling missing data is crucial for accurate analysis and modelling. Pandas offers methods like isnull(), fillna(), and dropna() for dealing with missing values effectively.

Conclusion

In conclusion, becoming proficient with Pandas gives data workers a flexible framework for effectively managing, analyzing, and visualizing structured data in Python. Pandas' extensive capability allows users to address a broad range of data-related tasks, from creating pivot tables and managing missing data to executing sophisticated methods like time series analysis and method chaining. It improves data analysis and manipulation efficiency by promoting improved decision-making, getting deeper insights, and optimising processes.

For Refering to Next Part of the Article:

Advancing with Pandas: Beyond the Basic

For Referring to Previous Part of the Article:

Pandas in Python: A Comprehensive Guide

Advancing with Pandas: Beyond the Basic

Thu, 27 Mar 2025 10:52:14 +0000

In this article, we will look into Pandas, a core Python package for data analysis and manipulation. Now, we'll explore more complex topics to improve your data handling skills, building on the fundamental concepts we covered in earlier sections. This thorough tutorial gives you the skills and information required to successfully handle challenging data problems, from performance optimization to integrating Pandas with external libraries and carrying out practical data analysis projects. Gaining proficiency in these sophisticated Pandas techniques will enable you to go deeper into your dataset analysis and extract even more value, whether you're an experienced data expert or just starting out.

Table of Contents:

Performance Optimization with Pandas
Pandas and External Libraries Integration
- Integrating Pandas with other Python libraries
- Converting between Pandas Data Frames and other data structures
Real-world Data Analysis Projects
Conclusion

Performance Optimization with Pandas

Enhancing the effectiveness and speed of data manipulation operations is known as performance optimization in Pandas, and it is particularly important when working with big datasets.

1. Techniques for improving performance

Vectorized Operations: To efficiently conduct element-wise operations on arrays, vectorized operations make use of the NumPy library. Compared to similar procedures carried out using loops, these operations are substantially faster.
Python loops should be avoided since they might be slow with huge datasets. For operations on complete columns or rows at once, use Pandas methods or vectorized operations instead.
The Effective Functions provided by pandas, such as pivot_table(), applymap(), groupby(), and map(), provide optimal performance and eliminate the need for additional functions and loops.

import numpy as np 

# Generate a random number between 0 and 1

random_num = np.random.rand()

print(random_num)

Tip: Use the built-in Pandas functions and NumPy's vectorized operations to efficiently execute element-wise operations without resorting to sluggish Python loops.

2. Memory Optimization

Pandas memory optimization is essential for managing big datasets effectively and minimizing memory consumption. There are various methods that can be used to accomplish this. Firstly, columns can be converted to more memory-efficient types like int32 or float32 without sacrificing precision by downcasting numeric data types using the 'astype()' method. Second, the Data Frame's memory footprint can be greatly reduced by eliminating any columns that are not needed for analysis. A different strategy is to use the category data type for categorical variables, which can significantly lower memory consumption—particularly for columns that have few unique values.

import pandas as pd 

import numpy as np

# Creating a large DataFrame

n_rows = 1000000

data = {'A': np.random.randint(0, 100, n_rows),

        'B': np.random.rand(n_rows)}

df = pd.DataFrame(data)

# Vectorized operations

df['C'] = df['A'] * 2 + df['B']  # Vectorized operation instead of a loop

# Using efficient Pandas methods

mean_value = df['A'].mean()  # Efficient computation of mean using Pandas method

# Memory optimization

df['A'] = df['A'].astype(np.int32)  # Downcasting to int32 for memory optimization

Caution: Be mindful of potential memory overhead generated by specific procedures while optimizing performance since this could result in higher memory usage and slower execution.

Pandas and External Libraries Integration

Pandas enhances Python's ability to analyze and manipulate data by integrating with a variety of external libraries with ease.

1. Integrating Pandas with other Python libraries

Pandas may be used for a wider range of data analysis jobs and has more capability when it is integrated with other Python packages. Since Pandas Series and Data Frames are constructed on top of NumPy arrays, there is a common integration with NumPy that enables smooth interoperability between the two libraries. Effective vectorized operations and mathematical calculations on Pandas data structures are made possible by this integration. Furthermore, Pandas easily interacts with Seaborn and Matplotlib for data visualisation, providing easy ways to generate informative charts from Data Frame data.

import numpy as np

import matplotlib.pyplot as plt

from sklearn.linear_model import LinearRegression

# Sample DataFrame and column names (replace with your own)

df = pd.DataFrame({

    'Date': pd.date_range(start='2024-01-01', periods=6),

    'Feature1': np.random.rand(6),

    'Feature2': np.random.rand(6),

    'Target': np.random.rand(6),

    'Value': np.random.rand(6)

})

# Replace 'Feature1', 'Feature2', and 'Target' with your actual column names

X = df[['Feature2', 'Feature1']].values

y = df['Target'].values

# Use Pandas plotting with Matplotlib

df.plot(x='Date', y='Value', kind='line', ax=plt.gca())

# Use Pandas DataFrame as input for Scikit-learn model

model = LinearRegression()

model.fit(df[['Feature1', 'Feature2']], df['Target'])

2. Converting between Pandas Data Frames and other data structures

Interoperability with different data formats is made possible by the frequent process of converting between Pandas DataFrames and other data structures. For this reason, Pandas offers simple ways that make data sharing and integration across various data processing workflows easier to accomplish.

DataFrame to NumPy array:

Use the 'values' attribute of the DataFrame to obtain a NumPy array representation.

import pandas as pd

# Create a DataFrame

df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})

# Convert DataFrame to NumPy array

array = df.values

NumPy array to DataFrame:

Pass the NumPy array as data to the DataFrame constructor, along with column names.

import pandas as pd

import numpy as np

# Create a NumPy array

array = np.array([[1, 4], [2, 5], [3, 6]])

# Convert NumPy array to DataFrame

df = pd.DataFrame(array, columns=['A', 'B'])

DataFrame to dictionary:

Use the 'to_dict()' method of the DataFrame to convert it into a dictionary.

import pandas as pd

# Create a DataFrame

df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})

# Convert DataFrame to dictionary

dictionary = df.to_dict()

Dictionary to DataFrame:

Use the 'from_dict()' method of the DataFrame to create a DataFrame from a dictionary.

import pandas as pd

# Create a dictionary

dictionary = {'A': {0: 1, 1: 2, 2: 3}, 'B': {0: 4, 1: 5, 2: 6}}

# Convert dictionary to DataFrame

df = pd.DataFrame.from_dict(dictionary)

Tip: Process data in smaller parts using methods like chunking or streaming when working with large datasets that don't fit in memory to prevent memory issues.

Real-world Data Analysis Projects

Practical challenges can be solved with Pandas, as demonstrated by real-world data analysis projects. Typically, these projects involve activities like modelling, data exploration, data cleaning, and visualization.

import pandas as pd

import matplotlib.pyplot as plt

from sklearn.linear_model import LinearRegression

# Load data

sales_data = pd.read_csv('/path/to/sales_data.csv')  # Provide your file path

# Clean data

sales_data.dropna(inplace=True)

# Explore data

total_sales = sales_data['Sales'].sum()

average_sales = sales_data['Sales'].mean()

# Visualize data

sales_data['Date'] = pd.to_datetime(sales_data['Date'])

sales_data.plot(x='Date', y='Sales', kind='line')

plt.title('Sales Over Time')

plt.xlabel('Date')

plt.ylabel('Sales')

# Model data

model = LinearRegression()

model.fit(sales_data[['Advertising']], sales_data['Sales'])

The given script is essential for comprehending sales trends and assessing the effectiveness of advertizing campaigns in a real-world data research project. The main concepts of the script are stated below:

Importing and Cleaning Data:

First, We have imported necessary libraries, such as scikit-learn, Matplotlib, and Pandas.
It loads the sales information from a CSV file, most likely containing past sales records and associated advertising costs.
In order to guarantee data integrity, data cleaning is carried out, and the dropna() method is used to eliminate missing values. This stage is essential to a precise analysis.

Analysing exploratory data:

Calculations are made to determine important metrics like average and total sales. These metrics help with subsequent analysis by offering preliminary insights into the dataset.

Data Visualization:

Using Matplotlib, the following example shows sales trends over time. Through the conversion of the date column to datetime format and the creation of a sales line plot, the evolution of sales across several time periods can be clearly visualized. The data's patterns, seasonality, and anomalies may be seen in this visualization.

Modeling:

The association between sales success and advertising cost is measured using linear regression modelling.
Using the LinearRegression class from scikit-learn, the example applies a linear regression model to the data. This makes it possible to estimate the effects of variations in advertising expenditure on sales.
Businesses can learn more about the efficacy of their advertising strategy and make data-driven decisions to maximise the performance of next campaigns by modelling the data.

FAQ
Q: How can I speed up my code when working with large datasets in pandas?
A: Avoid using loops whenever possible, as they can be slow. Instead, use vectorized operations or pandas' built-in methods, such as apply(), map(), and groupby(), which are optimized for performance.
Q: Can I use Pandas with database libraries like SQLAlchemy or psycopg2 for database operations?
A: Absolutely! Pandas supports integration with database libraries such as SQLAlchemy for interacting with SQL databases and psycopg2 for PostgreSQL databases. You can use Pandas to read data from databases into DataFrames, perform data manipulation and analysis, and write the results back to the database.
Q: Is it possible to integrate Pandas with libraries like TensorFlow or PyTorch for deep learning?
A: Yes, Pandas can be used alongside deep learning frameworks like TensorFlow and PyTorch. You can preprocess and manipulate data with Pandas before feeding it into neural networks built using TensorFlow or PyTorch. Pandas' data manipulation capabilities complement the data preprocessing requirements of deep learning tasks.

Conclusion

In conclusion, Pandas is a flexible Python data analysis and manipulation toolkit that provides strong capabilities for working with structured data. Pandas is a vital tool for data scientists, analysts, and researchers because of its user-friendly interface and wide range of features, which make it possible to efficiently clean, explore, and visualise datasets. You may improve your data workflows, glean insightful information, and facilitate well-informed decision-making by becoming proficient with Pandas.

For Referring to Previous Part1 of the Article:

Pandas in Python: A Comprehensive Guide

For Referring to Previous Part2 of the Article:

Mastering Data Analysis and Manipulation with Pandas: A Comprehensive Guide

Pandas v3.x Defaults Copy-on-Write Feature - Get Used to it Early

Tue, 07 Jan 2025 06:13:57 +0000

Pandas supports Copy-on-Write, an optimization technique that helps improve memory use, particularly when working with large datasets.

Starting from version 2.0 of Pandas, the Copy-on-Write (CoW) has taken effect but has not been fully implemented. Most of the optimizations that are possible through Copy-on-Write are supported.

Aim of Copy-on-Write

As the name suggests, the data will be copied when it is modified. What it means?

When a DataFrame or Series shares the same data as the original, it will initially share the same memory for the data rather than creating a copy. When the data of either the original or new DataFrame is modified, a new copy of the data is created for the DataFrame that is being modified.

This will efficiently save memory usage and improve performance when working with large datasets.

Enabling CoW in Pandas

It is not enabled by default, so we need to enable it using the copy_on_write configuration option in Pandas.

import pandas as pd

# Option1
pd.options.mode.copy_on_write = True
# Option2
pd.set_option("mode.copy_on_write" : True)

You can use any of the options to turn on CoW globally in your environment.

Note: CoW will be enabled by default in Pandas 3.0, so get used to it early on.

Impact of CoW in Pandas

The CoW will disallow updating the multiple pandas objects at the same time. Here's how it will happen.

import pandas as pd
df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})
subset = df["A"]
subset.iloc[0] = 10
df

With CoW, the above snippet will not modify df rather it modifies only the data of subset.

# df
AB
014
125
236

# subset
A
010
12
23

inplace Operations will Not Work

Similarly, the inplace operations will not work with CoW enabled, which directly modifies the original df.

df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})
df["A"].replace(1, 5, inplace=True)
df
--------------------
AB
014
125
236

We can see that df has remained unchanged and additionally, we will see a ChainedAssignmentError warning.

The above operation can be performed in two different ways. One method is to avoid inplace, and another is to use inplace to directly modify the original df at the DataFrame level.

# Avoid inplace
df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})
df["A"] = df["A"].replace(1, 5)
df
--------------------
    AB
054
125
236

# Using inplace at DataFrame level
df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})
df.replace({"A": {2: 34}}, inplace=True)
df
--------------------
    AB
014
1345
236

Chained Assignment Will Never Work

When we modify the DataFrame or Series using multiple indexing operations in a single line of code, this is what we call the chained assignment technique.

# CoW disabled
with pd.option_context("mode.copy_on_write", False):
    df = pd.DataFrame({"A": [1, 2, 3, 4], "B": [5, 6, 7, 8]})
    df["B"][df['A'] > 2] = 10
df

The above code snippet is trying to change column B from the original df where column A is greater than 2. It means the value at the 2nd and 3rd index in column B will be modified.

Since the CoW is disabled, this operation is allowed, and the original df will be modified.

But, this will never work with CoW enabled in pandas.

# CoW enabled
df = pd.DataFrame({"A": [1, 2, 3, 4], "B": [5, 6, 7, 8]})
df["B"][df["A"] > 2] = 10
df
--------------------
    AB
015
126
237
348

Instead, with copy-on-write, we can use .loc to modify the df using multiple indexing conditions.

# CoW enabled
df = pd.DataFrame({"A": [1, 2, 3, 4], "B": [5, 6, 7, 8]})
df.loc[(df["A"] == 1) | (df["A"] > 3), "B"] = 100
df

This will modify column B where column A is either 1 or greater than 3. The original df will look like the following.

Read-only Arrays

When a Series or DataFrame is accessed as a NumPy array, that array will be read-only if the array shares the same data with the initial DataFrame or Series.

df = pd.DataFrame({"A": [1, 2, 3, 4], "B": ['5', '6', '7', '8']})
arr = df.to_numpy()
arr
--------------------
array([[1, '5'],
       [2, '6'],
       [3, '7'],
       [4, '8']], dtype=object)

In the above code snippet, arr will be a copy because df contains two different types of arrays (int and str). We can perform modifications on the arr.

arr[1, 0] = 10
arr
--------------------
array([[1, '5'],
       [10, '6'],
       [3, '7'],
       [4, '8']], dtype=object)

Take a look at this case.

df = pd.DataFrame({"A": [1, 2, 3, 4], "B": [5, 6, 7, 8]})
arr = df.to_numpy()
arr

The DataFrame df has only one NumPy array (array of same data types), so arr shares the data with df. This means arr will be read-only and cannot be modified inplace.

print(arr.flags.writeable)
arr[0,0] = 10
arr
--------------------
False
ValueError: assignment destination is read-only

Lazy Copy Mechanism

When two or more DataFrames share the same data, the copies will not be created immediately.

df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})
df2 = df.reset_index(drop=True)

Both df and df2 shares the same reference in the memory as both share the same data. The copy mechanism will trigger only when any of the DataFrame is modified.

df2.iloc[0, 0] = 10
print(df2)
print(df)
--------------------
    A  B
0  10  4
1   2  5
2   3  6
   A  B
0  1  4
1  2  5
2  3  6

But this is not necessary, if we don't want initial df, we can simply reassign it to the same variable (df) and this process will create a new reference. This will avoid the copy-on-write process.

df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})
print("Initial reference: ",id(df))
df = df.reset_index(drop=True)
print("New reference: ",id(df))
df.iloc[0, 0] = 10
print(df)
--------------------
Initial reference:  138400246865760
New reference:      138400246860336
    A  B
0  10  4
1   2  5
2   3  6

This same optimization (lazy copy mechanism) is added to the methods that don't require a copy of the original data.

DataFrame.rename()

df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6]})
df.rename(columns={"A": "X", "B": "Y"})
--------------------
XY
014
125
236

When CoW is enabled, this method returns the original DataFrame rather than creating an entire copy of the data, unlike the regular execution.

DataFrame.drop() for axis=1

Similarly, the same mechanism is implemented for DataFrame.drop() for axis=1 (axis='columns').

df = pd.DataFrame({"A": [1, 2, 3], "B": [4, 5, 6], "C": [7, 8, 9]})
df.drop(["A"], axis=1)
--------------------
BC
047
158
269

Conclusion

Pandas will by default implement Copy-on-Write (CoW) in version 3.0. All these optimizations that are compliant with CoW will lead to efficient memory and resource management when working with large datasets.

This will reduce unpredictable or inconsistent behavior and greatly maximize performance.

Other articles you might be interested in if you liked this one

✅Merge, combine, and concatenate multiple datasets using pandas.

✅Find and delete duplicate rows from the dataset using pandas.

✅Find and Delete Mismatched Columns From DataFrames Using pandas.

✅Create temporary files and directories using tempfile module in Python.

✅Upload and display images on the frontend using Flask.

✅How does the learning rate affect the ML and DL models?

That's all for now
Keep Coding✌✌

Data Visualization with Python: Using Matplotlib and Seaborn

Sat, 06 Jul 2024 06:51:20 +0000

Effective data analysis and science depend on data visualization. It aids in the identification of patterns, developments, and linkages in data, enabling well-informed choices. Python packages like seaborn and matplotlib facilitate data visualization.

This article teaches you matplotlib and seaborn while highlighting the significance of data visualization. You'll discover how to extract insightful information, produce illuminating visualizations, and effectively convey data-driven conclusions.

Getting Started with matplotlib

Matplotlib is a well-liked Python package for making static, animated, and interactive visualizations.

Installing with pip

Use pip to install matplotlib by running the following command.

pip install matplotlib

After installation, use this command to view the Pandas version:

import matplotlib

print(matplotlib.__version__)

Tip: Use virtual environments to keep your matplotlib installs apart and your dependencies organised across many projects.

1. Basic Plotting:

Matplotlib offers various plot types, including: Line plots: plot(), Scatter plots: scatter(), Bar plots: bar().

A. Line plot:

import matplotlib.pyplot as plt

x = [1, 2, 3]

y = [2, 4, 6]

plt.plot(x, y)

plt.show()

B: Scatter Plot:

import matplotlib.pyplot as plt

x = [1, 2, 3, 4, 5]

y = [2, 3, 5, 7, 11]

plt.plot(x, y, color='blue', linestyle='-', marker='o', label='Line Plot')

plt.scatter(x, y, color='red', marker='o', label='Scatter Plot')

plt.show()

2. Customizing Plot Appearance:

Customize your plots with: Labels: xlabel(), ylabel(), Titles: title(), Colors: color, Styles: style.

import matplotlib.pyplot as plt

x = [1, 2, 3, 4, 5]

y = [2, 3, 5, 7, 11]

plt.plot(x, y)

plt.xlabel('X-axis', fontsize=12, fontweight='bold', color='blue')

plt.ylabel('Y-axis', fontsize=12, fontweight='bold', color='blue')

plt.xticks(fontsize=10, rotation=45)

plt.yticks(fontsize=10)

plt.legend(['Data'], loc='upper left', fontsize=10)

plt.text(3, 6, 'Important point', fontsize=10, color='red')

plt.show()

Exploring Advanced Features of matplotlib

We'll explore the essential procedures for using matplotlib's pyplot interface to create simple plots. Here's a thorough description and an example of the code:

Multiple Axes and Subplots

You can make complicated figures with numerous subplots and axes within a single figure using matplotlib's "Multiple Axes and Subplots" functionality.

plt.subplots(): This function is used to produce a figure with several subplots grouped in a grid format. It yields a multidimensional array of axis objects and a figure object. Axes objects are used to represent each subplot and can be used to customize or plot data for each subplot.

axes.plot(): After using plt.subplots() to generate subplots, you may plot data on particular axes by using the axes object's plot() method. With this approach, you may individually alter each subplot's appearance and attributes, including line styles, colours, markers, and labels.

import matplotlib.pyplot as plt

import numpy as np

fig, axs = plt.subplots(2, 2)

x = np.linspace(0, 2*np.pi, 400)

axs[0, 0].plot(x, np.sin(x))

axs[0, 1].plot(x, np.cos(x))

axs[1, 0].plot(x, np.sin(x**2))

axs[1, 1].plot(x, np.cos(x**2))

plt.show()

Saving Plots:

import matplotlib.pyplot as plt

x = [1, 2, 3, 4, 5]

y = [2, 3, 5, 7, 11]

plt.plot(x, y)

# Saving the plot as a PDF file

plt.savefig('plot.pdf')

plt.savefig('plot.svg')

plt.show()

Introduction to seaborn

Based on matplotlib, seaborn is a Python data visualization package that offers a high-level interface for creating visually appealing and educational statistical visuals. It makes it simple to produce educational and captivating visualizations by concentrating on statistical analysis visualization.

Installing with pip

Run the following command to install seaborn using pip.

pip install seaborn

After installation, use this command to view the Pandas version:

import seaborn

print(seaborn.__version__)

Advantages of seaborn over matplotlib

High-level interface: To create intricate visualizations, seaborn offers an easier-to-use API.
Statistical graphics: Seaborn is perfect for data exploration and visualization because it was created especially for statistical analysis.
Aesthetics: Seaborn's default colour schemes and style selections produce plots that are aesthetically pleasing and appear well-designed.
Integration with Pandas: Seaborn's smooth integration with Pandas DataFrames facilitates data processing and visualization.

Note: Python's intuitive statistical visuals might help you streamline your data visualization process.

Exploring seaborn's Capabilities

A variety of features are available from seaborn to improve your visualizations, such as:

Standard themes for a polished appearance
colour schemes for dependable and eye-catching hues
Exploring categorical data with categorical plots
Relationship charts to illustrate the connections between factors

1. Using Seaborn's Default Themes

Seaborn comes with several default themes.

import matplotlib.pyplot as plt

import seaborn as sns

# Set the default theme

sns.set_theme()

sns.stripplot(x=[1, 2, 3, 4])

plt.title("Default Theme")

plt.show()

2. Using Colour Palettes

Seaborn provides several colour palettes.

sns.color_palette("husl", 8)  # Use the husl palette with 8 colors

sns.color_palette("RdBu", 8)  # Use the RdBu palette with 8 colors

3. Categorical Plots

The sns.catplot() function from seaborn produces eye-catching and educational category graphs, such as: Swarm plots, Box plots, Violin plots. It makes it possible to specify the x and y data axes using, correspondingly, arguments like x and y. The kind argument determines the sort of categorical plot (e.g., "strip" or "swarm"), while the data parameter makes dataset provision easier.

Swarm Plot:

import seaborn as sns

import matplotlib.pyplot as plt

tips = sns.load_dataset("tips")

sns.catplot(x="sex", y="total_bill", data=tips, kind="swarm")

plt.show()

Box Plot:

import seaborn as sns

import matplotlib.pyplot as plt

tips = sns.load_dataset("tips")

sns.catplot(x="sex", y="total_bill", data=tips, kind="box")

plt.show()

Violin Plot:

import seaborn as sns

import matplotlib.pyplot as plt

tips = sns.load_dataset("tips")

sns.catplot(x="sex", y="total_bill", data=tips, kind="violin")

plt.show()

Relationship Plot:

Seaborn's sns.relplot() function visualizes relationships between variables. The dataset provisioning process also makes use of the data parameter. Data subsets are distinguished using parameters like hue, col, and row. Plot types like "scatter" and "line" are specified by the kind option, which makes it easier to visualize variable relationships.

Scatter Plot:

sns.relplot(x="x", y="y", data=df)

Line Plot:

sns.relplot(x="x", y="y", data=df , kind="line")

Advanced Data Visualization Techniques

1. Pair Plots

Pair plots are a great way to visualize relationships between multiple variables. Seaborn's sns.pairplot() function creates a matrix of pairwise plots, showing the relationship between each pair of variables.

import seaborn as sns

import matplotlib.pyplot as plt

# Load the iris dataset

iris = sns.load_dataset("iris")

# Create a pair plot

sns.pairplot(iris)

# Show the plot

plt.show()

2. Facet Grids

For more effective comparison, faceted grids are a fantastic tool for creating multi-plot grids. Seaborn's social network. A grid of plots is produced using the FacetGrid() function, each of which displays a distinct portion of the data.

import seaborn as sns

import matplotlib.pyplot as plt

# Load the tips dataset

tips = sns.load_dataset("tips")

# Create a facet grid

g = sns.FacetGrid(tips, col="sex", row="smoker")

# Map a plot to each subset

g.map(sns.histplot, "total_bill", alpha=0.7)

# Show the plot

plt.show()

3. Heatmaps

A useful tool for visualizing correlation matrices is a heatmap. A heatmap can be produced from a dataset or correlation matrix using seaborn's sns.heatmap() method.

import seaborn as sns

import matplotlib.pyplot as plt

import numpy as np

import pandas as pd

# Load the flights dataset

flights = sns.load_dataset("flights")

# Select only numerical columns

flights_num = flights.select_dtypes(include=[np.number])

# Create a heatmap of the correlation matrix

sns.heatmap(flights_num.corr(), annot=True, cmap="YlGnBu")

# Show the plot

plt.show()

Real-World Examples

Let's analyse and display some data from a CSV file that contains employee details.

import pandas as pd

# Read the CSV file into a DataFrame

data = pd.read_csv('data.csv')

# Display the first few rows of the DataFrame

print(data.head())

# Print information about the DataFrame

print(data.info())

# Display descriptive statistics of the DataFrame

print(data.describe())

# Plot a histogram of the 'age' column

plt.hist(data['age'])

# Plot a count of employees per department

sns.countplot(x='department', data=data)

# Plot a scatterplot of age vs salary

sns.scatterplot(x='age', y='salary', data=data)

# Calculate the average salary per department

dept_avg_salary = data.groupby('department')['salary'].mean()

# Select employees with salaries greater than $100,000

high_salary_employees = data[data['salary'] > 100000]

This example provides a basic demonstration, illustrating the use of seaborn and matplotlib for visualizing real-world datasets. There are a plethora of additional ways to study and visualize your data based on the particular questions you wish to address!

Additional visualization libraries

1. Plotly

Plotly is a flexible Python module that works well for interactive plotting, easily meeting the needs of web dashboards and data exploration. It is a well-liked solution for many applications due to its wide range of chart options, which support both static and interactive displays.

import plotly.express as px

import pandas as pd

df = pd.DataFrame({

    "x": [1, 2, 3, 4, 5],

    "y": [2, 4, 6, 8, 10]

})

fig = px.line(df, x="x", y="y", title="Plotly Line Chart")

fig.show()

2. Bokeh

Bokeh is a Python interactive visualization framework that works well for web-based plots and can be embedded into programs like Flask or Django. It includes capabilities for hovering, panning, and zooming.

from bokeh.plotting import figure, output_notebook, show

# Sample data

x = [1, 2, 3, 4, 5]

y = [2, 4, 6, 8, 10]

# Plotting with Bokeh

output_notebook()

p = figure(title="Bokeh Line Plot")

p.line(x, y)

show(p)

FAQ
Q: What are data visualization techniques?
A: Data visualization techniques include charts (line, bar, or pie), plots (bubble or scatter), diagrams, maps (heat maps, geographic maps, etc.), and matrices.

Conclusion

In this article, we've covered the principles of matplotlib and seaborn data visualization. We have looked at: Configuring and personalizing plots; displaying several data formats (numerical, categorical, and time series); developing interactive plots; and using data visualization on actual datasets.

Understanding and sharing insights is made easier with the help of data visualization. Your data may have hidden trends, patterns, and correlations that you can find by using matplotlib and seaborn. Don't be scared to experiment, investigate, and visualize; you might be surprised by the insights you find!

Reference

For more reference on matplotlib and seaborn:

Matplotlib Official Wensite:
Matplotlib Official Website

Seaborn Official Website:
Seaborn: statistical data visualization

Resolved: Attributeerror: 'dataframe' object has no attribute 'reshape'

Thu, 20 Jun 2024 21:46:10 +0000

Encountering the AttributeError 'DataFrame' object has no attribute 'reshape' is common when using Pandas in Python. It happens due to the absence of the reshape attribute within the DataFrame class, especially during reshaping operations. However, you can resolve this by using alternative methods like the reshape function from numpy or pandas methods like pivot_table or stack. These options effectively manipulate DataFrame objects, eliminating the AttributeError.

Why did this error occur?

The error occurred because the .reshape() method was erroneously applied to a DataFrame object, which doesn't support this operation. Attempting to reshape a single column selection without converting it into a DataFrame resulted in a Series object, which also lacks the reshape() method. Additionally, DataFrame selection processes that didn't maintain the DataFrame structure led to applying a method that doesn't exist for Series objects.

Recreating the Scenario

To recreate this scenario, try applying the .reshape() method directly to a DataFrame object or to a Series object resulting from a single column selection without proper DataFrame conversion. Additionally, ensure that the DataFrame selection process doesn't maintain the DataFrame structure, leading to the application of a method that doesn't exist for Series objects.

Example

Consider a scenario where you have a pandas DataFrame named df:

import pandas as pd
# Creating a DataFrame
data = {'A': [1, 2, 3],
        'B': [4, 5, 6]}
df = pd.DataFrame(data)

In the above code, shape of df(df.shape) is : (3, 2)

Now, let's say you want to reshape this DataFrame using the reshape() method:

# Trying to reshape the DataFrame
reshaped_df = df.reshape(2, 3)

Here's where the problem arises! When you run this code, you encounter the AttributeError: 'DataFrame' object has no attribute 'reshape' error.

OUTPUT ERROR

When you try to reshape a DataFrame using the reshape() method, pandas doesn't recognize reshape() as a valid method for DataFrames. The reshape() method is native to NumPy arrays, not pandas DataFrames.

What does the error mean?

Alright, so what's this error all about? Basically, when you see AttributeError: 'DataFrame' object has no attribute 'reshape', it's pandas politely saying, "Hey, I'm a DataFrame, not a NumPy array! I don't know what you want me to do with this reshape() thing." In simpler terms, it's like trying to teach your dog to juggle – it's just not gonna happen because that's not what dogs do!

In technical words, this error message indicates that the DataFrame object (df) doesn't have an attribute called reshape. In other words, you're trying to use a method (reshape()) that doesn't exist for DataFrame objects.

What are the causes of the error?

1. Dataframe Attribute Misinterpretation

This name encapsulates the scenario where a DataFrame object is treated as having an attribute (reshape) that it doesn't actually possess, leading to an AttributeError.

PROBLEM

import pandas as pd

# Creating a DataFrame
df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})

# Attempting to use reshape on a DataFrame directly
reshaped_df = df.reshape(2, 3)  # Causes AttributeError

SOLUTION

import pandas as pd
import numpy as np

# Creating a DataFrame
df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
print("df : \n",df)

# Convert DataFrame to numpy array
array_df = df.values
print("\narray_df : ", array_df)

# Reshape the numpy array
reshaped_array = array_df.reshape(2, 3)
print("\nreshaped_array : ",reshaped_array)

OUTPUT

Tip: Utilize NumPy's Reshape Function

2. Incorrect Method Chaining

This cause describes the situation where incorrect method chaining is attempted on a DataFrame, resulting in an AttributeError.

PROBLEM

import pandas as pd

# Creating a DataFrame
df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})

# Incorrect method chaining
reshaped_df = df.groupby('A').reshape(2, 3)  # Causes AttributeError

SOLUTION

import pandas as pd

# Creating a DataFrame
df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})

# Proper method chaining using pivot_table
reshaped_df = df.pivot_table(index='A', columns='B', aggfunc='size', fill_value=0)
print("reshaped_df : \n", reshaped_df)

Explanation

reshaped_df = df.pivot_table(index='A', columns='B', aggfunc='size', fill_value=0)

df: This is the DataFrame object that you're working with.
pivot_table: This is a method provided by pandas DataFrame objects for reshaping data. It allows you to create a pivot table from the DataFrame's data.
index='A': This specifies that the values in column 'A' will be used as the index (rows) of the pivot table.
columns='B': This specifies that the values in column 'B' will be used as the columns of the pivot table.
aggfunc='size': This specifies the aggregation function to be used. In this case, 'size' means that it will count the occurrences of each combination of 'A' and 'B' values.
fill_value=0: This specifies the value to be filled in for missing combinations of 'A' and 'B' values. In this case, it's set to 0.
reshaped_df: This is the variable where the resulting reshaped DataFrame will be stored.

Overall, this line of code takes the DataFrame df and creates a pivot table from its data based on the specified index ('A'), columns ('B'), aggregation function ('size'), and fill value (0 for missing combinations). The resulting pivot table is stored in the variable reshaped_df.

OUTPUT

Tip: Familiarize Yourself with Pandas Methods

3. Incorrect DataFrame Selection

This cause occurs when trying to reshape a DataFrame selection that results in a Series object rather than a DataFrame.

PROBLEM

import pandas as pd

# Creating a DataFrame
df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})

# Selecting a single column which results in a Series
selected_column = df['A']

# Attempting to reshape the Series
reshaped_df = selected_column.reshape(3, 1)  # Causes AttributeError

SOLUTION

import pandas as pd

# Creating a DataFrame
df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]})
print("df : \n", df)

# Selecting a single column and converting it to a DataFrame
selected_column = df[['A']]  # Using double brackets to select as DataFrame

# Reshaping the DataFrame
reshaped_df = selected_column.values.reshape(3, 1)
print("\nreshaped_df : \n",reshaped_df)

Explanation

We create a DataFrame df with two columns 'A' and 'B'.
We select the column 'A' using double brackets [['A']], which ensures that it's selected as a DataFrame rather than a Series.
We use the .values attribute to retrieve the underlying NumPy array from the selected DataFrame.
We reshape the array to have 3 rows and 1 column using the reshape() method.
The resulting reshaped array is assigned to the variable reshaped_df.

The use of double brackets [[]] ensures that the selected column remains a DataFrame, allowing us to apply the reshape() method without encountering the AttributeError.

OUTPUT

Tip: Check Documentation

Refer to the documentation of pandas and NumPy libraries for comprehensive guidance on reshaping operations and available methods. Documentation provides detailed explanations and examples that can aid in resolving errors and optimizing code.

Best Practices

DataFrame limitations
Keep in mind that not all numpy functions are directly applicable to DataFrame objects due to differences in their underlying data structures.
Method compatibility
Ensure compatibility between the methods used and the specific DataFrame structure and data types to avoid potential errors.
Data integrity
Be cautious when reshaping data, as it may impact the integrity and interpretability of the original dataset. Consider the implications of reshaping operations on downstream analysis and visualization.

Note: Be diligent in understanding the implications of DataFrame operations to ensure data integrity and accurate analysis.

Caution: Make sure DataFrame methods are compatible with your data to avoid messing it up.

Conclusion

In summary, the AttributeError: 'DataFrame' object has no attribute 'reshape' error occurs when attempting to use the reshape() method on a pandas DataFrame, which doesn't support this method. Understanding the difference between NumPy arrays and pandas DataFrames, as well as utilizing appropriate DataFrame methods like pivot() or melt(), can help avoid this error and efficiently manipulate data in pandas.

Now, armed with this knowledge, you can reshape your pandas DataFrames without stumbling into the reshape() method roadblock!

References

Pandas documentation:https://pandas.pydata.org/docs/
NumPy documentation: https://numpy.org/doc/

Pandas in Python: A Comprehensive Guide

Tue, 09 Apr 2024 21:57:18 +0000

Pandas is a powerful Python library for analysis and data manipulation. It offers user-friendly, high-performance data structures and tools for handling structured data. Pandas' versatility and extensive feature set make it a popular choice for data research and analysis tasks.

Series and DataFrame are the two primary data structures in Pandas.

A series is a one-dimensional object that resembles an array and is made up of an array of labels called the index and an array of data.
A data structure with two dimensions and labels that can have different kinds of columns is called a data frame. It can be compared to a SQL table or spreadsheet.

The Pandas Series adds labelled axes and missing data handling to NumPy arrays. Data Frames are similar to dictionaries in that they handle diverse data types and use column names as keys. They provide effective row and column operations for manipulating structured data in Python.
In this article we will look at the Pandas Python data analysis library. We will go over its main characteristics, including performance optimization, handling of missing data, and intuitive data structures. Let's learn about Pandas' capabilities and improve your knowledge of data analysis.

Table of Contents:

Installing Pandas
Panda Series
Pandas Data Frames
Data Manipulation with Pandas

Installing Pandas

Pip or Anaconda can be used to install Pandas.

1. Installing with pip

pip install pandas

2. Installing with Anaconda

Pandas comes pre-installed in the distribution if you use Anaconda. You may use conda to update it:

conda install pandas

Once installed, run the following command to see the Pandas version:

import pandas as pd

print(pd.__version__)

Tip: Use the robust indexing features of Pandas to effectively access and alter data.

Pandas Series

Python's Pandas Series provide a flexible and effective method for handling one-dimensional labelled data. Different data types, such as texts, floats, integers, and even custom objects, can be stored in series.

1. Creating Series Objects

You can use a variety of data structures, including Series and Data Frames, when constructing data objects in Python using Pandas. For instance, to create a Series object, you can use:

A. pd.Series()

It shows a labelled, one-dimensional array with indices that can hold several kinds of data used to handle data with flexibility and ease of usage, enabling effective data processing and analysis.

import pandas as pd

import numpy as np

# Creating a Pandas Series from a list

s = pd.Series([1, 3, 5, np.nan, 6, 8])

# Creating a Pandas Series from a NumPy array

arr = np.array([10, 20, 30, 40, 50])

s_np = pd.Series(arr)

# Creating a Pandas Series from a dictionary

data = {'a': 0, 'b': 1, 'c': 2}

s_dict = pd.Series(data)

2. Accessing and manipulating Series data

There are several ways to access and work with Pandas Series, including slicing, indexing, and simple arithmetic operations.

import pandas as pd

s = pd.Series([1, 3, 5])

# Accessing elements of a Series using index

print(s[0])  # Output: 1

# Slicing a Series

print(s[1:3])  # Output: [3, 5]

s_dict = pd.Series({'a': 2, 'b': 4, 'c': 6})

# Basic arithmetic operations on Series

s_add = s + s_dict  # Adding two Series with different indices

3. Methods for data alignment and missing data handling

When executing operations, Pandas automatically aligns data according to the index. Techniques like 'isnull()','fillna()' ,and 'dropna()'can be used to deal with missing data.

import numpy as np

s = pd.Series([1, 3, np.nan, 5])

# Checking for missing values

print(s.isnull())

# Filling missing values with a specified value

s_filled = s.fillna(0)

# Dropping missing values

s_dropped = s.dropna()

Pandas Data Frames

Pandas Data Frame is a two-dimensional labelled data structure that is widely used for data manipulation and analysis in Python. It can be thought of as a table with rows and columns, where each column represents a different variable and each row represents a different observation.

1. Creating DataFrames

The 'pd.DataFrame()' constructor can be used to create Data Frames from a variety of data sources, including CSV, Excel, SQL, and more, by providing in a dictionary or list of lists.

import pandas as pd

# Creating a DataFrame from a dictionary

data = {'Name': ['John', 'Anna', 'Peter', 'Linda'],

        'Age': [28, 35, 21, 45]}

df = pd.DataFrame(data)

print(df)

2. Understanding Data Frame structure

Data Frames are made up of rows, indexes, and columns. Rows hold the actual data, whereas columns represent variables and indices provide row labels.

import pandas as pd

data = {'A': [1, 2, 3], 'B': [4, 5, 6]}

df = pd.DataFrame(data)

# Accessing column names

print(df.columns)

# Accessing index

print(df.index)

# Accessing rows

print(df.iloc[0])  # Accessing the first row

3. Indexing and Selecting data

Many indexing techniques, such as label-based, position-based, and Boolean indexing, can be used to retrieve data in Data Frames.

import pandas as pd

data = {'Name': ['Alice', 'Bob', 'Charlie'], 'Age': [25, 30, 35]}

df = pd.DataFrame(data)

# Label-based indexing

print(df.loc[0, 'Name'])  # Accessing the 'Name' column of the first row

# Position-based indexing

print(df.iloc[0, 1])  # Accessing the element in the first row and second column

# Boolean indexing

print(df[df['Age'] > 30])  # Selecting rows where Age is greater than 30

Note: Keep data types consistent between columns to prevent unexpected behaviour when performing operations. Verify and convert data types as needed to guarantee accuracy and compatibility.

Data Manipulation with Pandas

Pandas can be used to manipulate data in a variety of ways, including grouping and aggregating data, filtering and sorting data, joining and merging Data Frames, and more.

1. Adding and Removing columns and rows

You can add or remove rows and columns with 'df.dropna()' and 'df.drop()', among other ways.

import pandas as pd

# Creating the initial DataFrame

data = {'Name': ['John', 'Anna', 'Peter', 'Linda'],

        'Age': [28, 35, 21, 45]}

df = pd.DataFrame(data)

# Adding a new row using pd.concat()

new_row = pd.DataFrame([['Tom', 33]], columns=['Name', 'Age'])

df = pd.concat([df, new_row], ignore_index=True)

print("DataFrame after adding a new row:")

print(df)

# Removing a row using pd.concat()

df = pd.concat([df.iloc[1:]], ignore_index=True)

print("\nDataFrame after removing the first row:")

print(df)

2. Filtering and sorting data

Functions like 'df.filter()' and 'df.sort_values()'can be used to filter and sort Data Frames.

import pandas as pd 

# Sample DataFrame

data = {'Name': ['Alice', 'Bob', 'Charlie', 'David'],

        'Age': [25, 30, 35, 40]}

df = pd.DataFrame(data)

# Filtering rows based on conditions

filtered_df = df[df['Age'] > 30]

# Sorting by a column

sorted_df = df.sort_values(by='Age', ascending=False)

print("Filtered DataFrame:")

print(filtered_df)

print("\nSorted DataFrame:")

print(sorted_df)

Caution: Large datasets may need computationally costly tasks like sorting, merging, or grouping. Employ the right Pandas techniques and algorithms to maximise performance.

3. Combining and Merging Data Frames

Data Frames can be joined together or merged using 'pd.concat()' and 'pd.merge()'.

import pandas as pd

import numpy as np

# Define DataFrame 1

data1 = {'Name': ['Alice', 'Bob', 'Charlie'],

         'Age': [25, 30, 22]}

df1 = pd.DataFrame(data1)

# Define DataFrame 2

data2 = {'Name': ['David', 'Emily'],

         'Age': [28, 35]}

df2 = pd.DataFrame(data2)

# Concatenating DataFrames horizontally

df_concat_horizontal = pd.concat([df1, df2], axis=1)

print(df_concat_horizontal)

# Merging DataFrames based on multiple columns

merged_df_multiple = pd.merge(df1, df2, on=['Name', 'Age'])

print(merged_df_multiple)

4. Grouping and Aggregating data

Aggregation functions such as count(), mean(), and sum(), as well as 'df.groupby()', can be used to group and aggregate data.

import pandas as pd 

# Creating a DataFrame

data = {'Category': ['A', 'B', 'A', 'B', 'A'],

        'Price': [10, 20, 15, 25, 12],

        'Quantity': [5, 3, 4, 2, 6]}

df = pd.DataFrame(data)

# Grouping and aggregating data

aggregated_df = df.groupby('Category').agg({'Price': 'sum', 'Quantity': 'mean'})

print("Aggregated Data:")

print(aggregated_df)

FAQ
Q: How do I perform data manipulation?
A: Pandas provides functions for selecting, filtering, merging, grouping, and aggregating data.
Q: What is the difference between loc and iloc?
A: loc is label-based indexing, used for selecting data by labels, while iloc is integer-location based indexing, used for selecting data by integer indices.

For Refering to Part2 of the Article:

Mastering Data Analysis and Manipulation with Pandas: A Comprehensive Guide

For Refering to Part3 of the Article:

Advancing with Pandas: Beyond the Basic

Numpy Tuitorial: Data Analysis for Data Science

Sun, 24 Mar 2024 09:48:00 +0000

1. Introduction to NumPy:

In the world of data science and machine learning, we analyze big sets of data to make smart choices. A really helpful tool for this is called NumPy. It's a part of Python, a popular programming language. NumPy helps us handle data in a special way. It's like a super tool that lets us organize and work with data really efficiently.

Importance and Usefulness:

NumPy efficiently handles large datasets, enabling tasks like data manipulation and statistical analysis. Mastery of NumPy streamlines workflows and enhances data analysis, crucial for building robust machine learning models.

This tutorial is a comprehensive guide to leveraging NumPy for data analysis. Whether you're a novice or a seasoned practitioner, it equips you with essential skills to harness NumPy effectively.

Lessons to Learn:

Understanding: Foundational concepts in data science facilitated by NumPy.
Crafting: Creating and manipulating NumPy arrays for data encapsulation.
Executing: Performing crucial calculations on NumPy arrays for data analysis.
Application: Applying NumPy skills to real-world data science challenges.

Prior Python experience is recommended. While a background in matrix mathematics is beneficial, it's not mandatory.

For more information, refer to the official NumPy documentation.

In this guide, we'll master data analysis with NumPy, an essential Python library. NumPy, short for Numerical Python, empowers analysts with multidimensional arrays and mathematical functions.

What is NumPy?

NumPy is like a superhero for handling numbers in Python! It's a special library that makes working with data super easy. Think of it as a powerful toolbox full of tools specifically designed for crunching numbers and doing math. With NumPy, we can create and manipulate arrays, which are like supercharged lists that can handle lots of data at once. This makes it perfect for tasks like data analysis, machine learning, and scientific computing.

Exploring NumPy and its capabilities:

NumPy's core object, the ndarray, serves as the foundation for its vast capabilities. This homogeneous multidimensional array allows for seamless manipulation and analysis of data. Whether it's performing mathematical computations or handling large datasets, NumPy provides the tools necessary for efficient data analysis.

2. Installing Numpy

NumPy Installation Methods:

NumPy can be installed using various methods, including package managers and environments like pip, Anaconda, or virtual environments. Below are some common methods for installing NumPy:

1. Using pip:

You can install NumPy using pip, which is the standard package manager for Python. This method is suitable for users who prefer to manage their Python packages using pip. It is also convenient for installing NumPy in virtual environments or when you need to install additional Python packages along with NumPy:

pip install numpy

2. Using conda (Anaconda or Miniconda):

If you're using Anaconda or Miniconda, you can install NumPy using conda. Conda is a package manager and environment manager that is particularly useful for managing Python packages in data science and scientific computing environments. This method is recommended for users working with Anaconda distributions or those who prefer to manage their Python environments and packages using conda:

conda install numpy

3. Using system package managers:

Some operating systems provide package managers that allow you to install Python packages, including NumPy. This method is useful if you prefer to install NumPy through your system's package manager, ensuring compatibility and integration with other system packages. It is recommended for users who want NumPy to be installed system-wide and managed alongside other system packages:

On Ubuntu/Debian:
For installing on Ubuntu, use,

sudo apt-get install python3-numpy
On Fedora:
For installing on Fedora, use,

sudo dnf install numpy

Tip: Consider using pip if you're working in a virtual environment or need to manage additional Python packages alongside NumPy.

3. Setting up the environment:

Installing NumPy is straightforward using pip, Python's package manager. With NumPy installed, you can swiftly import it into your scripts or interactive sessions, ready to harness its power for data analysis tasks.

pip install numpy

import numpy as np

4. NumPy Basics:

Performing basic array operations:

Once arrays are in place, performing basic operations such as arithmetic operations, indexing, and slicing become essential techniques for data manipulation. NumPy provides intuitive syntax and optimized functions for these operations, enabling efficient processing of large datasets.

CODE

a) Creating Arrays :

import numpy as np

# Creating arrays
arr1 = np.array([1, 2, 3])
arr2 = np.array([4, 5, 6])

b) Arithmetic Operations :

# Arithmetic operations
result = arr1 + arr2
print("\nResult : ", result)

c) Indexing and Slicing :

# Indexing and slicing
print(arr1[0])      # Accessing the first element
print(arr2[1:])     # Slicing from the second element onwards

OUTPUT

Tip: Utilize np.arange() for creating sequential arrays with specified start, stop, and step size easily.

Working with multidimensional arrays:

NumPy's support for multidimensional arrays is a cornerstone of its functionality. This feature enables data analysts to work with datasets of varying complexity, from simple matrices to high-dimensional tensors. With NumPy, you can reshape arrays, stack them together, and split them into smaller arrays, providing unparalleled flexibility in data manipulation.

CODE

a) Creating a 2D array :

Function Used - numpy.array

import numpy as np

# Creating a 2D array
arr = np.array([[1, 2, 3], [4, 5, 6]])
print("arr : ", arr)

b) Reshaping the array :

Function Used - numpy.reshape

# Reshaping the array
reshaped_arr = arr.reshape(3, 2)
print("\nreshaped array : ", reshaped_arr)

c) Transposing reshaped_arr to match the shape of arr :

Function Used - numpy.ndarray.T

# Transposing reshaped_arr to match the shape of arr
reshaped_arr_transposed = reshaped_arr.T
print("\nreshaped_arr_transposed : ", reshaped_arr_transposed)

d) Stacking arrays vertically :

Function Used - numpy.vstack

# Stacking arrays vertically
stacked_arr = np.vstack([arr, reshaped_arr_transposed])
print("\nstacked_array : ", stacked_arr)

e) Splitting the array horizontally :

Function Used - numpy.hsplit

# Splitting the array horizontally
split_arr = np.hsplit(stacked_arr, 3)
# Printing split_arr
print("\nsplit_array:")
for i, sub_array in enumerate(split_arr):
    print(f"Part {i + 1}:")
    print(sub_array)
    print()

5. Advanced NumPy Techniques:

Advanced NumPy techniques involve leveraging the full power of NumPy for complex data analysis tasks. These techniques include advanced array manipulation, broadcasting, and utilizing universal functions (ufuncs) for efficient element-wise operations on arrays of varying shapes and sizes.

Why Advanced NumPy Techniques?

While basic NumPy operations are essential for data analysis, mastering advanced techniques unlocks the full potential of NumPy, enabling you to efficiently handle more intricate datasets and perform sophisticated computations.

Expected Outcome:

By learning advanced NumPy techniques, you'll be equipped with the skills to:

Perform complex mathematical operations on arrays.
Efficiently manipulate and transform multidimensional arrays.
Apply broadcasting to perform element-wise operations on arrays of different shapes.
Utilize universal functions (ufuncs) for fast element-wise array operations.
Implement advanced array manipulation techniques for data preprocessing and feature engineering.

Let's look at some most used advanced NumPy techniques.

a) Broadcasting:

NumPy's broadcasting feature is a powerful tool for performing operations between arrays of different shapes. This capability eliminates the need for explicit looping over array elements, resulting in more concise and efficient code. Broadcasting enables data analysts to apply element-wise operations across arrays, even when their shapes differ, simplifying complex data manipulation tasks.

CODE

import numpy as np

# Broadcasting example
arr1 = np.array([[1, 2, 3], [4, 5, 6]])
arr2 = np.array([10, 20, 30])
print("\narr1 : ",arr1)
print("\narr2 : ",arr2 )

result = arr1 + arr2
print("\narr1 + arr2", result)

OUTPUT

b) Universal functions (ufuncs):

NumPy provides universal functions (ufuncs) for performing element-wise operations on arrays. These functions are highly optimized and vectorized, resulting in faster execution compared to traditional looping constructs. By leveraging ufuncs, data analysts can efficiently apply mathematical functions, such as exponentiation and trigonometric operations, to entire arrays, enhancing productivity and performance in data analysis workflows.

CODE

import numpy as np

# Universal functions example
arr = np.array([1, 2, 3])
print("arr : ", arr)

result = np.square(arr)
print("\nresult : ", result)

OUTPUT

c) Array manipulation:

NumPy offers advanced techniques for modifying array shapes and structures, providing data analysts with greater control over their data. Whether it's transposing arrays, flattening them, or rearranging their elements, NumPy's array manipulation capabilities enable flexible and efficient data processing. These techniques are invaluable for tasks such as data cleaning, preprocessing, and feature engineering, contributing to more robust and accurate data analyses.

CODE

import numpy as np

# Array manipulation example
arr = np.array([[1, 2, 3], [4, 5, 6]])
print("arr : \n", arr)

# Transposing the array
transposed_arr = arr.T
print("\ntransposed array : \n", transposed_arr)

# Flattening the array
flattened_arr = arr.flatten()
print("\nflattened array :  : ", flattened_arr)

OUTPUT

By mastering these basic and advanced techniques in NumPy, you'll be well-equipped to handle various data analysis tasks efficiently. Stay tuned for the next parts of this comprehensive guide, where we'll delve deeper into more advanced topics.

Tip: Harness NumPy's broadcasting for seamless element-wise operations between arrays of varying shapes.

6. Case Studies and Real-World Applications:

Practical examples demonstrating the use of NumPy and Pandas for data analysis:

NumPy and Pandas are extensively used in real-world scenarios to tackle various data analysis tasks. Let's explore some practical examples:

Financial Analysis:

Financial analysts use NumPy and Pandas to analyze stock market data, perform portfolio optimization, and calculate risk metrics. For example, they might use NumPy to calculate statistical measures like mean, standard deviation, and correlation coefficients, while Pandas is utilized for data manipulation tasks such as merging datasets and handling missing values.
Sentiment Analysis:

In sentiment analysis, which involves analyzing text data to determine sentiment or opinion, NumPy and Pandas are employed to preprocess textual data, extract features, and build machine learning models. For instance, sentiment analysis of customer reviews might involve tokenization, text vectorization using NumPy arrays, and model training with Pandas DataFrame.
Predictive Modeling:

Data scientists use NumPy and Pandas extensively in predictive modeling tasks such as regression, classification, and time series forecasting. They leverage NumPy arrays for feature engineering, model training, and evaluation, while Pandas facilitates data preprocessing, exploratory data analysis, and model validation.

Tip: Employ pandas_datareader for fetching real-time financial data directly into Pandas DataFrame for efficient financial analysis.

7. Conclusion:

Recap of the key concepts covered in the guide:

In this comprehensive guide, we have covered fundamental and advanced concepts of data analysis using NumPy. We explored NumPy's powerful capabilities for array manipulation, mathematical operations, and broadcasting. Additionally, we delved into practical examples and techniques for working with multidimensional arrays. Moreover, we discussed how Pandas, built on top of NumPy, extends data analysis capabilities with its high-level data structures and manipulation tools.

Note: Regular practice and experimentation are crucial for mastering data analysis with NumPy and Pandas and developing proficiency in real-world projects.

Encouragement for readers to apply their newfound knowledge:

Now that you've gained a solid understanding of NumPy and Pandas, it's time to apply your knowledge to real-world data analysis projects. Practice is key to mastering these concepts, so don't hesitate to work on projects that interest you. Experiment with different techniques, explore diverse datasets, and continue learning from online resources and communities. Remember, the more you practice, the more proficient you'll become in data analysis with NumPy and Pandas.

Tip: Practice your NumPy and Pandas skills on real-world datasets through Kaggle competitions, open-source projects, or personal projects to solidify your understanding.

8. Additional Resources and Further Reading (Optional):

Links to official documentation, tutorials, and online courses:

Here are some official resources to deepen your understanding of NumPy and Pandas:

Note: Explore official documentation, tutorials, and online courses for in-depth learning and stay engaged with online communities for continuous growth and support.

Recommended books and resources for mastering data analysis techniques:

Here are some recommended books and resources for further exploration:

"Python for Data Analysis" by Wes McKinney
"NumPy Cookbook" by Ivan Idris
"Pandas Cookbook" by Theodore Petrou
"Data Science from Scratch" by Joel Grus

FAQ:
Q. What is NumPy, and why is it important for data analysis?b
A. NumPy, short for Numerical Python, is a Python library used for numerical computing. It provides support for multidimensional arrays and mathematical functions, making it ideal for efficient data manipulation and analysis tasks.
Q. What is the difference between NumPy and Pandas?
B. NumPy focuses on numerical computing and provides support for multidimensional arrays, while Pandas is built on top of NumPy and offers high-level data structures and manipulation tools, particularly suited for data analysis tasks with labeled data.
Q. Is NumPy used for data analysis?
C. Yes, NumPy is commonly used for data analysis tasks due to its efficient handling of multidimensional arrays and mathematical functions.
Q. Should I use NumPy or pandas?
D. NIt depends on your specific needs. If you primarily work with numerical data and require efficient array operations, NumPy is a good choice. If you need to handle labeled data, perform data manipulation tasks, and work with tabular data, pandas is more suitable.
Q. Can you use NumPy and pandas together?
D. Yes, NumPy and pandas are often used together in data analysis workflows. While NumPy provides support for numerical operations and arrays, pandas offers high-level data structures like DataFrames and Series, allowing for easier data manipulation and analysis.

This comprehensive guide provides a structured approach to mastering data analysis with NumPy and Pandas, covering fundamental concepts, advanced techniques, and real-world applications.

NumPy in Python: An Advanced Guide

Wed, 13 Mar 2024 11:57:38 +0000

Table of Contents:

Array Iteration
NumPy Operations
Working with Mathematical Functions
NumPy Random Modules
- Generating Random Numbers
- Setting Seed for Reproducibility
File Input and Output with NumPy
NumPy and Pandas
- Converting between NumPy arrays and Pandas Data Frame
Performance Optimization with NumPy
Advanced NumPy Topics
Conclusion

Array Iteration

The process of iterating across an array in NumPy is called array traversal. Loops and the np.nditer() function are the two main ways to iterate over NumPy arrays.

1. Iterating with loops

NumPy arrays can be iterated over using conventional loops like for loops and list comprehensions. Nestled loops are usually used to iterate over a multi-dimensional array, navigating through each dimension. For example:

import numpy as np

arr = np.array([[1, 2], [3, 4]])

# Iterating over rows

for row in arr:

    print(row)

2. Iterating with np.nditer()

A more effective and versatile method of iterating over NumPy arrays is offered by the np.nditer() function, particularly when working with multi-dimensional arrays or arrays that contain several data types. Regardless of the array's dimensions or structure, it permits continuous iteration across its elements. For example:

import numpy as np

arr = np.array([[1, 2], [3, 4]])

# Iterating with np.nditer()

for x in np.nditer(arr):

    print(x)

NumPy Operations

Numpy facilitates broadcasting, which enables operations on arrays with various shapes, as well as vectorized operations, which act element-wise on arrays.

1. Vectorized Operations

NumPy's vectorized operations make it possible to efficiently operate element-wise on arrays without the use of explicit loops.NumPy uses optimised C implementations to execute computations quickly by applying functions or mathematical operations directly to full arrays. As a result, it is a good choice for numerical workloads in scientific computing, data analysis, and machine learning.

import numpy as np

arr = np.array([1, 2, 3])

result = arr * 2  # Multiply each element by 2

print(result)  # Output: [2 4 6]

2. Broadcasting

NumPy has a powerful method called broadcasting that makes it possible to mix arrays of different shapes for arithmetic operations. It removes the need for manual array copying and reshaping, simplifying code and improving efficiency. It is a key component of NumPy that makes it simple to use and effective for numerical calculations.

import numpy as np

arr1 = np.array([[1, 2, 3], [4, 5, 6]])

arr2 = np.array([1, 2, 3])

result = arr1 + arr2

print(result)

Tip: Use NumPy's broadcasting functionality to execute arithmetic operations on arrays with varying dimensions.

Working with Mathematical Functions

NumPy offers a range of mathematical functions for computing and manipulating arrays.

1. Mathematical Functions

With NumPy, you can compute and manipulate arrays more efficiently due to its extensive mathematical operations. The trigonometric function, the exponential and logarithmic functions, and many other mathematical processes are all covered by these functions.

import numpy as np 

arr = np.array([0, np.pi/2, np.pi])

# Trigonometric functions

print(np.sin(arr))  # Output: [0.0000000e+00 1.0000000e+00 1.2246468e-16]

# Exponential and logarithmic functions

print(np.exp(arr))  # Output: [ 1.          4.81047738 23.14069263]

print(np.log(arr))  # Output: [      -inf 0.45158271 1.14472989] (Note: -inf indicates negative infinity)

2. Linear Algebra Operations

NumPy is an essential tool for applications involving linear algebra in scientific computing, machine learning, and data analysis since it offers strong functions for carrying out these processes. These functions make it possible to work with matrices and vectors efficiently. Some of the operations that they can do include matrix multiplication, eigenvalue decomposition, and solving systems of linear equations.

import numpy as np 

arr1 = np.array([[1, 2], [3, 4]])

arr2 = np.array([[5, 6], [7, 8]])

# Matrix multiplication

result = np.dot(arr1, arr2)

print(result)

# Eigenvalues and eigenvectors

eigenvalues, eigenvectors = np.linalg.eig(arr1)

print("Eigenvalues:", eigenvalues)

print("Eigenvectors:", eigenvectors)

NumPy Random Module

For applications like simulation, statistical sampling, and random initialization in machine learning algorithms, the NumPy random module offers functions for creating random integers and random arrays. This module is a flexible tool for creating random data since it provides a large variety of distributions and ways to adjust the unpredictability.

1. Generating Random Numbers

np.random.rand(): The function np.random.rand() produces random numbers between 0 and 1 based on a uniform distribution.
np.random.randn(): Using a conventional normal distribution with a mean of 0 and a standard deviation of 1, np.random.randn() generates random integers.

import numpy as np 

# Generate a random number between 0 and 1

random_num = np.random.rand()

print(random_num)

2. Setting Seed for Reproducibility

np.random.seed(): Establishes the seed for producing random numbers, guaranteeing consistency in outcomes throughout several code executions.

import numpy as np 

np.random.seed(42)  # Set seed for reproducibility

random_num = np.random.rand()

print(random_num)

File Input and Output with NumPy

NumPy offers useful functions for reading and writing data in binary and text formats to and from files. This functionality is essential for effectively processing data from external files and combining it with NumPy arrays.

1. Loading and Saving text Files

The np.loadtxt() function in NumPy loads data into a NumPy array by means of a text file. It allows for customisation with respect to data formats, skip headers, and delimiters. Whereas, array data is saved to a text file with formatting choices like headers, footers, and delimiters by using np.savetxt().

import numpy as np

# Saving data to a text file

arr = np.array([[1, 2, 3], [4, 5, 6]])

np.savetxt('data.txt', arr)

# Loading data from a text file

loaded_arr = np.loadtxt('data.txt')

print(loaded_arr)

2. Loading and Saving Binary Files

The np.load() function in NumPy effectively loads data into a NumPy array while maintaining accuracy and structure from a binary file. On the other hand, NumPy arrays can be saved to binary files using np.save(), which preserves the array's data structure for np.load() loading at a later time.

import numpy as np 

# Saving data to a binary file

np.save('data.npy', arr)

# Loading data from a binary file

loaded_arr = np.load('data.npy')

print(loaded_arr)

NumPy and Pandas

NumPy is easily integrated with Pandas, a powerful Python data analysis system. It offers a tabular data format, similar to a spreadsheet or SQL table, with labelled rows and columns. Data manipulation operations like filtering, sorting, grouping, and aggregation can be effectively completed with Pandas DataFrame.

1. Converting between NumPy arrays and Pandas Data Frame

Here are few techniques to convert between Pandas Data Frame and NumPy arrays:

import numpy as np 

import pandas as pd

# Suppose 'arr' is a NumPy array

arr = np.array([[1, 2, 3], [4, 5, 6]])

# Converting NumPy array to DataFrame

df = pd.DataFrame(arr)

# Converting DataFrame to NumPy array

arr = df.to_numpy()

print(arr)

Performance Optimization with NumPy

NumPy's improved C implementation and effective array-based operations allow it to be significantly faster than native Python lists. NumPy provides faster code execution and cleaner code by using vectorized operations and avoiding explicit loops. It is an essential tool for machine learning algorithms, scientific computing activities, and numerical computations because of its memory layout and ufuncs, which further improve efficiency.

Advanced NumPy Topics

1. Memory layout and strides

Understanding the memory architecture and strides of NumPy is essential to comprehending how data is stored and accessed in arrays. The arrangement of elements in memory is known as the memory layout, and the amount of bytes needed to move to the next element along each dimension is known as the stride. Gaining an understanding of these ideas improves memory usage and access patterns, which boosts computation and array operations performance.

2. Universal functions (ufuncs) and vectorization

NumPy's vectorization and universal functions (ufuncs) are key components that allow for effective element-wise operations on arrays. Without the need for explicit loops, you may perform mathematical operations, trigonometric functions, logical operations, and more to entire arrays using ufuncs, which are functions that act element-wise on arrays. Vectorization is the application of ufuncs to arrays, which results in faster loop execution times when compared to loops that use scalar operations. NumPy allows for short and effective code for jobs involving numerical computations and data processing by using ufuncs and vectorization.

3. NumPy's datetime and timedelta functions

NumPy has timedelta functions to manipulate time intervals and methods to interact with dates and timings.

import numpy as np 

# Creating datetime arrays

dates = np.array(['2022-01-01', '2022-01-02', '2022-01-03'], dtype='datetime64')

print(dates)

# Calculating time differences

diff = np.diff(dates)

print(diff)

FAQ
Q: What are ufuncs in NumPy?
A: Universal functions (ufuncs) in NumPy are functions that operate element-wise on arrays, enabling efficient numerical operations without the need for explicit loops.

Conclusion

In conclusion, NumPy provides strong tools for effective mathematical operations and array manipulation. Data management is made easier by its support for array iteration, multidimensional slicing, and advanced indexing. NumPy's optimized efficiency and smooth integration with other Python libraries make it indispensable for numerical computing across a range of fields, including scientific computing and machine learning.

For Refering to Previous Part of the Article:

NumPy in Python: A Comprehensive Guide

NameError: name 'pd' is not defined in Python [Solved]

Thu, 29 Feb 2024 09:11:50 +0000

Introduction

If you are a newbie to Python and are working with Pandas module, you might get this error "NameError: name 'pd' is not defined". This error occurs when you are trying to work with Pandas without properly defining the alias as pd. The error is pretty straightforward, but it might leave you puzzled. No worries, in this article we will see what this error means, why it occurs, and how to solve it so let's get started.

Why did this error happen?

This error usually occurs while working with the Pandas library in Python. Consider the following code block.

 import pandas # Importing pandas
  df = pd.DataFrame({'Marks': [50, 20, 30],
              'Status': ['Pass', 'Fail', 'Pass']}) # Creating data frame
  print(df)

If you run the above code it will give you the below error.

The error "NameError: name 'pd' is not defined" suggests that the name pd is not defined which you are using in the code df = pd.DataFrame({...}). The Python interpreter cannot figure out what exactly pd means. Great, now you know why you are getting this error let's move further.

What is meant by an alias?

In the context of programming, an alias refers to an alternative name. It is a common practice for programmers to use aliases instead of actual names. Consider the following code.

import pandas # Importing pandas
  df = pandas.DataFrame({'Marks': [50, 20, 30],
              'Status': ['Pass', 'Fail', 'Pass']}) # Creating data frame
  print(df)

In the above code we are importing pandas and then using it to create a data frame pandas.DataFrame({...}). Now what if we don't want to use the name pandas instead we want to use a short name that's where aliases come into the picture. Consider the code below.

  import pandas as pd # Importing pandas
  df = pd.DataFrame({'Marks': [50, 20, 30],
              'Status': ['Pass', 'Fail', 'Pass']}) # Creating data frame
  print(df)

Now in the above code, we have defined an alias name pd as import pandas as pd, and later we are using pd.DataFrame({...}) instead of pandas.DataFrame({...}). Now you know what an alias is and why programmers prefer to use an alias instead of simple imports so let's see the causes of this error.

What are the causes of the error?

1. Module not imported

If you have not imported the module import pandas as pd then you are going to get this error "NameError: name 'pd' is not defined". Consider the code below.

df = pd.DataFrame({'name': ['ali', 'uzair', 'asim'], 'cgpa': [3.0, 3.5, 3.9]}) # Creating a new data frame
print(df)

In the above code, you are trying to create a data frame but you have not imported the pandas module so it will result in the following error.

2. Alias not defined

One reason for getting this error "NameError: name 'pd' is not defined" is the alias is not properly defined. Take a look at the following code block.

import pandas # Importing pandas
  df = pd.DataFrame({'Marks': [50, 20, 30],
              'Status': ['Pass', 'Fail', 'Pass']}) # Creating data frame
  print(df)

After running the above code you will get the following error.

You are getting this error because you have not defined the alias pd that you are using in the code.

3. Using local scope

Another reason for getting this error "NameError: name 'pd' is not defined" is you have imported pandas within a local scope that will restrict its use. Consider the following code block.

def create(): # Function for creating a data frame
    import pandas as pd
    pd.DataFrame({'name': ['ali','uzair','asim'], 'marks': [10,20,30]})
   

create() # Calling the create function
pd.DataFrame({'name': ['ali', 'uzair', 'asim'], 'cgpa': [3.0, 3.5, 3.9]}) # Creating a new data frame

By running the above code you will get the following error.

In this case, you have imported pandas with a proper alias import pandas as pd but still you are getting this error "NameError: name 'pd' is not defined" the reason is the local scope you have imported pandas inside a function and you are trying to create a data frame outside that function pd.DataFrame({'name': ['ali', 'uzair', 'asim'], 'cgpa': [3.0, 3.5, 3.9]}).

Caution: When working with functions be cautious about importing Pandas within a local scope. Importing Pandas locally can restrict its availability

How to solve the error?

1. Solution for a module not imported

If you haven't imported the panda's module and you are trying to access it then you will get this error "NameError: name 'pd' is not defined". To solve it you just need to import the module. Consider the code below.

import pandas as pd # Importing the pandas module

df = pd.DataFrame({'name': ['ali', 'uzair', 'asim'], 'cgpa': [3.0, 3.5, 3.9]}) # Creating a new data frame
print(df)

After importing the module import pandas as pd the error will be removed. Below is the output of the above code.

You can use try except blocks for proper error handling.

Tip: Import the module first before using it.

2. Solution for alias not defined

The error "NameError: name 'pd' is not defined" clearly says that the name pd is not defined which simply means you are using pd as an alias but haven't defined it which can be simply resolved by defining pd as an alias. Consider the following code.

  import pandas as pd # Importing pandas and defining alias as pd
  df = pd.DataFrame({'Marks': [50, 20, 30],
              'Status': ['Pass', 'Fail', 'Pass']}) # Creating a data frame
  print(df)

Output of the above code.

Tip: Define the alias properly before using it.

3. Solution for local scope

In the case of local scope try importing pandas as global by doing that you can access it anywhere in your code. Consider the following code block.

import pandas as pd # Importing pandas in global scope

def create(): # Function for creating a data frame
    pd.DataFrame({'name': ['ali','uzair','asim'], 'marks': [10,20,30]})
   
create() # Calling the create function
df = pd.DataFrame({'name': ['ali', 'uzair', 'asim'], 'cgpa': [3.0, 3.5, 3.9]}) # Creating a new data frame
print(df)

Output of the above code.

Tip: Import the module in the global scope to access it anywhere in your code.

4. Solving the error in Jupyter Notebook or Google Colab

Sometimes you might be working in Jupyter Notebook or Google Colab in this case if you are getting this error "NameError: name 'pd' is not defined" check whether the alias pd is defined in global scope and if not define it and rerun your code.
If the alias is defined properly and you are still getting this error then it might be the following case.

where the import statement is in a separate cell and you are creating the data frame in a separate cell. In this case, run the previous cell first where you have imported Pandas, and then the cell where you are creating the data frame.

FAQ Q. What does the error message "NameError: name 'pd' is not defined" mean?
A. The error suggests that you are using an alias name 'pd' which is not properly defined in your code.

Q. How can I resolve the "NameError: name 'pd' is not defined" error?
A. To resolve it try importing 'import pandas as pd' in a global scope.

Conclusion

In summary, the error indicates that you are using an alias pd which is not defined in your code. To resolve this problem try to define the alias import pandas as pd in global scope first before using it. If you are working in Jupyter Notebook or Google Colab you have to define the alias as previous and also try to rerun the cell where you have imported pandas.

Coder Legion - Recent posts tagged pandas

Building Decision Trees with Pandas and Scikit-learn

The Theory

The Code

Step 1: Load and Explore the data

Step 2: Transform the data

Step 3: Select Model type and train.

Step 4: Predict

Step 5: Validate

Step 6: Optimise

Trial by Fire: From Garbage Excel to Relational Graph with Python and Pandas

1. The Hook: Industrial Data Entropy

2. The ETL Pipeline (Extract, Transform, Load)

3. The Graph Challenge (Shattering the Flat Table)

4. The Golden Rule: Ingestion Idempotency

The Sandbox: Execute it Yourself

5. Closing & CTA (Towards the AI Brain)

Mastering Pandas — Part 4: Data Visualization with Matplotlib & Seaborn

From Clean Data to Clear Insight

Part 1 — Matplotlib: The Engine

What is Matplotlib?

Global Settings with plt.rcParams

Setting a Visual Theme with plt.style.use()

Plotting from a DataFrame with .plot()

Plotting Directly with plt.plot()

Decorating Any Chart

Multiple Subplots with Object-Oriented Style

Part 2 — Seaborn: Statistical Visualization

What is Seaborn?

Global Styling with set_theme()

Relational Charts — Showing Relationships Between Numbers

Distribution Charts — Showing How Values Are Spread

Categorical Charts — Comparing Groups

Matrix & Regression Charts — Correlations and Trends

Combining Seaborn with Matplotlib

Choosing the Right Tool

S&OP: Why Your Excel Is Lying to You (and How to Interrogate It with Python)

The Problem: Signal-to-Noise Ratio

The Visual Evidence

The Solution: "Quality Valve" Architecture

The Code: Statistics > Intuition

Open Kitchen: Try It Yourself

Production Architecture (Behind the Scenes)

Data Flow Visualization

Next Step: Scientific Prediction

Mastering Pandas — Part 3: Data Cleaning, Merging & Joining

Real Data Is Never Clean

Step 1 — Inspect Before You Touch Anything

Step 2 — Fix Column Names First

Step 3 — Handle Missing Values

Step 4 — Remove Duplicates

Step 5 — Fix Data Types

Step 6 — Clean Strings

Step 7 — Replace and Standardize Values

Now Combine — Merging & Joining DataFrames

1️⃣ merge() — The Main Tool

The how= Parameter — All Four Types

The on= Parameter — What to Match On

Handling Shared Column Names — _x and _y

Other Useful Parameters

2️⃣ concat() — Stacking DataFrames

axis= Parameter

join= Parameter

ignore_index= Parameter

keys= Parameter

verify_integrity= Parameter

merge() vs concat() — When to Use Which

3️⃣ join() — Index-Based Joining

Why merge() Is Almost Always Better

> Use join() only when your data is already indexed — for example, after a groupby() that produced a meaningful index. Otherwise, merge() is cleaner and more flexible.

Mastering Pandas — Part 1: Reading, Sorting & Displaying Data

What is Pandas and Why Does It Matter?

Part 1 — Reading Files into Pandas

CSV File — Most Common

Excel File

JSON File

HTML File — Web Scraping Made Easy

SQL Database

Text File

Other Formats

Global Settings with `plt.rcParams`

Setting a Visual Theme with `plt.style.use()`

Plotting from a DataFrame with `.plot()`

Plotting Directly with `plt.plot()`

Global Styling with `set_theme()`

1️⃣ `merge()` — The Main Tool

The `how=` Parameter — All Four Types

The `on=` Parameter — What to Match On

Handling Shared Column Names — `_x` and `_y`

2️⃣ `concat()` — Stacking DataFrames

`axis=` Parameter

`join=` Parameter

`ignore_index=` Parameter

`keys=` Parameter

`verify_integrity=` Parameter

`merge()` vs `concat()` — When to Use Which

3️⃣ `join()` — Index-Based Joining

Why `merge()` Is Almost Always Better

> Use `join()` only when your data is already indexed — for example, after a `groupby()` that produced a meaningful index. Otherwise, `merge()` is cleaner and more flexible.

Using `'update()'` method

Using `'|'` Operator

Using Set Union Method - `'union()'`