Coder Legion - Recent posts tagged dataanalytics

Forensic Aviation Analytics: Auditing Fuel Bleeds & Delay Cascades on a Regional A320neo Network

Mon, 28 Sep 2026 11:41:49 +0000

When evaluating airline performance, typical analytics dashboards rely blindly on passenger-facing "On-Time" arrival percentages. However, from an operational and financial engineering standpoint, these high-level metrics can easily mask millions of dollars in silent cash destruction.

Due to strategic schedule padding, a short-haul flight can sit idling its engines on a congested tarmac for 20 minutes, yet still be registered as "On-Time" to the customer. The passenger is happy, but the airline’s finance department just watched pure margin vaporize into thin air.

To expose these hidden friction layers, I engineered LCC-Ops: an end-to-end data warehouse auditing over 120,000 regional transaction legs modeled around the single-aisle network performance of flynas across key GCC hubs.

Here is the technical architecture and the forensic database methodologies behind the pipeline.

The Tech Stack & Architecture

Database Layer: MySQL Server 8.0 (Advanced CTEs, Mathematical Window Functions, Schema Migrations)
Business Intelligence: Power BI Desktop (Data Type Optimization, Cross-Table Iterators, UI Design)
Core Fleet Constants: Engineered around the Airbus A320neo (CFM LEAP-1A26 engine) utilizing official ICAO Engine Emissions Databank metrics. At a standard 7% ground idle taxi thrust, this engine burns exactly 0.0940 kg of fuel per second, per engine

The Database Engineering Breakdown

Pillar 1: Isolating Tarmac Cash Destruction (Domain A)To find the true cost of airport ground bottlenecks, the ingestion engine isolates rows inside the transactions table (fact_flights) where the Out-Gate to Runway Takeoff latency (taxi_out) exceeded a standard 4-minute engine stabilization and warming buffer.

By pulling the real-world ICAO fuel coefficients down via relational keys, the calculation bypasses text-based padding and computes raw physical fuel destruction.

SELECT 
    f.origin AS departure_hub,
    COUNT(f.tail_number) AS total_monitored_flights,
    ROUND(AVG(f.taxi_out), 2) AS avg_taxi_out_minutes,
    -- Formula: (Actual Taxi Time - 4 Min Buffer) * 60 Sec * ICAO Idle Flow Rate * 2 Engines
    ROUND(SUM((f.taxi_out - 4) * 60 * e.fuel_flow_idle_kg_sec * 2), 2) AS total_wasted_fuel_kg
FROM fact_flights f
JOIN dim_aircraft a ON f.tail_number = a.tail_number
JOIN dim_engine e ON a.engine_identification = e.engine_identification
WHERE f.reporting_airline = 'XY' AND f.taxi_out > 4
GROUP f.origin;

Pillar 2: Tracking Cross-Border Delay Cascades (Domain D)
When an international flight arriving from an airport like Karachi (KHI) experiences customs or airspace delays, that latency risks cascading into the aircraft’s subsequent domestic legs (e.g., RUH ➔ JED).

To track this chronologically across individual daily airframe lifecycles, I utilized advanced LEAD() mathematical window functions

WITH ranked_daily_schedule AS (
    SELECT 
        flight_date, tail_number, origin, dest, carrier_delay AS leg_1_delay,
        LEAD(origin, 1) OVER (PARTITION BY tail_number, flight_date ORDER BY taxi_out DESC) AS leg_2_origin,
        LEAD(carrier_delay, 1) OVER (PARTITION BY tail_number, flight_date ORDER BY taxi_out DESC) AS leg_2_delay
    FROM fact_flights
    WHERE reporting_airline = 'XY'
)

Pillar 3: Menu Logistics & Hangar Risk Automation (Domains B & C)
Catering Spoilage Optimization: Short flight windows on regional tracks mean cabin crews lack the time to complete complex perishable sales loops. The data models validate flynas’s lean, shelf-stable buy-on-board menu model, proving that switching to fresh profiles would trigger an immediate 54% inventory spoilage cash bleed

Automated Hangar Reorder Points (ROP): The warehouse tracks rolling airframe hours against specific tail numbers (HZ-NS1 to HZ-NS5) [Query-relevant Context]. If a component approaches its high-wear safety limit, the system triggers automated database alerts to protect the airline against catastrophic $25,000/day Aircraft on Ground (AOG) hangar penalties

UI Engineering in Power BI

To present these data layers to corporate stakeholders, I built an interactive application-style interface directly inside Power BI using flynas’s signature branding tones.

The biggest challenge during BI ingestion was data type conversions—empty strings or trailing spaces in the raw logs caused math functions to drop blank values (--). By implementing upfront string transformations (Uppercase/Trim) and embedding data validation wrappers like ISNUMBER(VALUE()) directly into row-level DAX iterators, the front-end dynamically scales across massive transaction volumes without data loss.

The entire production code schema and scripts are live on my GitHub Link. I'd love to hear how other data engineers manage real-time fuel-burn parameters or multi-leg network routing in your pipelines!

The Part of Shipping Software That Doesn't Show Up in a Portfolio Screenshot

Sun, 27 Sep 2026 16:15:09 +0000

A few weeks into building a live finance dashboard for a client, I learned something that had nothing to do with writing code.

What I built

The brief was simple to state and harder to deliver: a dashboard connected directly to the client's Excel workbook — no exports, no "send me the latest file," no waiting on me every time a number changed. Edit the sheet, save it, and the dashboard reflects it within seconds.

Under the hood, it's a Python (Flask) backend that reads straight from the workbook on every request, paired with a frontend that polls for changes and re-renders automatically. On top of that, an interactive map view lets the client click into any project location and see its P&L, purchases, and profitability pulled live from the sheet — no manual recalculation, no stale numbers.

The last step was packaging it so the client — who isn't technical — could just double-click one file and have it work. No Python install, no terminal, no code in sight.

Where it actually got hard

Technically, everything worked. Locally, on my machine, the dashboard ran exactly as intended.

Then I packaged it into a standalone .exe to hand off — and Windows flagged it as potentially unsafe before it ever reached the client.

My first instinct was to assume something was wrong with the code. It wasn't. The actual cause was Smart App Control — a Windows 11 feature that's cautious by default about any unsigned, unrecognized executable, regardless of what's inside it. A freshly built app with no signing history and no reputation looks suspicious to it, even when it's completely benign.

Knowing why it happened didn't make it "shippable," though. I couldn't just tell the client "ignore the warning." I submitted the file to Microsoft for analysis through their official review process, and the status sat at Pending for a stretch — there's no way to force that along, it just takes as long as it takes. When the analyst's response came back, it confirmed the file had been incorrectly blocked and had been remediated so it would no longer trigger the warning.

Only then was it actually ready to hand to someone else's computer.

The takeaway

If I ship more builds of this app over time, the same thing could happen again — a new build means a new file hash, and a new file hash means zero reputation, all over again. The durable fix, I learned, is code-signing: a certificate from a trusted authority that lets Windows (and Microsoft's own analysts) recognize the publisher quickly instead of treating every new build as an unknown.

But the bigger lesson wasn't technical. It was this: building something and shipping something to a real, non-technical user are two different bars. The first one is the part that shows up in a portfolio screenshot. The second one — the packaging, the trust signals, the "will this actually work on someone else's machine without me standing next to them" — is invisible until it isn't, and it's just as much a part of the job.

Looking for a feedback

Sun, 27 Sep 2026 06:46:52 +0000

I have 7+ years of experience as a full-stack and data engineer, and I’m currently transitioning into AI.

I’ve tried applying for projects through platforms such as Mercor, AfterQuery, DataAnnotation, Outlier, Turing, and Snorkel, but I haven’t had much success so far.

I’d really appreciate advice from developers or AI professionals who have experience with these platforms or similar AI-related opportunities.

I’d especially like to know:

How to improve my profile and applications
What skills or experience these platforms are currently looking for
How to stand out when applying for AI projects
Other platforms or opportunities worth exploring

Any practical advice or personal experience would be greatly appreciated.

Retail FMCG Sales Analytics: From 100,000 Transactions to Power BI, Machine Learning, and Streamlit

Sat, 12 Sep 2026 16:51:51 +0000

Introduction

Retail businesses generate large amounts of transactional data every day, but raw data alone does not provide much value unless it can be transformed into meaningful insights and actionable information.

For our Retail FMCG Sales Analytics project, we worked with a dataset representing one full year of retail activity during 2024 for a multi-city FMCG retailer in India.

The dataset contains 100,000 transaction records and 21 variables, covering product and brand information, customer characteristics, sales and financial metrics, stores and cities, sales channels, payment methods, loyalty status, and inventory information.

The objective was not simply to perform exploratory analysis. We wanted to build an end-to-end analytics solution that could move from:

Raw Data → Data Preparation → EDA → Business Analytics → Power BI → Machine Learning → Streamlit Deployment

The final solution combines Python-based analysis, interactive Power BI dashboards, regression modeling, and a Streamlit application that allows users to explore the data and generate transaction-level margin predictions.

1. Understanding the Dataset

The dataset contains several analytical dimensions that make it suitable for both business intelligence and predictive analytics.

The main dimensions include:

Product: Category and Brand
Customer: Age, Gender, and Loyalty Flag
Sales: Units, Selling Price, Revenue, Cost, Margin, and Margin %
Store: City and Store Format
Channel: Online, Offline, and Omnichannel
Payment: UPI, Card, Wallet, and Cash
Inventory: Stock on Hand, Reorder Level, and Lead Time
Time: Invoice Date and derived temporal features

The dataset includes eight FMCG categories: Beverages, Snacks, Dairy, Grocery, Personal Care, Home Care, Fruits, and Vegetables.

One of the main data quality challenges was Customer Age, where approximately 40% of the values were missing. The original dataset also did not include information about returns, wastage, or promotional campaigns, which limited some areas of analysis.

2. Data Cleaning and Preprocessing

Before performing analysis or machine learning, we first had to understand and prepare the dataset.

The preprocessing workflow focused on improving consistency while preserving the analytical value of the original data.

The main steps included:

Identifying missing values and data quality issues.
Validating customer age values.
Creating an indicator for missing customer age.
Handling missing customer age values.
Handling missing categorical values.
Extracting temporal features from Invoice_Date.
Creating additional analytical features.
Preparing categorical variables for machine learning.
Separating the target variable from the input features.

The project also considered the limitations of the dataset during interpretation. In particular, the missing customer-age values affect age-based segmentation, while the absence of promotion and wastage data limits more advanced FMCG analysis.

3. Exploratory Data Analysis

After cleaning the data, the next stage was Exploratory Data Analysis.

The purpose of EDA was not only to create visualizations, but to understand how different business dimensions interact.

We explored:

Sales and Profitability

Revenue, cost, margin, margin percentage, and units sold were analyzed to understand overall business performance.

Product and Category Performance

Category and brand-level analysis was used to examine differences between sales volume and profitability.

This distinction is particularly important in FMCG because the category with the highest sales volume is not necessarily the category generating the highest margin.

Sales Channels

The dataset contains online, offline, and omnichannel transactions, allowing us to compare the economic performance of different channels.

Customer Behavior

Customer age, gender, and loyalty status were analyzed to understand differences in customer behavior.

Inventory

Stock on hand, reorder level, and lead time were analyzed as operational indicators.

These variables allowed us to investigate questions around inventory availability and replenishment.

4. Business Intelligence with Power BI

After completing the exploratory analysis, we transformed the results into interactive dashboards using Microsoft Power BI.

The objective was to move beyond static charts and create a business-facing interface where users could interact with the results.

The dashboard focused on areas such as:

Revenue
Profitability
Profit margin
Units sold
Sales channels
Categories
Brands
Cities
Customer loyalty
Inventory performance
Sales & Profitability Dashboard

The first dashboard provides a high-level view of sales and profitability.

It combines KPI cards with charts that allow users to explore revenue and profit across different business dimensions.

The dashboard also provides interactive filtering, allowing users to investigate specific channels and categories.

Customers & Sales Channels

The customer-focused analysis looks at sales channels, customer age, loyalty, revenue, and margin.

This helps connect customer behavior with financial performance rather than treating customer analytics as a separate topic.

Power BI therefore became the business intelligence layer of the project, transforming analytical results into an interactive decision-support interface.

5. Machine Learning

The next stage was to introduce predictive analytics.

We formulated the machine learning problem as a supervised regression task, with transaction-level Margin as the target variable. Ten regression models were evaluated using standard metrics including:

MAE
MSE
RMSE
R²

The purpose was to compare different approaches rather than relying on a single algorithm.

The evaluated models included:

Linear Regression
Ridge Regression
Lasso Regression
Decision Tree
Random Forest
Gradient Boosting
XGBoost
LightGBM
CatBoost
Support Vector Regression

The final evaluation identified Linear Regression as the selected model based on the executed test results, with an R² value of approximately 1.00 and error values approaching zero.

The selected model was then integrated into the deployed application.

Why compare multiple models?

Model comparison gave us a broader understanding of how different regression approaches performed on the dataset.

Instead of assuming that a more complex algorithm would automatically produce the best result, we evaluated the models using the same evaluation framework and selected the final model based on the observed test performance.

6. Streamlit Deployment

One of the main goals of the project was to take the analysis beyond notebooks and dashboards.

For this reason, we developed a Streamlit web application that integrates the main analytical and machine learning components into one interface.

The deployed application contains five main sections:

Overview

Introduces the project, dataset, and analytical context.

Analytics

Provides interactive Python-based analysis across revenue, profit, margin, channels, categories, brands, cities, customers, loyalty, and inventory.

Data Quality

Provides visibility into important data quality characteristics, including missing customer-age values.

Machine Learning

Presents the machine learning workflow, evaluated models, and final model results.

Prediction

Allows users to enter transaction characteristics and generate a predicted transaction-level margin.

The application workflow is essentially:

Load Data → Load Model → Prepare Features → Accept User Inputs → Apply Transformations → Generate Prediction

This deployment creates a user-facing layer on top of the analytical workflow and removes the need for users to directly execute the original notebooks.

7. The Complete Architecture

The project can be viewed as several connected layers rather than separate tools.

                Retail FMCG Dataset
                       │
                       ▼
            Data Understanding
                       │
                       ▼
          Cleaning & Preprocessing
                       │
                       ▼
              Feature Engineering
                       │
                       ▼
                Exploratory EDA
                 /           \
                /             \
               ▼               ▼
      Business Analytics    ML Modeling
               │               │
               ▼               ▼
           Power BI       Model Evaluation
               │               │
               └───────┬───────┘
                       ▼
                  Streamlit
                       │
                       ▼
             Interactive Solution

This architecture allowed us to keep the project connected from the original dataset all the way to the final deployed application.

The final report describes the same lifecycle as a nine-stage workflow: dataset understanding, cleaning, feature engineering, EDA, business analytics, Power BI, machine learning, model evaluation, and Streamlit deployment.

8. Key Business Questions

The project was built around several business questions rather than analysis for its own sake.

Which categories perform better in terms of margin versus volume?

This allows management to distinguish between high-volume categories and highly profitable categories.

Which sales channels are more economically viable?

The combination of revenue, margin, and channel information allows online, offline, and omnichannel performance to be compared.

Do loyalty customers behave differently?

Customer loyalty can be analyzed alongside revenue, units, and margin.

How can inventory indicators support operational decisions?

Stock on hand, reorder level, and lead time provide indicators that can be used to investigate inventory and replenishment performance.

Which store formats are more efficient for perishable products?

The dataset contains store-format and category information that can support comparisons between Hyper, Super, and Express formats.

9. Challenges and Limitations

No analytical project should be presented without discussing its limitations.

The main limitations of this dataset were:

Missing Customer Age:
Approximately 40% of customer-age values were missing.

No Returns or Wastage:
This limits detailed analysis of shrinkage and perishables.

No Promotion Data:
The dataset does not include promotional campaigns or discounts, limiting promotion-effectiveness and price-elasticity analysis.

Limited Basket Analysis:
Each record represents a single transaction line, which limits the ability to reliably analyze products purchased together.

These limitations are important because they define what the dataset can—and cannot—tell us.

10. What We Built

At the end of the project, we had a complete analytical solution rather than a collection of disconnected notebooks.

The final solution includes:

Python
For data preparation, analysis, visualization, and machine learning.

Power BI
For interactive business intelligence dashboards.

Machine Learning
For comparing ten regression models and building a margin prediction capability.

Streamlit
For exposing the analytical and predictive functionality through an interactive web application.

GitHub
For organizing and sharing the complete project.

The final project therefore combines descriptive analytics, business intelligence, predictive analytics, and deployment into one workflow.

11. Final Takeaways

The main lesson from this project was that a successful data analytics project is not only about building charts or achieving a high machine learning score.

The real value comes from connecting the different stages.

A dataset becomes useful when it is properly understood.

Clean data enables reliable analysis.

EDA reveals patterns.

Business intelligence makes those patterns easier to explore.

Machine learning adds predictive capability.

And deployment makes the final solution accessible to people who may never open a Jupyter Notebook.

For us, the project was an opportunity to work through that entire process.

100,000 records → one analytical workflow → one deployed solution.

Project Resources

GitHub Repository:
https://github.com/Fadydesoky/Retail-FMCG-Analytics

Live Streamlit Application:
https://retail-fmcg-analytics.streamlit.app/

What's next?

There are several directions in which this project could be extended.

Future versions could include:

Demand forecasting
Inventory optimization
Promotion and pricing analysis
Customer segmentation
Basket analysis
Perishable-product analysis
More advanced machine learning approaches
Additional Streamlit functionality

These extensions are also consistent with the limitations identified in the project report.

Let's Connect

If you found this project interesting or would like to discuss data analytics, business intelligence, machine learning, or similar projects, feel free to connect with me.

LinkedIn:
https:www.linkedin.com/in/fadydesokysaeedabdelaziz

GitHub:

https:www.github.com/Fadydesoky

From GitHub Repositories to a Professional Data Analytics Portfolio Website

Sat, 08 Aug 2026 07:35:02 +0000

Building More Than Just Projects — Building My Professional Identity

Completing my Data Analyst Internship at Logic Stack was a significant milestone. Over four weeks, I worked on real-world business problems using Excel, Power BI, Python and SQL, successfully completing four practical analytics projects.

Like many learners, I initially thought that uploading my projects to GitHub would be enough.

After all, GitHub is the standard platform for developers and data professionals.

But then I asked myself a simple question:

If a recruiter visits my profile, will they truly understand my journey just by opening four separate repositories?

The answer was No.

That realization inspired me to build something much bigger than another GitHub repository.

It inspired me to build my own Production-Level Data Analytics Portfolio Website.

Why GitHub Alone Wasn't Enough

GitHub is excellent for storing source code and documentation.

However, recruiters usually spend only a few minutes reviewing a candidate's profile.

Opening multiple repositories one by one makes it difficult to understand:

The overall learning journey
Project progression
Skills gained
Business impact
Technical growth

I wanted to create a place where everything could be explored from a single professional dashboard.

The Idea

Instead of sharing multiple independent repositories, I designed a portfolio website that presents my internship as one complete journey.

Rather than saying,

"Here are my projects."

I wanted the website to tell a story:

How I started, what I learned, which challenges I faced, how I solved them and what I built by the end of the internship.

That became the foundation of the entire portfolio.

Designing the Portfolio

The goal was simple:

Create a clean, modern, recruiter-friendly website that highlights both my Data Analytics skills and my Web Development experience.

The portfolio includes carefully designed sections that guide visitors through my internship journey.

Home

A professional landing page introducing my internship, background, and career focus.

Projects

A dedicated section for all four internship projects.

Each project contains:

Project overview
Technologies used
GitHub Repository
LinkedIn post
YouTube explanation video

Everything is connected from one location.

Skills Learned

Instead of simply listing technologies, I organized everything I learned during the internship into one dedicated section.

The portfolio highlights skills including:

Microsoft Excel
Power BI
Python
SQL
Data Cleaning
Exploratory Data Analysis
KPI Development
Business Intelligence
Dashboard Design
Documentation

This gives recruiters a quick overview of my technical capabilities.

Challenges Solved

One of my favourite sections is the Challenges Solved page.

Instead of hiding the problems I encountered, I documented them professionally.

Examples include:

Python environment setup
Library installation issues
SQL query debugging
Data cleaning problems
Documentation challenges
GitHub repository organization

Each challenge is paired with the solution I implemented.

I believe problem-solving demonstrates professional growth more effectively than simply showing successful projects.

Certificate & Internship Documents

The portfolio also includes my internship certificate and professional documents.

To keep them organized, I implemented a simple password-protected section that allows authorized viewers to access these files when needed.

This keeps the homepage clean while still making important documents available.

Building with Modern Technologies

This project also became an opportunity to strengthen my web development skills.

The website was built using modern technologies including:

Next.js
TypeScript
Tailwind CSS
Framer Motion
Responsive Design
SEO Optimization
Structured Data (JSON-LD)
Robots.txt
Sitemap.xml

By combining these technologies, I transformed a simple portfolio into a production-ready web application.

SEO Matters Too

I didn't want the website to look professional only for people who already had the link.

I wanted search engines to understand it as well.

To improve visibility, I implemented:

Metadata optimization
Open Graph tags
Structured Data
Robots configuration
XML Sitemap
Performance optimization

Achieving high Lighthouse scores became another goal during development, ensuring the website follows modern web standards.

Combining Two Different Skill Sets

This project became much more than a portfolio.

It combined two different fields I enjoy:

Data Analytics

Business insights
SQL
Power BI
Python
Excel

Web Development

Next.js
TypeScript
UI Design
Responsive Layouts
SEO
Performance Optimization

Bringing these two disciplines together allowed me to create something that represents my complete technical profile.

Lessons Beyond the Internship

Building this website taught me several important lessons.

A portfolio is not just about displaying work.

It is about presenting your journey professionally.

I learned that:

Presentation matters.
Documentation matters.
User experience matters.
Personal branding matters.
Professional storytelling matters.

A recruiter should be able to understand who you are within minutes.

That became the guiding principle behind this project.

Final Thoughts

Looking back, I'm glad I didn't stop after uploading four repositories.

Creating this portfolio allowed me to organize my internship experience into one professional platform that showcases both my technical skills and my continuous learning journey.

This website represents more than projects.

It represents persistence.

It represents growth.

It represents every challenge solved throughout the internship.

Most importantly, it reminds me that learning never truly ends.

Every project becomes a stepping stone toward the next opportunity.

Explore My Complete Portfolio

🌐 Live Portfolio:
https://yasirawaninfo-data.vercel.app/

💻 GitHub:
https://github.com/YasirAwan4831

Thank you for taking the time to explore my journey. I hope it inspires other learners to build not only projects—but also a professional identity around their work.

#DataAnalytics #SQL #SQLite #PowerBI #Python #Panda #Numpy #Matplotlib #Excel #BusinessIntelligence #DataVisualization #Dashboard #DataCleaning #JupyterNotebook #ProblemSolving #GitHub #WebDevelopment #LearningInPublic #Portfolio #CareerGrowth ##MuhammadYasir #YasirAwan4831

Continuing My Data Analyst Journey: Week 3, 4 and Challenges I Faced During My Data Analyst Intern

Fri, 07 Aug 2026 19:22:22 +0000

In my previous blog, I shared my journey through Week 1 and Week 2, where I worked on Excel-based data cleaning, sales analysis, Pivot Tables and Power BI dashboards.

As the internship progressed, the learning curve became much steeper. The focus shifted from spreadsheet analytics to programming, databases and solving real-world business problems using professional data analysis tools.

Weeks 3 and 4 introduced completely new technologies, larger datasets and more technical challenges that pushed me far beyond my comfort zone.

These two weeks transformed my internship from learning business reporting into building complete end-to-end data analytics solutions.

Week 3: Supply Chain Analytics using Python & Power BI

Week 3 marked my first serious experience with Python for Data Analytics.

Unlike previous weeks, this project required setting up a complete Python development environment before writing a single line of analysis. I worked extensively with Pandas, NumPy, Matplotlib and Jupyter Notebook to clean, analyse and visualize a real-world supply chain dataset containing thousands of records.

After completing the Exploratory Ddata Analysis (EDA), I integrated the insights into Power BI, creating an interactive dashboard that highlighted sales performance, shipment modes, product groups, vendors and country-wise business metrics.

This project taught me that successful data analysis begins with clean, reliable data—and that Python is one of the most powerful tools for transforming raw datasets into meaningful business insights.

GitHub Repository: https://github.com/YasirAwan4831/week-3-python-powerbi-supply-chain-analytics

YouTube Video: https://youtu.be/SQk41XxxpsY?si=ElhxePDt916bh_VJ

LinkedIn Post: https://www.linkedin.com/posts/yasirawan4831_eda-dataabranalyst-fullabrstackabrdevel

Week 4: Funnel & Revenue Analysis using SQL & Power BI

The final week introduced one of the most important skills in modern data analytics: SQL.

For this project, I worked with SQLite to analyse customer funnel performance, revenue generation and conversion metrics directly from a relational database. Instead of relying on spreadsheets, I learned how businesses retrieve insights using SQL queries, aggregate functions, filtering, grouping and KPI calculations.

The extracted data was then connected to Power BI, where I built an interactive business dashboard presenting conversion rates, revenue trends and overall funnel performance.

This project completed my internship journey by combining database querying, business intelligence and visualization into one complete analytics workflow.

GitHub Repository: https://github.com/YasirAwan4831/week-4-sql-powerbi-funnel-analysis

YouTube Video: https://youtu.be/sdBYOKqT6YM?si=HfNw1CrgG71dHIzi

LinkedIn Post: https://www.linkedin.com/posts/yasirawan4831_sqlite-sqlabrbased-dataabranalys

From Learning Tools to Solving Problems

By the end of Week 4, I had worked with:
Microsoft Excel
Power BI
Python
SQL
Business Analytics
Data Cleaning
Exploratory Data Analysis (EDA)
KPI Dashboard Development
Funnel Analysis
GitHub Documentation

However, learning these tools was only part of the journey.
The real growth came from overcoming technical challenges, debugging errors, researching solutions and continuously improving every project.

10 Real Challenges I Faced During My Data Analyst Internship (And How I Solved Them)

Every Internship Teaches Skills — But Real Growth Comes From Solving Problems

When people look at a completed project, they usually see the dashboard, the charts, the SQL queries, or the final GitHub repository.

What they don't see are the countless hours spent fixing errors, learning unfamiliar tools, debugging environments, researching documentation and solving problems that never appeared in any tutorial.

During my one-month Data Analyst Internship at Logic Stack, I completed four industry-based projects covering Excel, Power BI, Python, SQL and Business Analytics.

However, the most valuable outcome wasn't just the projects themselves.

It was learning how to solve problems independently.

This blog shares the biggest technical and practical challenges I faced during my internship—and how each one made me a better Data Analyst.

Week 3 Project

Supply Chain Analytics using Python & Power BI

By Week 3, the internship shifted from Excel into Python-based analytics.

This project focused on cleaning, analysing and visualizing a real-world Supply Chain dataset using Python (Pandas, NumPy, Matplotlib) alongside Power BI.

Unlike previous weeks, this project introduced a completely new development environment, making it both exciting and challenging.

Week 4 Project

Funnel & Revenue Analysis using SQL & Power BI

The final week introduced SQL for business analytics.

The objective was to analyse conversion funnels, revenue performance and customer behaviour using SQL queries before presenting business insights inside Power BI dashboards.

This project simulated real-world reporting workflows where raw database queries become executive dashboards.

Challenge 1

Starting With Zero Practical Experience

Challenge

Although I had experience in web development, I had almost no practical exposure to professional Data Analytics workflows.

Everything—from Excel reporting to SQL analytics—was new.

Solution

Instead of trying to learn everything at once, I followed a structured approach:

Learning one technology at a time helped me build confidence while creating portfolio-ready projects every week.

Challenge 2

Building a Professional Excel Workflow

Challenge

Creating clean spreadsheets is easy.

Creating professional business reports is not.

I had to learn:

Structured Tables
Pivot Tables
Professional Formatting
Multiple Worksheets
Formula Organization

Solution

I researched Excel best practices and redesigned my workbook several times until it followed professional reporting standards.

The result was an organized multi-sheet Excel project that later became the foundation for my Power BI dashboard.

Challenge 3

Learning Power BI Dashboard Design

Challenge

Power BI wasn't difficult because of the software.

It was difficult because dashboards must answer business questions—not simply display charts.

Solution

I learned:

KPI Cards
DAX Basics
Dashboard Layout Principles
Visual Hierarchy
Business Storytelling

Rather than creating colourful visuals, I focused on dashboards that communicate insights clearly.

Challenge 4

Configuring the Python Development Environment

Challenge

This was one of the toughest technical challenges.

Issues included:

VS Code Python interpreter problems
Jupyter Kernel connection failures
Package installation errors
Environment configuration
Version conflicts

These problems had nothing to do with analytics—but they completely blocked development.

Solution

Instead of giving up, I researched official documentation, debugging guides and community discussions.

Eventually I:

Fixed interpreter paths
Installed required libraries
Connected Jupyter successfully
Stabilized the development environment

This experience taught me that environment setup is a real engineering skill.

Challenge 5

Cleaning a Real-World Dataset

Challenge

The Supply Chain dataset contained:

Missing values
Mixed data types
Invalid numeric values
Text inside number columns

Real business datasets are rarely clean.

Solution

Using Pandas, I:

Converted invalid values safely
Filled missing values
Removed inconsistencies
Validated cleaned data

Only after proper cleaning did visualization become meaningful.

Challenge 6

Installing SQLite & Building SQL Workflows

Challenge

Week 4 introduced SQL.

Before writing queries, I first had to configure SQLite and import datasets correctly.

Solution

I installed DB Browser for SQLite, verified database structure, imported CSV datasets correctly and prepared the environment before beginning SQL analysis.

This preparation prevented many future errors.

Challenge 7

Writing Complex SQL Queries

Challenge

Simple SELECT statements were easy.

Business reporting required:

GROUP BY
Aggregate Functions
Filtering
Revenue Calculations
Conversion Metrics

Solution

Instead of memorizing syntax, I practiced writing increasingly complex queries while documenting each command inside my repository.

This improved both my SQL knowledge and documentation skills.

Challenge 8

Turning Data Into Business Insights

Challenge

Anyone can generate numbers.

A Data Analyst must explain what those numbers actually mean.

Solution

For every project I focused on answering business questions:

Which products perform best?
Where is revenue growing?
Which funnel stage loses customers?
What actions should businesses take?

This mindset transformed technical work into business analysis.

Challenge 9

Professional GitHub Documentation

Challenge

Writing code is only half the job.

Presenting projects professionally is equally important.

Solution

I created consistent repositories with:

Detailed README files
Professional documentation
SVG animations
Shields.io badges
Folder organization
Screenshots
Business explanations

These repositories became portfolio-quality projects rather than simple assignments.

Challenge 10

Building My Own Data Analyst Portfolio Website

Challenge

Instead of simply uploading four GitHub repositories, I wanted to create something unique.

Solution

I designed and developed a complete interactive portfolio website using Next.js and TypeScript.

The website includes:

Internship Overview
Four Complete Projects
GitHub Repositories
LinkedIn Posts
YouTube Project Videos
Skills Learned
Challenges Solved
Password-Protected Certificate Section
Resume Download
Modern Responsive UI
SEO Optimization

This project combined my web development background with my newly acquired Data Analytics skills, creating a portfolio that reflects both technical implementation and professional presentation.

What This Internship Really Taught Me

This internship wasn't just about learning Excel, Python, SQL, or Power BI.

It taught me:

How to research independently.
How to debug unfamiliar problems.
How to organize professional projects.
How to document technical work.
How to communicate business insights.
How to keep learning when solutions aren't immediately obvious.

These lessons will remain valuable long after the internship ends.

Final Thoughts

Looking back, the dashboards and repositories represent only the visible outcome.

The real achievement was overcoming every obstacle that appeared along the way.

Every error message, every failed installation, every debugging session and every rewritten query contributed to my growth as a Data Analyst.

The internship has ended—but the learning journey continues.

Thank you, Logic Stack, for providing a practical environment where I could build real projects, solve real problems and gain valuable industry experience.

Connect With Me

🌐 Portfolio
https://yasirawaninfo-data.vercel.app/

💻 GitHub
https://github.com/YasirAwan4831

💼 LinkedIn
https://linkedin.com/in/yasirawan4831

Thank you for reading. If this journey inspires someone starting their own Data Analytics path, then sharing these experiences has been worthwhile.

#DataAnalytics #SQL #SQLite #PowerBI #BusinessIntelligence #DataVisualization #Dashboard #GitHub #Internship #LogicStack #DataAnalyst #LearningJourney #OpenToWork #Portfolio #YasirAwan4831

My Data Analyst Internship Journey: From Learning the Fundamentals to Building Production-Ready Anal

Fri, 07 Aug 2026 06:12:48 +0000

Introduction

Every learning journey begins with a single decision.

For me, that decision was to step into the world of Data Analytics—a field that combines problem-solving, business understanding, and technology to transform raw data into meaningful insights.

When I joined the Logic Stack Data Analyst Internship, I wasn't simply looking to complete another internship. My goal was much bigger: I wanted to build practical skills, work on real-world projects, document every step professionally and create a portfolio that genuinely reflected my growth.

Over the course of one month, I worked on multiple projects involving Excel, Python, SQL, Power BI, GitHub, technical documentation, and portfolio development. Every week introduced new concepts, new challenges and new opportunities to improve.

This blog shares my complete journey—from the first day of the internship to the completion of the final project. Rather than focusing only on the outcomes, I want to share the process, the lessons I learned and how each project helped me grow as an aspiring Data Analyst.

Why I Chose This Internship

The field of Data Analytics has become one of the fastest-growing domains in technology. Organizations across every industry rely on data to make better decisions, optimize operations, and understand customer behaviour.

I wanted hands-on experience with the tools and workflows used by professionals instead of learning only through theory.

This internship provided exactly that opportunity.

It focused on:

Practical business datasets
Real analytical workflows
Professional documentation
Dashboard development
Industry-standard tools
Portfolio building

Instead of completing isolated exercises, every week contributed to a larger learning journey.

About the Logic Stack Internship

The internship was designed as a structured four-week program where every week introduced a new challenge.

Rather than repeating similar exercises, each project focused on solving a different business problem using different analytical techniques.

Throughout the internship I worked with:

Microsoft Excel
Python
Pandas
NumPy
SQL (SQLite)
Power BI
GitHub
Technical Documentation

Every project required not only completing the technical work but also presenting it professionally through GitHub repositories, documentation, dashboards, screenshots and project reports.

My Goals Before Starting

Before beginning the internship, I set a few personal goals for myself:

Learn modern Data Analytics tools through practical work.
Improve my analytical thinking.
Build projects suitable for a professional portfolio.
Learn proper GitHub documentation.
Create dashboards that communicate business insights effectively.
Gain confidence working with real datasets.

Looking back now, these goals helped me stay focused throughout the internship.

Week 1 – Learning the Foundations

The first week focused on building a strong analytical foundation.

Instead of jumping directly into advanced visualization, the emphasis was placed on understanding datasets, identifying business questions, cleaning information and extracting useful insights.

This project taught me that successful data analysis begins long before dashboards are created.

Some of the concepts I worked on included:

Understanding structured datasets
Data cleaning
Exploratory analysis
Identifying patterns
Basic business insights
Professional project documentation

More importantly, I learned how to organize a project professionally from the beginning—a habit that became extremely valuable during the following weeks.

GitHub Repository: https://github.com/YasirAwan4831/week-1-retail-sales-excel-analysis

YouTube Video: https://youtube.com/shorts/-vCaOvUj-DA?si=4DzXsE1v3Idt84xs

LinkedIn Post: https://www.linkedin.com/posts/yasirawan4831_excel-logicstack-dataanalyst-activity-

Week 2 – Building Analytical Confidence

The second week pushed my skills further.

The datasets became larger, the questions became more analytical and the expectations shifted from simply calculating values to explaining business performance.

This project strengthened my ability to:

Interpret business metrics
Compare trends
Identify opportunities
Present findings clearly
Think from a business perspective instead of only a technical perspective

One important lesson from this week was that data alone has very little value unless it is converted into actionable insights.

That mindset completely changed how I approached the remaining internship projects.

GitHub Repository: https://github.com/YasirAwan4831/week-2-excel-powerbi-sales-dashboard

YouTube Video: https://youtube.com/shorts/JLg-B-rs0gE?si=kCWG7KHxdxw2d-Pa

LinkedIn Post: https://www.linkedin.com/posts/yasirawan4831_weekabr2-excel-powerabrbi-activity-

Continuing the Journey...

By the end of the first two weeks, I had gained confidence working with datasets, understanding business requirements, and documenting projects professionally.

However, the most challenging—and most rewarding—part of the internship was still ahead.

In the next part of this journey, I'll cover:

Week 3: Supply Chain Analytics with Python & Power BI
Week 4: SQL + Power BI Funnel Analysis
Building production-quality GitHub repositories
Creating a modern Data Analytics Portfolio Website
Earning my internship completion certificate
Key lessons and reflections from the entire experience

The second half of this journey represents the point where I transitioned from completing internship tasks to building projects that reflect my professional capabilities as a Data Analyst.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - ---------------------------------------------------
#DataAnalytics #Portfolio #SQL #PowerBI #Python #Excel #GitHub #WebDevelopment #LearningInPublic #CareerGrowth #OpenToWork Muhammad Yasir #MuhammadYasir #YasirAwan4831

Smart Movie Recommender using Linear SVC

Sun, 19 Apr 2026 14:49:45 +0000

Project Overview

This project focuses on building an end-to-end Machine Learning pipeline to analyze user sentiment from IMDB movie reviews. Unlike standard binary classifiers, this system incorporates Decision Logic to identify neutral feedback, providing a more nuanced recommendation: "Recommend," "Do Not Recommend," or "Maybe Watch."

How I Built It (Technical Workflow)

Advanced Text Preprocessing:
Raw text is often noisy. I developed a cleaning pipeline using NLTK and Regex:
- Noise Removal: Stripped HTML tags and non-alphabetic characters.
- Normalization: Converted all text to lowercase for consistency.
- Stopword Removal & Lemmatization: Removed common fillers (e.g., "the", "a") and reduced words to their dictionary root (e.g., "watched" becomes "watch") to reduce dimensionality.
Feature Engineering (TF-IDF):
To translate text into a language machines understand, I used TF-IDF (Term Frequency-Inverse Document Frequency). I limited the vocabulary to the top 5,000 features to ensure the model focuses on the most impactful words while ignoring rare noise.
Model Implementation:
I chose Linear SVC (Support Vector Classification). It is highly efficient for high-dimensional text data and provides a clear decision boundary between sentiments.

Challenges & Solutions

1. The Neutral Data Gap

The Challenge: The IMDB dataset is strictly binary (Positive/Negative). This meant the model had never "seen" a neutral review, causing it to force-classify "average" movies as either good or bad.
The Solution: Instead of using model.predict(), I utilized model.decision_function(). By calculating how close a review lies to the decision boundary (Thresholding), I was able to manually trigger a Neutral (0) classification for reviews that lacked strong positive or negative conviction.

2. Balancing Overfitting

The Challenge: Initially, the model showed high training accuracy but lower performance on new data.
The Solution: I leveraged L2 Regularization (inherent in Linear SVC). By monitoring the gap between training and testing scores, I ensured the model generalized well. Limiting the TF-IDF features also helped prevent the model from "memorizing" specific review noise.

How To Use This Project

Setup: Ensure scikit-learn, pandas, and nltk are installed in your Python environment.
Input: Pass a raw string (movie review) into the get_recommendation function.

Inference Example:

review = "The visuals were stunning, but the pacing was a bit slow and mediocre."
print(get_recommendation(review)) 
    
# Result -> Score: 0.04 | Sentiment: 0 | Action: Maybe watch (Neutral)

Final Performance

Test Accuracy: 88%
Generalization: The model maintains a balanced F1-score across both classes, proving it is robust against bias and ready for real-world unseen data.

Movie Review Sentiment Analysis

Sun, 19 Apr 2026 14:48:21 +0000

Project Overview

How I Built It (Technical Workflow)

Advanced Text Preprocessing:
Raw text is often noisy. I developed a cleaning pipeline using NLTK and Regex:
- Noise Removal: Stripped HTML tags and non-alphabetic characters.
- Normalization: Converted all text to lowercase for consistency.
- Stopword Removal & Lemmatization: Removed common fillers (e.g., "the", "a") and reduced words to their dictionary root (e.g., "watched" becomes "watch") to reduce dimensionality.
Feature Engineering (TF-IDF):
To translate text into a language machines understand, I used TF-IDF (Term Frequency-Inverse Document Frequency). I limited the vocabulary to the top 5,000 features to ensure the model focuses on the most impactful words while ignoring rare noise.
Model Implementation:
I chose Linear SVC (Support Vector Classification). It is highly efficient for high-dimensional text data and provides a clear decision boundary between sentiments.

Challenges & Solutions

1. The Neutral Data Gap

The Challenge: The IMDB dataset is strictly binary (Positive/Negative). This meant the model had never "seen" a neutral review, causing it to force-classify "average" movies as either good or bad.
The Solution: Instead of using model.predict(), I utilized model.decision_function(). By calculating how close a review lies to the decision boundary (Thresholding), I was able to manually trigger a Neutral (0) classification for reviews that lacked strong positive or negative conviction.

2. Balancing Overfitting

The Challenge: Initially, the model showed high training accuracy but lower performance on new data.
The Solution: I leveraged L2 Regularization (inherent in Linear SVC). By monitoring the gap between training and testing scores, I ensured the model generalized well. Limiting the TF-IDF features also helped prevent the model from "memorizing" specific review noise.

How To Use This Project

Setup: Ensure scikit-learn, pandas, and nltk are installed in your Python environment.
Input: Pass a raw string (movie review) into the get_recommendation function.

Inference Example:

review = "The visuals were stunning, but the pacing was a bit slow and mediocre."
print(get_recommendation(review)) 
    
# Result -> Score: 0.04 | Sentiment: 0 | Action: Maybe watch (Neutral)

Final Performance

Test Accuracy: 88%
Generalization: The model maintains a balanced F1-score across both classes, proving it is robust against bias and ready for real-world unseen data.

Sentiment Analysis Using NLP: Visualizing Emotions in Text with Python and Power BI

Fri, 17 Apr 2026 00:26:59 +0000

Introduction

Sentiment analysis is one of the most practical applications of Natural Language Processing (NLP).

In this project, I explore how to perform sentiment analysis on a short story and visualize emotional patterns using Python and Power BI.

The goal is simple: transform raw text into meaningful insights.

In this guide, we’ll walk through how to perform sentiment analysis using NLP step-by-step.

What is Sentiment Analysis in NLP?

Sentiment analysis is a technique used in NLP to determine whether a piece of text expresses a positive, negative, or neutral emotion.

It is widely used in:

Customer feedback analysis
Social media monitoring
Product reviews

In this case, I applied sentiment analysis to a literary text to understand how emotions evolve throughout a story.

Project Idea

Most people read stories…

But what if we could analyze how a story feels?

I used “The Tell-Tale Heart” by Edgar Allan Poe and broke it into smaller segments, then applied sentiment analysis to each part.

This allowed me to track emotional changes across the narrative.

How to Analyze Text Using NLP

The workflow for this project was:

Load and clean the text
Split the text into segments
Apply sentiment analysis
Generate a structured dataset
Visualize the results

Here’s a simplified Python example:

from textblob import TextBlob

text = "I felt nervous and terrified."
blob = TextBlob(text)

sentiment = blob.sentiment.polarity
print(sentiment)

This returns a sentiment score that helps classify the emotional tone of the text.

Building a Sentiment Analysis Dataset

After processing the text, I created a dataset like this:

   Segment  | Label   | Score
   1        | Positive| 0.91
   2        | Negative| 0.45

This step is critical because it transforms unstructured text into structured data that can be analyzed and visualized.

Visualizing Sentiment Analysis Results Using Power BI

Once the dataset was ready, I built a dashboard using Power BI to visualize:

Emotional trends over time
Distribution of positive vs negative segments
Key moments of emotional intensity

Sentiment analysis dashboard using NLP and Power BI

This helped turn raw numbers into a clear emotional narrative.

Key Insights from the Analysis

The story appears mostly positive overall (~56%)
However, it contains frequent sharp emotional drops
These drops align with moments of tension and psychological intensity

This shows that even if a story seems balanced overall, emotional spikes reveal its most impactful moments.

What I Learned

How to apply sentiment analysis using NLP in a real project
How to transform text into structured datasets
How to visualize insights using Power BI
The importance of storytelling in data analysis

Conclusion

This project demonstrates how sentiment analysis using NLP can go beyond traditional use cases and be applied to creative domains like literature.

By combining NLP with visualization tools, we can better understand how emotions evolve in any text.

Project Links

GitHub Repository: https://github.com/Fadydesoky/ai-story-sentiment-analysis

LinkedIn: https://www.linkedin.com/in/fadydesokysaeedabdelaziz

If you're interested in sentiment analysis using NLP or building real-world data projects, feel free to explore the full project on GitHub or connect with me on LinkedIn.

Mastering Pandas — Part 3: Data Cleaning, Merging & Joining

Wed, 18 Mar 2026 23:24:55 +0000

Real Data Is Never Clean

In Part 2, you learned how to group and slice a DataFrame with precision. But in real projects, the data you receive rarely arrives ready to analyze. Column names have spaces. Ages are negative. Phone numbers mix letters with digits. Duplicate rows sneak in. And your data often lives across multiple files that need to be combined before you can do anything useful.

This article covers the two skills that bridge raw data and real analysis:

Data Cleaning — detecting and fixing everything wrong with your data before it corrupts your results
Merging & Joining — combining separate DataFrames into a single, unified dataset

These topics flow naturally into each other. You clean each source first, then you combine them. That's the order every data analyst follows in practice — and it's the order we'll follow here.

We'll use this sample dataset throughout the cleaning section:

import pandas as pd
import numpy as np

data = {
    "Name":             ["Alice", "bob", "Carol", "Alice", "  dave  ", "Eve"],
    "Age":              [25, -5, 200, 25, 30, None],
    "Phone":            ["055-123-4567", None, "55123x", "055-123-4567", "055-987-0000", "055abc"],
    "Email":            ["*Emails are not allowed*", "*Emails are not allowed*", None, "*Emails are not allowed*", "*Emails are not allowed*", "not-an-email"],
    "Paying_Customer":  ["Y", "N", "Yes", "Y", "N", "y"],
    "Score":            [85, 110, 72, 85, -10, 90],
}
df = pd.DataFrame(data)

Step 1 — Inspect Before You Touch Anything

The most important rule in data cleaning: look before you act. Jumping straight into fixes without understanding the full picture leads to mistakes you won't catch until much later.

These are the inspection tools you should run on every new dataset:

df.shape          # (rows, columns) — how big is this?
df.info()         # column names, data types, non-null counts
df.head()         # first 5 rows
df.tail()         # last 5 rows
df.describe()     # min, max, mean, std — spot impossible values here

For missing values specifically:

df.isna().sum()           # count NaN per column
df.isnull().sum()         # same as isna() — both work
df[df["Phone"].notna()]   # rows that actually have a phone number

For duplicates and inconsistent categories:

df.duplicated()                         # True/False per row
df[df.duplicated()]                     # show only the duplicate rows
df["Paying_Customer"].value_counts()    # reveals: Y, N, Yes, y — all meaning the same thing

Always run describe() on numeric columns — the min and max alone will tell you immediately if something is wrong. A minimum age of -5 or a score of 110 out of 100 is a red flag you can't miss.

Step 2 — Fix Column Names First

Messy column names make everything harder. Clean them before anything else, so every subsequent operation uses predictable, consistent names.

Rename specific columns:

df.rename(columns={"Old Name": "New_Name", "phone number": "Phone"})

Clean all column names at once — the standard one-liner:

df.columns = df.columns.str.strip().str.lower().str.replace(" ", "_")

This strips surrounding spaces, converts to lowercase, and replaces spaces with underscores in a single chain. After this, "First Name" becomes "first_name", " Age " becomes "age", and so on.

Before:              After:
"First Name"    →    "first_name"
"  Age "        →    "age"
"Phone Number"  →    "phone_number"
"ZIP Code"      →    "zip_code"

Step 3 — Handle Missing Values

Missing values appear as NaN in Pandas. Before deciding what to do with them, you need to understand why they're missing — because the right fix depends on the cause.

Fill missing values with a fixed value:

df["Phone"].fillna("000-000-0000")    # placeholder for missing phone
df["Age"].fillna(0)                   # fill numeric with 0
df["Name"].fillna("Unknown")          # fill text with label
df["Score"].fillna(df["Score"].mean()) # fill with column average

Fill missing numeric values by estimating from neighbors:

df["Sales"] = df["Sales"].interpolate()

interpolate() fills NaN by calculating the midpoint between the values above and below it. This is ideal for time series or sensor data where values change gradually — it would be wrong to use for random datasets.

Before:     After interpolate():
100         100
NaN    →    150   ← estimated midpoint
200         200
NaN    →    225   ← estimated midpoint
250         250

Remove rows with missing values:

df.dropna()                            # drop any row with at least one NaN
df.dropna(subset=["Phone"])            # drop only if Phone is NaN
df.dropna(thresh=3)                    # drop rows that have fewer than 3 non-NaN values

inplace rule: use either inplace=True or variable assignment — never both. They do the same thing, and combining them produces a bug where df becomes None.
df.dropna(subset=["Phone"], inplace=True)   # ✅ modifies df directly
df = df.dropna(subset=["Phone"])            # ✅ returns new df
df = df.dropna(subset=["Phone"], inplace=True)  # ❌ df becomes None

Step 4 — Remove Duplicates

Duplicate rows silently inflate counts, averages, and totals. Always check for them.

df.drop_duplicates()                         # remove fully identical rows
df.drop_duplicates(subset=["Name"])          # remove rows with duplicate Name only
df.drop_duplicates(subset=["Name", "Email"]) # remove rows where both columns match
df.drop_duplicates(keep="last")              # keep the last occurrence instead of the first

Before:                        After drop_duplicates():
Name   Email                   Name   Email
Alice  *Emails are not allowed*          Alice  *Emails are not allowed*
Bob    *Emails are not allowed*      →     Bob    *Emails are not allowed*
Alice  *Emails are not allowed* ← dup    Carol  *Emails are not allowed*
Carol  *Emails are not allowed*

Step 5 — Fix Data Types

Pandas sometimes reads numeric columns as strings, or dates as plain text. Operations on wrong types either crash or silently produce wrong results.

Change column type directly:

df["Zip_Code"] = df["Zip_Code"].astype(str)     # treat as text, not a number
df["Age"] = df["Age"].astype(int)               # convert to integer
df["Price"] = df["Price"].astype(float)         # convert to decimal

Handle messy numbers that contain symbols or text:

df["Price"] = pd.to_numeric(df["Price"], errors="coerce")

errors="coerce" converts anything that can't be parsed into NaN instead of crashing. This is the safe way to clean columns like "$1,200" or "N/A" that can't be directly cast with astype().

Before:           After pd.to_numeric(..., errors="coerce"):
"1200"    →       1200.0
"$850"    →       NaN     ← couldn't parse
"700.5"   →       700.5
"N/A"     →       NaN     ← couldn't parse

Convert to datetime:

df["Date"] = pd.to_datetime(df["Date"])
df["Date"] = pd.to_datetime(df["Date"], format="%d/%m/%Y")  # specify format if needed

Once a column is proper datetime, you unlock sorting by date, filtering by year/month, calculating time differences, and much more.

Step 6 — Clean Strings

String columns are almost always the messiest part of a real dataset. Pandas provides a full .str accessor with everything you need.

Strip unwanted characters from edges:

df["Name"] = df["Name"].str.strip()           # remove spaces from both sides
df["Name"] = df["Name"].str.strip(" ./-")     # remove spaces, dots, slashes, dashes
df["Name"] = df["Name"].str.lstrip()          # left side only
df["Name"] = df["Name"].str.rstrip()          # right side only

Replace characters anywhere in the string:

df["Phone"] = df["Phone"].str.replace(r"[a-zA-Z]", "", regex=True)   # remove all letters
df["Phone"] = df["Phone"].str.replace(r"\D", "", regex=True)          # keep digits only
df["Name"]  = df["Name"].str.replace(r"[./]", "", regex=True)         # remove dots and slashes

Standardize text case:

df["Name"] = df["Name"].str.lower()     # all lowercase → "alice"
df["Name"] = df["Name"].str.upper()     # all uppercase → "ALICE"
df["Name"] = df["Name"].str.title()     # title case    → "Alice"

Check text content:

df[df["Email"].str.contains("@")]           # filter valid-looking emails
df[df["Phone"].str.len() != 12]             # find rows where phone length is wrong
df[df["Name"].str.startswith("A")]          # names starting with A
df[df["Name"].str.endswith("e")]            # names ending with e

Extract a pattern using regex:

df["Zip"] = df["Address"].str.extract(r"(\d{5})")        # pull 5-digit zip code
df["Area_Code"] = df["Phone"].str.extract(r"^(\d{3})")   # pull first 3 digits

Split one column into multiple columns:

df[["Street", "State", "Zip"]] = (
    df["Address"]
    .str.split(",", n=2, expand=True)
)

n=2 limits the split to 2 times, producing exactly 3 parts regardless of how many commas the string contains. expand=True spreads those parts into separate columns.

Before:                              After:
"123 Main St, Texas, 75001"   →   Street="123 Main St"  State="Texas"  Zip="75001"

Step 7 — Replace and Standardize Values

Raw data often has the same value written multiple ways: "Y", "Yes", "yes", "y" — all meaning the same thing. Standardize them.

Replace whole cell values using a dictionary:

df["Status"] = df["Status"].replace({"Y": "Yes", "N": "No", "y": "Yes"})

# Apply to multiple columns at once
df[["Paying_Customer", "Do_Not_Contact"]] = (
    df[["Paying_Customer", "Do_Not_Contact"]]
    .replace({"Y": "Yes", "N": "No", "y": "Yes", "n": "No"})
)

Overwrite invalid values using loc[]:

df.loc[df["Age"] < 0,   "Age"] = None     # negative age → invalid
df.loc[df["Age"] > 120, "Age"] = None     # age over 120 → invalid
df.loc[df["Price"] < 0, "Price"] = None   # negative price → invalid

Keep valid values, replace everything else with NaN:

df["Salary"] = df["Salary"].where(df["Salary"] > 0)
# Rows where Salary <= 0 become NaN; the rest are untouched

Now Combine — Merging & Joining DataFrames

Once each source is clean, the next challenge is combining them. In the real world, data almost never lives in a single file. You might have orders in one table, customers in another, and products in a third. Merging is how you bring those pieces together.

Pandas provides three main tools for this:

Tool	What it does
`merge()`	Joins two DataFrames by matching on a shared column
`concat()`	Stacks DataFrames vertically or side by side
`join()`	Joins on the index instead of a column

We'll use these two DataFrames for all the examples:

customers = pd.DataFrame({
    "ID":        [1, 2, 3, 4],
    "Name":      ["Alice", "Bob", "Carol", "Dave"],
    "Country":   ["USA", "UK", "India", "USA"],
})

orders = pd.DataFrame({
    "ID":        [1, 2, 3, 5],
    "Product":   ["Laptop", "Phone", "Tablet", "Monitor"],
    "Price":     [1200, 800, 600, 400],
})

Notice that ID=4 (Dave) exists in customers but not in orders, and ID=5 exists in orders but not in customers. This is exactly the kind of mismatch you encounter in practice — and why the how= parameter matters so much.

1️⃣ `merge()` — The Main Tool

merge() works like a SQL JOIN. It aligns two DataFrames by finding rows where a shared column has the same value.

Basic syntax:

customers.merge(orders, how="inner", on="ID")

The `how=` Parameter — All Four Types

# INNER → only rows that match in both DataFrames (default)
customers.merge(orders, how="inner", on="ID")
# Result: IDs 1, 2, 3 only — Dave (4) and Monitor (5) are excluded

# LEFT → all rows from the left + matched rows from the right
customers.merge(orders, how="left", on="ID")
# Result: all 4 customers — Dave gets NaN for Product and Price

# RIGHT → all rows from the right + matched rows from the left
customers.merge(orders, how="right", on="ID")
# Result: all 4 orders — Monitor (ID=5) gets NaN for Name and Country

# OUTER → all rows from both, NaN where no match
customers.merge(orders, how="outer", on="ID")
# Result: all 5 unique IDs — gaps filled with NaN on both sides

`how=`	Keeps from Left	Keeps from Right	NaN Possible
`inner`	Matched only	Matched only	No
`left`	All rows	Matched only	Right side
`right`	Matched only	All rows	Left side
`outer`	All rows	All rows	Both sides

CROSS merge — every row with every row:

customers.merge(orders, how="cross")
# Produces 4 × 4 = 16 rows — every customer paired with every order
# No on= parameter — it will raise an error if you include it

The `on=` Parameter — What to Match On

# Match on a single column
customers.merge(orders, how="inner", on="ID")

# Match on multiple columns — both must match simultaneously
customers.merge(orders, how="inner", on=["ID", "Country"])

# Auto-detect — Pandas finds all shared column names automatically
customers.merge(orders)

Be careful with auto-detect. If both DataFrames happen to share a column that you don't intend to join on (like a generic "Name" column), Pandas will include it in the match condition and produce unexpected results. Explicit is safer.

Handling Shared Column Names — `_x` and `_y`

If both DataFrames have a column with the same name, but you only join on ID, Pandas keeps both copies and renames them automatically:

# Both DataFrames have a "Name" column, but you only merge on "ID"
customers.merge(orders, how="inner", on="ID")
# Result: Name_x (from customers), Name_y (from orders)

Three ways to fix this:

# Option 1 — merge on all shared columns so no duplicates appear
customers.merge(orders, how="inner", on=["ID", "Name"])

# Option 2 — use custom suffixes instead of _x and _y
customers.merge(orders, how="inner", on="ID", suffixes=("_customer", "_order"))

# Option 3 — drop the shared column from one DataFrame before merging
orders_clean = orders.drop(columns=["Name"])
customers.merge(orders_clean, how="inner", on="ID")

Other Useful Parameters

# left_on / right_on — when the key column has different names in each DataFrame
customers.merge(orders, how="inner", left_on="CustomerID", right_on="OrderID")

# indicator — adds a column showing where each row came from
customers.merge(orders, how="outer", on="ID", indicator=True)
# _merge column values: "left_only", "right_only", "both"

The indicator=True parameter is especially useful for auditing: after a merge, you can immediately see which rows matched, which came only from the left, and which came only from the right.

2️⃣ `concat()` — Stacking DataFrames

While merge() combines DataFrames horizontally by matching values, concat() simply stacks them — either rows on top of rows, or columns side by side.

pd.concat([df1, df2])        # stack vertically (default)
pd.concat([df1, df2], axis=1)  # stack horizontally

`axis=` Parameter

# axis=0 → stack rows vertically (append one DataFrame below another)
monthly_data = pd.concat([january, february, march], axis=0)

# axis=1 → stack columns side by side
combined = pd.concat([names_df, scores_df], axis=1)

`join=` Parameter

# outer (default) → keep all columns, NaN where a column doesn't exist in one source
pd.concat([df1, df2], join="outer")

# inner → keep only columns that exist in all DataFrames
pd.concat([df1, df2], join="inner")

`ignore_index=` Parameter

# Without ignore_index — original indices are preserved, which can repeat
pd.concat([df1, df2])
# Index: 0, 1, 2, 0, 1, 2  ← 0, 1, 2 repeats

# With ignore_index=True — fresh sequential index
pd.concat([df1, df2], ignore_index=True)
# Index: 0, 1, 2, 3, 4, 5  ← clean

Always use ignore_index=True after stacking unless you specifically need the original indices.

`keys=` Parameter

# Label each source so you can identify where each row came from
combined = pd.concat([df1, df2], keys=["source_a", "source_b"])

This creates a MultiIndex where the first level is your label and the second is the original row index — useful when you need to trace data back to its source after combining.

`verify_integrity=` Parameter

# Raises an error if any duplicate index values are found after concatenation
pd.concat([df1, df2], verify_integrity=True)

`merge()` vs `concat()` — When to Use Which

	`merge()`	`concat()`
Direction	Horizontal (by matching)	Vertical or horizontal (by stacking)
Needs a shared key column	Yes	No
Best for	Joining related tables	Combining same-structure data
Equivalent to	SQL JOIN	SQL UNION

A simple rule: if you're combining columns from different tables, use merge(). If you're stacking more rows of the same structure, use concat().

3️⃣ `join()` — Index-Based Joining

join() works like merge(), but it joins on the index instead of a regular column. You need to set the index first.

customers_indexed = customers.set_index("ID")
orders_indexed    = orders.set_index("ID")

customers_indexed.join(orders_indexed, how="inner")

The how= options are the same as merge(): inner, left, right, outer.

Why `merge()` Is Almost Always Better

	`merge()`	`join()`
Join on any column	✅ Yes	❌ No — index only
Join on index	✅ Yes	✅ Yes
Requires `set_index()` first	❌ No	✅ Yes
Simpler syntax	✅ Yes	⚠️ Extra steps

> Use `join()` only when your data is already indexed — for example, after a `groupby()` that produced a meaningful index. Otherwise, `merge()` is cleaner and more flexible.

This is Part 3 of the Pandas for Data Science series. Next up: Part 4 — Data Visualization with Matplotlib & Seaborn

To access the full article, please visit the link below or directly access the repository via the links below.

Refrences

GitHub Repo : https://github.com/Hu8MA/Mastering-Pandas-Reference
Website : https://pandas.pydata.org/
GitHub Library : https://github.com/pandas-dev/pandas
Course : https://youtu.be/Mdq1WWSdUtw
Full article : https://hussein16mahdi.medium.com/mastering-pandas-part-3-data-cleaning-merging-joining-817a070147ca

Why Most Data Projects Fail Before the First Model Is Built

Thu, 05 Mar 2026 18:45:16 +0000

Many organizations invest in AI, analytics, and dashboards — yet most data projects fail before the first model is even built.

When people think about data projects, they often imagine machine learning models, predictive algorithms, and complex pipelines.

But in reality, most data initiatives fail long before any model is trained.

Not because the algorithms are weak.

But because the foundation is broken.

The Data Illusion

Organizations today generate enormous amounts of data.

They store logs, transactions, operational records, and performance metrics.

On paper, everything looks ready for analytics.

But when teams actually start working with the data, they quickly encounter problems:

Missing values
Inconsistent formats
Conflicting sources
Undefined metrics
Poor documentation

Suddenly, the project shifts from analysis to data archaeology.

Data Science Starts with Data Reliability

Before any meaningful analysis can begin, teams must answer fundamental questions:

What is the source of truth?
Who owns the dataset?
How frequently is it updated?
What transformations are applied?
Are definitions consistent across departments?

Without clear answers, even the most advanced models produce misleading insights.

The Hidden Cost of Poor Data Foundations

Many organizations invest heavily in analytics tools, dashboards, and AI platforms.

But without strong data foundations, these investments create an illusion of intelligence.

Dashboards become visually impressive but operationally misleading.

Models generate predictions, but the inputs themselves are unstable.

This leads to one of the most dangerous outcomes in data work:

False confidence.

Decisions start relying on numbers that appear precise but are fundamentally unreliable.

Data Engineering Is the Real Backbone

In practice, the majority of effort in data projects is not modeling.

It is:

Data cleaning
Validation
Schema alignment
Pipeline reliability
Monitoring data quality

That is why experienced teams often say:

“80% of data science is data preparation.”

And the better the data infrastructure, the faster meaningful insights appear.

A Simple Rule for Data Teams

Before building any model, ask three questions:

Is the data trustworthy?
Is the definition of the metric consistent?
Can the pipeline reproduce the same dataset tomorrow?

If the answer to any of these is unclear, the problem is not analytical.

It is architectural.

Final Thought

Good data teams do not start with models.

They start with reliability.

Because in data systems, accuracy is not created by algorithms.

It is created by architecture.

If you're interested in systems thinking, data architecture, and enterprise optimization, feel free to connect.

LinkedIn: https://www.linkedin.com/in/fadydesokysaeedabdelaziz

GitHub: https://github.com/fadydesoky

Beyond the Basics: 5 Game-Changing Secrets of SQL Joins and Window Functions

Mon, 02 Mar 2026 07:40:25 +0000

1. Introduction: The Data Relationships Hook

Think of a database as a digital filing cabinet. In this architecture, information is organized into drawers—or schemas—such as HR, Finance, or Sales. Within these drawers sit the documents, which represent our individual tables.

While this structure ensures clean organization, real-world business intelligence rarely lives in a single drawer. To answer strategic questions, you must bridge the gaps between disparate tables without creating a "data mess". Whether you are joining sales records with inventory levels or HR data with finance budgets, mastering these relationships is what separates a basic query-writer from a Senior Architect. This guide distills complex SQL operations into high-impact takeaways, focusing on maintaining relational integrity while scaling your analysis.

2. The Horizontal vs. Vertical Divide: Joins vs. Unions

The most fundamental distinction in data architecture is how you choose to expand your result set.

Joins combine tables horizontally. They add columns based on a shared key (like a Primary or Foreign Key), making the result set "wider".
Unions combine tables vertically. They stack rows from one dataset on top of another, making the result set "longer".

For a UNION to be architecturally sound, the query must meet three strict structural requirements: the columns must be in the same order, have the same count, and—most importantly—possess matching data types. As an architect, you must remember that the database engine is indifferent to your column aliases but ruthless regarding data types.

"The database doesn't care about naming but cares about data types in Unions... If a VARCHAR is matched with an INT, the database will throw a mismatch error." — Data with Baraa

The Mental Model: Distinct vs. All From a strategy perspective, think of UNION as a DISTINCT operation for stacked rows; it automatically removes duplicates. If you need the "data as is" without the overhead of deduplication, UNION ALL is your preferred tool.

Sample Query: Vertical Stacking

``-- Combining customer and employee names into a single master list
SELECT first_name, last_name FROM customers
UNION
SELECT first_name, last_name FROM employees
LIMIT 100;

``

3. The "Workhorse" Joins: Why Left is Often Better than Right

In relational logic, the Inner Join is the default state. When you use the JOIN keyword, SQL is explicitly looking for matching values in the ON statement. If a record in the first table—for example, a customer who hasn't placed an order yet—has no match in the second table, that record "disappears" from the final result set.

For high-level reporting where data integrity is paramount, the Left Join is the industry workhorse. It preserves every record from the left (first) table and fills missing data from the right table with NULL values. This ensures you don't accidentally drop crucial business entities (like customers or products) simply because they lack transaction history.

Architectural Insight: The Right Join is functionally redundant. Any Right Join can be reframed as a Left Join by simply reordering the tables. In production environments, Left Joins are the standard because they align with a left-to-right reading flow, making queries significantly easier to audit and maintain.

Sample Query: Production-Ready Left Join
``-- Retrieving all customers and any associated event data
SELECT
c.customer_id,
c.first_name,
e.event_name
FROM customers AS c
LEFT JOIN events AS e
ON c.customer_id = e.customer_id
WHERE c.customer_id IS NOT NULL
LIMIT 500;

``

4. The Cartesian Chaos: Understanding the Cross Join

A Cross Join represents the "Cartesian Product" of two tables. Unlike other joins, it ignores matching values entirely, attaching every row of the second table to every row of the first. If you join a 1,000-row table with a 500-row table, you will generate a massive result set of 500,000 rows.

In standard relational reporting, Cross Joins are often avoided because they lack the primary/foreign key bond that defines logical relationships. Joining a car table and an item table by a shared color attribute "wouldn't make any sense" for data integrity. However, from a strategic standpoint, Cross Joins are powerful for generating permutations—such as creating a master grid of every possible product-color combination for an inventory audit.

5. Window Functions: Grouping Without the "Collapse"

The true "game-changer" for intermediate learners is the OVER() clause. Traditional GROUP BY operations "collapse" your data, rolling multiple rows into a single aggregate level. While useful for summaries, this loses the individual row detail.

Window Functions allow you to perform calculations across a result set while keeping every unique row intact. This allows you to view an individual's salary right next to the department average, or calculate a Rolling Total.

Strategic Use Case: The Rolling Total In finance and healthcare, tracking cumulative sums is vital. By adding an ORDER BY clause inside the OVER() window, you transform a static sum into a running balance.

Sample Query: Rolling Totals and Averages
-- Calculating a cumulative salary sum and a static department average SELECT first_name, gender, salary, SUM(salary) OVER(PARTITION BY gender ORDER BY employee_id) AS rolling_total, AVG(salary) OVER(PARTITION BY gender) AS avg_dept_salary FROM employees;

6. The Ranking Trio: Row Number, Rank, and Dense Rank

Sequencing data is a core requirement for leaderboards and performance tracking. SQL offers three nuances for handling ties within a window:

- Row Number: Assigns a unique, sequential integer to every row (1, 2, 3, 4). Even in the event of a tie, the numbers will not repeat.
**

- Rank: Assigns the same number to ties but skips the next position based on the count of duplicates. If two rows tie for 1st, the next is 3rd (1, 1, 3). This is "positional" ranking.
**-

- Dense Rank: Assigns the same number to ties but keeps the next number sequential (1, 1, 2).

This is "numerical" ranking.
Architectural Preference: DENSE_RANK is typically preferred for professional reporting. It ensures there are no "gaps" in your leaderboard, maintaining a clean hierarchy regardless of how many entities share the same value.7.

Conclusion: Levelling Up Your Query Game

Moving from a beginner to an advanced analyst is less about learning syntax and more about understanding the logic of relationships. Every time you approach a new dataset, you must make a strategic choice: do you need to collapse your data into an aggregate summary with a Join or Group By, or do you need to partition it with a Window Function to maintain row-level detail?

The absolute grounding rule of SQL is that it is a language of relationships. By mastering the nuances of how data stacks vertically via Unions or expands horizontally via Joins, you ensure the relational integrity of your architecture.

Hydrolix buys Quesma tech to keep Kibana dashboards running on cost-efficient data lakes.

Tue, 23 Sep 2025 13:11:07 +0000

Hydrolix Secures Kibana Support by Acquiring Quesma Gateway Technology

Acquisition ensures seamless dashboard integration as data lake platform expands ecosystem reach.

Hydrolix has made a strategic move to protect its customers' existing workflows by acquiring intellectual property from Quesma, specifically the gateway proxy technology that enables Kibana integration. This acquisition comes after Quesma announced it would discontinue support for the product, potentially leaving Hydrolix customers without their familiar dashboard interface.

Why This Matters for Developers

The acquisition addresses a common challenge in the data analytics space: maintaining familiar tools while upgrading underlying infrastructure. Many organizations have invested heavily in Kibana dashboards, queries, and workflows. When they migrate to more cost-effective platforms like Hydrolix, they want to preserve these investments rather than rebuilding everything from scratch.

"Many of our customers rely on Kibana as a familiar and powerful interface for monitoring and alerting," said Marty Kagan, CEO of Hydrolix. "By bringing Quesma's gateway proxy IP in-house, we can guarantee that customers will continue to enjoy seamless interoperability between Kibana dashboards and Hydrolix data."

The Technical Solution

Quesma's gateway proxy acts as an intelligent intermediary between Kibana and Hydrolix. When users run queries through Kibana, the gateway translates these requests into the appropriate format for Hydrolix's streaming data lake platform. This translation happens transparently, so users can continue working with their existing dashboards while benefiting from Hydrolix's cost advantages and performance improvements.

The technology has been integrated into Hydrolix since 2024, giving the company nearly a year of operational experience with the solution. This integration allows organizations to redirect queries from expensive Elasticsearch clusters to Hydrolix's more efficient storage and query engine without changing their front-end tools.

Beyond Kibana Integration

This acquisition reflects Hydrolix's broader ecosystem strategy. The company doesn't just support Kibana—it also integrates with Splunk, Databricks, Microsoft Fabric, and various machine learning exploration tools. This approach recognizes that different teams within organizations often prefer different tools, and forcing standardization can create adoption barriers.

The move also demonstrates how Hydrolix is positioning itself in the competitive data lake market. Rather than requiring customers to abandon their existing toolchains, the company is building bridges that make migration easier and less risky.

Market Context and Timing

The acquisition comes at a time when organizations are increasingly focused on data infrastructure costs. Traditional log analytics platforms like Elasticsearch can become expensive at scale, particularly when organizations need to retain data for extended periods or handle high-volume ingestion.

Hydrolix has built its platform specifically to address these cost concerns. The company's streaming data lake can handle up to 20 million events per second while keeping data "hot" and queryable for 15 months or more—something that would be prohibitively expensive with traditional solutions.

Customer Impact

For existing Hydrolix customers using the Kibana integration, this acquisition means business as usual. They won't need to worry about losing access to their dashboards or having to rewrite queries. The transition should be seamless, with Hydrolix taking over support and development responsibilities.

New customers evaluating Hydrolix can also feel confident that their Kibana investments are protected. This removes a significant barrier to adoption, especially for organizations with large teams already trained on Kibana.

What's Next for Quesma

While Quesma is exiting the gateway proxy business, the company isn't disappearing entirely. According to Jacek Migdał, founder and CEO of Quesma, the transaction allows his company to focus on its next chapter of product development. The partnership between the companies will continue, though in a different capacity.

Industry Implications

This acquisition highlights a broader trend in the data infrastructure market: the importance of ecosystem integration. Companies that build walls around their platforms often struggle to gain adoption, while those that embrace interoperability can capture market share more effectively.

It also shows how smaller, specialized companies like Quesma can have outsized impact by solving specific integration challenges. Even though Quesma is stepping back from this particular product, their technology will continue to benefit users through Hydrolix.

The Bottom Line

For developers and data engineers working with log analytics, this acquisition removes uncertainty about Kibana support on the Hydrolix platform. It also demonstrates Hydrolix's commitment to protecting customer investments in existing tools and workflows.

As organizations continue to grapple with rising data infrastructure costs, solutions that offer both economic advantages and tool compatibility become increasingly valuable. By securing the Quesma technology, Hydrolix has strengthened its position in this competitive market while ensuring its customers can continue using the tools they know and trust.