Coder Legion - Recent posts tagged statistics

Statistical Arbitrage & Cointegration in LATAM Equities (BVC) with Python

Tue, 13 Oct 2026 13:33:11 +0000

Two stocks can rise together for years and still be untradeable as a pair. Correlation measures co-movement; pairs trading needs cointegration — a stationary, mean-reverting spread. This post builds the standard test pipeline in Python and shows where most LATAM pairs fail.

Educational content: methodology, no signals, no return claims.

1. Correlation Lies, Cointegration Trades

Correlated prices drift apart permanently (both trend up, spread widens forever). A cointegrated pair has a spread that reverts to a stable mean, which is what a long/short entry bets on. The Engle–Granger (1987) two-step procedure tests exactly this:

Regress Y on X by OLS → hedge ratio β.
Run an Augmented Dickey–Fuller (ADF) test on the residual spread. Only a stationary spread (low ADF p-value) is tradeable.

import numpy as np
import statsmodels.api as sm
from statsmodels.tsa.stattools import coint, adfuller

# y, x: aligned log-price series of the two candidate assets
score, pvalue, _ = coint(y, x)          # Engle-Granger cointegration test
print(f"EG p-value: {pvalue:.4f}")      # tradeable only if clearly < 0.05

res = sm.OLS(y, sm.add_constant(x)).fit()
beta = res.params[1]                    # hedge ratio: short beta units of X per 1 of Y
spread = y - beta * x

adf_stat, adf_p, *_ = adfuller(spread)[:2]

A p-value above conventional significance is not a "weak pair" — it is a rejection. Discard it. Most random pairs fail; that is the expected outcome, not bad luck. (Our falsification-first protocol for backtests is documented in Why Perfect Backtests Fail.)

2. Half-Life: How Fast Does It Revert?

A stationary spread that reverts once a year is untradeable. Estimate the half-life with a simple AR(1) regression of spread changes on lagged spread:

d_spread = np.diff(spread)
lag = spread[:-1]
beta_hl = np.polyfit(lag, d_spread, 1)[0]
halflife = -np.log(2) / beta_hl
print(f"Half-life: {halflife:.1f} bars")

Practical rule: half-life should sit comfortably between transaction-cost frequency (too fast = costs eat everything) and your holding horizon (too slow = capital sits idle). Both extremes fail — for opposite reasons.

3. Entries on the Z-Score, Exits on Discipline

z = (spread - spread.mean()) / spread.std()
# Enter when |z| > 2.0 (long the laggard, short the leader),
# exit when z crosses back through ~0.25 — never "wait for full reversion".

Hard stops still apply: cointegration breaks (mergers, regime changes, delistings). A pair that held for three years can decouple in a week. Size every pair as if the spread can go to 3–4σ and stay there.

4. Why LATAM (BVC) Pairs Are Harder

Applying this to Colombian equities (BVC) adds three frictions US textbook examples ignore:

Liquidity: wider spreads and thin books mean your backtest fill assumptions are fiction unless you model slippage explicitly. Test with volume-capped fills.
Costs: commissions, exchange fees and taxes compound on a strategy that trades often. Recompute the half-life economics after costs — many "profitable" spreads die here.
Short access: pairs trading needs both legs. If you cannot reliably short one side, the strategy is long-only spread timing, a different (usually worse) trade. Verify borrow availability before any capital.

No specific pairs are recommended here — universes change, and any named pair would be stale on arrival. Run the pipeline above on current data instead.

5. Validation Checklist

In-sample cointegration + out-of-sample spread stability (Walk-Forward, not one fit).
Half-life compatible with costs and horizon.
Break scenario sized and stop-defined before the first trade.
60+ days paper trading the exact execution path.

Study material under Colombia SFC Decreto 2555 de 2010 — financial-engineering education, not investment advice. No signals, no managed accounts.

By Mahdi Goodarzi (g.dev/mahdigoodarzi), Gueta Quant — research notes at guetaquant.com/blog. Related: Pine Script v6 guide.

The Parts Of Scientific Work That Are Really Software Problems

Sun, 30 Aug 2026 21:45:55 +0000

Most of us here write software for products, but a surprising amount of the same work happens in labs and research groups, written by people who were never taught to write code. If you have ever inherited a research script and wondered why it does what it does, this one is for you.

Where Research Code Actually Breaks

Research code fails differently from product code. It rarely crashes in a way anyone notices. It quietly produces a number, and the number is wrong because a filter was applied twice, or a merge silently dropped half the rows, or a random seed moved between runs.

The habits that catch this are the ones we already use: keep the raw data immutable, make every transform a named step you can rerun from scratch, and check row counts before and after every join. A pipeline that prints how many records survived each stage catches more errors than any amount of careful reading.

Statistics Is A Modeling Problem, Not A Library Problem

Calling a t-test is one line in any language. Knowing that a t-test is the wrong tool for your data is the actual work, and no library will tell you.

The questions that matter come before the function call: what is the unit of observation, are the observations independent, how many comparisons are you running, and what effect size would actually mean something. Get those wrong and a perfectly correct implementation returns a perfectly meaningless p value.

Reading A Paper Like A Code Review

A paper is a claim plus the evidence for it, and it can be reviewed the same way you review a pull request. Read the methods before the results, because the methods tell you what the results are allowed to mean.

Then check the sample size, look for the comparison that is missing, and see whether the conclusion in the abstract is the same one the data supports. This is a skill, not a talent, and it improves fast once you start doing it deliberately. We keep the practical side of all of this collected in one place, the tools and methods behind real science, covering Python for scientific work, statistics, scientific computing, algorithms, experiment design, research methods and how to read a paper properly.

The Takeaway

Scientific work and software work fail in the same places: unclear inputs, unstated assumptions, and results nobody can reproduce. If you already have the engineering instincts, most of what separates you from doing real analysis is vocabulary, and that part is learnable in an afternoon.

Skaylar - Analyze datasets

Mon, 24 Aug 2026 05:49:07 +0000

Skaylar

Find what doesn't add up

Skaylar is a simple, web-based tool for forensic data analysis using Benford's Law to analyze and visualize first-digit distributions in a dataset.
Paste the data, click Run Test, and instantly see if there are anomalies.

Benford's Law showing population of countries (1970, 1980, 1990, 2000, 2010, 2015, 2020, 2022)

What is Benford's Law?

In real financial data, the digit "1" appears as the first digit ~30% of the time, "2" ~18%, etc.
If your data deviates too much, it can signal fraud, errors, or manipulation.
Skaylar checks this for you in seconds.

How to Use

1. Visit the website or download the repo (offline use):

git clone https://github.com/Temi-Tade/Skaylar.git
or click Code > Download ZIP and extract files on your computer.

2. Open

Double-click index.html
Works in Chrome, Edge, Firefox, and on mobile.

3. Analyze

Input/Paste your numbers in the textbox or Click Select Dataset to select/upload a .json dataset. Numbers in the textbox should be delimited by commas, spaces or new lines
Click Run Test
View the results table and graph

What You Get

Results Table: Expected % vs Actual % for digits 1-9 and deviations
Graph: Visual comparison to spot deviations fast
Analytics: Max deviation and Chi square value.

Height distribution (Does not follow Benford's Law)

Fibonacci series (Follows Benford's Law)

What you can do

Customize Graph: Change colour of expected and actual curves on the graph. Also, you can scale the graph to different zoom levels.
Save Datasets: Save a Dataset that you manually pasted/inputted in the text box as a .json file that you can share with other people.

Save manually-inputted data as a .json Dataset file.

Customize curve colours on the graph and adjust zoom levels.

Contribute to Skaylar

Skaylar is open source and is open to contribution from developers, data scientists/analysts, forensic accountants, auditors and anyone interested in making it better.

Contribution Guidelines

Fork the repo and make your improvements/additions/fixes.
Check out the issues tab on the repo if you are confused in where to start from.
Create a Pull Request from your forked repo.
Your PR will be reviewed and merged accordingly.

You can also contribute by adding datasets for users to test Skaylar. The datasets are .json files and should strictly follow the format below:

{
    "name": "Name of the dataset", // recommended: separate words with underscores 
    "description": "Description of the dataset",
    "source": "Verified and official source", // You can put your name if you scrapped the data yourself 
    "data": [120, 300] // array of the numbers in the dataset
}

Juniors Don't Love Rust — You Just Can't Separate Age From Year From Cohort

Wed, 22 Jul 2026 17:09:35 +0000

Originally published at https://vibeagentmaking.com/blog/juniors-don-t-love-rust-you-just-can-t-separate-age-from-yea/

An APC teardown of the Stack Overflow Developer Survey. The numbers are real; the story about who was always optional, and one line of century-old arithmetic shows exactly why a survey like this can never pin it down.

In June 2022, Stack Overflow published its annual developer survey and handed the internet a headline it had printed six times before: Rust was the "most loved" programming language, for the seventh year in a row, with 87% of its users saying they wanted to keep using it. When the survey retired "loved" for the sterner "admired" in 2023, Rust just kept winning: 83% admired in 2024 ("for the second year in a row," as Stack Overflow's own write-up put it), 72% and still #1 in 2025.

You have read the think-piece this figure generates. You may have written it. It goes: young developers love Rust. Juniors chase memory safety the way their elders chased garbage collection; the kids grew up on the borrow checker; Gen Z devs are built different. It comes with a chart, the chart goes up and to the right, and the conclusion feels like it is sitting right there in the data.

Here is the uncomfortable thing I want to show you, with arithmetic: that conclusion is not in the data. Not because the sample is too small, or self-selected (though it is), or because correlation is not causation. Something sharper. The dataset, any dataset shaped like this one, is mathematically incapable of telling you whether young people drive a trend. The proof is one line long, it is about a century old in demography, and once you see it you will spot undeclared versions of it in half the trend pieces you read.

One line of arithmetic, three incompatible stories

The Stack Overflow survey is what statisticians call a repeated cross-section: every year, a fresh pile of respondents, each row carrying an age bucket and a survey year. From those two fields you can derive a third: roughly, birth year, or, closer to what tech punditry actually means, the year this person entered the field. Demographers call these three clocks age, period, and cohort, and every generational claim is a claim about which clock is doing the work:

Age effect: juniors try new things; people cool on novelty as they age. ("Juniors love Rust.")
Period effect: a moment pulled everyone in at once, regardless of age. ("2023 changed everything.")
Cohort effect: the class that entered around 2021 imprinted on the tool and will carry it forever. ("This generation is different.")

The trap is that the three clocks are not three measurements. Cohort = period − age. Exactly. Know any two and you know the third, which means a model trying to estimate all three is asking the data to split one number three ways.

This is the age-period-cohort identification problem, and it is not a folk worry; it is a theorem about the geometry of the question. Put a variable for age, a variable for survey year, and a variable for cohort into one regression and the design matrix is rank-deficient by exactly one: its columns are linearly dependent, and infinitely many different coefficient vectors reproduce the observed data identically, not approximately but identically. More respondents do not help. A bigger survey sharpens every one of the competing answers equally and leaves them exactly as tied. The ambiguity lives in the question, not the noise.

Running it for real

Claims like that deserve a demonstration, so let us run one on the survey's cleanest curve. Stack Overflow started asking about AI tools in 2023, and the topline is famous: 70% of respondents using or planning to use AI tools in 2023, 76% in 2024, 84% in 2025.

Fit the linear age-period-cohort model and ask the standard software for the answer, three different runs, three different constraint choices. Read the rows as headlines. One says the trend is all age and cohort: the generational story, +7 points a step each. One says it is all the moment: ChatGPT year, everyone at once, no generation involved. The diplomatic compromise a fancy estimator produces splits it 2.33 / 4.67 / 2.33 and looks, to the untrained eye, like a finding.

The punchline: across every cell of the grid, the three fits differ by zero. Not "within confidence intervals." Zero, to machine precision. They are the same surface wearing three stories, because the reallocation between them (add δ to the age slope, subtract δ from the period slope, add δ to the cohort slope) cancels exactly, for any δ, forever. That is the rank deficiency made flesh: four columns, rank three.

In 2004, Yang, Fu and Land published the Intrinsic Estimator in Sociological Methodology, a principled-sounding fix that uses a pseudoinverse to pick, out of the infinite family of equally-fitting answers, the unique one orthogonal to the null space of the design matrix. It has lovely statistical properties. It is also a choice: one member of the tied family, selected by a criterion that has nothing to do with how developers actually adopt tools. In 2013, Andrew Bell and Kelvyn Jones published a paper in Social Science & Medicine whose title states the thesis with admirable bluntness: "The impossibility of separating age, period and cohort effects." No estimator solves this, because the problem is inherent to the real-world process, not the statistics. The fanciest estimator is not the one that found the answer. It is the one that best disguised the assumption.

What the data can say

Within one survey year, age comparisons are fine. If 2025's 18-to-24-year-olds admire Rust more than 2025's 45-year-olds, that is an observable fact, a cross-sectional gradient, no identification problem at all. What you cannot do is attribute the drift across years to any one clock.

Curvature survives. Straight lines don't. The unidentifiable piece is exactly the shared linear trend, the smooth drift. Kinks, spikes, accelerations, the second-difference structure, are identified, because reallocating a straight line among three clocks cannot manufacture or absorb a corner. In the AI curve, 70 to 76 to 84 is +6 then +8. The acceleration, +2 points, came out identical in every fit, under every constraint. The near-vertical jump the year ChatGPT broke is real, attributable, and visibly a period shock: everyone, every age bucket, at once.

Sit with the irony of that. The one thing this dataset can actually pin down, "a moment moved everybody," is the least generational story available. The moment a narrative becomes interesting ("this cohort is different") is precisely the moment it slides into the unidentifiable linear subspace. The seductiveness and the unprovability are the same mathematical property.

And there is a fourth clock nobody models. Stack Overflow's 2024 write-up notes that respondents aged 35 and up were 31% of the sample in 2022, 35% in 2023, and 39% in 2024. This is a self-selected convenience sample that re-draws itself every year. A composition shift like that can manufacture, mask, or reverse any apparent trend on any of the three clocks, before we even reach the theorem.

The grown-up in the room already blinked

In May 2023, Pew Research Center published a methodological statement titled "How Pew Research Center Will Report on Generations Moving Forward." In it, they concede the core point: when younger adults answer differently than older ones, "it may be driven by their demographic traits rather than the fact that they belong to a particular generation." Their new house rules: generational claims require decades of comparable historical data, and where the label is not earned, they will group by decade or event instead.

A century of theory sits behind that retreat. Karl Mannheim's 1928 essay "The Problem of Generations" was already more careful than the genre it spawned: sharing birth years creates only a potential for shared consciousness, and no generation is a homogeneous block. When a survey house of Pew's stature stops writing "Gen Z believes…" headlines from single-year cross-sections, that is a detector being honest about what it can detect.

What to actually do with this

The practical insight is not "never trust surveys." It is a one-question audit you can run on any trend claim, others' or your own, in about ten seconds:

"Which clock did you zero, and where did you say so?"

Every attribution of a repeated-cross-section trend to age, generation, or moment has zeroed at least one clock. There is no exception; the algebra does not permit one. From there, three habits:

Downgrade smooth stories, respect kinks. A gradual "each cohort more than the last" drift is exactly the unattributable shape. A sharp everyone-at-once jump carries real, identifiable period signal.
Check the composition clock first. A survey whose 35+ share climbs 31 to 35 to 39 in two years can produce "trends" without a single human changing their mind.
When you must attribute, declare the assumption and defend it from outside the data. "We attribute this to cohort because switching costs lock tool choices in the first two working years" is an argument: checkable, arguable, honest.

And if you are the one writing the analysis: say the null proudly. "This survey cannot tell whether juniors drive Rust adoption" is not a failure to find a result. It is the result. The numbers are real: 87%, seven years running, 83%, 72%, 70-76-84. What is optional, what was always optional, is the story about who.

Anyone who tells you otherwise has made an assumption. The good ones tell you which.

Sources

Stack Overflow, 2022 Developer Survey — Rust "most loved" for the seventh consecutive year, 87%.
Stack Overflow Blog (Jan 2025), "Developers want more, more, more: the 2024 results" — Rust 83% admired; respondents 35+ = 31% (2022) to 35% (2023) to 39% (2024); AI use 76%.
Stack Overflow, 2025 Developer Survey — Rust most admired at 72%; 84% using or planning to use AI tools.
Stack Overflow, 2023 Developer Survey — 70% of all respondents using or planning to use AI tools.
Bell, A., & Jones, K. (2013). "The impossibility of separating age, period and cohort effects." Social Science & Medicine, 93, 163–165.
Yang, Y., Fu, W. J., & Land, K. C. (2004). "A Methodological Comparison of Age-Period-Cohort Models." Sociological Methodology, 34, 75–110.
Pew Research Center (May 2023). "How Pew Research Center Will Report on Generations Moving Forward."
Mannheim, K. (1928/1952). "The Problem of Generations."

Originally published at https://vibeagentmaking.com/blog/juniors-don-t-love-rust-you-just-can-t-separate-age-from-yea/

Humble proof of Machine Learning & Deep Leaning: Across 10 families and 100 Models.

Fri, 13 Feb 2026 18:55:40 +0000

"Elite Engineering in less than 36 hours!"

Best for: Experienced developers who value raw performance and terminal-heavy builds.

Subject: Architecting the "King Model" Gauntlet: Stop letting AI get lazy.

I’ve just pushed Project A-Z to my repo. It’s an end-to-end autonomous ML pipeline, but with a twist that most "Auto-ML" tools miss: Transparency through destruction.

Early in the build, the AI engine was trying to cut corners—treating the stack like a simple script. I had to turn the tables. I shifted the entire architecture to a Terminal-First Tournament.

If you run this, your PowerShell becomes a graveyard. You’ll see the "Death Row" of rejected models—KNNs, Naive Bayes, SVMS—logged with their failure metrics before the GUI even initializes. We’re moving past the "black box" and back to Scientific Benchmarking.

The Spec:

Wrangling: MICE + Isolation Forest for outlier pruning.

Taxonomy: A gauntlet of 100+ model variants (Bagging, Boosting, Kernels).

Visualization: Streamlit-powered post-mortem and "What-If" causal simulator.

The GUI is just the cosmetics; the PowerShell is where the war is won. Check the proofs in the repo.

Stay lethal.

https://github.com/usman19zafar/Project-A-Z-Cognitive-Pipeline-End-to-End-Autonomous-ML-Engineering-by-Gemini/tree/main

Data Analysis - Basic Visualizations

Wed, 07 Jan 2026 20:55:59 +0000

Building on the concepts from our post on "Descriptive Statistics," we will now explore the most common charts for data visualization: "Descriptive Statistics", below, we will know the most basic ways to visualize the data or information to be analyzed. Specifically:

Pie chart
Line chart
Treemap & Subburst
Histograms & Bar charts
Density plot
Boxplots
Violin plots
Scatter plots
Pareto chart

Note: All the examples (except for the line graphs due to its features) are based on the train.csv dataset file, which is a subset of 891 representative individuals of the population of the Titanic used for the training of models of Machine Learning.

Pie charts

Pie charts are useful to represent the numerical proportions of the values of the data of a variable or feature with respect to its total. This type of graphs present a vast catalogue of variants and are widely used due to their simplicity of preparation and interpretation. However, these charts are often overused and misused. For more on this topic, see this excellent article on why pie charts are so controversial about the right use of pie charts and why these charts are so hated).

Example: Proportions of Titanic passengers by socioeconomic status

Line charts

Line charts are ideal for displaying how a variable changes over a continuous interval, such as time. Generally, they are used to represent trends or relationships (in a group with more variables) between the data during a continuous time interval.

Note: In order to visualize a use case in a continuous time interval, the following dataset from the Open data initiative of the Government of Spain has been used [the population data by Autonomous Communities from 1997 to 2017.](https://datos.gob.es/es/catalogo/ea0010587-poblacion-por-comunidades-y-ciudades-autonomas-y-tamano-de-los-municipios-anual-comunidades- autonomas-official-figures-of-population-of-the-Spanish-municipalities-revision-of-padron-municipal-identifier-api-2915) *

Example: Spanish population since 1997
Example: Spanish population by Autonomous Communities since 1997

Treemaps & Sunbursts

Treemaps and sunburst graphs are visualizations to represent hierarchical data and summarize the information in areas proportional to the corresponding values.

In the case of treemaps, the representation consists of rectangles with an area proportional to the value of the corresponding data with respect to the total.

Example: Proportion of passengers according to their socioeconomic status classified by gender (treemap)

In the case of sunburst, the representation consists groups of rings according to the level represented, which are divided into proportional sections corresponding to the value of the data they represent.

Example: Proportion of passengers according to their socioeconomic status classified by gender (sunburst)

On the example represented, you can quickly see information about the hierarchical data represented as for example that the number of male passengers was almost 2 / 3 of the crew, as well as that male passengers and low socioeconomic status represented something more than 1 / 3 of the crew.

Histograms & Bar charts

Looking over the information already collected in the post "Descriptive Statistics", where these two type of representations widely used in data analysis are explained, we can check that their main objective is to represent the distribution of the data with respect to a variable or characteristic of the dataset. Thus, the histograms allow to visualize continuous data and the bar diagrams represent discrete data.

Example: Passenger distribution by price of the ticket

Example: Passenger distribution by socioeconomic status

As we saw in the post "Descriptive Statistics", representing on these graphs measures of descriptive statistics (such as the mean or the median), we can characterize the distribution of the variable under analysis and thereby extract more information about it.

Density plot

As known as Kernel density plots, they serve to visualize the distribution of data in a continuous interval. This type of graphics is a variation of the histograms where the shape of the distribution is smoothly represented.

Example: Distribution of the age of Titanic passengers

Boxplots

The boxplots or whisker plots serve to represent a variable or characteristic depending on its range and its quartiles.

Note: The quartiles are the values that divide a list of ordered numbers into 4 parts in equal percentages

The boxplot consists of a box that includes the data from the first quartile (Q1 - 25% of the data under it) to the third quartile (Q3 - 75% of the data under it) of the values of the variable represented (Interquartile Range - IQR). Within said box, the median is represented with an horizontal line and optionally, the average is represented by an asterisk or a star.

In addition to the box, this plot also shows 2 horizontal lines (or "fences") that represent the maximum and minimum typical values of our variable, and from which the outliers of the distribution are visualized using points.

Statistical note: The outliers are those that are numerically distant from the rest of the data of the variable. Taking the Interquartile Range (IQR), outliers are considered those that are below Q1 - 1.5 IQR or above Q3 + 1.5 IQR.

Example: Distribution of Titanic passengers by age

Violin plots

Violin plots are used to visualize simultaneously the distribution of the data and its probability density. These visual graphs consist of a combination of a boxplot and a density plot. In this plot, the Intercuartile range is represented by a thick black line or small box, the typical data are represented with a thin black line and the median is visualized with a white point.

Example: Distribution and density of Titanic passengers by age

Scatter plots

Scatter plots consist of visualizations that show the relationship of 2 variables of a dataset on two Cartesian axes. They are quite useful since they allow extracting information about the correlation between these variables.

Example: Representation of ticket price by age and gender of the passenger

Pareto charts

The Pareto charts serve to visually represent those distributions or datasets related to the Pareto principle or the 80 / 20 rule.
Pareto principle: also known as the 80 / 20 rule, establishes that generally in a dataset related by a cause-effect relationship, approximately 80% of the results or consequences come from 20% of the causes.

Example: Representation of income data for tickets sold for the socioeconomic status

Example: Representation of income data for tickets sold for passengers

Regarding the example, we can see the usefulness of this graph given that it is quickly visualized that 90% of the ticket sales revenues of the Titanic come from tickets sold to passengers of high socioeconomic status. Or in a more detailed way about the second graph, approximately 80% of the revenue from ticket sales came from ~ 20% of the crew's passengers.

Sources of interest:

Kaggle – Titanic Machine Learning from Disaster
Open data initiative of the Government of Spain – datos.gob.es
Medium (Clément Viguier) – The hate of pie charts harms good data visualization...
Course “Essential Statistics for Data Analysis using Excel” perteneciente al Microsoft Data Science Program
The Data Visualisation Catalogue - The Data Visualisation Catalogue

Descriptive Analysis - Basics

Wed, 07 Jan 2026 00:19:35 +0000

As a starting point in data analysis, it is necessary to know some mathematical and statistical foundations in order to work with the information to be analyzed.

Thus, we will start learning some basic concepts:

Population, Samples, Individuals and Variables:

Population: The set of all the elements under study
Sample: A subset of elements of the population (it should be representative of the population)
Individuals: Every individual element in the popularion
Variables: Characteristics of the individuals

Example: Titanic Dataset

Population: 2.224 individuals
Samples: files:
- train.csv: Set of 891 representative individuals of the population used for the training of models of Machine Learning.
- test.csv: Set of 418 representative individuals of the population used for the testing of models of Machine Learning.
Individuals: Each passenger and crew of the population (every row of the sample or population data collected).
Variables: The characteristics collected for each individual (every column of the sample or population data, ej.: survival (survivals), sex (sex) o Age (age)).

Types of Variables

The variables can be classified depending on the type of data they contain and, more specifically:

Numerical: the data is represented by numbers that are metric or measures of said variable. According to their values, these could be classified in:
- Continuous: they could take a countless or indeterminate number of values. For example, in the data set of the Titanic example, the variable fare could be considered a continuous variable by taking different values with up to 4 decimals of precision.
- Discretes: they could take a countable number within a list of values (the data of continuous variables can be truncated to discrete variables for easier analysis). For example, the variable Sibsp (# of siblings) would be a discrete variable.
Categorical: the data is represented by texts or numbers whose meaning is not a metric, but associated with a specific datum or category. Depending on the relationship between their values, the data would be classified in:
- Nominals: data that have no relation or intrinsic order in themselves. An example of this type would be the embarked variable that represents the port of embarkation of passengers.
- Discretes: data with a natural order or classification among themselves. As for example it could be the variable Pclass that represents the class in the Titanic according to the social status.

Note of interest: The typology of the variables is not always clear and they could be considered of different types depending on the criteria of each person and the objectives sought. An example could be the representation of age, which could be correctly considered both numerical (continuous or discrete depending on the precision) and ordinal, depending on whether you want to work with formulas to treat age as a continuous value or simply work with specific age ranges according to the relationship of the rest of the variables.

Histograms and Bar charts

Although I will make a specific post for the data visualization, it is important to briefly mention the histograms and bar graphs. These are two of the most useful representations in the data analysis and help to understand how the data are distributed with respect to a variable or characteristic of the dataset (continuous in the case of histograms and discrete in the case of bar graphs).

Example: Bar chart

Example: Histogram

Thus, significant information about the data distribution can be extracted, classifying it in symmetric or asymmetric, as we will see below.

Measures of descriptive statistics

A key step when analyzing a dataset is what is called "descriptive statistics", that is, in describing the data through a series of measures that allow to characterize its distribution and its dispersion.

Measures of central tendency

The measures of central tendency serve to give us an idea of the distribution of our data.

Arithmetic Mean (Average)

This measure seeks to obtain the average value of a dataset. That is, all the values in the dataset are summed and divided by the total number of values in that data.

Population mean:
\[\mu = \frac{1}{N}\displaystyle\sum_{i=1}^Nx_i\]
Sample mean:
\[\bar x = \frac{1}{n}\displaystyle\sum_{i=1}^Nx_i\]

Being N the total of values in the population and n the total of values in the sample.

Example:
If we have the following set of 5 data: [1, 2, 2, 5, 10], the average or mean of these values is their sum (20) divided by the total of data (5), that is, the average or mean is 4.
Example of the Titanic dataset:
Following up on our example of the Titanic dataset, and taking the Age variable in the sample train.csv, we visualize its distribution and its average or mean:

Median

This measure represents the middle value in the dataset. That is, for its calculation, we take the data point in the middle of all the dataset x_m:

\[\displaystyle x_{m} = x (\frac{n+1}{2})\]

The median, due to its dependence on the position of the values and not on their values, is less dependent on extreme values (outliers) than the arithmetic mean. That is why in many common statistics (for example, the average salary of the population) it would be more convenient to use the median instead of the average to avoid the presence of very high values that could misalign or mislead the analysis.

Example:
If we have the following set of 5 data: [1, 2, 2, 5, 10], the median of these values is the data n the middle of the dataset. That is, the value in the third position, the number 2.
Example of the Titanic dataset:
Following up on our example of the Titanic dataset, and taking the Age variable in the sample train.csv, we visualize its distribution and its median:

Mode

The mode in statistics represents the most frequently accruing value in the dataset.

Example:
Continuing with our set of 5 data: [1, 2, 2, 5, 10], the mode is the most repeated value, that is 2.
Example of the Titanic dataset:
Following with our example of the Titanic dataset, and taking the Age variable in the sample train.csv, we visualize its distribution and its mode:

Analysis of central tendency

As the examples show, first conclusions can be drawn regarding the distribution of the analyzed variable and its most representative values. Therefore, and comparing these values, we can characterize the distribution of a variable to be symmetric (those cases where the mean, median and mode are similar), or to be positive asymmetric (more values are below the mean and in that case the mean is greater than the median and this greater than the mode) or negative asymmetric (more values are above the mean and the mean is lower than the median and this lower than the mode).

As we have seen in our example of the Titanic and the Age of its passengers we have a certain positive asymmetry (Mean = 29.69 > Median = 28 > Mode = 24) which indicates that there were more people below 29.69 years that above.

Meaures of variability

The measures of variability serve to give us an idea about how spread apart the data values are.

Range

The range is simply a way to express full distribution of the data and it's calculated substracting the minimum value in a set from the maximum one.

\[\displaystyle Range = Max (i) - Min (i)\]

Example:
Analyzing our dataset of 5 values: [1, 2, 2, 5, 10], the range is 10 - 1 = 9.
Example of the Titanic dataset:
Following with our example of the Titanic dataset, the range in the Age variable is 80-0,42 = 79,58 years.

Variance

The variance is a measure of the dispersion of the values with respect to its average or mean value. It is calculated by obtaining the average value of the square of its residuals (differences of each value with respect to the average). That is:

Variance of the population:
\[\sigma^{2} = \frac{\displaystyle\sum_{i=1}^N (x_i-\mu)^2}{N}\]
Unbiased variance of the sample:
\[s^{2} = \frac{\displaystyle\sum_{i=1}^N (x_i-\bar x)^2}{n - 1} \]

The calculation is made on the square of the differences because if we add all their differences the result would be 0.

Regarding the difference in the denominator of both calculations, this is because we are using what is known as unbiased variance of the sample. This modification is based on the fact that we are calculating a statistical value of a sample that could be leaving the population mean outside or far from our values, if we calculate the variance of the sample dividing by n, we could be underestimating the real value of the population variance (more information can be found at Kahn Academy: Explanation about why we divide by n-1 for the unbiased variance of the sample

Example:
Continuing with our set of 5 data: [1, 2, 2, 5, 10], the variance of the population is 10,8.
Example of the Titanic dataset:
Following with our example of the Titanic dataset and the Age variable in the sample train.csv our unbiased variance is 211,01.

Standard deviation

The standard deviation is a measure of the dispersion of the values of the analyzed variable represented in their same units. Its calculation is made by the square root of the variance and its representation is simply sigma or s depending on whether it represents the population or the sample.

Example:
Continuing with our set of 5 data: [1, 2, 2, 5, 10], the standard deviation is 3,67.
Example of the Titanic dataset:
Following with our example of the Titanic dataset and the Age variable in the sample train.csv our, the standard deviation of the sample is 14,52 years.

Standard error

The standard error serves to measure how well a sample represents the population. The calculation is made by dividing the standard deviation by the square root of the total values of the sample.

\[\displaystyle SE_{\bar x} = \frac{s} {\sqrt n}\]

Example of the Titanic dataset:
Following with our example of the Titanic dataset and the Age variable in the sample train.csv our, the standard error of the sample is 0,54.

Analysis of the variability

As seen from the examples seen, these measures help to draw first conclusions regarding the dispersion or variability of our distribution values. This is useful when comparing datasets on variables with those we have worked with previously.

In addition, and in combination with the measures of central tendency, these measures are of special interest in order to extract information about the distribution of the data of our variable, and in this way, characterize said variable in one way or another.

Basic distributions

As final subject in this first article of the Fundamentals series, it is interesting to visualize the distributions of the data because they can give more clear information than the one extracted from their calculated values. According to the data treated, the most typical distributions that we can find could be:

Based on the asymmetry coefficient (according to the measures of the central tendency), and as mentioned above, we could distinguish:

Symmetric distribution:

The values of the mean, median and mode are similar

Asymmetric distribution:

Positive asymmetry:

More values are concentrated below the average or mean. Being this greater than the median and this greater than fashion.

Negative asymmetry:

More values are concentrated above the average or mean. Being this smaller than the median and this smaller than fashion.

According to the shape of the distribution, the most typical cases of representation would be:

Uniform distribution:

Almost all values are identical throughout the range of the variable. This type of distributions could be due to the fact that the representation intervals are too small or too large, and because the variables represented consist of a combination of several variables. Therefore, it is advisable to vary the way of representing the data to detect changes that induce more information:

Random distribution:

Those distributions without a clear pattern of representation where different modes or peaks can be distinguished. These could be due to the same cases as those indicated for the normal distribution, so it is recommended to try to make changes in their representation:

Bimodal distribution:

Distributions with 2 modes or peaks, which generally indicates that they are affected by 2 sources of variation that should try to be studied independently. It could be the case that the measures of central tendency and variability are similar, so their visualization is convenient for a better understanding.

Normal distribution:

The normal or Gaussian distribution consists of the desired distribution for all our data. Although it will have a specific post for analyzing its characteristics, briefly explained, it is a symmetrical bell-shaped function that allows us to model numerous phenomena of nature. The great advantage of having variables whose data are assimilated to this type of distribution is that their distribution can be characterized according to their standard deviation in the following way:

Sources of interest:

Kaggle – Titanic Machine Learning from Disaster