Coder Legion - Recent posts tagged datascience

Advanced SQL Analytics for Data Analysis.

Sun, 18 Oct 2026 13:54:35 +0000

Advanced SQL analytics for running a hospital database.

Introduction

Structured Query Language is essential and common in most industry sectors. In healthcare reporting, SQL plays a critical role in analytics. PostgreSQL lets data scientists take those transaction records and pull out operational insights, so you can make decisions. In this guide we go through the main SQL techniques, using a hospital database schema; patients , doctors, departments , wards, diagnoses, procedures, and visits.

First, before anything else we need to create our database schema. Connect to PostgerSQL DBMS and run a new sql script to start.

Create a schema hospital

create schema hospital;

Check to which schema you are connected.

show search_path;

If connected to a different schema, set connection to your hospital schema.

set search_path to hospital;

Now you are set to begin creating your tables. Run the following codes to create the different tables and insert data into the newly created tables:

Doctors Table

--create doctors table

create table hospital.doctors (
doctor_id int primary key,
doctor_name varchar(50) not null,
doctor_gender varchar(20),
doctor_specialty varchar(20),
doctor_experience_years int
);

select * from hospital.doctors;

-- insert data into doctors

insert into hospital.doctors (doctor_id, doctor_name, doctor_gender, doctor_specialty, doctor_experience_years)
values 
(101, 'DR.Brian Mwangi', 'Male', 'Cardiologist', 12),
(102, 'Dr. Sarah Wanjiku', 'Female', 'General Physician', 9), 
(103, 'Dr. Peter Otieno', 'Male', 'Orthopedic Surgeon', 15),
(104, 'Dr. Faith Chebet', 'Female', 'Pediatrician', 8);

Departments Table

-- create departments table

create table hospital.department (
department_id int primary key,
department_name varchar (50) not null
);


select * from hospital.department;

-- insert data into departments

insert into hospital.department (department_id, department_name)
values 
(10, 'Cardiology'),
(20, 'General Medicine'),
(30, 'Orthopedics'),
(40, 'Pediatrics');

Departments Table

-- create diagnosis table

create table hospital.diagnosis (
diagnosis_id int primary key,
diagnosis_name varchar(50),
diagnosis_category varchar(50)
);

select * from hospital.diagnosis;

--insert data into diagnosis table

insert into hospital.diagnosis (diagnosis_id, diagnosis_name, diagnosis_category)
values 
(501,'Hypertension', 'Chronic Disease'),
(502, 'Malaria', 'Infectious Disease'),
(503, 'Fracture', 'Injury'),
(504, 'Pneumonia', 'Respiratory Disease'),
(505, 'Diabetes', 'Chronic Disease'),
(506, 'Typhoid', 'Infectious'),
(507, 'Asthma', 'Respiratory Disease'),
(508, 'Back Pian', 'Musculoskeletal'),
(509, 'Gastritis', 'Digestive Disease'),
(510, 'Heart Failure', 'Chronic Disease'),
(511, 'Flu', 'Respiratoey Disease'),
(512, 'Kidney Infection', 'Infectious Disease'),
(513, 'Anaemia', 'Blood Disorder'),
(514, 'Sprain', 'Injury'),
(515, 'Gastroenteritis', 'Digestive Disease'),
(516, 'Dislocation', 'Injury'),
(517, 'Migraine', 'Neurological Disorder'),
(518, 'Chest Pain', 'Cardiovascular Disease');

Wards Table

-- create wards table 

create table hospital.wards (
ward_id int primary key, 
ward_name varchar (50)
);

-- insert data into wards
insert into hospital.wards (ward_id, ward_name)
values 
(100, 'Cardiac Care Unit'),
(200, 'General Ward'),
(300, 'Surgical Ward'),
(400, 'Childrens Ward');

select * from hospital.wards;

Patients Table

-- create patients table 
create table hospital.patients (
patient_id serial primary key, 
patient_name varchar(50) not null,
patient_gender varchar(20),
patient_age int,
county varchar(30),
sub_county varchar(30)
);

--insert data into patients
insert into hospital.patients (patient_name, patient_gender, patient_age, county, sub_county)
values 
('Amina Otieno', 'Female', 34, 'Nairobi', 'Westlands'),
('John Kariuki', 'Male', 42,' Kiambu', 'Thika'),
('Grace Naliaka', 'Female', 28, 'Kakamega', 'Lurambi'),
('David Mutua', 'Male', 55, 'Machakos', 'Athi River'),
('Esther Achieng', 'Female', 63, 'Kisumu', 'Kisumu Central'),
('Mohamed Ali', 'Male', 19, 'Mombasa', 'Nyali'),
('Lucy Wambui', 'Female', 7, 'Nakuru', 'Naivasha'),
('Samuel Kiptoo', 'Male', 39, 'Uasin Gishu', 'Eldoret East'),
('Mary Wairumu', 'Female', 31, 'Murang''a', 'Kiharu'),
('Peter Ouma', 'Male', 46, 'Siaya', 'Bondo'),
('Brian Omondi', 'Male', 24, 'Kisumu', 'Kisumu East'),
('Catherine Musyoka', 'Female', 52, 'Kitui', 'Kitui Central'),
('Daniel Maina', 'Male', 16, 'Nyeri', 'Othaya'),
('Eunice Moraa', 'Female', 44, 'Kisii', 'Bonchari'),
('Francis Njuguna', 'Male', 60, 'Laikipia', 'Nyanyuki'),
('Hellen Jepchirchir', 'Female', 37, 'Bomet', 'Sotik'),
('Isaac Kamau', 'Male', 29, 'Nairobi', 'Kasarani'),
('Jane Muthoni', 'Female', 22, 'Meru', 'Imenti North'),
('Kevin Barasa', 'Male', 48, 'Bungoma', 'Webuye'),
('Lilian Adhiambo', 'Female', 26, 'Homa Bay', 'Rangwe'),
('Moses Cheruiyot', 'Male', 33, 'Kericho', 'Ainamoi'),
('Nadia Hussein', 'Female', 5, 'Garissa', 'Dadaab'),
('Patrick Muli', 'Male', 51, 'Makueni', 'Wote'),
('Ruth Nekesa', 'Female', 41, 'Trans Nzoia', 'Kitale'),
('Josephine Wekesa', 'Female', 36, 'Busia', 'Matayos');

select * from hospital.patients;

Procedures Table

create table hospital.procedures (
procedure_id int primary key,
procedure_name varchar (50) not null,
procedure_category varchar (20)
);

--insert data into procedures
insert into hospital.procedures (procedure_id, procedure_name, procedure_category)
values
(701, 'Blood Pressure Monitoring', 'Diagnostic'),
(702, 'Malaria Test', 'Laboratory'),
(703, 'X-ray', 'Imaging'),
(704, 'Chest X-ray', 'Imaging'),
(705, 'Blood Sugar Test', 'Laboratory'),
(706, 'Full Blood Count', 'Laboratory'),
(707, 'Nebulization', 'Treatment'),
(708, 'Physiotherapy', 'Treatment'),
(709, 'Endoscopy', 'Diagnostic'),
(710, 'ECG', 'Diagnostic'),
(711, 'Consultation', 'Consultation'),
(712, 'Joint Support Bandage', 'Treatment'),
(713, 'IV Fluids', 'Treatment'),
(714, 'Closed Reduction', 'Treatment'),
(715, 'Pain Management', 'Treatment');


select * from hospital.procedures;

Visits Table

-- create visits table (Fact Table)

create table hospital.visits (
visit_id int primary key,
visit_date date not null,
patient_id int not null references hospital.patients(patient_id),
doctor_id int not null references hospital.doctors(doctor_id),
department_id int not null references hospital.department(department_id),
ward_id int references hospital.wards(ward_id),
diagnosis_id int references hospital.diagnosis(diagnosis_id),
procedure_id int references hospital.procedures(procedure_id),
admission_type varchar(20) not null
    check (admission_type in ('Outpatient', 'Inpatient', 'Emergency')),
length_of_stay_days int default 0
    check (length_of_stay_days >= 0),
triage_level varchar(10)
    check (triage_level in ('Red', 'Orange', 'Yellow', 'Green')),
insurance_provider varchar(50),
total_bill_kes numeric(10,2) not null
    check (total_bill_kes >= 0),
payment_method varchar(20),
follow_up_required varchar(3)
    check (follow_up_required in ('Yes', 'No')),
patient_satisfaction_score numeric(3,1)
    check (patient_satisfaction_score between 0 and 5)
);

--insert data into visits table
insert into hospital.visits (visit_id, visit_date, patient_id, doctor_id, department_id, ward_id, diagnosis_id,
procedure_id, admission_type, length_of_stay_days, triage_level, insurance_provider, total_bill_kes, payment_method,
follow_up_required, patient_satisfaction_score)
values
(1001, '2026-06-01', 1, 101, 10, 100, 501, 701, 'Outpatient', 0, 'Green', 'NHIF', 3500, 'Cash', 'Yes', 4.5),
(1002, '2026-06-01', 2, 102, 20, 200, 502, 702, 'Outpatient', 0, 'Yellow', 'Britam', 2500, 'Insurance', 'No', 4.1),
(1003, '2026-06-02', 3, 103, 30, 300, 503, 703, 'Inpatient', 3, 'Red', 'AAR', 28000, 'Insurance', 'Yes', 4.3),
(1004, '2026-06-02', 4, 104, 40, 400, 504, 704, 'Inpatient', 4, 'Red', 'None', 42000, 'Cash', 'Yes', 3.9),
(1005, '2026-06-03', 5, 101, 10, 100, 505, 705, 'Outpatient', 0, 'Yellow', 'CIC', 4000, 'Mobile Money', 'Yes', 4.7),
(1006, '2026-06-03', 6, 102, 20, 200, 506, 706, 'Outpatient', 0, 'Green', 'None', 3000, 'Cash', 'No', 4.0),
(1007, '2026-06-04', 7, 104, 40, 400, 507, 707, 'Outpatient', 0, 'Yellow', 'NHIF', 5500, 'Insurance', 'Yes', 4.6),
(1008, '2026-06-04', 8, 103, 30, 300, 508, 708, 'Outpatient', 0, 'Green', 'None', 6500, 'Cash', 'Yes', 4.2),
(1009, '2026-06-05', 9, 102, 20, 200, 509, 709, 'Outpatient', 0, 'Yellow', 'Jubilee', 15000, 'Insurance', 'No', 3.8),
(1010, '2026-06-05', 10, 101, 10, 100, 510, 710, 'Inpatient', 5, 'Red', 'AAR', 76000, 'Insurance', 'Yes', 4.4),
(1011, '2026-06-06', 1, 101, 10, 100, 505, 705, 'Outpatient', 0, 'Green', 'NHIF', 4200, 'Mobile Money', 'Yes', 4.8),
(1012, '2026-06-06', 11, 102, 20, 200, 502, 702, 'Outpatient', 0, 'Green', 'None', 2300, 'Cash', 'No', 4.0),
(1013, '2026-06-07', 12, 101, 10, 100, 501, 710, 'Outpatient', 0, 'Yellow', 'Britam', 7000, 'Insurance', 'Yes', 4.5),
(1014, '2026-06-07', 13, 104, 40, 400, 511, 711, 'Outpatient', 0, 'Green', 'None', 1800, 'Cash', 'No', 4.1),
(1015, '2026-06-08', 14, 103, 30, 300, 503, 703, 'Inpatient', 2, 'Red', 'CIC', 24000, 'Mobile Money', 'Yes', 4.6),
(1016, '2026-06-08', 15, 102, 20, 200, 512, 706, 'Inpatient', 3, 'Orange', 'Jubilee', 36000, 'Insurance', 'Yes', 3.7),
(1017, '2026-06-09', 16, 101, 10, 100, 510, 710, 'Inpatient', 6, 'Red', 'AAR', 82000, 'Insurance', 'Yes', 4.2),
(1018, '2026-06-09', 17, 102, 20, 200, 509, 711, 'Outpatient', 0, 'Green', 'None', 2000, 'Cash', 'No', 4.0),
(1019, '2026-06-10', 18, 104, 40, 400, 507, 707, 'Outpatient', 0, 'Yellow', 'NHIF', 5200, 'Mobile Money', 'Yes', 4.7),
(1020, '2026-06-10', 19, 103, 30, 300, 508, 708, 'Outpatient', 0, 'Green', 'Britam', 6500, 'Insurance', 'No', 4.1),
(1021, '2026-06-11', 2, 101, 10, 100, 501, 701, 'Outpatient', 0, 'Yellow', 'Britam', 3800, 'Insurance', 'Yes', 4.3),
(1022, '2026-06-11', 20, 102, 20, 200, 513, 706, 'Outpatient', 0, 'Green', 'None', 3200, 'Mobile Money', 'No', 4.4),
(1023, '2026-06-12', 21, 103, 30, 300, 514, 712, 'Outpatient', 0, 'Yellow', 'NHIF', 7200, 'Cash', 'Yes', 4.0),
(1024, '2026-06-12', 22, 104, 40, 400, 515, 713, 'Inpatient', 2, 'Orange', 'None', 18500, 'Cash', 'Yes', 4.5),
(1025, '2026-06-13', 5, 101, 10, 100, 501, 710, 'Outpatient', 0, 'Yellow', 'CIC', 7600, 'Insurance', 'Yes', 4.6),
(1026, '2026-06-13', 23, 102, 20, 200, 506, 706, 'Inpatient', 2, 'Orange', 'Jubilee', 29000, 'Insurance', 'Yes', 3.9),
(1027, '2026-06-14', 24, 103, 30, 300, 516, 714, 'Inpatient', 1, 'Red', 'AAR', 33000, 'Insurance', 'Yes', 4.2),
(1028, '2026-06-14', 7, 104, 40, 400, 511, 711, 'Outpatient', 0, 'Green', 'NHIF', 1900, 'Mobile Money', 'No', 4.8),
(1029, '2026-06-15', 25, 102, 20, 200, 517, 715, 'Outpatient', 0, 'Yellow', 'Britam', 4800, 'Cash', 'Yes', 4.1),
(1030, '2026-06-15', 10, 101, 10, 100, 518, 710, 'Emergency', 1, 'Red', 'AAR', 22000, 'Insurance', 'Yes', 4.4);

select * from hospital.visits;

After creating our database schema, we will begin modelling the data using different advanced SQL functions.

1. SQL Window Functions

Window functions do computations on related rows in a table while maintaining the individuality of each rows.

OVER, PARTITION BY, ORDER BY, and Window Frames

The OVER clause describes the rows for the window function operation.

PARTITION BY defines different partitions for each result set.
ORDER BY defines row order in each partition.
Window frames define different limits such asROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW for cumulative sums and ROWS BETWEEN 2 PRECEDING AND CURRENT ROW for moving averages.

Ranking Window Functions

Ranking functions provide numeric ordering according to specified criteria.

ROW_NUMBER provides a distinct numeric integer value to each row within a partition.
RANK provides identical rank values to ties, but skips other numbers.
DENSE_RANK provides identical rank values to ties, but does not skip numbers.
NTILE divides rows in partitions into buckets of equal size.

Question: How can hospital management rank inpatient visits by total bill within each admission type?

select 
    v.visit_id,
    p.patient_name,
    v.admission_type,
    v.total_bill_kes,
    row_number() over (partition by v.admission_type order by v.total_bill_kes desc) as row_num,
    rank() over (partition by v.admission_type order by v.total_bill_kes desc) as rnk,
    dense_rank() over (partition by v.admission_type order by v.total_bill_kes desc) as dense_rnk,
    ntile(2) over (partition by v.admission_type order by v.total_bill_kes desc) as bill_tier
from hospital.visits v
join hospital.patients p on v.patient_id = p.patient_id
where v.admission_type = 'Inpatient'
limit 5;

Expected Results

Explanation

ROW_NUMBER gives values from 1 to 5.
The values obtained by RANK and DENSE_RANK will be same as there are no duplicate bill values.
NTILE assigns the best bills to Tier 1.

Navigation functions retrieve values based on relative rows.

LAG retrieves values from previous row.
LEAD retrieves values from next row.
FIRST_VALUE gives the first value of the expression in the frame.
LAST_VALUE gives the last value of the expression in the frame bounds like ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING.

Question: How can we monitor patient billing process through multiple hospital visits?

select 
    v.visit_id,
    v.patient_id,
    v.visit_date,
    v.total_bill_kes,
    lag(v.total_bill_kes, 1, 0.00) over (partition by v.patient_id order by v.visit_date) as prev_bill,
    lead(v.total_bill_kes, 1, 0.00) over (partition by v.patient_id order by v.visit_date) as next_bill,
    first_value(v.total_bill_kes) over (partition by v.patient_id order by v.visit_date) as initial_bill,
    last_value(v.total_bill_kes) over (
        partition by v.patient_id 
        order by v.visit_date 
        rows between unbounded preceding and unbounded following
    ) as final_bill
from visits v
where v.patient_id in (1, 2)
order by v.patient_id, v.visit_date;

Expected Results

Explanation

LAG retrieves previous visit invoices and LEAD retrieves future visit invoices.
FIRST_VALUE and LAST_VALUE store the first and last visit invoices respectively.

Aggregate Window Functions

Aggregate functions SUM, AVG, COUNT, MIN, and MAX calculate moving sums and moving averages when combined with OVER.

Question: In what way do we determine cumulative income, rolling average billings, categories, and billings for departments on each visit?

select 
    v.visit_id,
    v.visit_date,
    v.total_bill_kes,
    sum(v.total_bill_kes) over (order by v.visit_date, v.visit_id) as cumulative_revenue,
    avg(v.total_bill_kes) over (
        order by v.visit_date 
        rows between 2 preceding and current row
    ) as rolling_avg_bill,
    count(*) over (partition by v.admission_type) as total_admission_type_count,
    min(v.total_bill_kes) over (partition by v.admission_type) as min_admission_bill,
    max(v.total_bill_kes) over (partition by v.admission_type) as max_admission_bill
from hospital.visits v
order by v.visit_date, v.visit_id
limit 5;

Expected Results

Explanation

SUM forms a running total.
AVG calculates a 3 row moving average.
COUNT, MIN, and MAX present group statistics without collapsing rows.

2. SQL Joins

Joins combine columns from multiple tables using related key columns.

INNER JOIN

INNER JOIN returns rows with matching key values in both tables.

Question: How do we generate a visit report combining patient names, attending doctors, and clinical departments?

select 
    v.visit_id,
    p.patient_name,
    d.doctor_name,
    dept.department_name,
    v.total_bill_kes
from hospital.visits v
inner join hospital.patients p on v.patient_id = p.patient_id
inner join hospital.doctors d on v.doctor_id = d.doctor_id
inner join hospital.department dept on v.department_id = dept.department_id
limit 5;

Expected Results

Explanation

INNER JOIN Matches keys across visits, patients, doctors, and department tables.

LEFT JOIN

The LEFT JOIN produces all the rows from the left table and the matched rows from the right table. Fields in the right table without a match produce NULL.

Question: How can we study the number of visits and expenses of patients without losing those that have no visits at all?

select 
    p.patient_id,
    p.patient_name,
    count(v.visit_id) as total_visits,
    coalesce(sum(v.total_bill_kes), 0.00) as total_spent_kes
from hospital.patients p
left join hospital.visits v on p.patient_id = v.patient_id
group by p.patient_id, p.patient_name
order by total_visits desc, p.patient_id
limit 5;

Expected Results

Explanation

All patient records are retained using LEFT JOIN.
Patients having no visits will have COUNT as 0 while COALESCE will convert NULL sums to 0.00.

RIGHT JOIN

RIGHT JOIN retrieves all records from right table and matching records from left table.

Question: How can hospital administration ensure that all physical wards are reflected in admission records?

select 
    w.ward_id,
    w.ward_name,
    v.visit_id,
    v.total_bill_kes
from hospital.visits v
right join hospital.wards w on v.ward_id = w.ward_id
where v.visit_id is null or v.visit_id <= 1003;

Expected Results

Explanation

All wards are kept in RIGHT JOIN.
The Childrens Ward will be with NULL visits if no match occurs.

FULL OUTER JOIN

FULL OUTER JOIN will display all records if there is a match in either left or right record.

Question: How do we perform auditing of doctor and department records for unassigned doctors or unmapped departments?

select 
    d.doctor_name,
    dept.department_name
from hospital.doctors d
full outer join hospital.department dept on d.doctor_id = dept.department_id;

Expected Results

Explanation

Null values will be returned in the rows where there is no relationship between doctor and department IDs.

CROSS JOIN

CROSS JOIN results in a Cartesian product where each row in the first table is paired up with each row of the second table.

Question: How can hospital managers get a complete list of all pairings of physicians with hospital wards for coverage planing?

select 
    d.doctor_name,
    w.ward_name
from hospital.doctors d
cross join hospital.wards w
order by d.doctor_name, w.ward_name
limit 5;

Expected Results

Explanation

Joining 4 doctors with 4 wards results in 16 combination rows.

SELF JOIN

SELF JOIN involves joining a table to itself for comparing rows in the same table.

Question: In what way can readmission of patients be identified by matching their first visit to their next visit?

select 
    v1.patient_id,
    v1.visit_id as original_visit,
    v1.visit_date as original_date,
    v2.visit_id as return_visit,
    v2.visit_date as return_date
from hospital.visits v1
inner join hospital.visits v2 on v1.patient_id = v2.patient_id and v1.visit_id < v2.visit_id
order by v1.patient_id
limit 5;

Expected Results

Explanation

Linking visits on patient ID where v1.visit_id is less than v2.visit_id matches distinct visits per patient.

Join Summary

INNER JOIN matches rows in both tables.
LEFT JOIN keeps all left table rows.
RIGHT JOIN keeps all right table rows.
FULL OUTER JOIN keeps all rows from both tables.
CROSS JOIN produces Cartesian pairs.
SELF JOIN compares rows within one table.

3. Common Table Expressions (CTEs)

A CTE is a named temporary table created within a WITH clause.

Simple CTEs

Question: What approach should be used to identify costly inpatient treatments and connect them with patients and their physicians?

with high_cost_visits as (
    select 
        visit_id,
        patient_id,
        doctor_id,
        total_bill_kes,
        admission_type
    from hospital.visits
    where total_bill_kes >= 35000.00
)
select 
    h.visit_id,
    p.patient_name,
    d.doctor_name,
    h.total_bill_kes
from high_cost_visits h
join hospital.patients p on h.patient_id = p.patient_id
join hospital.doctors d on h.doctor_id = d.doctor_id;

Expected Results

Explanation

CTE will filter out high cost visits first to keep the subsequent join clean.

Multiple Chained CTEs

Question: How do we find out department revenues, overall average, and top performing departments?

with dept_revenue as (
    select 
        department_id,
        sum(total_bill_kes) as total_revenue,
        count(visit_id) as total_visits
    from hospital.visits
    group by department_id
),
dept_benchmark as (
    select 
        avg(total_revenue) as benchmark_avg_revenue
    from dept_revenue
)
select 
    d.department_name,
    dr.total_revenue,
    dr.total_visits,
    round(db.benchmark_avg_revenue, 2) as benchmark_avg
from dept_revenue dr
join hospital.department d on dr.department_id = d.department_id
cross join dept_benchmark db
where dr.total_revenue > db.benchmark_avg_revenue;

Expected Results

Explanation

dept_revenue calculates the total revenue for each department, dept_benchmark calculates the average and the last SELECT statement filters departments whose revenues are greater than the average.

Recursive CTEs

Recursive CTEs create a sequence by means of an anchor query and recursive query linked by UNION ALL operator.

Question: How can the hospital management create a biweekly follow up schedule?

with recursive follow_up_schedule as (
    select 
        1 as cycle_number,
        cast('2026/07/01' as date) as scheduled_date
    union all
    select 
        cycle_number + 1,
        cast(scheduled_date + interval '14 days' as date)
    from follow_up_schedule
    where cycle_number < 4
)
select * from follow_up_schedule;

Expected Results

Explanation

Cycle number 1 starts from July 1 2026 as the anchor date, and the recursive query keeps adding 14 days till cycle number is 4.

CTEs vs. Subqueries

CTEs provide well-defined top-level modules, which can be used again and again, while subqueries within the WHERE or FROM clause tend to complicate code and cannot be executed recursively.

4. SQL Subqueries

A subquery is a query nested inside another SQL query.

Subquery Types

Scalar Subquery fetches one row and one column.
Non-correlatedSubquery executes only once and independently.
Correlated Subquery uses columns of outer query and executes for every row of outer query.
Nested Subquery uses subqueries in subqueries.

Subqueries in SELECT Clauses

Question: How can individual patient bills be shown along with the total hospital average bill?

select 
    visit_id,
    total_bill_kes,
    (select round(avg(total_bill_kes), 2) from hospital.visits) as overall_avg_bill
from hospital.visits
where visit_id between 1001 and 1005;

Expected Results

Explanation

Scalar subquery evaluates to average 16266.67 for all rows.

Using Subqueries in FROM Clauses (Derived Table)

Question: How do we query the pre-aggregated metrics of patients using an inline derived table?

select 
    patient_summary.patient_id,
    patient_summary.visit_count,
    patient_summary.total_spent
from (
    select 
        patient_id,
        count(visit_id) as visit_count,
        sum(total_bill_kes) as total_spent
    from hospital.visits
    group by patient_id
) as patient_summary
where patient_summary.visit_count > 1;

Expected Results

Explanation

The FROM subquery groups the number of visits per patient into table alias patient_summary, subject to WHERE.

Subqueries in WHERE Clause

Question: What is the correlated subquery to find visits where the total bill is greater than the average bill of that particular department?

select 
    v1.visit_id,
    v1.patient_id,
    v1.department_id,
    v1.total_bill_kes
from hospital.visits v1
where v1.total_bill_kes > (
    select avg(v2.total_bill_kes)
    from hospital.visits v2
    where v2.department_id = v1.department_id
)
limit 5;

Expected Results

Explanation

The inner query calculates the average billing amount for each department based on outer query department IDs.

Question: Which physicians have admitted any patients as inpatients?

select 
    d.doctor_id,
    d.doctor_name,
    d.doctor_specialty
from hospital.doctors d
where exists (
    select 1 
    from hospital.visits v 
    where v.doctor_id = d.doctor_id 
      and v.admission_type = 'inpatient'
);

Expected Results

Explanation

EXISTS returns true when there is a row for a visit that matches.

Question: How can we find visits where the billing value is greater than all the outpatient visit bills?

select 
    visit_id,
    patient_id,
    total_bill_kes,
    admission_type
from hospital.visits
where total_bill_kes > all (
    select total_bill_kes
    from hospital.visits
    where admission_type = 'outpatient'
)
limit 5;

Expected Results

Explanation

The internal query locates outpatient charges up to 15000.00. ALL filters visits more than 15000.00.

Conclusion

Understanding the concepts of window functions, joins, common table expressions, and subqueries helps in creating strong PostgreSQL queries that enable healthcare workers to review their operations and enhance financial efficiency. Such information enables healthcare providers to offer quality pateint care which meets patient outcomes.

GitHub

For the completed project visit my GitHub project Hospital Database

A Thousand Cleared AI Devices: What the FDA's Own List Says About How AI Gets Approved

Wed, 14 Oct 2026 12:14:24 +0000

Originally published at vibeagentmaking.com.

The FDA publishes a spreadsheet of every AI-enabled medical device it has ever authorized. One row per device: a name, a company, a clearance date, a review pathway, a medical specialty. As of late March 2026 it holds 1,524 rows, and it is one of the strangest artifacts in the whole AI discourse, because it is the rare place where a decade of arguments about "AI regulation" collapses into something you can actually count.

So this piece counts. The commentary around this list makes three standard claims: the growth curve is a hockey stick, radiology dominates, and almost everything comes through a fast lane called 510(k). All three are checkable, and here is the honest headline: two of them check out cleanly, the third is understated, and the most interesting object in the whole dataset turns out not to be in the dataset at all.

One bookkeeping note first, because it sets the tone for everything else. If you go looking for the total, you will find two. The FDA's own list, as published in late March 2026, carries 1,524 entries. An independent tracker's cumulative count through December 2025 says 1,451. The 73-entry gap is not an error, exactly. The list is a living document; the agency adds entries retroactively as well as forward, so any total is a snapshot with a date on it, and any article that tells you "over a thousand AI devices" without saying which snapshot is quoting a number that was already moving when they wrote it down. Both figures here come from the sources named at the bottom, with their dates. Keep that habit in mind; it is the whole method of this essay in miniature.

The three claims, computed

The hockey stick is real. In 2015 the FDA authorized 6 AI/ML devices. In 2025 it authorized 295. That is roughly a 49x increase in annual volume across a decade, and it is the least surprising fact in the file. Everybody asserts the curve; the curve computes. Fine.

Radiology's share computes too, and that is the more interesting result. The commonly cited figure is "about three-quarters," and the cumulative numbers through 2025 say 1,104 of 1,451 devices: 76.1 percent. In 2025 alone, 255 radiology AI devices cleared via 510(k). So the folk number survives contact with the data, which is worth saying out loud in an essay premised on checking rather than repeating.

But sit with the pairing of those first two facts, because together they say something neither says alone. The field grew 49x, and its composition barely moved. A decade of explosive growth in "AI in medicine" did not diversify medicine's AI; it deepened one specialty's lead. Growth stories usually broaden as they scale: new entrants find new niches, and the long tail thickens. This one didn't. If you build or buy in this space, the list is telling you that the regulatory, technical, and commercial machinery for getting an AI device cleared is overwhelmingly machinery for getting an imaging device cleared, and every other specialty is still traveling a much thinner path.

The pathway claim is understated. The standard line is that "most" AI devices arrive via 510(k), the premarket notification route. For machine-learning-enabled Class II devices authorized in 2024, the measured split was 94.6 percent via 510(k) and 5.4 percent via De Novo. That is not "most." 510(k) is nearly the only door.

Which means you cannot understand this list without understanding what that door is.

Cleared by analogy

A 510(k) clearance does not ask "is this device safe and effective?" in the way outsiders assume. It asks: is this device substantially equivalent to a device already on the market (a "predicate") in intended use and technological characteristics, without raising new questions of safety or effectiveness? Demonstrate equivalence and you are cleared. The heavyweight route that runs its own clinical evidence from scratch is the exception; for AI devices, at one door in twenty, it is nearly a rounding error.

If you are a developer, you already know this structure intimately, because it is transitive trust. It is the dependency tree. Nobody audits the whole stack; you trust your package because it declares a dependency on a package you trusted last year, which was trusted because of what it depended on before that. The 510(k) system is a dependency graph of safety arguments, and a clearance is an edge in it: this device is vouched for by that one.

And that raises the question the essay brief called the sharpest one: what do the chains look like? An AI device cleared as equivalent to an earlier device, which was cleared as equivalent to an earlier one still. How far back do these ancestries run, and what sits at the root?

The sharpest object is not in the file

Here is the finding about the finding: you cannot compute predicate chains from the AI list, because the list does not contain the predicate field. Clearance date, pathway, company, specialty: yes. Ancestry: no. Predicate relationships live in the individual 510(k) summaries and the separate 510(k) database. Walking a chain means joining datasets and reading documents one hop at a time, because, as a 2021 PLOS One genealogy study put it plainly, "the full genealogy of ancestral relationships connecting devices cleared through the 510(k) framework is not reported (only the immediate predicates of a device are listed)."

The regulator's own public record shows each device's parents, never its lineage. The genealogy is an inference, always and for everyone, including the agency's own readers. In dependency terms: every package declares its direct imports, and there is no lockfile. If you want the transitive closure, you crawl.

People have crawled. This is not a new idea, and pretending otherwise would be theft: the PLOS One team mapped predicate genealogies across cleared devices generally in 2021, and a study in The Lancet Digital Health built the predicate networks specifically for AI/ML devices cleared from 2019 to 2021. What those walks found is the part worth carrying to a general reader.

More than a third of cleared AI/ML devices trace back to a non-AI/ML device in the first generation. The equivalence argument crosses the category boundary in a single hop: a system built on a learned model, vouched for by a device that does not learn.

And the ancestors are old. In the Lancet study's 2019–2021 cohort, the most recent ancestor that itself carried an AI/ML component dated to 2008 in cardiovascular devices, and to 2001 in radiology and haematology. A radiology AI cleared in 2021 could sit in a chain whose last AI-bearing ancestor predates AlexNet by eleven years, the deep learning era in its entirety, and every architecture the 2021 device is actually made of. Whatever that 2001 device did, it was not doing what a 2021 convolutional network does. In radiology specifically, the study also found the AI/ML task changed repeatedly along predicate networks, a drift with its own name in the literature, "predicate creep," documented in a 2023 PLOS One case study of a robotic surgical system.

Bound this honestly, because the point is sharper when it is fair. "Substantially equivalent" is a statutory term of art. It does not mean "identical" and never claimed to; it means comparable intended use and characteristics raising no new safety questions, and each individual hop in these chains presumably met that bar. The question the genealogy work raises is not whether anyone broke the rules. It is what a chain of pairwise comparisons is capable of carrying across twenty years: whether "no new questions, twenty times in a row" can quietly accumulate into a very new answer. Any engineer who has watched a codebase drift one reasonable refactor at a time knows the shape of that question.

The row is a snapshot of something licensed to move

There is a second, quieter way the list undersells what is happening, and most commentary misses it entirely. Roughly 10 percent of 2025 clearances, per secondary summaries, included a Predetermined Change Control Plan (a PCCP). A PCCP lets the manufacturer specify, in advance and with the regulator's sign-off, how the model may be updated after clearance without filing a new submission.

Think about what that does to the spreadsheet. The list records a clearance date, one row, one moment. A device with a PCCP is licensed to stop being the exact device that was cleared, within pre-agreed bounds. The row is a photograph of something designed to keep moving after the shutter closed. This is arguably the correct regulatory answer to the awkward fact that ML systems retrain. The alternative is freezing models in 2019 amber or re-reviewing every weight update. But it changes what a clearance date means, and it means one in ten recent rows is best read the way you read a dependency pinned to a version range rather than a hash: accurate on the day, guaranteed to drift.

The gate and the outcomes

Everything so far is about the gate. The list's deepest silence (and the brief's own caution, which deserves to be kept) is that it records nothing about what happened after the gate. Clearance is not efficacy. A 1,524-row list of authorizations contains zero rows about performance in the field.

But an outcomes dataset does exist, and it puts a clock on that silence. A study of 950 AI-enabled devices authorized through November 2024 (reported in JAMA-affiliated coverage) found 60 devices associated with 182 recall events. Two findings stand out.

First, the skew: publicly traded companies made 53.2 percent of the devices and accounted for more than 90 percent of the recall events, a 5.9x association between public-company status and recalls. That runs exactly opposite the intuitive story, in which under-resourced startups cut corners and incumbent giants have quality systems. The coverage also reports that devices lacking clinical validation were recalled more often than those with trials or retrospective studies behind them, a direction I will state without printing the per-segment percentages that circulate, because the denominators in the secondary coverage are murky and the underlying JAMA paper is where those numbers should be read before anyone repeats them.

Second, the timing, which is the number that closes this essay's argument without overclaiming: roughly 43 percent of those recall events came within one year of authorization. That statistic says nothing about whether any individual clearance was wrong. What it says is that the gate and the outcome are separated by months, not decades, and that when the answer-by-analogy misses something, the miss tends to surface fast, in a system the list never records. The spreadsheet's silence about outcomes is a short silence.

What to do with this

For a reader who builds, buys, or signs off on AI systems, medical or otherwise, the list teaches a portable discipline: when someone shows you an approval, ask what the approval is. Concretely, three questions.

Which door? "FDA-cleared" spans a 94.6-percent-of-the-time equivalence argument and a from-scratch evidence review, and the phrase does not tell you which you got. Every certification regime you will ever meet (SOC 2, ISO, app-store review) has the same split between demonstrated-directly and inherited-by-analogy. Find out which one you are holding.

What predicate? An equivalence argument is an inheritance, and inheritances have ancestors. For a medical device, the immediate predicate is public record even though the full chain is not. For anything else (a model card citing a benchmark, a vendor citing a framework) the same question applies: what, exactly, is this thing claiming to be like, and how old is that thing?

What happens after the gate? Is the artifact frozen or licensed to change (the PCCP question), and is anyone measuring outcomes on the other side (the recalls question)? A green light with no post-gate telemetry is a very different object from one with a feedback loop, and the two look identical on a certificate.

And if you are the data-minded sort: the predicate networks have been walked for 2019 through 2021, and, in everything surveyed for this piece, nobody has published the walk for the 2024 or 2025 cohorts, the two biggest years in the list's history. The documents are public. The chains are sitting there, one hop per PDF, waiting for someone with a scraper and a free weekend. That is not a metaphor for anything. It is just true, and it is the most concrete invitation this spreadsheet extends.

The list itself will keep doing what it does: recording, one row at a time, a decade of the question "is it safe?" being answered by pointing at a previous answer. It is a useful public artifact, more transparent than what most industries publish about their gates. Just read it as what it is: a record of doors opened, with the doors' reasons filed elsewhere and the rooms beyond them unrecorded.

Sourcing notes: every figure is from the named public sources; none was computed from the raw list, which was not downloaded and row-counted for this piece. A reader needing one authoritative total should count the published list on a named date. The 2001/2008 ancestors and the non-AI-predicate share are the Lancet study's, 2019–2021 cohort only. The PCCP ~10 percent figure is secondary-sourced, unverified against a primary. The per-segment validation-absence percentages circulating in recall coverage were deliberately omitted pending the primary JAMA paper; the recall counts, public-company skew, and ~43 percent first-year figure are as the cited trade coverage reports them.

Sources

FDA, Artificial Intelligence-Enabled Medical Devices list (the dataset under discussion). https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-enabled-medical-devices
IntuitionLabs, FDA AI medical device tracker (1,524-entry snapshot, late March 2026; 2015 and 2025 annual counts). https://intuitionlabs.ai/articles/fda-ai-medical-device-tracker
The Imaging Wire, "AI-enabled medical devices granted FDA marketing authorization," December 2025 (cumulative and radiology counts through 2025). https://theimagingwire.com/2025/12/10/ai-enabled-medical-devices-granted-fda-marketing-authorization/
"Machine Learning-Enabled Medical Devices Authorized by the FDA in 2024: Regulatory Characteristics, Predicate Lineage, and Transparency Reporting" (the 94.6%/5.4% 2024 pathway split). https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12730494/
"FDA-cleared artificial intelligence and machine learning-based medical devices and their 510(k) predicate networks," The Lancet Digital Health (the 2019–2021 predicate networks; the 2001/2008 ancestors; the non-AI predicate share). https://www.thelancet.com/journals/landig/article/PIIS2589-7500(23)00126-7/fulltext
"Mapping the genealogy of medical device predicates in the United States," PLOS One, October 2021 (the one-generation reporting limit, quoted). https://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0258153
"Identification of predicate creep under the 510(k) process," PLOS One, 2023. https://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0283442
MedTech Dive, "AI medical devices with no clinical validation tied to more recalls" (the 950-device recalls study coverage). https://www.medtechdive.com/news/ai-medical-devices-no-validation-more-recalls-jama/758991/
TechTarget, "Public companies linked to 92% of AI medical device recalls" (the public-company skew; the first-year recall timing). https://www.techtarget.com/healthtechanalytics/news/366630073/Public-companies-linked-to-92-of-AI-medical-device-recalls
FDA, "Premarket Notification 510(k)" (the substantial-equivalence standard). https://www.fda.gov/medical-devices/premarket-submissions-selecting-and-preparing-correct-submission/premarket-notification-510k

Where the lineage would live

The discipline this list teaches is provenance: not is it approved but what is the approval, what is it inherited from, and how old is that ancestor. The FDA's own record shows each device's immediate parent and never its lineage, so the chain has to be reconstructed by hand, one hop per document.

Chain of Consciousness keeps that record for agent decisions: what was read, in what order, to arrive at what was claimed. The ancestry is written down rather than inferred.

pip install chain-of-consciousness and npm install chain-of-consciousness

Hosted Chain of Consciousness

Introducing Native UUIDs in ThingsDB

Fri, 09 Oct 2026 06:28:57 +0000

In the latest release, ThingsDB introduces native support for UUIDs alongside a powerful new typed dictionary pattern (dict<uuid:Type>) that simplifies high-performance key-value lookups.

While ThingsDB relies on auto-incrementing integer IDs under the hood for maximum internal efficiency, integers aren't always ideal for public application interfaces. Exposing sequential integer IDs can expose business metrics or open your API to enumeration attacks. Native UUID support solves this without sacrificing speed or developer experience.

How ThingsDB Handles UUIDs

ThingsDB uses UUID version 7 for its native UUID implementation. Because UUID v7 embeds a Unix timestamp in its leading bits, generated IDs remain time-ordered (chronologically sortable) while ensuring global uniqueness across your cluster:

Sequential Efficiency: Since UUIDs sort naturally by creation time, database indexes stay performant without the index fragmentation typically caused by purely random UUIDs (like UUID v4).
Guaranteed Uniqueness: Generation accounts for node identification, precise millisecond timestamps, and cryptographically sound randomness, so no two nodes will ever generate colliding UUIDs.

Internally, ThingsDB processes and stores UUIDs as efficient 128-bit integers. However, when interacting with the API or serializing data to JSON, ThingsDB exposes them as standard, lower-case hyphenated strings:

"00f3c1b0-7d12-7000-8b1e-213456789abc"

Building a User Management System with UUIDs

Let's walk through a practical example: creating a custom User type that uses UUIDs instead of default ThingsDB integer IDs.

1. Define the User Type

We'll start by defining the User type with two properties: a uuid field named id and a string name. We'll also pass the HID flag to hide ThingsDB's default internal integer ID from API responses.

// Define the User type.
// We use the `HID` flag to suppress the default internal integer ID.
set_type('User', {
    id: 'uuid',
    name: 'str',
}, HID);

2. Set Up Collection Storage

Instead of using a generic dictionary, we can define a dedicated root App type to anchor our collection data. We'll use the new dict<uuid:User> type constraint to ensure the dictionary map strictly pairs uuid keys with User instances.

// Define the App root structure with a typed user lookup dictionary
set_type('App', {
    users: 'dict<uuid:User>',
}, HID);

// Convert the collection to the App type (automatically initializes `users`)
.to_type('App');

3. Implement Procedures for CRUD Operations

To maintain clean architecture, we'll write reusable procedures.

By convention, using CamelCase for custom procedures helps distinguish them from built-in ThingsDB functions.

Adding Users (`NewUser`)

Creating a user is straightforward. ThingsDB automatically generates a valid UUID if one isn't explicitly provided in the payload:

new_procedure('NewUser', |object| {
    user = User(object);
    .users[user.id] = user;
});

To run this procedure, use the ! operator (or wse()) to tell ThingsDB that the call produces side effects:

NewUser({
    name: 'Alice',
})!;

Response:

{
    "id": "c9bf9e57-1685-4c89-bafb-ff5af830be8a",
    "name": "Alice"
}

Note: If you already have a UUID generated on the client side, you can pass it directly in the payload ({id: "...", name: "Alice"}), and ThingsDB will parse and store it automatically.

Fetching Users (`GetUserById`)

While you can iterate through dictionary values, accessing users directly by key using the new uuid() cast function offers O(1) fast lookup performance:

new_procedure('GetUserById', |id| .users[uuid(id)]);

Now, querying a user by string ID is clean and efficient:

GetUserById('c9bf9e57-1685-4c89-bafb-ff5af830be8a');

Calling Procedures from Your Application

Once your procedures are defined, you can invoke them directly from your application using any native client library or standard HTTP requests.

Using Native SDKs (Python, Go, C#, Java, JavaScript etc.)

Native clients make procedure execution seamless. For example, using the Python client:

# Execute the GetUserById procedure using the native client
user = await client.run('GetUserById', 'c9bf9e57-1685-4c89-bafb-ff5af830be8a')

Using the REST API

ThingsDB also exposes procedure execution over HTTP POST requests, making it easy to integrate with any language or web service:

curl --location --request POST 'http://<HOSTNAME:PORT>/collection/<COLLECTION_NAME>' \
  --header 'Content-Type: application/json' \
  --header 'Authorization: Bearer <TOKEN>' \
  --data-raw '{
    "type": "run",
    "name": "GetUserById",
    "args": ["c9bf9e57-1685-4c89-bafb-ff5af830be8a"]
  }'

Conclusion

With native 128-bit UUID support and typed key-value dictionaries, ThingsDB gives you the security and convenience of string UUIDs on the outside, paired with the storage performance of fast integer lookups on the inside.

Give it a try in the latest release and let us know what you think!

# Power BI Data Modelling: A Great Path to Great Analysis

Thu, 08 Oct 2026 03:59:58 +0000

Data Modeling in Power BI

Introduction

At a time when data is rapidly generated by different activities ranging from experiments to business, more often do we get data in different formats and structures. The data may or may not be related depending on the source. Related data in different formats and structure need to be re-organized to establish existing relationships for accurate analysis and reporting. To establish data model relationships and properties we need advanced tools that can connect to these sources. Then load the different datasets from source, and perform modelling to connect the links.

Data Modeling therefore, becomes very important. Data Modeling is the process of defining the data structure, properties, and relationships within a Data Model. Through organizing tables and columns, defining relationships between tables in the model, setting cardinality and cross-filter direction we achieve the best model which supports accurate calculations and reporting.

A Data Model is the logical representation of how data is structured and related within the data model.

Before modeling ensure that your data is clean and up to date, ready for modeling.

Why Data Modeling

Improved Performance and Scalability - join optimization, filters, and aggregations lead to a good model that is efficient and reliable. However, poorly designed models with complex relationships and redundant records cause delays, making queries slow.
Accuracy and Consistency - Guarantees the accuracy and consistency of the data used, making it reliable in reporting and drawing precise insights and improved decision making.
Supports Data Exploration - Users can use hierarchies and drill-down paths to support efficient data exploration.
Ease of Maintenance - Any changes in one model or report are reflected everywhere in real time enabling reusability and scalability, reducing the effort required to maintain reports and models as the business grows.

Components of Power BI Data Model

a. Tables and Sources

Tables are the raw building blocks of data models. They contain fields and records of particular data from different sources.
Sources, where we get our data from can vary depending on the context. We can get our data from SQL Database servers, Excel files, Data warehouse, SharePoint, and Power BI Dataflows.

Designing Source Tables

i. Fact Tables

Fact tables contain quantitative data, numerical data that can be aggregated and analyzed. Examples the Sales, Orders, or Invoices. This information is about specific events, business transaction, or measurements at the core of the data model.

Fact tables keeps one record for each transaction or event, includes only keys and metrics, and they avoid storing descriptive attributes.

Fact tables contain:

Foreign Key(FK) - a link to the Dimension table i.e. DateKey, ProductKey, e.t.c.
Measures(Numeric values) - Quantitative data you can aggregate. i.e. Quantity, UnitPrice, TotalAmount etc.
Degenerate dimensions - Descriptive data that does not warrant its dimension table. i.e. OrderNumber, InvoiceNumber.

ii. Dimension Tables

Dimension tables provide context or additional details to the data in Fact Tables. They contains fewer rows while fact tables can contain a large number of rows and they are on growing.

Relationship are used to establish links and connections between the fact and dimension tables in Power BI.

Dimension table contain:

Primary Key(PK) - Unique identifier e.g. ProductID
Descriptive attributes - Text, categories, hierarchies e.g. ProductName
Hierarchy Columns - Columns that form drill-down paths e.g. Country -> City -> Town.
Natural key - original identifier from the source system e.g. ProductCode

iii. Reference and Lookup Tables

They are small but critical tables in supporting the integrity and interpretation of data. They support additional mappings or categorizations. Example: DiscountTypes or StatusCodes.

Loading Data

Open Power BI
On the Ribbon Home tab, Click on Get Data from the Data group option.

From the drop down list, choose the source of your data. i.e. Text/CSV if your data is text or comma separated file.

Choose the path to your file i.e. C:/Users/User/Downloads/file.csv and click on Open.
Click on Connect on the Power Query editor window.
Your data is extracted to Power Query for cleaning. Click on Transform - Data to clean explore and clean it on the Power Query editor.

Finally, on the Power Query Editor Window Ribbon, Click on Close & Apply from the Home tab.

Changes are loaded and applied to Power BI. We will load all our Toy Store E-Commerce Database data set for modeling using the procedure above. The data is already clean, we are going to use it for demonstrating relationships in data modeling.

b. Relationships

Relationships are an important aspect of data modeling in Power BI. Relationships are for establishing connections and links between tables of a data model. They allow related tables to communicate with each other. They are the link between Fact Tables and Dimension Tables.

Relationships determine how data in the model is filtered, aggregated, combined, and visualized. They're critical to the accuracy and performance of queries in Power BI.

Cardinality

Cardinality type defines each relationship in a model. Cardinality refers to the number of unique values in one table related to the number of unique values in another.

Types of Cardinality in Relationships

One-to-One(1:1) - Every single record in one table is related to another single record in another table. Commonly used when splitting a large table for performance and security reasons.
One-to-Many(1:N) - Refers to when a single record in one table is related to many records in another table(s). Very common type of relationship in Power BI models. The Fact table is on the Many side while the Dimension table is on the One side when linking Fact tables and Dimension tables.
Many-to-Many(N:N) - Refers to when many records in the first table are related to many other records in the second table.

Cross Filter Directions

Used to determine the direction which filters propagate.

Single - When applied, only one of the tables in the relationship can filter data across the other table. Filtering data on the other table does not affect the data shown on the first table.

Bi-Directional - When applied, data can be filtered on either direction between the tables. Filtering data in one table affects data shown on the other table and vice versa. It is less often used and can impact performance negatively and create ambiguous filters.

Cross-filter options are largely dependent on the cardinality type.

Cardinality	Cross-Filter Options
1:1	Bi-Directional
1:*	Single or Bi-Directional
:	Single or Bi-Directional

Table 1: A table showing cardinality and their cross-filter options.

Creating a Relationship

To create relationships, first you need to:

Identify the matching columns(keys).
Choose the right cardinality.
Decide on the Filter direction.

The Model View Interface

To manage relationships in Power BI Desktop:

Click the Model icon in the left sidebar (the icon with three connected boxes)

You'll see all your tables displayed as boxes with their columns listed

Lines between tables represent relationships
The line style indicates cardinality: 1 on one end and * on the many end

Auto-Detect Relationships

When you load tables, Power BI automatically recognizes columns with same data type and name and creates relationships. However, you shoul always create relationships manually or review those created automatically.

Just like in our case, Power BI automatically creates relationships between tables in our model. We will review them.

When auto-detect works well:

Column names match exactly (e.g., ProductID in both tables)
Data types are compatible
Values actually correspond between the tables

Creating Relationships Manually

Using the Managing Relationship Dialog

On the Home tab → Manage Relationships

Click New

Select the first table and column, From Table
Select the second table and column, To Table
Configure cardinality and cross-filter direction

Click Save

Step by Step for Creating Relationships for our Toy Store E-Commerce Database

Load all tables:

 ```
     products
     orders
     order_items
     order_item_refunds
     website_sessions
     website_pageviews
 ````

Open Model view and delete any autodetected relationship.

To delete a relationship:

In Model view, right-click on the line joining the products and order_items tables
From the options displayed, Click on Delete

Save your work

Create the following relationships:

From Fact Table	To Dimension Table	Cardinality	Cross-Filter
products[product_id]	order_items[product_id]	One-to-Many	Single
orders[order_id]	order_items[order_id]	One-to-Many	Single
orders[order_id]	order_item_refunds[order_id]	One-to-Many	Single
website_sessions[website_session_id]	orders[website_session_id]	One-to-One	Bi-Direction
website_sessions[website_session_id]	website_pageviews[website_session_id]	One-to-Many	Single
order_items[order_item_id]	order_item_refunds[order_item_id]	One-to-one	Bi-Direction

Table 2: Table showing relationships to be created.

Creating a relationship between the products and order_items tables

On the Home tab → Manage Relationships
Click New
Select the products table and specify the column product_id
Select the order_items table and specify the column product_id
Set cardinality to One-to-Many and the cross-filter as Single

Click Save

Complete creating all the other relationships following the procedure. You will get something similar to:

Verify by checking that filters from any dimension table correctly filter the fact table.

Conclusion

Data modeling is very crucial to building powerful reports and visualizations. It is key to great analysis because it establishes relationships, links, and connections existing in a data model making it wholesome and complete. Further, it improves query performance increasing efficiency and reliability of queries. Correct data modeling can lead to accurate reporting and result reliability, thus informed decision making. To view the complete model and data for your practice on GitHub visit Toy Store E-Commerce.

The Forgetting Curve Everyone Cites Is Ebbinghaus's Own Column, Relabeled. The 90% Is Not in It.

Tue, 06 Oct 2026 13:19:27 +0000

Originally published at vibeagentmaking.com.

The "half in an hour, 70% in a day" figures that training blogs attribute to Ebbinghaus are his own column IV, a share of relearning time for lists of nonsense syllables, with the unit dropped. The 90% is not in his table; an exponential fits it five times worse than his 1885 formula; two replications confirm the shape.

On June 8, 2026, MeltingSpot's blog published a long post on the cost of forgetting in corporate training. Its first section, "What the Ebbinghaus forgetting curve actually tells us," reports what Hermann Ebbinghaus found: "roughly 56% is forgotten within one hour of learning, about 66% within one day, and around 75% within six days." Those figures are close to numbers Ebbinghaus printed. Two paragraphs later, in the same section, the post says that if a team finishes a Monday morning training and gets no review for the rest of the week, "cognitive science predicts that 90% of the material will be gone by Friday."

So 75% at six days became 90% at four, two paragraphs apart. The post then puts a price on it. Citing the Association for Talent Development's estimate that US organizations spend an average of $1,254 per employee per year on training, it says: "If 90% of that training content is not retained past the first week, the effective return on investment is not just low. It is catastrophic." The worked figure that follows is "roughly $940" wasted per employee. That is 75% of $1,254, the six-day number the post gave in its first section. Ninety percent would be $1,129.

The same figures run through the training industry's explainers. TalentLMS's blog says Ebbinghaus "found that within an hour of learning new information people tend to forget up to 50% of it. Within 24 hours, this can increase to 70%. By the end of the week, people tend to retain only about 25% of what they've learned." Further down, citing "studies" that illustrate the curve, it says people "tend to forget up to 90% of what they've learned within a month." The post dates from March 2023; the page was last modified on September 21, 2026. Today's Class says "on average we forget 50% of new information within an hour, and about 70% within one day," and "Within 30 days, we forget up to 90% of what we learned." TrainMeUK's headline from December 2025 is "Why Employees Forget 70% of Training in 24 Hours (The Ebbinghaus Effect)," and its introduction calls the forgetting "a biology problem."

We published a version of it too. An essay on this site in April said: "Hermann Ebbinghaus established the lower bound in 1885. Without reinforcement, humans lose roughly half of newly learned information within an hour and most of it within a week." This piece corrects that sentence, and the one after it, which went further.

What he measured

Ebbinghaus was his own and only subject. The seventh chapter of his book Memory, in the 1913 English translation by Henry Ruger and Clara Bussenius, gives the design: "The investigations in question fell in the year 1879-80 and comprised 163 double tests." Each double test meant learning eight series of 13 nonsense syllables (six, in 38 of the tests) until he could recite each series twice without error, then relearning them to the same standard after a fixed interval. He was 29.

He did not count the syllables he could still recall. He timed the relearning and took the time saved, as a share of the original learning time, as his measure: "This saving in work is each time the measure for the amount remembered at the end of the interval." If relearning a list took 60% as long as learning it, the saving was 40%.

He was also careful about whom the results described. Early in his discussion of them he writes that "it would be just as reasonable to be surprised at this initial rapidity and later slowness as they come to light here under the definite conditions of our experiment for a certain individual, and for a series of 13 syllables. One hour after the end of the learning, the forgetting had already progressed so far that one half the amount of the original work had to be expended before the series could be reproduced again; after 8 hours the work to be made up amounted to two thirds of the first effort."

That is where "half in an hour" comes from. Its unit is work.

The table

The summary table that closes Section 28 of the chapter has seven rows. Its second column is headed "So much of the series learned was retained that in relearning a saving of Q% of the time of original learning was made." Its fourth column is headed "The amount forgotten was thus equivalent to v% of the original in terms of time of learning," and every entry in it is 100 minus the second. The formula table in Section 29 gives the intervals in minutes, and they are less round than the summary suggests: the "1 hour" row was 64 minutes, and the "8.8 hours" row was 526.

Interval	Saving (column II)	Amount forgotten "in terms of time of learning" (column IV)
20 minutes	58.2%	41.8%
64 minutes	44.2%	55.8%
8.8 hours	35.8%	64.2%
1 day	33.7%	66.3%
2 days	27.8%	72.2%
6 days	25.4%	74.6%
31 days	21.1%	78.9%

Set the retellings beside it and most of their numbers turn out to be his. "Half in an hour" is 55.8% in column IV. "Seventy percent in a day" is 66.3%, rounded up. MeltingSpot's 56, 66 and 75 are column IV to the nearest point. TalentLMS's "about 25%" retained by the end of the week is column II at six days, 25.4%, copied correctly. Ebbinghaus made the conversion from savings to "forgotten" himself, in print.

What changed on the way is the thing the percentage is a percentage of. Column IV is the share of the original learning time that had to be spent again to relearn a list of nonsense syllables. The retellings make it a share of "new information," of "training content," of "what they've learned." A saving of 44.2% after an hour does not mean 55.8% of anything was lost. It means relearning took 55.8% as long as learning had.

Tom McDowall's critique in his Instructional Design Tips newsletter (November 2025) makes the savings point well, and says "you cannot legitimately convert savings scores into 'percentage forgotten.'" Ebbinghaus did convert them. He printed the result as column IV, with the unit in the heading. What later writers dropped was the unit.

The 90% has no source in the table. The largest value in column IV is 78.9%, at 31 days. At six days it is 74.6%. MeltingSpot's Friday, four days after a Monday session, falls between the two-day row (72.2%) and the six-day row (74.6%). TrainMeUK's "80% within a month" is near his 78.9%. The "up to 90%" within a month on TalentLMS and Today's Class is 11 points past it.

The shape

The retellings also name a shape. MeltingSpot: "memory of new material follows an exponential decay curve." Today's Class: "the percentage of information retained declined exponentially over time without reinforcement." McDowall's critique calls it "Ebbinghaus's exponential decline curve" too. An exponential loses a fixed fraction of what remains in each equal stretch of time.

We fitted that shape and two others to his seven savings values by least squares, with time in minutes. His own formula from 1885, with constants he set "with merely approximate estimates, not involving exact calculation by the method of least squares," misses his data by 1.73 percentage points, root mean square. A power law misses by 1.91. The best exponential misses by 9.10, more than five times the error of his hand-set formula.

The exponential cannot follow a curve that falls fast and then nearly stops. The best one starts at 40% saved and stays close to it through the first day: 39.9% at 20 minutes, where he measured 58.2%, and 38.8% at one day, where he measured 33.7%. It is too high at a week (32.4%, against 24.5% from his formula) and too low at a month (16.3%, against 21.3%). His own numbers drop 24.5 points between 20 minutes and one day, and only 4.3 points between the sixth day and the thirty-first. He named the pattern himself: "this initial rapidity and later slowness."

A 1996 survey points the same way. David Rubin and Amy Wenzel assembled 210 published data sets and fitted each "to 105 different 2-parameter functions." Their summary: "The best fits were to the logarithmic function, the power function, the exponential in the square root of time, and the hyperbola in the square root of time." The plain exponential is not among them.

Three people who ran it again

One subject invites the obvious question, whether his memory was unusual. In 2015 Jaap Murre and Joeri Dros of the University of Amsterdam repeated the experiment with Dros, then 22, as the only subject. "One subject spent 70 hours learning lists and relearning them," at Ebbinghaus's intervals, and "The results are similar to Ebbinghaus' original data." They set their results beside a German replication from 1991 by Heller, Mack and Seitz, with two subjects, which they describe as published only in German, without an English abstract, and never cited in international journals in English.

Their comparison table, in savings:

Interval	Ebbinghaus (1879–80)	Mack (1991)	Seitz (1991)	Dros (2015)
20 minutes	0.582	0.544	0.442	0.472
1 hour	0.442	0.432	0.325	0.373
9 hours	0.358	0.285	0.270	0.276
1 day	0.337	0.316	0.270	0.317
2 days	0.278	0.365	0.286	0.230
6 days	0.254	0.309	0.205	0.168
31 days	0.211	0.258	0.201	0.041

The shape holds for all four: a fast early drop, then a long shallow slope. At six days the largest loss in column-IV terms is Dros's, 83.2%. At 31 days, Mack is at 74.2% and Seitz at 79.9%, on either side of Ebbinghaus's 78.9%.

Dros at 31 days is the one number here that cuts against this essay. His saving of 0.041 is 95.9% in column-IV terms, past 90%. The authors treat it with care. His learning time rose by an average of 2.67 seconds a day per list over the 75 days of the experiment, which mostly affects the longest interval, and with a correction for that drift "the corrected savings measure would be 0.137 for the 31 day interval instead of 0.0410," which is 86.3%. That, they add, "is still well below the values for the three others." The Dros column above matches savings computed from the repetition counts in the paper's Table 1 at all seven intervals; computed from the seconds in its list-by-list Table 2, the 31-day saving is 0.090, or 91.0%. So there is a 90% in this literature: one subject, at one month, before the authors' correction for drift.

The replication also found something Ebbinghaus nearly discarded. In three of the four columns the one-day saving is at least as high as the nine-hour saving, which a curve that only runs downhill does not allow. The authors conclude that the curve "is not completely smooth but most probably shows a jump upwards starting at the 24 hour data point," and point to sleep: "The current body of research on sleep and memory would predict such a boost after one or two nights and attribute it to sleep, though for this particular type of experiments this has to established more firmly in further experiments." In Ebbinghaus's own table the one-day value sits 3.3 points above his formula, the largest miss of the seven, and he distrusted it. He wrote that it "would fit in well with the other observations to consider the number 33.7 per cent for the relearning after 24 hours as somewhat too large," and then that "it is upheld by observations to be stated presently, so that I am in doubt about it." He left it in.

What a saving is

The unit matters even though the shape replicates, because a saving and a recall score are different quantities. Murre returned to the measure in 2023 with Antonio Chessa. Ebbinghaus, they write, "measured memory retention in terms of the learning time saved in subsequent study trials relative to the time spent on the first learning trial," and they "prove mathematically that Ebbinghaus' savings measure is independent of initial encoding strength, learning time, and relearning times." They contrast it "with often used forgetting functions based on recall probability." A saving says how much faster the second learning went. It does not say what share of the material could still be produced on demand, which is what every retelling above takes it to say. And the material was lists of nonsense syllables.

What decades look like

Material that means something runs on a different clock. Harry Bahrick tested 733 people on the Spanish they had learned in school, across a span of 50 years, and published the results in 1984. "The analysis yields memory curves which decline exponentially for the first 3-6 years of the retention interval. After that retention remains unchanged for periods of up to 30 years before showing a final decline." Most had not kept it up: "The great majority of subjects rehearse so little that the data reveal no significant rehearsal effects." Retention over the whole period was "predictable on the basis of the level of original training," and "Large portions of the originally acquired information remain accessible for over 50 years in spite of the fact the information is not used or rehearsed."

Bahrick measured something different again, with tests of reading comprehension, recall and recognition, so his curves neither confirm nor overturn Ebbinghaus's. They show how far a table about nonsense syllables sits from a language learned in school.

Our April sentence

Here is the passage we published on April 26, 2026:

Hermann Ebbinghaus established the lower bound in 1885. Without reinforcement, humans lose roughly half of newly learned information within an hour and most of it within a week. Modern memory research has refined the curve in countless ways, but the directional finding has held up across 140 years of experiment. "We trained the team on the policy last quarter" is, neurologically, close to indistinguishable from "we never trained them."

The direction holds: both replications above found the fast early drop and the long flattening. The rest does not. "Humans" was one man, and he said so. "Newly learned information" was the time needed to relearn lists of nonsense syllables. "Most of it within a week" is true only in his unit, where it was 74.6% at six days. The chapter the numbers come from reports an average curve for one person and describes no lower bound. The last sentence is contradicted by the table it leans on: a month after learning, relearning still took 21.1% less time than learning had. Nothing in the experiment was neurological either; what he measured was time. And Bahrick's former students, most of whom rehearsed too little for it to show in the data, kept large parts of their Spanish for decades.

What the passage should have said: In 1879 and 1880, Hermann Ebbinghaus timed himself learning and relearning lists of nonsense syllables. An hour after learning a list, relearning it took about 56% as long as learning it had; a month later, about 79%. Replications in 1991 and 2015 found curves of the same shape.

Measuring your own

The part of Ebbinghaus a trainer can use is the method, and it needs a stopwatch. Time how long a group takes to reach a standard the first time: a procedure done without error, or a test passed. Some weeks later, before any refresher, time how long the same group takes to reach the same standard again. The time saved, as a share of the first session, is his measure for your material and your people. If the second session takes 70% as long as the first, the saving is 30%, and it is a number about your training.

Anyone who still wants to quote the 1885 figures can, with the qualifier Ebbinghaus attached to them: "for a certain individual, and for a series of 13 syllables."

The curve fits come from refit_ebbinghaus_c6733.py, which fits three shapes to the seven savings values in Ebbinghaus's summary table by least squares. Its first check is that his formula, as coded, reproduces the calculated values he printed in his formula table; it does, to within 0.1 point, but only with the exact 64 minutes from that table in place of a rounded hour, and that check is how the 64 minutes surfaced. Every quotation and figure above is checked against saved copies of its source by figures_c6733.py, which also recomputes the column-IV equivalents, the MeltingSpot arithmetic and the Murre and Dros savings from their Tables 1 and 2. Both scripts and their output are in one folder, with a list of every saved source the check reads, its public address, when it was saved and a checksum of the saved copy; the pages themselves are linked in the Sources below rather than republished. Ebbinghaus's tables are scanned images in the online edition; the values above were read off the scans, and his savings column agrees with Murre and Dros's transcription to the digit.

Part of Where the Number Came From, on how a published number is a fact about the way it was measured: Stanford says 12% to 66%, but 12% of what? · The safety score is a fact about the test rig · Tracing the 2026 AI-failure statistics to a primary

Sources:

Hermann Ebbinghaus, Memory: A Contribution to Experimental Psychology (1885), translated by Henry A. Ruger and Clara E. Bussenius (Teachers College, Columbia University, 1913), Chapter VII: Section 28 (the design; the definition of Q; the summary table that closes the section) and Section 29 (paragraphs 1 to 3, with the formula table of observed and calculated values). Classics in the History of Psychology, York University. https://psychclassics.yorku.ca/Ebbinghaus/memory7.htm
Jaap M. J. Murre and Joeri Dros, "Replication and Analysis of Ebbinghaus' Forgetting Curve," PLOS ONE 10(7): e0120644 (2015): abstract; Methods (the intervals, the subject, the differences from Ebbinghaus and from Heller et al.); Tables 1, 2 and 3; the paragraph after Table 3 (the drift correction); Discussion. https://doi.org/10.1371/journal.pone.0120644 (data: https://osf.io/6kfrp/)
O. Heller, W. Mack and J. Seitz, "Replikation der Ebbinghaus'schen Vergessenskurve mit der Ersparnismethode: 'Das Behalten und Vergessen als Function der Zeit'," Zeitschrift für Psychologie 199 (1991): 3–18. Values as reported in Murre and Dros, Table 3; the paper itself was not read.
Jaap M. J. Murre and Antonio G. Chessa, "Why Ebbinghaus' savings method from 1885 is a very 'pure' measure of memory performance," Psychonomic Bulletin & Review 30(1) (2023): 303–307, abstract. https://doi.org/10.3758/s13423-022-02172-3
Harry P. Bahrick, "Semantic memory content in permastore: Fifty years of memory for Spanish learned in school," Journal of Experimental Psychology: General 113(1) (1984): 1–29, abstract. https://doi.org/10.1037/0096-3445.113.1.1
David C. Rubin and Amy E. Wenzel, "One hundred years of forgetting: A quantitative description of retention," Psychological Review 103(4) (1996): 734–760, abstract (p. 734). https://doi.org/10.1037/0033-295X.103.4.734
Benoit Chatelier, "Forgetting curve corporate training: the real cost," MeltingSpot blog, June 8, 2026: the sections "What the Ebbinghaus forgetting curve actually tells us" and "The real financial cost of forgetting in L&D," and the paragraph before the second. https://meltingspot.io/en/blog/forgetting-curve-corporate-training
Athena Marousis, "What is the Ebbinghaus forgetting curve? Examples and strategies for overcoming it," TalentLMS Blog, March 30, 2023 (modified September 21, 2026): "What is the forgetting curve?" and the section after it. https://www.talentlms.com/blog/ebbinghaus-forgetting-curve/
David Boyes, "Fighting the Forgetting Curve," Today's Class blog, May 5, 2023 (modified April 28, 2025): the sections "What Is the 'Forgetting Curve'?" and "How Much Time Has Passed." https://www.todaysclass.com/blog/fighting-the-forgetting-curve
TrainMeUK, "Why Employees Forget 70% of Training in 24 Hours (The Ebbinghaus Effect)," December 3, 2025: headline and introduction. https://trainmeuk.co.uk/resources/ebbinghaus-effect-employees-forget-training-24-hours
Tom McDowall, "Sunday L&D Myth #4: People Forget 70% of What They Learn Within 24 Hours," Instructional Design Tips, November 9, 2025. https://idtips.substack.com/p/sunday-l-and-d-myth-4-people-forget
"We Wrote 25 Reminders and Made the Same Mistake Every Time," this site, April 26, 2026: the paragraph beginning "Hermann Ebbinghaus established the lower bound." https://vibeagentmaking.com/blog/we-wrote-25-reminders-and-made-the-same-mistake-every-time/

Keep the trail back to where the number was measured.

Every number in this piece survived because a script checks it against saved copies of its sources, so the trail from the figure back to the page it was measured on stays intact. A statistic that loses that trail, as "half in an hour" lost its unit, keeps its digits and changes its meaning. If you run agents that report numbers, keep the trail: Chain of Consciousness keeps a tamper-evident record of an agent's actions, so a figure it published can be traced back to what it actually read.

pip install chain-of-consciousness
npm install chain-of-consciousness

Or start without installing anything: Hosted Chain of Consciousness.

Building a Real-Time Social Listening & Brand Sentiment Engine with Python, Gemini 1.5, and DuckDB

Sun, 04 Oct 2026 16:57:35 +0000

Enterprise social listening platforms like Brandwatch or Sprout charge upwards of $800 to $2,000/month per seat. Under the hood, many still rely on brittle dictionary lookup tables (VADER-style positive/negative lexical scoring) or generic regex filters. When a developer tweets "Oh fantastic, another breaking API change deployed straight to prod on a Friday!", classical sentiment classifiers mark it as overwhelmingly positive due to "fantastic".

Meanwhile, writing your own real-time monitoring service often degrades into a distributed systems mess: rate limits across Reddit and Hacker News, unstructured LLM hallucinations, deduplication bottlenecks, and ballooning database costs.

Here is how to build an asynchronous, production-ready brand intelligence engine using Python, Google's Gemini 1.5 Flash structured outputs, and DuckDB—running for less than $1/month in API costs.

1. The Architecture

The engine is designed around a decoupled, non-blocking ingestion and evaluation pattern:

graph TD
    A[Reddit API / HN Firebase / Bluesky Firehose / RSS] -->|Async Polling| B[Ingestion Layer: Asyncio + HTTPX]
    B -->|Content Hash SHA-256| C[(DuckDB Local Store - Deduplication)]
    C -->|New Items| D[Gemini 1.5 Flash Engine]
    D -->|Pydantic Structured JSON| E[Alert Dispatcher]
    E -->|Urgency >= High or Polarity <= -0.6| F[Slack / Discord Webhook]
    E -->|Daily Rollup & Trend Metrics| G[Parquet / Analytical Store]

Core Pillars:

Async Multi-Source Ingestion: Pulls from Reddit (asyncpraw), Hacker News (Firebase REST API), Bluesky (AT Protocol firehose), and custom RSS feeds asynchronously.
Deduplication Engine: Generates an identity fingerprint (sha256(platform + post_id)) checked against local embedded DuckDB before hitting LLM inference.
Structured Gemini 1.5 Flash Parsing: Uses schema-constrained JSON outputs to extract polarity (-1.0 to 1.0), urgency classification (low, medium, critical), entity references, and sarcasm detection.
Downstream Alert Dispatcher: Routes critical events directly to Slack/Discord webhooks while appending all metadata into DuckDB for fast OLAP rollups.

2. Ingestion & Deduplication via DuckDB

Rather than spinning up Postgres or Redis for transient social data, we use DuckDB. DuckDB runs in-process, provides zero-latency ACID writes, and allows vector-like analytical SQL on JSON columns.

import duckdb
import hashlib

class StorageManager:
    def __init__(self, db_path: str = "brand_intel.duckdb"):
        self.con = duckdb.connect(db_path)
        self._init_schema()

    def _init_schema(self):
        self.con.execute("""
            CREATE TABLE IF NOT EXISTS raw_mentions (
                id VARCHAR PRIMARY KEY,
                platform VARCHAR,
                author VARCHAR,
                content VARCHAR,
                url VARCHAR,
                created_at TIMESTAMP,
                processed BOOLEAN DEFAULT FALSE
            );
            CREATE TABLE IF NOT EXISTS sentiment_results (
                id VARCHAR PRIMARY KEY,
                polarity FLOAT,
                urgency VARCHAR,
                is_sarcastic BOOLEAN,
                entities JSON,
                summary VARCHAR,
                evaluated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
                FOREIGN KEY (id) REFERENCES raw_mentions(id)
            );
        """)

    def is_duplicate(self, platform: str, external_id: str) -> bool:
        uid = hashlib.sha256(f"{platform}:{external_id}".encode()).hexdigest()
        res = self.con.execute("SELECT 1 FROM raw_mentions WHERE id = ?", [uid]).fetchone()
        return res is not None

    def insert_mention(self, platform: str, external_id: str, author: str, content: str, url: str, created_at):
        uid = hashlib.sha256(f"{platform}:{external_id}".encode()).hexdigest()
        self.con.execute("""
            INSERT INTO raw_mentions (id, platform, author, content, url, created_at)
            VALUES (?, ?, ?, ?, ?, ?)
            ON CONFLICT (id) DO NOTHING
        """, [uid, platform, author, content, url, created_at])
        return uid

3. Structured Gemini 1.5 Flash Evaluation

To eliminate hallucinations and avoid parsing regex out of freeform LLM completions, we leverage Google GenAI's native Pydantic schema validation. We use gemini-1.5-flash due to its sub-second latency and $0.075 per 1M tokens cost tier.

The Evaluation Schema

from pydantic import BaseModel, Field
from typing import List, Literal
import google.generativeai as genai
import os

genai.configure(api_key=os.environ["GEMINI_API_KEY"])

class SentimentEvaluation(BaseModel):
    polarity: float = Field(
        description="Polarity score strictly between -1.0 (extremely negative/hostile) and 1.0 (enthusiastic/positive)."
    )
    urgency: Literal["low", "medium", "high", "critical"] = Field(
        description="'critical' indicates data breaches, service downtime, or legal threats; 'high' indicates severe bugs or churn threats; 'medium' is regular feedback; 'low' is passive mention."
    )
    is_sarcastic: bool = Field(description="True if the statement implies the opposite of its literal words.")
    entities: List[str] = Field(description="List of specific products, competitor names, or people mentioned.")
    summary: str = Field(description="1-sentence technical digest of the user's issue or praise.")

Inference Function with Enforced JSON Schema

import json

def evaluate_mention(brand_name: str, text: str) -> SentimentEvaluation:
    prompt = f"""
    Analyze the following social mention referencing the brand: '{brand_name}'.
    Contextualize sarcasm, tech nuances, and developer sentiment.
    
    Content:
    """{text}"""
    """
    
    model = genai.GenerativeModel("gemini-1.5-flash")
    
    response = model.generate_content(
        prompt,
        generation_config=genai.GenerationConfig(
            response_mime_type="application/json",
            response_schema=SentimentEvaluation,
            temperature=0.1,
        ),
    )
    
    return SentimentEvaluation.model_validate_json(response.text)

4. Threshold Alerting & Webhook Routing

When a negative spike occurs (e.g., a critical authentication vulnerability drops on Hacker News), latency matters. The dispatcher inspects the structured output and fires immediate payloads to Slack, Discord, or an incident management endpoint like PagerDuty.

import httpx
import asyncio

SLACK_WEBHOOK_URL = os.environ.get("SLACK_WEBHOOK_URL")

async def dispatch_alert(eval_data: SentimentEvaluation, platform: str, url: str, content: str):
    # Alert trigger: high/critical urgency OR heavily negative sentiment
    if eval_data.urgency in ["high", "critical"] or eval_data.polarity <= -0.5:
        color = "#FF0000" if eval_data.urgency == "critical" else "#FFA500"
        
        payload = {
            "attachments": [
                {
                    "color": color,
                    "title": f"🚨 Brand Alert [{eval_data.urgency.upper()}] on {platform}",
                    "title_link": url,
                    "fields": [
                        {"title": "Summary", "value": eval_data.summary, "short": False},
                        {"title": "Polarity", "value": f"{eval_data.polarity:.2f}", "short": True},
                        {"title": "Sarcasm Detected", "value": str(eval_data.is_sarcastic), "short": True},
                        {"title": "Entities", "value": ", ".join(eval_data.entities) or "None", "short": True},
                    ],
                    "text": f"*Snippet:*
> {content[:280]}...",
                }
            ]
        }
        
        async with httpx.AsyncClient() as client:
            resp = await client.post(SLACK_WEBHOOK_URL, json=payload)
            resp.raise_for_status()

5. Deployment, Concurrency & Rate Limits

Running high-frequency pollers against varied APIs presents two main issues:

Rate Limit Exhaustion: Reddit restricts standard OAuth to 60 requests/minute. Gemini 1.5 Flash has a free-tier limit of 15 RPM and Tier-1 pay-as-you-go limit of 1,000 RPM.
Event Loop Starvation: Blocking HTTP calls freeze background listeners.

The Concurrency Pool Pattern

Use an asyncio.Queue paired with worker pools to guarantee your workers respect rate bounds:

async def analysis_worker(queue: asyncio.Queue, db: StorageManager, brand_name: str):
    while True:
        item = await queue.get()
        uid, platform, author, content, url = item
        try:
            # Non-blocking run for Gemini evaluation
            eval_res = await asyncio.to_thread(evaluate_mention, brand_name, content)
            
            # Save analytics to DuckDB
            db.con.execute("""
                INSERT INTO sentiment_results (id, polarity, urgency, is_sarcastic, entities, summary)
                VALUES (?, ?, ?, ?, ?, ?)
            """, [uid, eval_res.polarity, eval_res.urgency, eval_res.is_sarcastic, 
                  json.dumps(eval_res.entities), eval_res.summary])
            
            # Route Alerts
            await dispatch_alert(eval_res, platform, url, content)
            
        except Exception as e:
            print(f"[ERROR] Failed to process {uid}: {e}")
        finally:
            queue.task_done()

Analytical Daily Rollups via DuckDB

To view aggregated brand health over the past 24 hours without loading heavy Pandas workflows:

SELECT 
    date_trunc('hour', evaluated_at) as hour_bucket,
    AVG(polarity) as avg_sentiment,
    COUNT(CASE WHEN urgency IN ('high', 'critical') THEN 1 END) as critical_count,
    COUNT(*) as total_mentions
FROM sentiment_results
WHERE evaluated_at >= NOW() - INTERVAL '24 HOURS'
GROUP BY 1
ORDER BY 1 DESC;

6. Conclusion & Turnkey Template

With under 200 lines of modern Python, you can replace a legacy multi-thousand-dollar monitoring subscription with a real-time system that catches sarcasm, tags entities, writes to low-footprint local storage, and pings your on-call channel in real time.

You can manually implement this architecture using the snippets above, wire your own OAuth credentials, and maintain the worker pipelines yourself.

If you prefer to deploy a battle-tested, production-ready codebase immediately—complete with pre-configured connectors for Reddit, Bluesky, Hacker News, RSS feeds, an automated CLI dashboard, and native Docker Compose files:

Instant Access on Whop
Direct Download on Gumroad (Use promo code EARLYBIRD for 20% off)

Have questions about tuning the schema for specific B2B niche contexts? Drop a comment below!

Reading a rental-yield CSV without inventing missing data

Thu, 01 Oct 2026 06:55:49 +0000

A small CSV can be easy to load and surprisingly easy to misinterpret. This walkthrough uses dataHabibi's owned June 2026 Dubai community rental-yield snapshot to show how to preserve units, missing values and the distinction between a community and a city aggregate.

Disclosure: I am Haron Merzaie, co-founder of dataHabibi alongside Ibrahim Qorraj. The source platform is https://datahabibi.ae/. This is a historical data-reading example, not a live feed or an investment recommendation.

Start with the file's actual schema

The CSV has six columns: Community, Gross rental yield percent, Buildings tracked, Yield sample size, Snapshot, and Source. There are twelve named community rows and one row labelled Dubai city average.

The dataset and its field documentation are available in the public repository datahabibi-bot/dubai-community-yields-june-2026 on GitHub. Download dubai-community-yields-june-2026.csv from that repository and put it beside your Python script.

For example, the International City row records 9.98 in the yield field, 358 buildings tracked and a yield sample size of 220. The city-average row records 8.16 and 2,561 buildings, but has a blank sample-size field. These are values in this particular snapshot; they do not describe today's market.

Preserve percentages and missing values

The value 9.98 means 9.98 percent. If a later calculation needs a decimal fraction, convert it explicitly by dividing by 100. Do not silently switch between percentage points and fractions.

The blank city-average sample size means the source did not supply it. Converting it to zero would change the meaning of the file. Represent it as None instead.

This standard-library example preserves those distinctions:

import csv
from decimal import Decimal

with open("dubai-community-yields-june-2026.csv", newline="", encoding="utf-8") as source:
    rows = list(csv.DictReader(source))

records = []
for row in rows:
    sample_text = row["Yield sample size"].strip()
    records.append({
        "community": row["Community"],
        "yield_percent": Decimal(row["Gross rental yield percent"]),
        "buildings_tracked": int(row["Buildings tracked"]),
        "yield_sample_size": int(sample_text) if sample_text else None,
        "snapshot": row["Snapshot"],
    })

communities = [r for r in records if r["community"] != "Dubai city average"]
city_rows = [r for r in records if r["community"] == "Dubai city average"]

for record in communities:
    print(record["community"], record["yield_percent"], record["yield_sample_size"])

Decimal keeps the recorded decimal representation explicit. The original CSV remains unchanged, so another reader can distinguish source values from transformations in your code.

Do not manufacture an aggregation method

Separating city_rows prevents the city aggregate from being treated as a thirteenth community. It also avoids accidentally including the aggregate in an average of community rows.

The snapshot does not specify the sample unit, weights or aggregation method. Buildings tracked therefore cannot be assumed to be the yield denominator. The displayed community rows cannot independently reconstruct the city-average yield. A chart can show recorded values, but its caption should name the June 2026 period and explain that limitation.

The CSV also does not include the underlying rents and property prices needed to recompute yields. Gross rental yield is different from net yield or total return. Service charges, vacancy, maintenance, financing and transaction costs are not fields in this file. It contains no price forecast or guaranteed future return.

Make reuse traceable

The owned dataset is released under CC BY 4.0. When sharing an adaptation, credit dataHabibi, identify the June 2026 snapshot, include the licence and indicate changes. The dataset release does not include rights to the logo or commercial platform source code.

The practical lesson is small: retain the source file, describe its units, preserve blanks, separate aggregate rows and state what the data cannot establish. Those choices make a simple CSV analysis easier for another developer to inspect and reproduce.

FMCSA Served 16,414 Suspension Notices in Four Months. Five of the 280 I Checked Owned No Trucks.

Thu, 01 Oct 2026 06:11:36 +0000

Originally published at vibeagentmaking.com.

On January 16, 2026, the last piece of a federal rule about freight brokers came into force. Under 49 CFR 387.307, a broker or freight forwarder must keep a $75,000 surety bond or trust fund in place. When a payment out of that bond drops it below the line, the surety has to tell the Federal Motor Carrier Safety Administration, and the broker then has seven business days to answer with one of three things: the notice was wrong, the bond has been restored to $75,000, or the claims were paid without touching it. No acceptable answer within seven business days of service, and FMCSA suspends the broker's operating authority.

That is a real power, and the trade press read it as one: a clock that can take a broker out of business in a week and a half. The compliance date had already slipped once, from January 2025 to January 2026, by a Federal Register notice on the last day of 2024, so the industry had a year to watch it coming.

The obvious follow-up question is how many brokers the government has actually suspended since. The government publishes the file that answers it. What the file says is not what the coverage implied.

The count

FMCSA's suspension record on data.transportation.gov holds 18,269 orders. Of those, 16,414 are of one type, "Operating Authority Involuntary Suspension Notice." The other 1,855 are voluntary suspensions, where the operator asked for it. The serve dates run from May 18 to September 12, 2026, a span of 118 days.

Sixteen thousand involuntary suspensions in 118 days is 139 a day. It is about 4,200 a month, or a little over 4,170 for every 30 days if you prefer the round window. Ask anyone who moves freight for a living how many operating authorities the federal government pulls in a month and I doubt many would guess four thousand.

So the January rule sits inside a suspension programme running at industrial scale. The question is whether the rule is what's driving it.

Who is being suspended

The suspension file has a limitation the coverage never mentions: it identifies each entity by USDOT number and carries no docket number. A USDOT number tells you nothing about what kind of business holds it. A trucking company, a broker and a freight forwarder all have one. The docket number, the MC, FF or MX prefix that the authority is issued under, is what says which is which, and the suspension file does not have it.

The registry does. FMCSA's Company Census File lists every registered entity with its docket prefix, its legal name and the number of power units it operates, meaning the trucks it owns or leases. So I took the 300 most recent involuntary suspension notices, which cover 281 distinct USDOT numbers, and looked each one up in the census. Two hundred and eighty matched, a join rate of 99.6 percent.

docket prefix of the suspended entity	count	share of the 280
MC (motor carrier or property broker)	264	94.3%
no docket in the census record	14	5.0%
FF (freight forwarder)	1	0.4%
MX (Mexico-domiciled carrier)	1	0.4%

One freight forwarder in 280. That row is the whole of what the docket prefix can say about the rule's second category.

For brokers the prefix cannot say anything on its own, because MC covers both property brokers and motor carriers. The census has a second column that can. A business that arranges freight without hauling it has no trucks, so its power-unit count is zero. Of the 280 matched entities, five have zero power units. That is 1.8 percent.

Those five are named in the public record, and a reader who wants to check the join can open the rows one at a time: Get A-Long Logistics LLC, USA Airfish Logistic LLC, Champion Transportation and Logistics LLC, Solidify Logistics LLC, Dorpojo Express LLC. Naming them says nothing about why any of them was suspended. The file does not record a reason, and neither do I. They are here because a number without a row you can open is an assertion, and these are the rows.

The other 275 entities own trucks. Whatever suspended them, it was not a broker bond.

What that makes the January rule

On this evidence the broker-suspension power is not driving the volume. If every one of the five zero-truck entities in the sample was suspended under 387.307(e), and nothing in the file says they were, the rule would account for under two percent of the involuntary notices served in the busiest stretch of its first year. The remaining 98 percent are pointed at trucking companies.

What suspends a trucking company's operating authority thousands of times a month is, in the industry's standard account, insurance. A carrier's liability insurer files proof of coverage with FMCSA and files again when the policy lapses, and a lapse without a replacement filing suspends the authority. I want to be careful about the status of that sentence. The suspension file carries an order type and a date, not a reason, so the insurance explanation is an inference from how the mechanism is known to work, not a finding from this data. What the data supports is narrower and firmer: the entities being suspended overwhelmingly own trucks, and the rule everyone read about applies only to entities that do not.

What the record cannot tell you

Three gaps, each of them measured rather than assumed.

The suspension history begins on May 18, 2026. The rule commenced on January 16. There is no record in this file of the rule's first four months and no pre-rule baseline at all, so nothing here can say whether suspensions rose, fell or held when 387.307(e) arrived. Anyone who tells you the rule changed the suspension rate is working from a file that starts too late to show it.

The revocation file, a separate record of a separate order, has deep history: 1,529,083 rows. It can be cut by docket prefix, and 5,944 of its rows are FF dockets. Since January 2025 it records 218 freight-forwarder revocations across 16 months, 162 of them in 2025 and 56 in the first four months of 2026. Then the series stops. Its last month is April 2026, five months before the day I read it. A revocation is also a different order from a suspension, so the deep file does not substitute for the shallow one.

And the docket prefix only half-separates the population. FF isolates forwarders cleanly. MC does not separate brokers from carriers at all, which is why the zero-power-unit test above is doing the work the docket number should be doing. That test is a proxy. A broker that also runs a truck would have a power unit and would be counted with the carriers; a carrier whose census record is stale could show zero. Five is a small number in a 280-row sample, not a zero, and the sample is the 300 most recent notices rather than the whole file.

The operator's takeaway

Every trade article about 387.307 was, in effect, a warning to brokers about a week-and-a-half clock. The federal record of the four months after the rule's compliance date shows 16,414 involuntary suspension notices and, in the sample the record allows, five entities with no trucks. The lever exists. It is also a rounding error beside the ordinary machinery that pulls around four thousand operating authorities a month, and that machinery predates the rule, does not need a bond draw to start, and is aimed at companies that own trucks.

If you run a fleet, the thing most likely to suspend your authority is the boring thing, and it was already there before anyone wrote about the new one.

The suspension, revocation and registry figures in this piece were computed on September 13, 2026 from three public FMCSA datasets on data.transportation.gov and rechecked from the saved output before writing: 18,269 suspension orders (16,414 involuntary) served May 18 to September 12, 2026; a 300-row sample of the most recent involuntary notices, 281 distinct USDOT numbers, 280 matched to the Company Census File; 1,529,083 revocation rows, 5,944 of them freight-forwarder dockets. Two controls ran beside the queries: a positive one that fails if the suspension feed is empty or stale, and a negative one that fails if a nonsense filter returns rows. Both passed. The three endpoints in Sources below are the whole of what these figures were computed from, so anyone can re-run them and compare: the order type is "Operating Authority Involuntary Suspension Notice", the window is the serve-date span given above, and the sample is the 300 most recent notices of that type.

Sources:

FMCSA, Motus RevokeSuspend, All With History (suspension orders by USDOT number, order type and serve date). https://data.transportation.gov/resource/wb4f-neki.json, read September 13, 2026.
FMCSA, Revocation, All With History (revocation orders by docket number). https://data.transportation.gov/resource/sa6p-acbp.json, read September 13, 2026.
FMCSA, Company Census File (docket prefix, legal name, power units, status). https://data.transportation.gov/resource/az4n-8mr2.json, read September 13, 2026.
49 CFR § 387.307, Property broker surety bond or trust fund, paragraphs (a) and (e); the section states it is effective January 16, 2026. https://www.law.cornell.edu/cfr/text/49/387.307
FMCSA, Broker and Freight Forwarder Financial Responsibility, final rule, Federal Register Vol. 88 No. 220, November 16, 2023. https://www.govinfo.gov/content/pkg/FR-2023-11-16/html/2023-25312.htm
FMCSA, Broker and Freight Forwarder Financial Responsibility; Extension of Compliance Date, 89 FR 107021, December 31, 2024 ("Brokers, freight forwarders, surety providers, and financial institutions must comply with all the provisions of Sec. 387.307 beginning on January 16, 2026."). https://www.govinfo.gov/content/pkg/FR-2024-12-31/html/2024-30509.htm

Forensic Aviation Analytics: Auditing Fuel Bleeds & Delay Cascades on a Regional A320neo Network

Mon, 28 Sep 2026 11:41:49 +0000

When evaluating airline performance, typical analytics dashboards rely blindly on passenger-facing "On-Time" arrival percentages. However, from an operational and financial engineering standpoint, these high-level metrics can easily mask millions of dollars in silent cash destruction.

Due to strategic schedule padding, a short-haul flight can sit idling its engines on a congested tarmac for 20 minutes, yet still be registered as "On-Time" to the customer. The passenger is happy, but the airline’s finance department just watched pure margin vaporize into thin air.

To expose these hidden friction layers, I engineered LCC-Ops: an end-to-end data warehouse auditing over 120,000 regional transaction legs modeled around the single-aisle network performance of flynas across key GCC hubs.

Here is the technical architecture and the forensic database methodologies behind the pipeline.

The Tech Stack & Architecture

Database Layer: MySQL Server 8.0 (Advanced CTEs, Mathematical Window Functions, Schema Migrations)
Business Intelligence: Power BI Desktop (Data Type Optimization, Cross-Table Iterators, UI Design)
Core Fleet Constants: Engineered around the Airbus A320neo (CFM LEAP-1A26 engine) utilizing official ICAO Engine Emissions Databank metrics. At a standard 7% ground idle taxi thrust, this engine burns exactly 0.0940 kg of fuel per second, per engine

The Database Engineering Breakdown

Pillar 1: Isolating Tarmac Cash Destruction (Domain A)To find the true cost of airport ground bottlenecks, the ingestion engine isolates rows inside the transactions table (fact_flights) where the Out-Gate to Runway Takeoff latency (taxi_out) exceeded a standard 4-minute engine stabilization and warming buffer.

By pulling the real-world ICAO fuel coefficients down via relational keys, the calculation bypasses text-based padding and computes raw physical fuel destruction.

SELECT 
    f.origin AS departure_hub,
    COUNT(f.tail_number) AS total_monitored_flights,
    ROUND(AVG(f.taxi_out), 2) AS avg_taxi_out_minutes,
    -- Formula: (Actual Taxi Time - 4 Min Buffer) * 60 Sec * ICAO Idle Flow Rate * 2 Engines
    ROUND(SUM((f.taxi_out - 4) * 60 * e.fuel_flow_idle_kg_sec * 2), 2) AS total_wasted_fuel_kg
FROM fact_flights f
JOIN dim_aircraft a ON f.tail_number = a.tail_number
JOIN dim_engine e ON a.engine_identification = e.engine_identification
WHERE f.reporting_airline = 'XY' AND f.taxi_out > 4
GROUP f.origin;

Pillar 2: Tracking Cross-Border Delay Cascades (Domain D)
When an international flight arriving from an airport like Karachi (KHI) experiences customs or airspace delays, that latency risks cascading into the aircraft’s subsequent domestic legs (e.g., RUH ➔ JED).

To track this chronologically across individual daily airframe lifecycles, I utilized advanced LEAD() mathematical window functions

WITH ranked_daily_schedule AS (
    SELECT 
        flight_date, tail_number, origin, dest, carrier_delay AS leg_1_delay,
        LEAD(origin, 1) OVER (PARTITION BY tail_number, flight_date ORDER BY taxi_out DESC) AS leg_2_origin,
        LEAD(carrier_delay, 1) OVER (PARTITION BY tail_number, flight_date ORDER BY taxi_out DESC) AS leg_2_delay
    FROM fact_flights
    WHERE reporting_airline = 'XY'
)

Pillar 3: Menu Logistics & Hangar Risk Automation (Domains B & C)
Catering Spoilage Optimization: Short flight windows on regional tracks mean cabin crews lack the time to complete complex perishable sales loops. The data models validate flynas’s lean, shelf-stable buy-on-board menu model, proving that switching to fresh profiles would trigger an immediate 54% inventory spoilage cash bleed

Automated Hangar Reorder Points (ROP): The warehouse tracks rolling airframe hours against specific tail numbers (HZ-NS1 to HZ-NS5) [Query-relevant Context]. If a component approaches its high-wear safety limit, the system triggers automated database alerts to protect the airline against catastrophic $25,000/day Aircraft on Ground (AOG) hangar penalties

UI Engineering in Power BI

To present these data layers to corporate stakeholders, I built an interactive application-style interface directly inside Power BI using flynas’s signature branding tones.

The biggest challenge during BI ingestion was data type conversions—empty strings or trailing spaces in the raw logs caused math functions to drop blank values (--). By implementing upfront string transformations (Uppercase/Trim) and embedding data validation wrappers like ISNUMBER(VALUE()) directly into row-level DAX iterators, the front-end dynamically scales across massive transaction volumes without data loss.

The entire production code schema and scripts are live on my GitHub Link. I'd love to hear how other data engineers manage real-time fuel-burn parameters or multi-leg network routing in your pipelines!

Monte Carlo Permutation Testing in Python: How Random Trade Order Falsifies Your Equity Curve

Mon, 28 Sep 2026 06:17:46 +0000

Your backtest equity curve is merely one historical trajectory out of millions of equally probable alternatives.

The total profit of your trading system is invariant to order: 100 closed trades will yield the exact same ending dollar return whether the winning trades arrive first or last. However, path-dependent survivability metrics—Maximum Drawdown, Ulcer Index, Margin Call Probability, and Time-to-Recovery—are dominated by trade sequence.

If a cluster of 6 normal consecutive losses strikes at trade #1 instead of trade #70, an account operating under a strict 10% risk floor (common in institutional mandates and proprietary evaluation rules) is liquidated before the positive edge ever materializes.

In this article, we formulate and implement a vectorized Monte Carlo permutation engine in Python to rigorously falsify equity curves before committing live capital.

1. Permutation vs. Bootstrap: The Critical Distinction

Many trading platforms market "Monte Carlo analysis" without disclosing their resampling methodology. In quantitative finance, the distinction is fundamental:

Dimension	Permutation Testing (Without Replacement)	Bootstrap Resampling (With Replacement)
Mechanism	Random shuffle of historical trade order	Random draw where any trade can be selected $k$ times
P&L Distribution	100% Identical to sample (same mean, win rate, skewness)	Creates synthetic distributions with altered win rates
Statistical Target	Isolates sequence risk alone	Estimates sampling error under $i.i.d.$ assumption
Vulnerability	Assumes trades are conditionally independent	Heavily distorts tail risk if outliers are oversampled

For validating an existing trading journal or strategy backtest, Permutation Testing is the strict falsification baseline. It asks the minimal, humble question: "Given the exact trades we actually took, what percentage of alternate timelines would have breached our risk budget?"

2. The 4 Path Metrics That Expose Hidden Ruin

Looking solely at backtest Sharpe ratio or nominal net profit hides ruin. When running $N=5,000$ permutations, evaluate these four metrics:

Probability of Ruin ($P_{\text{ruin}}$): The percentage of shuffled paths whose equity breaches the hard drawdown barrier (e.g., $9,000$ on a $\$10,000$ starting balance) at any point along the horizon.
Median Max Drawdown ($MDD_{50}$): The central tendency of drawdown. If your backtest showed a 7% drawdown but the median shuffled drawdown is 16%, your historical curve was an unusually lucky sequence.
95th Percentile Max Drawdown ($MDD_{95}$): The stress-test boundary. 95% of simulated timelines experienced a drawdown less severe than this number; 5% suffered worse. This is your true operational capital requirement.
Drawdown Dispersion Ratio: $MDD{95} / MDD{50}$. A high ratio indicates severe tail-sequence vulnerability.

3. Vectorized Python Implementation (NumPy)

Iterating across 5,000 simulations using Python loops is computationally inefficient. Below is a high-performance, fully vectorized implementation using numpy.random.default_rng().permuted:

import numpy as np
from typing import Dict, Any

def run_monte_carlo_permutation(
    trades: np.ndarray,
    initial_capital: float = 10000.0,
    ruin_capital: float = 9000.0,
    num_simulations: int = 5000,
    seed: int = 42
) -> Dict[str, Any]:
    """
    Vectorized Monte Carlo Permutation Test (Resampling Without Replacement).
    Evaluates sequence risk and ruin probability on trade P&L arrays.
    """
    trades = np.asarray(trades, dtype=np.float64)
    n_trades = len(trades)
    if n_trades < 10:
        raise ValueError("At least 10 trades required for meaningful permutation testing.")

    rng = np.random.default_rng(seed)
    
    # 1. Broadcast and independently shuffle across simulations (without replacement)
    sim_matrix = np.tile(trades, (num_simulations, 1))
    shuffled_trades = rng.permuted(sim_matrix, axis=1)

    # 2. Vectorized cumulative equity trajectories
    cumulative_pnl = np.cumsum(shuffled_trades, axis=1)
    initial_col = np.full((num_simulations, 1), initial_capital)
    equity_paths = np.hstack([initial_col, initial_capital + cumulative_pnl])

    # 3. Peak equity and path drawdowns
    running_peaks = np.maximum.accumulate(equity_paths, axis=1)
    drawdowns_dollar = running_peaks - equity_paths
    drawdowns_pct = (drawdowns_dollar / running_peaks) * 100.0

    # 4. Max drawdown per trajectory
    max_mdd_pct = np.max(drawdowns_pct, axis=1)

    # 5. Ruin evaluation (breaching floor at any point in time)
    min_equity_per_path = np.min(equity_paths, axis=1)
    ruin_paths = np.sum(min_equity_per_path <= ruin_capital)
    prob_ruin = (ruin_paths / num_simulations) * 100.0

    return {
        "n_trades": n_trades,
        "initial_capital": initial_capital,
        "ruin_capital": ruin_capital,
        "nominal_net_pnl": float(round(np.sum(trades), 2)),
        "num_simulations": num_simulations,
        "prob_ruin_pct": float(round(prob_ruin, 2)),
        "mdd_p05_pct": float(round(np.percentile(max_mdd_pct, 5), 2)),
        "mdd_p50_median_pct": float(round(np.percentile(max_mdd_pct, 50), 2)),
        "mdd_p95_worst_pct": float(round(np.percentile(max_mdd_pct, 95), 2)),
        "falsification_verdict": "REJECT_EXCESSIVE_RUIN" if prob_ruin > 5.0 else "PASS_ROBUST_SEQUENCE"
    }

4. Empirical Case Study: The "Profitable" Strategy That Blows Up

Consider a 100-trade sample with a 52% win rate and positive net expectation:

52 winning trades (mean profit ~\$170)
48 losing trades (mean loss ~-\$170, with occasional -\$350 tail losses)
Nominal Net P&L: +$505.39 on a \$10,000 account (profitable on paper).

When we execute run_monte_carlo_permutation over 5,000 shuffles:

# Empirical verification run
result = run_monte_carlo_permutation(
    trades=trades_sample,
    initial_capital=10000.0,
    ruin_capital=9000.0, # 10% maximum drawdown ceiling
    num_simulations=5000,
    seed=42
)

print(result)

Exact Output:

{
  "n_trades": 100,
  "initial_capital": 10000.0,
  "ruin_capital": 9000.0,
  "nominal_net_pnl": 505.39,
  "num_simulations": 5000,
  "prob_ruin_pct": 42.64,
  "mdd_p05_pct": 11.3,
  "mdd_p50_median_pct": 16.77,
  "mdd_p95_worst_pct": 25.5,
  "falsification_verdict": "REJECT_EXCESSIVE_RUIN"
}

The Institutional Reality:

Even though the original single backtest ended with +$505.39 net profit, 42.64% of all possible alternate histories breached the 10% drawdown barrier.
In 95% of histories, the drawdown reached up to 25.50%—more than double the allowable risk budget.
Without Monte Carlo permutation, a developer deploying this system would falsely attribute failure to "market regime change" or "bad luck", when in reality the strategy was statistically insolvent against basic sequence risk from day one.

5. Methodological Boundaries (What Monte Carlo Cannot Do)

A rigorous quant must understand the falsification limits of permutation testing:

Serial Autocorrelation Breakdown: Permutation randomly destroys the temporal ordering of trades. If your strategy has positive autocorrelation in losses (e.g., clustered losses during high-volatility macro announcements), permutation may underestimate clustering severity.
Out-of-Distribution Shocks: Shuffling only explores combinations of events that already occurred. It cannot simulate a 5-sigma liquidity flash crash if none was present in your historical log.
Journal vs. Market: Monte Carlo evaluates the statistical fragility of the trade sequence, not the underlying market microstructure.

6. Open Source Implementation & References

Scientific DOI: Reproducible quant code registered under CERN Zenodo: 10.5281/zenodo.22012203.
GitHub Repository: guetaquant-byte/guetaquant-tools (AGPLv3).
Interactive Tool: Browser-based Monte Carlo permutation is integrated into the client-side Local-First Trading Journal.
In-Depth Study: Full mathematical derivation and percentile bands at Gueta Quant Monte Carlo Analysis.

Aviso Regulatorio (SFC Colombia — Decreto 2555 de 2010): Este artículo tiene un propósito 100% pedagógico, educativo y de investigación en ingeniería de software financiero. Gueta Quant no presta asesoría financiera ni emite señales de inversión.

By Mahdi Goodarzi (g.dev/mahdigoodarzi), Founder of Gueta Quant.

$1.45 Billion in Federal AI Contracts, and the Ledger Can Account for 17% of It

Fri, 25 Sep 2026 15:09:49 +0000

On October 31, 2019, the Department of Commerce signed an award with Accenture Federal Services to "improve patent search with artificial intelligence." USAspending.gov, the federal government's public ledger of what it spends, records the award at $44,076,744.60. Its period of performance ended on April 30, 2023. The same ledger has a column for what was actually paid on the award. That column reads 0.00.

Not blank. Zero, to the cent, on a contract that finished more than three years ago.

One row is an anecdote. So I pulled every federal contract award whose description contains the words "artificial intelligence" and that was signed or active between October 1, 2024, and September 11, 2026. There are 491 of them, across 29 awarding agencies, carrying $1,449,720,817.76 in obligations. The Department of Defense holds $937.6 million of that, just under two thirds. Then I added up the paid column, the one the Accenture row leaves at zero. Across all 491 awards it comes to $249,293,997.21.

That is 17.2 percent. For the other 83 percent of the money, the public record of federal AI spending cannot say what has been paid.

The objection, and what survives it

The obvious answer is that obligations are not payments. An obligation is a promise: the government has committed the money to a contract. An outlay is the check. A contract signed last spring has had no time to bill, so its outlay is zero and there is nothing wrong with that. If the population were mostly new awards, a low paid fraction would be a description of the calendar, not of the ledger.

It is partly the calendar. Of the 491 awards, 184 started in 2025 and 99 in 2026. Nobody should expect those to have paid out. So the honest test is not the total. It is the shape of the column, award by award, and in particular what it says about the awards that have had years to bill.

Here is the shape. USAspending's award-level outlay field answers the question "what was paid" in five different ways on this population.

what the outlay column says	awards	obligations	share of dollars
the field is absent entirely	269	$488,560,429	33.7%
exactly zero	98	$409,146,653	28.2%
a partial figure	77	$389,522,445	26.9%
equal to the obligation, to the cent	45	$160,754,616	11.1%
more than the obligation	2	$1,736,674	0.1%

The first two rows are the finding. A third of the dollars sit on awards where the ledger has no outlay field at all, not a zero, nothing. Another 28 percent sit on awards where the field exists and reads exactly zero. Only the bottom three rows, 38 percent of the money, carry a number that could be read as a payment, and the last of those rows is a number that cannot be right.

The zeros that had time

Take only the awards that began before 2024, which have had at least twenty months inside this window to bill. Twenty-four of them report exactly zero outlays, with $100.9 million obligated between them. The Accenture patent-search award is the largest. Another Accenture award for the same agency, $13.3 million, ran from May 2023 to June 2024 and reports zero. A DARPA award to Kitware, $4.8 million, ran from November 2021 to the end of 2024 and reports zero. Two more DARPA awards from the same week, to SRI International and to RTX's BBN unit, together $8.8 million, report zero. A MITRE award for Commerce, $4.2 million, ran from September 2021 to November 2024 and reports zero.

A further 24 pre-2024 awards, $47.1 million, have no outlay field at all.

These are not contracts that have not had time. Several of them are over. The work was, by the government's own dates, delivered and closed, and the column that is supposed to record the money going out records either nothing or none.

The largest single award in the population points the same way from a different angle. ECS Federal's Army contract to build prototype AI and machine-learning algorithms, signed in April 2020 and running to March 2027, carries $120,575,059.35 in obligations and $4,671,161.19 in outlays. That is 3.9 percent, six years into a seven-year contract with 47 subawards under it. Either almost nothing has been paid on the Army's biggest named AI contract, or the column does not know.

The documentation dates the problem

USAspending is candid about the mechanism, and it puts a date on it. Its "About the Data" guide says: "Beginning in Fiscal Year 2022, all agencies are required to submit their financial data on a monthly basis, including outlay data for award spending. Any outlay data before this period were optional for agencies to report, and thus may be incomplete."

Read that against the table. Optional reporting before fiscal 2022 explains an empty field on a 2019 award. It does not explain a zero on an award whose entire performance ran after the requirement took effect, and it does not explain the 2025 award to ECS Federal, $72.8 million for AI research and deployment, which sits in the mandatory era with an outlay of 0.00. Nor does the calendar. The requirement has been in force for four fiscal years. The awards with the largest gaps between promise and recorded payment are the ones that spent the most time inside it.

The site's methodology page describes how this happens. Agencies report award-level obligations and outlays in a file the system calls File C, and that file is linked to the contract records that give an award its description and its recipient. When the two do not link, the award still appears in searches, with its obligation, and the outlay side is simply not there. The Government Accountability Office reviewed federal spending-transparency data quality in 2022 and found the agency plans for it had gaps, outlay tracking among them. None of this is hidden. It is documented, dated, and then routinely ignored by everyone who quotes the obligation figure as if it were the spend.

Two awards that paid more than they promised

The cheapest demonstration that the outlay column is not a clean measurement is the last row of the table. Aperio Global's award 70SBUR24F00000223 shows $1,538,850.00 obligated and $1,539,034.52 paid. HCC Consulting's award 72MC1025C00003 shows $197,824.49 obligated and $198,319.43 paid. Between them, the ledger reports $679.46 in payments on money that was never committed.

Small amounts. Possibly a modification recorded on one side and not the other. But a column that can exceed its own ceiling is a column being filled by hand, from different systems, on different schedules, and that is the same column the 17.2 percent comes from. The 45 awards where outlays equal obligations to the cent deserve the same suspicion in the other direction. Tuknik Government Services' Interior award for AI support to the Pentagon's Joint Artificial Intelligence Center shows $106,725,460.68 obligated and $106,725,460.68 paid. That can be true. It is also exactly what a field looks like when someone copied the obligation into it.

The arithmetic corrected itself once

A piece about a ledger that cannot add up should show its own sums being checked. The first pass at these numbers used the 501 rows the search endpoint returned. Only 491 of them were distinct awards. Ten were repeats of rows already returned on an earlier page, a Stevens Institute award among them, and the uncorrected obligation total was $1,496,023,651.74, which is $46.3 million too high. Every figure here is computed with the duplicates dropped on the award's own identifier, and the count of dropped rows is reported rather than quietly applied, so anyone re-running the query can tell whether the endpoint's behaviour has changed rather than inheriting a silent fix.

The correction also moved a number in the research file this essay was written from. That file counted 26 pre-2024 awards with zero outlays and $111.8 million between them; on the deduplicated table the count is 24 and the sum $100.9 million. The essay carries the deduplicated figures, and the difference is two rows that the API served twice.

What the number is, and what it is being read as

Every procurement announcement, agency press release and chart of "federal AI spending" uses obligations, because obligations are the number that exists for every award. The column that answers the question a taxpayer actually asks, what has been paid, is populated well enough on this population to answer for 17.2 percent of the money, and for nearly a third of it the field does not exist.

That matters more for artificial intelligence than for most categories, because these are the contracts cited as evidence of a federal AI build-out. A $1.45 billion figure that is 62 percent unaccounted for in the payment column is a measurement of intent. It is being read as a measurement of activity. The distinction can be recovered, but only by reading the column that is mostly empty, and by saying which awards it is empty on.

None of this says the money was wasted or the work not done. An absent outlay is a reporting fact. The Accenture patent-search contract may have delivered exactly what it promised; the ledger has no way to tell you, and neither do I. What the record supports is narrower and, I think, more useful: when someone tells you how much the federal government is spending on artificial intelligence, ask which column they read.

The federal award figures in this piece were computed from the USAspending API v2 on 12 September 2026 and rechecked from the saved response before writing: 491 distinct contract awards, $1,449,720,817.76 obligated, $249,293,997.21 in the award-level outlay column. The query is the one in Sources below, so anyone can re-run it and compare. Two controls ran beside it: a positive one, in which the ECS Federal award must appear at its known obligation, and a negative one, in which a nonsense description filter must return nothing. Ten duplicate rows were dropped and counted. The population is awards whose description contains "artificial intelligence"; it will include boilerplate mentions and exclude AI work described as machine learning or autonomy, and the claim is about what the ledger says of the awards that call themselves AI, not about all of them.

Sources:

USAspending API v2, POST /api/v2/search/spending_by_award/, filters: award types A, B, C, D; time period 2024-10-01 to 2026-09-11; description "artificial intelligence". Run 12 September 2026. https://api.usaspending.gov/api/v2/search/spending_by_award/
USAspending API v2, GET /api/v2/awards/CONT_AWD_W911QX20C0023_9700_-NONE-_-NONE-/ (the ECS Federal award detail, including the subaward count and period of performance).
USAspending.gov, About the Data, last updated 19 November 2025, the section on agency financial data submissions (the FY2022 monthly outlay requirement and the note that earlier outlay data were optional). https://www.usaspending.gov/data/about-the-data-download.pdf
USAspending.gov, Data Sources & Methodology for Agency Submission Statistics (File C as the award-level obligation and outlay file, and unlinked File C awards). https://www.usaspending.gov/submission-statistics/data-sources
U.S. Government Accountability Office, GAO-22-105427, Federal Spending Transparency (2022). https://www.gao.gov/assets/gao-22-105427.pdf

Let the LLM Write the Parser, Never Run It: How labparse Handles Unknown Instrument Files

Thu, 24 Sep 2026 12:58:05 +0000

If you have ever written a one off script to read a CSV that was not really a CSV, this one is for you. It is a small open source Python package built around a pattern more of us should borrow: the model writes code once, and deterministic code does every run after that.

The Problem: Every Instrument Has Its Own Export Format

Lab instruments are a great stress test for parsers. A plate reader writes a metadata header, then a grid of wells per reading cycle, then maybe a temperature column, all in a layout that changes between vendors and sometimes between firmware versions. Most labs end up with a folder of throwaway scripts, one per instrument, each quietly breaking the next time someone changes an export setting.

Recipes Instead of Scripts

labparse splits the job in two. A format is described by a recipe, a small JSON file that says where the header ends, how the grid is laid out, and how time and cycle numbers are read. A deterministic engine executes the recipe. It ships with recipes for Tecan ASCII exports, SoftMax Pro and SpectraMax kinetic files, generic wide and long CSV or TSV tables, and Excel workbooks with a plate grid.

Every parse produces the same tidy schema whatever the instrument: well, row, col, cycle, time_s and value, plus any extra columns like temperature.

import labparse
result = labparse.parse("growth.asc")
result.data        # tidy pandas DataFrame
result.wide()      # one column per well
result.save_csv("tidy.csv")

Where the LLM Comes In, and Where It Does Not

When labparse meets a format it has no recipe for, it shows the raw file to your own model once and asks it to write a new recipe. That is the only job the model gets. The recipe is then run by the same engine against your actual file, and it is only accepted if the parse succeeds. A wrong recipe is rejected and retried, so a bad guess can never turn into silently wrong numbers.

Accepted recipes are cached in your home directory, so each new format costs at most one model call ever, and cached formats parse offline. Recipes are plain JSON, which means you can read them, fix them by hand, and share them with the rest of your lab.

For the model it checks, in order: the Claude CLI, the Codex CLI, an Anthropic API key, then an OpenAI API key, and a base URL setting points it at any OpenAI compatible server, local models included. With no model configured, everything with a bundled or cached recipe still works, and an unknown format gets a clear message instead of a guess.

Why This Pattern Travels

The part worth taking to other projects is the split. The model proposes, the deterministic engine disposes, and your data only ever flows through code you can inspect. It fits log formats, bank statement exports, legacy reports, anything where the variety of inputs is the real problem and the correctness of the output is not negotiable.

The Takeaway

Let the LLM write the parser, verify it against real input, cache the result, and never let it touch the numbers directly. labparse is MIT licensed, installs with pip install labparse, and the project page above links to the source and issue tracker.

Two Complaints in Two Years: New York City's AI Hiring Law Made a Record Nobody Can Read

Thu, 24 Sep 2026 05:51:26 +0000

The state Comptroller found two complaints in twenty-four months. Cornell's team found 14 audit reports among 267 employers. Both measurements hit the same wall: a disclosure rule whose scope is set by the discloser produces a record that cannot tell silence from exemption.

Between July 2023 and June 2025, New York City's Department of Consumer and Worker Protection received two complaints about employers using automated hiring tools without the bias audit the city requires. Two, in twenty-four months. The count comes from the New York State Comptroller, whose auditors spent that window's records deciding whether the department had "designed and implemented an effective system to enforce compliance with Local Law 144." Their answer, published on December 2, 2025, was that the department's complaint process "is ineffective in ensuring that all complaints related to non-compliance with LL144 are routed to DCWP."

The auditors then did the thing a complaint count cannot do for itself: they went and looked. The department had surveyed 32 companies about their use of these tools and found one non-compliance issue. The auditors examined the same 32 companies and found 17 potential instances. And the department's own words for the limit it works under, quoted in the audit, are these: "If an employer does not take these steps, it's difficult to identify non-compliance."

That sentence is the subject of this piece. It sounds like an admission of under-resourcing. It is a description of the law's design, and it means the near-empty record the city has built cannot be read as good news or as bad.

What the law asked for

Local Law 144 of 2021 prohibits employers and employment agencies from using an automated employment decision tool unless the tool "has been subject to a bias audit within one year of the use of the tool, information about the bias audit is publicly available, and certain notices have been provided to employees or job candidates." Enforcement began on July 5, 2023. The penalties are small per unit and multiply by time: not more than $500 for a first violation, $500 to $1,500 for each subsequent one, and each day a tool is used in violation counts as a separate violation.

Read the requirement again and notice what kind of law it is. It does not tell employers to do something to candidates. It tells them to publish something about themselves. The audit summary must be public. The notice must be given. Compliance leaves a paper trail by definition, which means the law creates its own public record, and anyone with a browser can go and count what is in it.

The first count, 2023

Somebody did. In the autumn of 2023 a team from Cornell's Citizens and Technology Lab, Data & Society and Consumer Reports recruited 155 student investigators and sent them through employer career sites between October 24 and November 9, modelling what a job seeker would see. The paper that came out of it, "Null Compliance: NYC Local Law 144 and the Challenges of Algorithm Accountability," presented at FAccT 2024, reports in its section 6.1: "Among 267 employers posting open jobs in NYC, we found 14 audit reports and 12 transparency notices."

Fourteen of 267 is 5.2 percent. Twelve of 267 is 4.5 percent. The paper's own abstract gives a different figure, 18 audit reports among 391 employers; that is the full set of employers the students recorded rather than the subset with live New York postings, and it works out to 4.6 percent. The two figures are not rival estimates of one quantity, and I am not going to average them. Keep the denominators attached. That habit is going to matter in a moment.

So a few months after enforcement began, roughly one employer in twenty had put anything in it. The obvious reading is that nineteen in twenty were breaking the law. The authors refuse that reading, and the refusal is their contribution.

Null compliance is not non-compliance

The paper names the state it found: "Null compliance describes a state in which the absence of evidence of compliance cannot be ascertained as non-compliance because the investigator lacks the information to determine if the regulated party's actions or products are in scope of the regulation."

The mechanism sits in the law's trigger. Local Law 144 covers tools that substantially assist or replace the human decision. Whether a given piece of software does that is not a property of the software. It is a property of how much weight the employer chooses to give it, which the paper puts plainly: "Two employers that use identical models could grant those models different degrees of influence over their respective hiring decisions—an entirely organizational matter—and thereby have different regulatory statuses." The regulated party decides whether it is in the population.

The survey the researchers ran beside the site walk shows what that decision looks like from the inside. Of the 26 firms that answered, 23 said the law did not apply to them. Eighty-eight percent of a small, self-selected group, so not a rate to build on, but a clean illustration. An employer that posts nothing may be using no such tool, or using one it has decided does not substantially assist anyone, or using one it should have audited and did not. From the career page all three look identical. The absence of a disclosure is the same absence in every case.

So the 5.2 percent is not a compliance rate. It is the share of employers who both fell inside the law by their own reckoning and did what it asked. Nobody outside those employers knows the size of the first group. The denominator that would make the number readable is the one number nobody holds.

The second count, 2025, by the state

Two years later the Comptroller's office measured the same record from the other side, the enforcer's, and hit the same wall. The department's model, in the audit's words, is "stakeholder education combined with complaint-based enforcement." A complaint-based design asks candidates to report a violation. The violation here is an absence: an audit summary that was never posted, a notice that was never given. A candidate who was screened by an unaudited tool and never told is, by construction, a person who does not know it happened. The two complaints in two years are not a measure of how often the law was broken. They are closer to a count of how often a violation of this kind is visible to the person it happened to.

That is why the department's sentence about difficulty is not a resourcing complaint. If an employer does not post, the employer is either outside the law or breaking it, and from the outside there is no third thing to look at. The auditors' 17 potential instances against the department's one, in the same 32 companies, is the only place in either measurement where a second, closer look changed the count. It took the state's access to get it, and it is still labelled potential.

Put the two measurements side by side. Researchers walking the public record in 2023 found it nearly empty and said they could not tell why. State auditors reading the enforcement record in 2025 found it nearly empty and, in effect, said the same. Neither found that the law is working. Neither found that it is failing. Both found that the instrument they were holding could not answer the question, and it is the same instrument: a record whose population is defined by the people in it.

What would make the zero legible

None of this says employers are breaking the law, and I want to be exact about that, because converting "null compliance" into "non-compliance" is the error the paper exists to name. It also does not say the law failed; the state audit examined the department's enforcement system and addressed its six recommendations to the department, not to the statute. And I have counted nothing myself. The counts are the researchers' and the auditors'. The only arithmetic here is division, and the script that does it ships beside this essay with each figure's locator.

What the two measurements do establish is a condition. A disclosure rule whose scope is decided by the party disclosing produces a record that cannot distinguish silence from exemption. Any count taken from that record, high or low, inherits the ambiguity. Two complaints, fourteen audits, one issue found, seventeen suspected: every one of those numbers has the same missing denominator, which is the number of employers who were actually inside the law.

A zero becomes readable when someone other than the regulated party defines the population. A register of employers using such tools, or a duty to declare use rather than only to audit it, would turn the next walk through the career sites into a measurement. Whether New York should want either is a policy question this piece is not placed to settle. What can be said is narrower and holds either way. The city built a public record to find out whether hiring software was being checked for bias. Two years and two instruments in, the record's clearest finding is that it cannot be read.

Reproduction script, published with this piece and runnable with no API key or network access: rates_c5500.py. Every percentage and ratio above is computed there from the published counts, each row carrying its source locator, and the 14-of-267 and 18-of-391 figures are kept as separate populations on purpose.

Sources: Lucas Wright, Roxana Mika Muenster, Briana Vecchione, Tianyao Qu, Pika (Senhuang) Cai, Alan Smith, Jacob Metcalf and J. Nathan Matias, "Null Compliance: NYC Local Law 144 and the Challenges of Algorithm Accountability," FAccT 2024, arXiv:2406.01399, DOI 10.1145/3630106.3658998, sections 2.2, 3, 5.1 and 6.1; Office of the New York State Comptroller, "Enforcement of Local Law 144 – Automated Employment Decision Tools," audit published December 2, 2025, covering July 2023 through June 2025; NYC Department of Consumer and Worker Protection, "Automated Employment Decision Tools" (the requirement and the July 5, 2023 enforcement date); New York City Administrative Code § 20-872 (the penalty schedule); Citizens and Technology Lab, "Studying How Employers Comply with NYC's New Hiring Algorithm Law" (the 18-of-391 figure, cited as the write-up's own).

A disclosure record is only as readable as its population

Local Law 144 asks employers to publish something about themselves, then lets each employer

    decide whether it is covered. The result is a record where an absence proves nothing. The same
    shape shows up wherever an agent's behaviour is self-reported: a claim with no population
    behind it. Chain of Consciousness keeps a record of what an agent actually did, over traffic
    someone else can check, so a rate you publish carries the denominator it was measured over.

pip install chain-of-consciousness
npm install chain-of-consciousness

Or start without installing anything: Hosted Chain of Consciousness.

← Back to blog

Neural Networks: Inspired by the Brain, Built for Data

Wed, 23 Sep 2026 14:01:57 +0000

When I first started learning about neural networks, I found the idea both fascinating and slightly intimidating.

The phrase “neural network” makes it sound like something extremely complex, and honestly, once you start looking at layers, weights, activation functions, forward propagation, backpropagation, and optimization, it can become overwhelming pretty quickly.

But beneath all of that mathematics and code is a surprisingly simple idea:

A neural network learns patterns from data by adjusting itself based on its mistakes.

That idea is what made neural networks interesting to me.

As I continue exploring data science and machine learning, understanding neural networks has helped me see machine learning from a different perspective. Instead of simply telling a computer what rules to follow, we can give it data and allow it to learn patterns that may be difficult for us to define manually.

In this article, I'll break down where the inspiration for neural networks came from, what they are, and the major components that make them work.

Where Did Neural Networks Come From?

The inspiration behind artificial neural networks comes from something we already have:

the human brain.

Our brains contain billions of neurons that communicate with one another. A biological neuron receives signals, processes them, and passes a signal forward when certain conditions are met.

Artificial neural networks don't replicate the human brain perfectly. Instead, they borrow a simplified idea from how biological neurons work.

A biological neuron can be thought of roughly as:

receive signals → process them → produce a response

An artificial neuron follows a similar conceptual process:

receive inputs → calculate a weighted sum → apply an activation function → produce an output

This was one of the things I found interesting when learning about neural networks.

The goal isn't to recreate the brain inside a computer.

The goal is to take a useful idea from biology and turn it into a mathematical model that can learn from data.

So, What Exactly Is a Neural Network?

A neural network is a machine learning model made up of interconnected nodes, commonly called neurons.

These neurons are organized into layers.

A basic neural network usually contains:

Input layer
One or more hidden layers
Output layer

A simple representation looks like this:

Input Layer        Hidden Layer        Output Layer

   x₁  ─────────►    ○
                    │
   x₂  ─────────►    ○ ─────────►     ŷ
                    │
   x₃  ─────────►    ○

The input layer receives the data.

The hidden layers transform the data and learn patterns.

The output layer produces the final prediction.

The interesting part is that we don't manually program the network to recognize every possible pattern.

Instead, the network learns the parameters needed to recognize those patterns from examples.

1. The Input Layer

Everything starts with the input.

Suppose we want to predict whether a student is likely to pass an exam.

Our dataset might contain:

Hours studied
Attendance
Previous test scores
Assignment completion

These become our input features.

For example:

Hours Studied     = 6
Attendance        = 85%
Previous Score    = 72
Assignments       = 9

The neural network receives these values through the input layer.

One important thing I learned is that neural networks generally work best when the input data has been appropriately prepared.

That can involve:

Handling missing values
Encoding categorical variables
Scaling numerical features
Removing irrelevant features
Splitting data into training and testing sets

So even though neural networks are powerful, data preparation still matters.

Garbage in can still mean garbage out.

2. Neurons

The basic building block of a neural network is the artificial neuron.

A neuron takes several inputs, assigns different importance to them using weights, adds a bias, and then passes the result through an activation function.

Mathematically, we can represent this as:

\[ z = w_1x_1 + w_2x_2 + ... + w_nx_n + b \]

Where:

$x$ = input
$w$ = weight
$b$ = bias
$z$ = weighted sum

The neuron then applies an activation function:

\[ a = f(z) \]

The result becomes the neuron's output.

This might look complicated initially, but the basic idea is straightforward:

The neuron combines information from its inputs and decides how strongly to activate.

3. Weights

Weights determine how important different inputs are to a neuron.

Imagine we're predicting whether a student will pass.

Perhaps hours studied is more predictive than the number of assignments completed.

The network can learn this by assigning different weights to those features.

For example:

Hours studied       → weight = 0.8
Attendance          → weight = 0.5
Previous score      → weight = 0.9
Assignments         → weight = 0.3

These numbers are only examples.

During training, the neural network continuously adjusts its weights to improve its predictions.

This is one of the key ideas behind learning in neural networks.

The model isn't explicitly told what the correct weights are. It learns them from data.

4. Bias

Another component is the bias.

Bias gives the neuron some flexibility in determining when it should activate.

Without going too deep into the mathematics, you can think of the bias as an additional parameter that allows the activation function to shift.

The basic equation becomes:

\[ z = wx + b \]

The weights control the influence of the inputs, while the bias helps adjust the overall output.

Both weights and biases are learned during training.

5. Activation Functions

This was one of the concepts that really helped me understand why neural networks can learn complex patterns.

After calculating the weighted sum, a neuron applies an activation function.

Why?

Because without activation functions, stacking multiple layers of neurons would still result in a fundamentally linear transformation.

Activation functions introduce non-linearity.

And real-world data is rarely perfectly linear.

Some commonly used activation functions include:

Sigmoid

The sigmoid function produces values between 0 and 1.

\[ \sigma(x) = \frac{1}{1 + e^{-x}} \]

This makes it useful in certain binary classification problems, particularly as an output activation.

For example:

0.91 → high probability of class 1
0.12 → low probability of class 1

ReLU

ReLU stands for Rectified Linear Unit.

It is defined as:

\[ ReLU(x) = \max(0,x) \]

So:

x = -3 → 0
x =  2 → 2
x =  7 → 7

ReLU became particularly important in deep learning because it is simple and computationally efficient and helps neural networks model nonlinear relationships.

Softmax

Softmax is commonly used in the output layer for multi-class classification.

Suppose we're classifying an image as:

Cat
Dog
Bird

Softmax can convert the model's outputs into probabilities that sum to 1.

For example:

Cat  → 0.10
Dog  → 0.82
Bird → 0.08

The model would predict Dog.

6. Hidden Layers

Now we get to one of the most important parts of a neural network.

The hidden layers.

These layers sit between the input and output layers.

A network with only a few layers might look like:

Input → Hidden → Output

A deeper network could look like:

Input
  ↓
Hidden Layer 1
  ↓
Hidden Layer 2
  ↓
Hidden Layer 3
  ↓
Hidden Layer 4
  ↓
Output

Each layer can learn different representations of the data.

This is where the term deep learning comes from.

A neural network with many hidden layers is generally described as a deep neural network.

The interesting part is that the network can learn increasingly complex representations as information moves through the layers.

For example, in an image recognition task:

Pixels
  ↓
Edges
  ↓
Shapes
  ↓
Patterns
  ↓
Objects

The network isn't necessarily programmed with these exact rules.

It learns useful representations from the training data.

7. Forward Propagation

So how does a neural network actually make a prediction?

This happens through forward propagation.

The input data moves through the network from the input layer toward the output layer.

At each neuron:

Inputs are multiplied by weights.
The results are added together.
A bias is added.
An activation function is applied.
The result is passed to the next layer.

Eventually, the network produces a prediction.

Conceptually:

Input
  ↓
Weighted Sum
  ↓
Activation
  ↓
Next Layer
  ↓
Weighted Sum
  ↓
Activation
  ↓
Prediction

This happens extremely quickly when implemented computationally.

8. Loss Functions

The network makes a prediction.

But how does it know whether the prediction is good or bad?

This is where the loss function comes in.

A loss function measures the difference between the model's prediction and the actual target.

For example:

Actual value      = 1
Predicted value   = 0.3

There is a significant difference.

The loss function quantifies that error.

Different problems use different loss functions.

Some common examples include:

Mean Squared Error (MSE)
Binary Cross-Entropy
Categorical Cross-Entropy

The general goal during training is simple:

Minimize the loss.

9. Backpropagation

This is probably one of the most important concepts in neural networks.

After making a prediction and calculating the loss, the network needs to figure out:

Which weights contributed to the error, and how should they change?

This is where backpropagation comes in.

The error is propagated backward through the network.

Using calculus—particularly the chain rule—the network calculates gradients that indicate how changes in the parameters would affect the loss.

Conceptually:

Prediction
    ↓
Calculate Loss
    ↓
Propagate Error Backward
    ↓
Calculate Gradients
    ↓
Update Weights

Then the process repeats.

Again.

And again.

This is how the neural network gradually improves.

10. Optimizers

Backpropagation tells us how the parameters should change.

The optimizer determines how those changes are actually made.

One of the most well-known optimization algorithms is Gradient Descent.

The basic idea is to move the model's parameters in a direction that reduces the loss.

A simplified update rule looks like:

\[ w_{new} = w_{old} - \eta \frac{\partial L}{\partial w} \]

Where:

$w$ = weight
$\eta$ = learning rate
$L$ = loss

The learning rate controls how large each update should be.

If the learning rate is too large, the model may overshoot useful solutions.

If it is too small, training may take a very long time.

Other optimizers include:

Adam
RMSprop
Adagrad
SGD with momentum

Adam is particularly common in practical deep learning applications.

11. Epochs and Batches

Neural networks don't normally learn from the entire dataset in a single step.

Instead, training data is often divided into smaller groups called batches.

For example, if we have:

10,000 training samples

we might use:

Batch size = 32

The network processes 32 samples at a time.

Once it has processed the entire training dataset, we have completed one epoch.

So:

Dataset
   ↓
Batch 1
Batch 2
Batch 3
...
Batch N
   ↓
One Epoch

The model typically trains for multiple epochs.

12. Learning Rate

The learning rate determines how aggressively the model updates its parameters.

Think of it as the size of the steps the model takes while trying to minimize its loss.

A very large learning rate might look like:

Too large
    ↓
Overshoot
    ↓
Unstable training

A very small learning rate might look like:

Too small
    ↓
Tiny updates
    ↓
Very slow learning

Finding an appropriate learning rate can make a significant difference in training performance.

13. Training, Validation, and Testing

Another important concept is understanding how we evaluate a neural network.

We generally don't want to train and evaluate the model on exactly the same data.

Instead, data can be divided into:

Dataset
   │
   ├── Training Data
   │
   ├── Validation Data
   │
   └── Test Data

Training data

Used to learn the model's parameters.

Validation data

Used to monitor performance and help with choices such as hyperparameters and model architecture.

Test data

Used to evaluate how well the final model performs on unseen data.

This is important because a model can perform extremely well on its training data but poorly on new data.

That brings us to one of the biggest challenges in machine learning.

14. Overfitting

A neural network can become too good at memorizing the training data.

This is known as overfitting.

Imagine studying for an exam by memorizing the answers to last year's exact questions.

You might perform extremely well if the same questions appear.

But if the questions change, your performance may drop.

A similar thing can happen with machine learning models.

The model learns the training examples extremely well but fails to generalize to unseen data.

Some techniques used to reduce overfitting include:

Dropout
Regularization
Early stopping
Data augmentation
More training data
Simpler model architectures

Why Are Neural Networks So Powerful?

What makes neural networks particularly interesting is their ability to learn nonlinear relationships and complex representations.

Traditional programming often looks like:

Rules + Data → Output

Machine learning changes the approach:

Data + Expected Outputs → Learned Rules

And neural networks take this further by learning multiple levels of representation.

This is why they have become so useful in areas such as:

Computer vision
Natural language processing
Speech recognition
Recommendation systems
Time-series forecasting
Fraud detection
Medical imaging
Generative AI

From Neural Networks to Deep Learning

A neural network with many layers is generally referred to as a deep neural network.

Deep learning is essentially about using these deeper architectures to learn increasingly complex representations from data.

This has led to architectures designed for different kinds of problems.

For example:

Convolutional Neural Networks (CNNs)

Often associated with image and computer vision tasks.

Recurrent Neural Networks (RNNs)

Designed to work with sequential data, although many modern sequence tasks now use transformer-based architectures instead.

Transformers

Transformers have become extremely important in modern AI, particularly for language and other sequence-based problems.

They power many of the systems behind today's generative AI applications.

What I Take Away From Learning Neural Networks

One thing I appreciate about neural networks is that they changed how I think about machine learning.

At first, the equations can make the subject feel intimidating.

You see:

\[ z = wx+b \]

then activation functions, gradients, derivatives, optimization algorithms, layers, epochs, and suddenly it feels like you're drowning in mathematics.

But when you break the process down, the basic idea becomes much easier to understand.

A neural network:

takes data → makes a prediction → measures its error → adjusts itself → tries again.

That cycle is at the heart of learning.

And I think that's the part that makes neural networks so fascinating.

Final Thoughts

I'm still learning neural networks, and I don't think understanding them means memorizing every equation or being able to build a state-of-the-art architecture from scratch.

For me, understanding the fundamentals is more important.

I want to know:

What is happening to the data?
Why do we need weights?
What does a bias actually do?
Why do we need activation functions?
How does the model know that it made a mistake?
How does backpropagation help?
What does an optimizer actually change?
How do we know whether the model is learning or simply memorizing?

Once these pieces start fitting together, neural networks stop looking like a mysterious black box.

They become a collection of understandable ideas working together.

And that's probably the most important lesson I've taken from studying them so far:

Complex models become much less intimidating when you understand the simple ideas underneath them.

I'm still on that learning journey and neural networks are definitely one of the areas I'm excited to explore more deeply.

Reverse-Engineering Messy Databases: My End-to-End Data Audits Across 17,000+ Relational Schema Logs

Wed, 23 Sep 2026 11:56:39 +0000

From Raw Logs to Business Strategy: How I Taught Myself Data Engineering & Analytics

I didn’t take the traditional path into data. Over the past 12 months, I bypassed the university lecture halls to teach myself relational database architecture, pipeline optimization, and growth analytics by diving straight into raw datasets.

I look at data through an engineering and business lens. Instead of building passive charts that just look pretty, I query transaction logs and user event data to find hidden operational friction, map out user behavior patterns, and isolate exactly where systems are dropping value. I build end-to-end data systems: raw SQL schemas, data sanitization scripts, interactive Power BI dashboards, and single-page executive reports.

Below are the architectural breakdowns, technical challenges, and business impact metrics of the four core data infrastructure projects I have engineered from scratch.

Project 1: StockPulse — Multi-Regional Supply Chain Audit

Technical Stack: MySQL (Mathematical Window Functions), Power BI, Advanced Excel
GitHub Codebase & Dashboard Layout: [LINK]

The Operational Bottleneck

Horizon Marketplace faced shrinking profit margins from slow-moving warehouse stock, while simultaneous supplier shipping delays caused high-demand items to go out of stock on our mobile application. Traditional procurement tracking models failed because they relied on simple historical averages, which immediately miscalculate when data hits high seasonal sales volatility spikes.

The Technical Implementation

Audited 17,646 operational and sales ledger transactions using MySQL window functions to calculate cumulative revenue contributions per SKU.
Coded a predictive restocking matrix script utilizing sales volatility standard deviations and rolling manufacturer lead timelines.
Formulated dynamic Safety Stock buffers and programmed automated database Reorder Point triggers to flag supplier order alerts before shelves hit zero.

The Business Impact Metrics

ABC inventory segmentation proved that exactly 23 Class A items drive 80% of total marketplace revenue, showing that the remaining catalog space was tying up critical capital.
Quantified a hidden revenue bleed of $63,897.26 caused by a single high-demand apparel product sitting entirely out of stock for 42 days due to upstream logistics bottlenecks.
Transitioned a multi-regional operation from manual, time-intensive spreadsheets to an automated, predictive data framework.

Project 2: FunnelOpt — HabitArcade B2C Telemetry Audit

Technical Stack: MySQL (Session Overrides), Funnel Analysis, User Lifecycle Mapping
GitHub Codebase & Database Logs: [LINK]

The Operational Bottleneck

A B2C application tracking engine suffered from severe data execution drops where product behavioral logs completely failed to map user activation patterns. The relational schema dropped child telemetry records because they were importing prior to the absolute database finalization of the parent account index logs.

The Technical Implementation

Audited full-funnel growth logs across 6 independent acquisition pipelines to analyze customer conversion latency and drop-offs.
Structured specific MySQL session overrides to temporarily bypass structural foreign key constraints.
Forced clean data synchronization across child records to completely restore parent-child index synchronization without corrupting historical system logs.

The Business Impact Metrics

Isolated a systemic “Streak-Breaking Crash Tax” where application sync errors wiped 30-day user milestones, triggering immediate 1-star reviews and an active 6% to 7% monthly subscriber bleed.
Exposed an “Inverted Value Curve” revealing that high-value annual checkout tiers underperformed monthly subscriptions due to silent gateway billing processor failures blocking top-priced subscription cash settlement.

Project 3: LeakStop — Olist B2B Revenue Leakage & Onboarding Audit

Technical Stack: MySQL (Schema Engineering), Power BI, Business Intelligence
GitHub Codebase & Analytical Readme: [LINK]

The Operational Bottleneck

Cross-vendor data duplication across digital marketplace ledgers was heavily skewing gross reporting metrics. Standard relational database joins double-counted total transactions when a customer order spanned multiple independent business vendors in a single checkout loop.

The Technical Implementation

Engineered robust MySQL ingestion and data sanitization scripts to enforce strict primary key integrity down to individual item vectors, removing all reporting bias.
Processed user account lifecycles to calculate onboarding completion latency across specific B2B verticals.
Programmed a real-time database filter to flag slipping user accounts precisely on Day 16 of transaction inactivity.

The Business Impact Metrics

Isolated an annual revenue leakage of R$ 600,600 (approximately $119,175 USD) stemming from 462 merchant accounts that stalled out completely during the onboarding phase.
Exposed an invisible latency anomaly where logistically complex business profiles stalled for an average of 112 days in domain routing and tax mapping phases.
Implemented a Day 16 early-warning database filter to flag at-risk accounts before they reached a terminal, zero-recovery retention cliff.

Project 4: CohortPace — SaaS Behavioral Retention Engine

Technical Stack: MySQL (Subqueries & Relational Joins), Cohort Modeling, Data Analytics
GitHub Codebase & Technical Report: [LINK]

The Operational Bottleneck

Initial application dashboards focused blindly on feature exploration depth, which completely masked why healthy-looking active user profiles were suddenly canceling subscriptions. Traditional active-user metrics acted as a lagging indicator, hiding friction because churning accounts remain active inside broken workflows right up to their exit timelines.

The Technical Implementation

Modeled simulated user cohort transactions across multi-platform SaaS behavioral profiles to analyze the core drivers of long-term Monthly Recurring Revenue (MRR).
Used advanced MySQL subqueries and relational joins to reconstruct full lifecycle interaction sequences.
Mapped user interaction velocity trends against subscription lifecycles to isolate clear behavioral churn signatures.

The Business Impact Metrics

Proved through data pacing sending signals that churning users maintain consistent interaction velocity close to their exit date, showing churn is driven by late-stage workflow utility limits rather than gradual disengagement.
Demonstrated that long-term user monetization is strictly dependent on habit formation within a single core workflow rather than broad, multi-feature platform exploration.

Conclusion: Turning Code Into Business Strategy

I don’t need a playbook to get started, and I don’t need a manager to hand me a static checklist. I find the question, ingest the data, clean the schemas, model the answer, and present the metric that directly protects business margins.

I am actively available for analytical remote contract roles, standalone data audits, and freelance pipeline optimization. If your startup is scaling fast and dealing with messy relational pipelines, let’s connect and fix your backend infrastructure loops.

Interactive Portfolio Hub: [LINK]
Contact: motiwala603@gmail.com
Connect with me: [LinkedIn]

Open to analyst remote roles across product, growth, and business analytics. Based in Pakistan, available worldwide.

Why Perfect Backtests Fail: Walk-Forward Optimization and Deflated Sharpe in Python

Wed, 09 Sep 2026 10:54:50 +0000

Why Perfect Backtests Fail: Walk-Forward Optimization and Deflated Sharpe in Python

Every quantitative trader and algorithmic developer knows this pattern: you design an intraday or swing strategy, optimize parameters in MetaTrader, TradingView, or Python, obtain a near-perfect upward equity curve with a Sharpe Ratio of 2.5... and the moment you deploy live capital, the strategy collapses into a severe drawdown.

Why does this happen mathematically? The answer lies in curve-fitting (overfitting) and selection bias from multiple testing.

In this engineering guide, we implement two foundational quantitative tools in Python to falsify backtests before risking capital:

Walk-Forward Optimization (WFO) and the Walk-Forward Efficiency (WFE) ratio proposed by Robert Pardo.
Deflated Sharpe Ratio (DSR) by Bailey and Marcos López de Prado to adjust performance for the number of tested trials $N$.

Full empirical research and 5-market cross-validation study available at GuetaQuant.

1. The Mathematics of Walk-Forward Optimization (WFO)

Unlike traditional backtesting that optimizes parameters across an entire historical sample, Walk-Forward Analysis splits data into sequential sliding windows:

In-Sample (IS): Historical lookback window used to optimize strategy parameters.
Out-of-Sample (OOS): Forward blind window where optimized parameters are evaluated without further tuning.

The Walk-Forward Efficiency (WFE) ratio measures how much annualized return survives in out-of-sample execution:

\[\\text{WFE} = \\frac{\\text{Annualized OOS Return}}{\\text{Annualized IS Return}} \\times 100\]

$\text{WFE} \ge 50\%$: The algorithmic strategy shows adaptive out-of-sample persistence.
$\text{WFE} < 50\%$: High probability of memorized curve-fitting.

2. Python Implementation: Walk-Forward Efficiency

import numpy as np
import pandas as pd

def calculate_wfe(in_sample_returns: pd.Series, out_of_sample_returns: pd.Series) -> dict:
    """
    Computes Walk-Forward Efficiency (WFE) ratio.
    """
    ann_is = in_sample_returns.mean() * 252
    ann_oos = out_of_sample_returns.mean() * 252

    wfe = (ann_oos / ann_is) * 100.0 if ann_is > 0 else 0.0

    return {
        "annualized_in_sample_return_pct": round(ann_is * 100, 2),
        "annualized_out_of_sample_return_pct": round(ann_oos * 100, 2),
        "walk_forward_efficiency_pct": round(wfe, 2),
        "verdict": "ROBUST_ADAPTIVE" if wfe >= 50.0 else "OVERFITTED_REJECTED"
    }

3. Deflated Sharpe Ratio (DSR): Correcting for Multiple Testing

If you test 100 moving average combinations and pick the single best result, the observed Sharpe ratio is not a standard normal random variable: it is the maximum of 100 variables.

According to Extreme Value Theory, the expected maximum Sharpe under the null hypothesis of zero skill ($H_0$) is:

\[E[\\max_N \\{SR_n\\}] \\approx \\sqrt{V} \\left[ (1 - \\gamma) \\Phi^{-1}\\left(1 - \\frac{1}{N}\\right) + \\gamma \\Phi^{-1}\\left(1 - \\frac{1}{N \\cdot e}\\right) \\right]\]

Where $\gamma$ is the Euler-Mascheroni constant ($0.5772...$) and $N$ is the total number of parameter combinations tested.

You can calculate DSR using the open-source library gueta-risk:

pip install gueta-risk

And execute in Python:

from gueta_risk import deflated_sharpe_ratio

# Suppose you ran N=100 parameter permutations
# and your best backtest showed an annualized Sharpe of 1.40 across 252 daily bars:
result = deflated_sharpe_ratio(
    observed_sr=1.40,
    num_trials=100,
    sample_length=252,
    skewness=-0.1,
    kurtosis=3.2
)

print(result)
# {
#   observed_sharpe: 1.4,
#   num_trials_tested: 100,
#   expected_max_sharpe_threshold: 2.505,
#   deflated_sharpe_ratio: 0.0035,
#   is_significant_95pct: False,
#   falsification_verdict: FALSIFIED_OVERFITTING
# }

Conclusion: A Sharpe of 1.40 appears attractive at first glance, but after 100 trials, the expected false discovery threshold is 2.505. Because DSR is 0.0035 (< 0.95), the strategy is rejected by statistical falsification.

4. Open-Source Quantitative Ecosystem

All our empirical research, validation scripts, and risk engines are open-source under AGPLv3:

Full Empirical Research Paper: Walk-Forward Optimization Guide on GuetaQuant
44 Open Source Quantitative Tools: GitHub Repository
CERN Zenodo DOI Scientific Registry: DOI 10.5281/zenodo.22012203
Interactive MT5 Position Sizer: GuetaQuant Risk Tool

About the Author: Mahdi Goodarzi is the founder and quant developer at Gueta Quant. Developer profile: g.dev/mahdigoodarzi.

The Parts Of Scientific Work That Are Really Software Problems

Sun, 30 Aug 2026 21:45:55 +0000

Most of us here write software for products, but a surprising amount of the same work happens in labs and research groups, written by people who were never taught to write code. If you have ever inherited a research script and wondered why it does what it does, this one is for you.

Where Research Code Actually Breaks

Research code fails differently from product code. It rarely crashes in a way anyone notices. It quietly produces a number, and the number is wrong because a filter was applied twice, or a merge silently dropped half the rows, or a random seed moved between runs.

The habits that catch this are the ones we already use: keep the raw data immutable, make every transform a named step you can rerun from scratch, and check row counts before and after every join. A pipeline that prints how many records survived each stage catches more errors than any amount of careful reading.

Statistics Is A Modeling Problem, Not A Library Problem

Calling a t-test is one line in any language. Knowing that a t-test is the wrong tool for your data is the actual work, and no library will tell you.

The questions that matter come before the function call: what is the unit of observation, are the observations independent, how many comparisons are you running, and what effect size would actually mean something. Get those wrong and a perfectly correct implementation returns a perfectly meaningless p value.

Reading A Paper Like A Code Review

A paper is a claim plus the evidence for it, and it can be reviewed the same way you review a pull request. Read the methods before the results, because the methods tell you what the results are allowed to mean.

Then check the sample size, look for the comparison that is missing, and see whether the conclusion in the abstract is the same one the data supports. This is a skill, not a talent, and it improves fast once you start doing it deliberately. We keep the practical side of all of this collected in one place, the tools and methods behind real science, covering Python for scientific work, statistics, scientific computing, algorithms, experiment design, research methods and how to read a paper properly.

The Takeaway

Scientific work and software work fail in the same places: unclear inputs, unstated assumptions, and results nobody can reproduce. If you already have the engineering instincts, most of what separates you from doing real analysis is vocabulary, and that part is learnable in an afternoon.

Weather Models Solved Honest Error Bars Before The Rest Of Us Did

Tue, 25 Aug 2026 22:03:01 +0000

Most of us ship a model that returns one number and then bolt a confidence score onto it afterwards, usually because somebody asked. Operational meteorology went the other way decades ago: the distribution is the product, and the single run is just one sample from it. The pattern is worth stealing, so here is how it actually works.

The Physics Chain Underneath Every Forecast

Weather is not a bag of correlations, it is a solved-forward physics problem. The Sun heats Earth unevenly, temperature differences become pressure differences, pressure differences become wind, wind moves moisture, and moisture becomes clouds and precipitation. Earth's rotation deflects the moving air through the Coriolis effect, which is why cyclones spiral instead of collapsing straight into the low.

The energy accounting matters more than it looks. Evaporating water stores a large amount of latent heat in the vapor, and when that vapor condenses inside a cloud the heat dumps back into the surrounding air. That release is the fuel that turns an ordinary updraft into a thunderstorm and a tropical disturbance into a hurricane.

How A Numerical Model Actually Runs

Numerical weather prediction slices the atmosphere into a three dimensional grid and integrates the governing equations forward at every point: conservation of mass, momentum and energy, plus the ideal gas law. ECMWF's global system runs at roughly 9km grid spacing. Regional models at 3km or finer stop parameterizing convection and start resolving individual thunderstorms.

The inputs come from about 10,000 surface stations, roughly 900 radiosonde launches every 12 hours, plus radar, aircraft, buoys and satellites. Geostationary satellites give you the same hemisphere every few minutes, polar orbiters give you higher resolution on a slower revisit.

Why The Ensemble Is The Real Output

Every one of those observations carries error, and the system is chaotic, so two nearly identical initial states diverge. Lorenz found this inside a weather model in the first place, which means the field inherited the problem and the workaround at the same time.

The workaround is ensemble forecasting: run the same model many times from slightly perturbed initial conditions and read the spread across members. Tight clustering means high confidence. Wide scatter means several outcomes are genuinely open. A "40 percent chance of rain" is that scatter converted into a number, not a forecaster hedging. This is what pushed medium range forecasts, the 3 to 10 day window, from unusable to genuinely decision grade.

What To Take Back To Your Own Stack

If your model is deterministic and your users need to act on it, you are shipping the least informative part of what you computed. Perturb the inputs at the level of their real measurement error, run the thing N times, and report the spread. It costs N times the compute and it is usually worth it, because the honest answer to "how sure are you" is a distribution rather than a trained confidence head.

The full walkthrough of the physics, from dew point and cloud formation through fronts, supercells and hurricanes to how the forecast pipeline is assembled, is in our complete guide to how weather works.

Curious whether anyone here has actually shipped an ensemble instead of a point estimate, and what the compute bill looked like.

A SQL interview is a translation test, not a syntax test

Sat, 22 Aug 2026 14:27:55 +0000

Four-Leaf's SQL Interview Guide

Key takeaways

A SQL interview is a translation test. The syntax is the easy half, and most candidates who fail were fluent in SQL. What sinks them is starting to type before the question is pinned down, or writing a correct query that answers something nobody asked. The round scores three things: whether you clarify before you write, whether your query is correct on the edge cases, and whether you can explain your reasoning while your hands are moving. It also looks meaningfully different depending on whether you're interviewing as a data analyst, a data scientist, or an engineer.

What does a SQL interview test?

The round tests whether you can turn an underspecified business question into a query that survives contact with real data. Interviewers rarely hand you a clean specification, because the job never does either.

SQL is worth preparing for on volume alone. The Stack Overflow 2025 Developer Survey found 58.6% of developers had used SQL in the past year, making it the third most-used language behind JavaScript at 66% and HTML/CSS at 61.9%, and ahead of Python at 57.9%. It shows up in loops for roles that aren't nominally data roles at all, which is part of why candidates underprepare for it.

The tell that separates a strong candidate is what happens in the first sixty seconds. Weak candidates read the prompt and start typing. Strong candidates restate the question, ask what counts as an active user or a completed order, confirm whether the answer should include rows with nulls, and only then write. That opening exchange is often worth more to the interviewer than the query itself, because it's the part of the job that can't be looked up.

What kinds of SQL questions come up?

Three shapes cover most of what gets asked, and they escalate in a predictable order.

The first is a join and aggregation question. Given two or three tables, produce a count or a sum grouped by something. These look trivial and catch people on join type, because an inner join silently drops the users who never ordered, and the question usually wanted them counted as zero.

The second is a window function question. Rank purchases per customer, find each user's second transaction, compute a running total, or calculate a month-over-month change. This is where a lot of loops separate candidates, because window functions are the boundary between people who write SQL occasionally and people who use it as a primary tool.

The third is an open business question against a schema you've just been shown. "Tell me whether retention improved after the March release." There's no single correct query. The interviewer is watching how you decompose the question, what you decide retention means, and whether you notice that the March cohort has less time to churn than the February one.

How is a SQL interview scored?

Three dimensions, and they map to those question shapes. It helps to run each one against the questions the round keeps returning to: give me a count by group, rank something per user, and answer this vague business question.

Do you clarify before you write? A passing answer asks one question about the schema. A strong answer names the ambiguity that would change the query and resolves it. "Does an active user mean any event in the window, or a purchase?" is the difference between two very different numbers, and interviewers plant that ambiguity on purpose.

Is your query correct on the edges? A passing answer runs. A strong answer accounts for nulls, duplicates, ties in a ranking, and rows on the boundary of a date range. Say the edge case out loud when you handle it, because a silent LEFT JOIN looks identical to a lucky one.

Can you narrate while you write? A passing answer goes quiet and produces a query. A strong answer talks through the plan first, writes, then checks the result against a rough expectation out loud. Interviewers score reasoning they can hear, and a correct query delivered in silence gets a weaker write-up than a slightly imperfect one that was explained.

How does a SQL interview differ for a data analyst versus a data scientist?

Same language, meaningfully different round, and mixing them up is the most common preparation mistake in the function.

Data analyst. The emphasis is breadth and business translation. Expect more questions, less depth per question, and heavy weighting on whether your numbers would hold up in front of a stakeholder. Definitional precision matters more than query elegance. You're likely to be asked what a metric should mean before you're asked to compute it, and an answer that flags a misleading denominator scores higher than one that optimizes a join.

Data scientist. The emphasis shifts toward experiment and cohort logic. Expect fewer questions with more depth, and expect at least one that touches an A/B test readout, a cohort definition, or a sampling problem hiding inside the SQL. A question about whether a March cohort had less time to churn than a February one is a data scientist question. Correctness under a statistical framing is what's being scored, not query volume.

Analytics engineer. The round moves toward modeling. Expect questions about how you'd structure the model rather than write a one-off answer, including idempotency, incremental logic, and what happens when the query is rerun.

Software engineer. SQL usually appears inside a broader technical round rather than on its own, and the framing is performance and correctness at scale. Expect indexing, query plans, and the consequences of a full table scan rather than business definitions.

Product management. Increasingly common and usually light. The bar is whether you can pull your own numbers without asking an analyst, so expect one aggregation question and no window functions.

Four-Leaf covers the wider data loop in how to prepare for a data science interview.

What is overrated

Memorizing exotic syntax. Recursive CTEs and pivot tricks appear in practice sets far more often than in interviews, and time spent there is time not spent on window functions, which appear constantly.

Speed is the other overrated thing. Candidates rush because they assume the clock is the test, then miss a join type. The interviewer is almost always more interested in whether you caught the null case than in whether you finished ninety seconds early.

Using an AI assistant to get through this round is a losing trade. CodeSignal reported in February 2026 that cheating and fraud attempts on proctored assessments more than doubled, from 16% in 2024 to 35% in 2025. Interviewers noticed. A 2025 interviewing.io survey of 67 of them, 52 at FAANG companies, found 81% suspected candidates of using AI and about a third had caught someone at it. The visible response has been more live, narrated rounds, which is precisely the format where a candidate who can't explain their own query falls apart.

A five-step playbook for a SQL round

Drill window functions until they're automatic. ROW_NUMBER, RANK, DENSE_RANK, LAG, LEAD, and a running sum. These carry more interview weight per hour of study than anything else in SQL.

Practice restating the question out loud before writing. Give yourself a rule that you don't type until you've named one ambiguity.

Build a null and duplicate checklist. Before you call a query done, ask what happens to rows with nulls, rows that appear twice, and rows exactly on the boundary date.

Do at least three problems on a whiteboard or plain text editor. Autocomplete hides gaps that a bare editor exposes, and many live rounds use a bare editor.

Narrate a solved problem end to end. Explaining a query you already understand is a separate skill from writing it, and it's the one being scored. A voice mock interview is a reasonable way to rehearse the narration, since the failure mode is going quiet under pressure rather than not knowing the syntax.

Where this is heading

SQL rounds are getting more conversational, and the reason is the same force reshaping every technical loop. As assistants make it trivial to produce a syntactically correct query, interviewers move the bar to the part that's harder to fake, which is deciding what to compute and defending the choice. Four-Leaf's AI-Era Hiring Index, a study of 3,502 open roles at 16 top tech employers, found LLM or foundation-model experience listed in 57% of data and machine learning job descriptions against 21% of engineering ones, so the expectation that data candidates work alongside these tools is already written into the postings.

That direction rewards the candidate who understood the question. The syntax was never the hard part, and it's about to matter even less. Where this round sits in the wider sequence is covered in how an onsite interview loop actually works. To rehearse it out loud rather than read about it, start with eight coding interview platforms, compared.

Beyond Binary Classification: Engineering an Autonomous Probabilistic Forecasting AI

Tue, 18 Aug 2026 20:34:18 +0000

In the realm of applied data science, building a model that performs well on static datasets is a standard rite of passage. But deploying an AI to predict outcomes in a chaotic, high-variance environment-such as professional sports-is an entirely different engineering challenge.

When you introduce human variability, real-time tactical shifts, and unpredictable physical elements into your feature space, traditional machine learning models break down. They overfit to historical noise, fail to account for variance, and ultimately output brittle predictions.

I recently set out to solve this exact data science problem. The objective was to build a system that moves beyond simple binary classification and acts as a true mathematical oracle. The result is Matchmind Oracle, a fully autonomous, ML-driven probabilistic forecasting engine.

Here is a surgical breakdown of the data science architecture, the feature engineering process, and the mathematical modeling required to build an AI that actually understands chaos.

The Failure of Binary Output in Chaotic Systems

Most entry-level predictive models in sports analytics treat the problem as a standard binary classification task: Team A vs. Team B, Output = 1 or 0. A developer might throw a Random Forest or a basic deep neural network at the problem, optimize for accuracy on a historical dataset, and call it a day.

This approach is fundamentally flawed. In chaotic environments, predicting a definitive "winner" is statistically dishonest.

Instead of asking the model to output a binary decision, I engineered Matchmind Oracle to function as a probabilistic forecasting engine. Rather than outputting a single prediction, the model generates a probability density function across a spectrum of possible outcomes. It calculates the exact implied probability of an event, the expected value (EV), and the statistical variance. It doesn’t tell you what will happen; it tells you the exact mathematical likelihood of every possibility.

High-Dimensional Feature Engineering

A machine learning model is effectively blind until you teach it how to see the data. For Matchmind Oracle, simply feeding raw box scores into a neural network would result in garbage outputs. The feature engineering had to be surgical.

The Oracle ingests massive amounts of structured match data via automated pipelines. However, before that data reaches the inference engine, it passes through a rigorous transformation layer:

Temporal Weighting: Recent performance is mathematically more significant than data from three years ago. I implemented time-decay functions on historical data points, ensuring the model heavily weights current form while still maintaining a baseline understanding of long-term franchise or player tendencies.
Contextual Vectorization: How do you encode a sudden injury or a weather shift? I built dynamic feature handlers that vectorize contextual data. When an anomaly occurs (e.g., a star player is benched), the model instantly recalculates the matrix, cross-referencing thousands of historical vectors where similar anomalies occurred.
Dimensionality Reduction: Sports APIs provide an overwhelming amount of raw data. To prevent the model from drowning in high dimensionality, I utilized Principal Component Analysis (PCA) to isolate the statistical features that actually correlate with outcome variance, dropping the "noise" that leads to overfitting.

The Training Loop: Calibrating Confidence

Training an AI on sports data requires a delicate balance. If you train it too aggressively, it overfits to the past and fails to generalize to the future.

To solve this, I built extensive historical evaluation scripts. These are not simple backtests; they are rigorous cross-validation pipelines. The Oracle runs through years of historical data, makes its probabilistic forecasts, and then measures its output against the actual results using logarithmic loss (log-loss) functions.

Because the system is probabilistic, evaluating its success is highly nuanced. If the Oracle states there is a 30% chance of an underdog winning, and the underdog wins, the model wasn't "wrong"-it correctly identified a high-variance edge. The evaluation scripts continually calibrate the model's confidence intervals. If the model is too confident in highly volatile scenarios, the loss function penalizes it, forcing the algorithm to dynamically adjust its variance parameters.

From Jupyter to Autonomous Inference

Many data scientists build brilliant models that never leave their Jupyter Notebooks. Taking a complex mathematical framework and turning it into a live, autonomous product requires a bridge between data science and software architecture.

Matchmind Oracle operates continuously without manual intervention. The inference engine is containerized and runs as a persistent background service on a virtual private server. Whenever new data hits the ingestion pipelines, the model recalculates the probability matrices in real-time.

But the true innovation lies in the delivery mechanism. I wanted the outputs to be instantly accessible, avoiding the friction of web dashboards. To achieve this, the entire Python-based ML backend is integrated via webhooks into a Telegram bot infrastructure.

When the model detects a statistically significant edge or a high-probability variance, the automated agent formats the complex data into digestible, mathematically grounded insights and pushes them directly to the user's device.

Open Source Exploration

Building the Matchmind Oracle has been one of the most rewarding data science projects of my career. It bridges the gap between deep mathematical modeling, rigorous data engineering, and frictionless product delivery. It is a testament to the fact that with clean data and surgical algorithm design, we can quantify chaos.

I am a strong advocate for the developer community and believe that AI architectures should be explored openly. If you are a machine learning engineer, a data scientist, or a developer fascinated by probabilistic forecasting, I am opening up the system's outputs.

I have set up a dedicated Telegram channel where the Oracle automatically posts its mathematically driven possibilities and real-time analytical insights for the community to analyze, test, and use logically.

Join the Matchmind Oracle Telegram community here to see the ML engine in action:
👉 https://t.me/+6qJiFtze8sc0ZjM0

Firecrawl vs Apify vs Quorel

Sat, 25 Jul 2026 21:23:40 +0000

Firecrawl and Apify both do the hard part well. They render JavaScript, dodge bot detection, and hand you back clean markdown or structured JSON. If you need a page turned into text, either one will do that job.

But most of the time the point isn't a single page. It's a page you need to check again tomorrow, and the day after, and the day after that: a price, a listing, a status, a set of rows that might quietly change. And that's where the workflow, not the extraction, starts costing you.

Firecrawl bills per credit, and the credit math is not what the pricing page implies. Scrape is 1 credit. Add JSON extraction and Enhanced Mode — both of which you probably want for structured data — and it's 9 credits per page. A $16/mo Hobby plan advertised at 3,000 credits works out closer to 300–500 pages once you're actually extracting fields, not just fetching markdown. Credits also don't roll over, so a quiet month doesn't bank anything for a busy one.

Apify flips the complaint around. Pricing is compute-unit based (RAM × time) plus separate line items for proxy and storage. It's flexible and it scales, but you can't look at a number on the pricing page and know what a job will cost until you've run it. For teams that want a predictable bill, that unpredictability is the whole problem.

Neither tool keeps history by default. You get the current state of the page. If you want yesterday's version to compare against today's, that's on you to build — a database, a diffing script, a place to store snapshots, a way to roll back if a refresh comes back wrong.

Quorel starts from a different question: not "how do I fetch this page," but "how do I keep a page as structured, versioned data over time." You describe the fields you want in plain English. Every refresh (nightly by default, or on demand via a ping URL, webhook, or your own cron) is saved as its own immutable version. Nothing is overwritten. You can diff any two versions to see exactly what changed, and roll back in one click if a refresh comes back bad.

Pricing is flat, not metered. Free tier is one dataset, 20 URLs, full version history included. Pro is $19/mo for 5 datasets and 100 URLs, with on-demand refresh and webhooks. No credit multipliers, no compute-unit math, no surprise bill.

This isn't a claim that Quorel replaces Firecrawl or Apify at everything they do. If you need to fetch an arbitrary one-off page as clean markdown for an LLM right now, Firecrawl is built for exactly that. If you need thousands of site-specific scrapers off a marketplace, Apify's Actor ecosystem has years of head start. Quorel is built for the narrower, recurring job: turning a page you check regularly into a structured, versioned API you can trust and query later.

Every dataset ships with an MCP server on every plan, free included, so an agent can query it directly — or pull the raw rows, fix nulls, and publish a new version itself.

If you've been burned by a scraper that silently broke, or wished you'd kept last week's data to compare against, that's the specific gap Quorel is built to close.

I'm building Quorel — happy to answer questions on versioning, pricing, or how the MCP integration works.

How I built a computational AMP screening pipeline: from 24,000 sequences to 47 drug candidates

Sat, 04 Jul 2026 22:33:41 +0000

Antimicrobial resistance is projected to kill 10 million people annually by 2050 - more than cancer. Antimicrobial peptides (AMPs) are one of the most promising therapeutic classes to fight it. They're cationic, membrane-disrupting, and bacteria struggle to develop resistance against them.

The problem: there are thousands of candidate sequences in databases. How do you identify the ones worth synthesising?

In this post I'll walk through a full computational screening pipeline I built from raw database download through machine learning activity prediction to ColabFold structure prediction - ending with 47 high-confidence drug candidates.

The full code is on GitHub: amp-colabfold

The pipeline at a glance

Five stages:

Data curation - 24,076 sequences from DRAMP 4.0 down to 12,435 after filtering
Feature engineering - 16 physicochemical features, 0% missing
Activity modelling - gradient boosting classifier, ROC-AUC 0.77
Candidate selection - 131 structured peptides sent to ColabFold
Structure prediction - 69 high-confidence structures, 47 very high confidence

Stage 1: Data curation

I pulled sequences from DRAMP 4.0 (the Database of Antimicrobial Peptides)
using their REST API - no manual downloads needed.

Three filters applied in sequence:

# Length: 10-50 aa (therapeutic window for AMPs)
df = df[df["sequence"].str.len().between(10, 50)]

# Canonical amino acids only - no modified residues
CANONICAL = set("ACDEFGHIKLMNPQRSTVWY")
df = df[df["sequence"].apply(lambda s: set(s).issubset(CANONICAL))]

# 90% identity clustering — Python reimplementation of CD-HIT
# (CD-HIT is Linux-only; I wrote a greedy longest-first algorithm in pure Python)
df_curated = cluster_cdhit_python(df, identity=0.90)

The CD-HIT reimplementation was the most interesting part of this stage.
CD-HIT's greedy algorithm is conceptually simple: sort sequences longest-first, then for each sequence check if it's ≥90% identical to any existing cluster representative. If yes, discard it. If no, it becomes a new representative. I added 3-mer pre-filtering to skip obviously dissimilar pairs, matching CD-HIT's default behaviour.

Result: 24,076 → 12,435 sequences

Stage 2: Feature engineering

For each of the 12,435 sequences I computed 16 physicochemical features using
the peptides Python package:

features = {
    "net_charge_pH7":           pep.charge(pH=7.0, pKscale="Lehninger"),
    "hydrophobicity_eisenberg":  pep.hydrophobicity(scale="Eisenberg"),
    "hydrophobic_moment":        pep.hydrophobic_moment(window=11, angle=100),
    "boman_index":               pep.boman(),
    "aliphatic_index":           pep.aliphatic_index(),
    "instability_index":         pep.instability_index(),
    # + Chou-Fasman helix/sheet/turn propensities computed manually
    # + aromaticity, fraction_positive, fraction_negative, length, pI, MW
}

One methodological note: aromaticity() was removed from newer versions of
peptides, so I computed it directly from residue frequencies
(F + Y + W) / length — same definition as Biopython, just implemented inline.

Result: 16 features, 0% missing across all 12,435 sequences

Stage 3: Activity modelling

Binary classification: antibacterial (confirmed MIC data) vs
general/natural (uncharacterised or broad labels).

clf = GradientBoostingClassifier(
    n_estimators=300,
    learning_rate=0.05,
    max_depth=3,        # shallow trees → interpretable SHAP values
    subsample=0.8,
    random_state=42,
)

ROC-AUC: 0.77 on a length-decile-stratified holdout set.

The class imbalance (28.9% positive) is real and intentional — confirmed
antibacterial sequences with MIC data are rarer than general database entries.

I didn't oversample, because the model's conservative predictions are
appropriate for a screening tool: better to miss some true positives than
flood the downstream ColabFold stage with false ones.

What drives the predictions?

The SHAP beeswarm tells a clean biological story:

Length dominates - longer peptides push positive. This reflects the
antibacterial subset containing more full-length defensins and cathelicidins.
fraction_helix - high helix propensity → predicted antibacterial.
Consistent with alpha-helical membrane disruption.
fraction_positive - cationic residues push positive. Textbook
electrostatic attraction to anionic bacterial membranes.
fraction_negative - anionic residues suppress prediction. Expected.

This is not a black box. Every prediction is traceable to a physical property with a known biological mechanism. That's the point of using shallow trees with SHAP rather than a deep model.

Stage 4: Candidate selection

From the classifier output I selected peptides with ab_proba ≥ 0.938 and
then applied a low-complexity filter:

def is_low_complexity(seq: str, threshold: float = 0.5) -> bool:
    # Flag homopolymers (KKKKKK, RRRRRR) and tandem repeats
    max_freq = max(seq.count(aa) for aa in set(seq)) / len(seq)
    if max_freq > threshold:
        return True
    for i in range(len(seq) - 2):
        kmer = seq[i:i+3]
        if seq.count(kmer) * 3 / len(seq) > 0.6:
            return True
    return False

This step is important. Homopolymeric sequences (poly-K, poly-R) score high
on fraction_positive and fool the classifier - they're real AMPs but
structurally trivial and not useful for ColabFold. The filter removed 44
low-complexity sequences.

Result: 131 structured candidates (15–44 aa) sent to ColabFold

Stage 5: Structure prediction with ColabFold

I ran ColabFold 1.6.1 on Google Colab (free T4 GPU) in single-sequence mode appropriate for short peptides where MSA depth is limited anyway.

colabfold_batch \
  colabfold_input.fasta \
  colabfold_output/ \
  --model-type alphafold2_ptm \
  --num-recycle 3 \
  --num-models 1 \
  --msa-mode single_sequence

For 131 peptides of 15-44 aa, the full run took ~15 minutes on a T4.

Parsing results locally

def parse_scores_json(json_path):
    with open(json_path) as f:
        data = json.load(f)
    return {
        "mean_plddt": float(np.mean(data["plddt"])),
        "ptm":        float(data["ptm"]),
        "plddt_per_residue": data["plddt"],
    }

pLDDT distribution

69 of 131 candidates (52.7%) have pLDDT ≥ 70, indicating meaningful structural confidence. For short peptides - which are inherently more disordered than globular proteins - this is a strong result.

Structural confidence vs activity prediction

The scatter shows no strong correlation between classifier confidence and
structural confidence - which is biologically expected. A peptide can be
highly predicted antibacterial (high ab_proba) but disordered in solution
(low pLDDT), because many AMPs adopt structure only upon membrane contact.

Final results

Applying both filters - pLDDT ≥ 90 AND ab_proba ≥ 0.95 - gives
47 very high-confidence candidates:

AMP ID	Length	pLDDT	pTM	ab_proba
AMP_02388	30 aa	97.3	0.54	0.952
AMP_01454	36 aa	97.2	0.60	0.985
AMP_02689	28 aa	97.2	0.50	0.954
AMP_01282	37 aa	97.1	0.61	0.981
AMP_01129	38 aa	97.0	0.62	0.959

Full list: candidate_amps.csv

What I'd do next

Hemolysis screening - filter candidates through HemoPI2 to remove cytotoxic sequences before any wet-lab work
ESM-2 embeddings - train a second classifier on sequence embeddings and compare against the physicochemical feature model
MD simulation - for the top 5 candidates, run short molecular dynamics to assess membrane-binding stability
Experimental validation - MIC assays against E. coli and S. aureus

Key lessons

1. Reproducibility over convenience. The CD-HIT reimplementation in pure Python adds ~100 lines but means the entire pipeline runs on any OS without binary dependencies.

2. Class imbalance is a feature, not a bug. The 3:1 negative:positive
ratio reflects reality. Don't oversample your way out of it.

3. Interpret your model before trusting it. The SHAP beeswarm told me
the model was learning real biology (charge, helix propensity) rather than
spurious correlations. Without that check I wouldn't have been confident
sending its predictions to a GPU.

4. pLDDT ≠ activity. Low structural confidence doesn't mean an AMP is
inactive - it may just be disordered until it hits a membrane. Keep both
metrics and let them inform each other.

GitHub: github.com/Farhan89082/amp-colabfold

If you found this useful or have questions about any stage, drop a comment below.

How I Used Python to Analyse 40,000 Human Gut Cells and Uncover What Makes Crohn's Disease Different

Sat, 04 Jul 2026 20:19:33 +0000

A step-by-step walkthrough of my multi-sample single-cell RNA sequencing project - written for anyone curious about how computational biology actually works

Before we start - what is this article actually about?

Imagine being able to take a tiny sample of tissue from a patient's gut, and instead of just knowing "there are cells here", you could read the activity of every single gene inside every single individual cell - thousands of cells at once.

That's what single-cell RNA sequencing (scRNA-seq) does. And in this article, I'll walk you through how I used Python to analyse data from 40,000 human gut cells across 18 patients to understand what makes two similar gut diseases - Crohn's disease and ulcerative colitis - biologically different from each other.

No biology PhD required. I'll explain every term as we go.

The biological problem - two diseases that look the same but aren't

Crohn's disease (CD) and ulcerative colitis (UC) are both types of Inflammatory Bowel Disease (IBD). Both cause chronic inflammation in the gut, both cause pain and discomfort, and both are lifelong conditions. Doctors have known for decades that they're different diseases, but at a surface level they're easy to confuse.

The key difference is where and how they inflame:

Crohn's disease can affect any part of the digestive tract, tends to cause deep, patchy inflammation, and is driven largely by a type of immune cell called a myeloid cell (think macrophages - the "pac-man" cells of the immune system)
Ulcerative colitis affects only the colon and rectum, causes continuous surface inflammation, and is driven more by antibody-producing cells called plasma cells

Understanding these differences at the level of individual cells - not just tissue - is critical for developing better, more targeted treatments. That's where single-cell sequencing comes in.

The data - 18 patients, 40,000 cells, one big challenge

I used a publicly available dataset from the CZ CELLxGENE platform containing colonic mucosa (colon lining) biopsies from 18 donors:

6 patients with Crohn's disease
6 patients with ulcerative colitis
6 healthy controls

Each donor contributed thousands of cells, giving us 46,700 cells total (40,084 after quality filtering).

But here's the problem: when you combine data from 18 different people, collected at different times, processed slightly differently in the lab - the data gets messy. The technical differences between samples (called batch effects) can be so strong that they hide the real biological differences you actually care about.

Think of it like trying to compare photos taken by 18 different cameras with different colour settings. The same object might look completely different just because of the camera, not because the object actually changed.

This is where Harmony comes in.

What is Harmony and why does it matter?

Harmony is a batch correction algorithm. In plain English: it's a mathematical tool that looks at all 18 samples, figures out which differences between samples are due to technical variation (the "camera settings"), and removes those differences — leaving only the real biological signal.

Here's what the data looks like before Harmony:

PCA plot coloured by donor - you can see individual donors clustering separately, meaning donor identity is driving the structure more than the actual biology.

And after Harmony:

UMAP plot coloured by donor - all 18 donors are now mixed uniformly within each cell type cluster. The batch effects are gone.

But crucially - when we colour the same post-Harmony UMAP by disease group instead of donor:

UMAP coloured by disease - Crohn's, UC, and normal cells now separate into distinct regions. The biology is preserved, the noise is removed.

This before/after comparison is the core technical contribution of this project. Without Harmony, any findings could be explained by "oh, that's just because donor 3 was processed differently." With Harmony, we can be confident the differences are real.

Setting up the pipeline in Python

The full analysis uses Scanpy - the standard Python library for single-cell analysis - along with harmonypy for the integration step.

import scanpy as sc
import harmonypy

# Load the dataset
adata = sc.read_h5ad('data/ibd_dataset.h5ad')
# 46,700 cells × 32,354 genes

The data is stored in an AnnData object - think of it as a very specialised spreadsheet where rows are cells, columns are genes, and there's extra space to store metadata like disease status, donor ID, and cell type labels.

Step 1 - Quality control

Not every cell in a sequencing experiment is a real, healthy cell. Some are damaged, some are empty droplets that got accidentally captured, and some are doublets (two cells mistakenly counted as one). We filter these out using three metrics:

# Remove low quality cells
adata = adata[adata.obs['n_genes_by_counts'] > 200]   # too few genes = empty droplet
adata = adata[adata.obs['n_genes_by_counts'] < 6000]  # too many genes = doublet
adata = adata[adata.obs['pct_counts_mt'] < 30]        # high mitochondrial % = dying cell

Why mitochondrial genes? When a cell is dying or stressed, the nucleus breaks down and releases its RNA - but mitochondria (the cell's energy factories) have their own separate DNA and RNA that stays intact longer. So a high percentage of mitochondrial gene reads is a reliable sign of a low-quality cell.

After filtering: 40,084 cells remain.

Step 2 - Normalisation

Different cells capture different amounts of RNA simply due to technical variation in the sequencing process. A cell with 10,000 RNA molecules captured will look like it expresses every gene more than a cell with only 2,000 captured — even if their true biology is identical.

We normalise by scaling every cell to have the same total count (10,000), then apply a log transformation to reduce the influence of very highly expressed genes:

sc.pp.normalize_total(adata, target_sum=1e4)
sc.pp.log1p(adata)

Step 3 - Finding highly variable genes

Out of 32,354 genes, most are either not expressed at all or expressed at the same level in every cell (housekeeping genes that keep basic cell functions running). These genes add noise without adding information.

We select only the highly variable genes - genes whose expression varies meaningfully between cells - for downstream analysis. We found 2,873 of these, using batch_key='donor_id' to ensure we pick genes that are variable across all donors, not just one:

sc.pp.highly_variable_genes(adata, min_mean=0.0125, max_mean=3,
                             min_disp=0.5, batch_key='donor_id')
# Result: 2,873 highly variable genes

Step 4 - PCA and Harmony integration

PCA (Principal Component Analysis) reduces our 2,873-gene matrix into 50 dimensions that capture the most important variation. Then Harmony corrects for batch effects within this PCA space:

sc.tl.pca(adata, svd_solver='arpack')
sc.external.pp.harmony_integrate(adata, key='donor_id', max_iter_harmony=20)
# Harmony converged in just 6 iterations

The fact that Harmony converged in only 6 out of 20 possible iterations is a good sign - it means the batch effects were relatively structured and correctable, and the biological signal is strong.

Step 5 - UMAP visualisation

UMAP (Uniform Manifold Approximation and Projection) takes the Harmony-corrected embeddings and projects them into 2D for visualisation. Similar cells end up close together, different cells end up far apart:

sc.pp.neighbors(adata, use_rep='X_pca_harmony', n_neighbors=10)
sc.tl.umap(adata)

What we found - the biology

Finding 1: Five major cell populations in the gut

The UMAP revealed five well-separated clusters corresponding to the major cell types in the colon:

Plasma cells (15,633 cells) - antibody factories
Colon epithelial cells (12,347 cells) - the cells lining the gut wall
T cells (12,128 cells) - immune soldiers
Myeloid cells (3,771 cells) - macrophages and related immune cells
Stromal cells (2,821 cells) - structural support cells

Finding 2: UC and CD have completely different cellular makeups

This is where the biology gets interesting. When we look at what proportion of cells belong to each type across disease groups:

Plasma cells:

Normal: ~27%
Crohn's disease: ~30%
Ulcerative colitis: ~52%

UC has nearly double the plasma cell proportion of healthy tissue. Plasma cells make antibodies - this confirms that UC is primarily driven by antibody-mediated (humoral) immunity, a well-established finding that we independently reproduced from raw data.

Myeloid cells:

Normal: ~6%
Ulcerative colitis: ~8%
Crohn's disease: ~13%

CD has more than double the myeloid cell proportion. Myeloid cells include macrophages - the cells responsible for the granulomatous (nodule-forming) inflammation that's the hallmark of Crohn's disease.

Epithelial cells:

Normal: ~29%
Crohn's disease: ~19%
Ulcerative colitis: ~3%

This is striking. UC patients have almost no epithelial cells left - the gut lining is severely disrupted. This explains why UC causes such pronounced mucosal damage and bleeding.

Finding 3: The S100A8/S100A9 signature - a clinical biomarker reproduced at single-cell resolution

To find which specific genes are driving the myeloid difference in Crohn's disease, we ran a differential expression analysis comparing CD myeloid cells against normal myeloid cells using the Wilcoxon rank-sum test.

The top upregulated genes were:

Gene	What it does
S100A8	Subunit of calprotectin - a protein released by activated immune cells
S100A9	The other subunit of calprotectin
CXCL8	Also known as IL-8 - a chemical signal that recruits more immune cells to the inflammation site
IL1RN	An anti-inflammatory signal - the body trying to dampen its own response
BCL2A1	Keeps immune cells alive longer in the inflamed environment

Here's why S100A8 and S100A9 matter: together they form a protein called calprotectin. When gut inflammation is active, immune cells release calprotectin into the stool. Doctors measure this in a routine test called the faecal calprotectin test - one of the most common non-invasive ways to monitor IBD disease activity in clinic.

By analysing raw single-cell data, we independently identified the exact genes behind this clinical test - at the resolution of individual cells. That's the kind of validation that confirms the analysis is biologically meaningful, not just a statistical artefact.

What I learned - technical takeaways

1. Batch correction is non-negotiable for multi-sample studies. Without Harmony, the donor-to-donor variation would swamp the disease signal. Any "finding" could be explained by technical noise. Harmony is now a standard step in any multi-patient single-cell study.

2. Choosing highly variable genes with batch_key matters. If you find HVGs without accounting for batches, you risk selecting genes that are variable only because of technical differences between samples. Using batch_key='donor_id' ensures you're finding genes that are genuinely biologically variable.

3. Clinical relevance is the best validation. When your computational analysis independently reproduces a well-established clinical biomarker (faecal calprotectin), it gives you confidence that the pipeline is working correctly and the findings are real.

4. Public datasets are powerful. This entire analysis used freely available data from CZ CELLxGENE - no lab access required. The tools (Scanpy, harmonypy) are free and open source. Computational biology has an exceptionally low barrier to entry compared to wet lab science.

The full code

The complete annotated notebook is available on GitHub:
👉 https://github.com/Farhan89082/ibd-harmony-integration

The README includes all figures, biological interpretation, and instructions for reproducing the analysis from scratch.

What's next?

This project is part of a series of three single-cell RNA sequencing analyses I've built for my computational biology portfolio:

Alzheimer's Disease - microglial activation and mitochondrial dysfunction in human brain cells → https://github.com/Farhan89082/alzheimers-scrna-analysis
NSCLC Tumour Microenvironment - T cell exhaustion trajectories and macrophage polarisation in lung cancer → https://github.com/Farhan89082/nsclc-tumour-microenvironment
IBD Harmony Integration - this article

If you're a Python developer curious about getting into computational biology, or a biology student learning to code, I hope this walkthrough shows that the barrier is lower than it looks. The tools are excellent, the data is freely available, and the biology is genuinely fascinating.

Have questions about the analysis or want to discuss the methodology? Drop a comment below - I'd love to hear from you.

T Cells, Tumour Macrophages, and Why Lung Cancer Evades Your Immune System

Sat, 04 Jul 2026 19:50:20 +0000

One of the most frustrating puzzles in cancer biology: some lung cancer patients respond brilliantly to immunotherapy. Others don't respond at all. The tumour microenvironment (TME), the ecosystem of immune, stromal, and cancer cells that surrounds a tumour is a big part of why.

I wanted to understand what that ecosystem actually looks like at the resolution of individual cells. So I built a single-cell RNA sequencing (scRNA-seq) analysis of non-small cell lung cancer (NSCLC) tissue using Scanpy, working with a publicly available dataset of 111,683 cells from CZ CELLxGENE, spanning 38 distinct cell states.

Here's what the data revealed about T cell exhaustion, macrophage behaviour, and what makes some tumours more immune-suppressive than others.

Background

NSCLC accounts for approximately 85% of all lung cancer cases. Immune checkpoint inhibitors have transformed treatment for some patients — but a large proportion still don't respond. The leading explanation is that the TME is actively suppressing anti-tumour immunity, rather than the immune system simply being absent.

To understand this, you need to look not just at whether immune cells are present, but at what functional state those cells are in. That's exactly what single-cell RNA sequencing makes possible.

The Dataset

The data comes from a study characterising the cellular and molecular identities of histologic subtypes in lung adenocarcinoma:

111,683 cells after QC (117,266 raw)
57,398 genes
38 distinct cell states across the TME
4 histologic subtypes: Acinar/Papillary (A/P), A/P + Solid, Micropapillary (MP), and Solid
Sequenced with 10× Chromium 3' v3

The Pipeline

1. Quality Control

Cells were filtered to retain those with 200–8,000 detected genes, fewer than 100,000 UMI counts, and less than 25% mitochondrial content. This removed around 5,583 low-quality cells.

2. Normalisation

Standard scRNA-seq approach: normalise to 10,000 counts per cell, log1p transform, identify 2,544 highly variable genes.

3. Dimensionality Reduction

PCA (50 components) → neighbourhood graph → UMAP. The resulting embedding resolved all 38 cell states, with clear separation between tumour epithelial cells, immune populations, and stromal cells.

The Findings

1. T Cell Exhaustion Is Not an End State - It's a Trajectory

This was the most biologically interesting result. The CD8+ T cell data revealed three functionally distinct states forming a clear exhaustion trajectory:

T.CD8.Naive → T.CD8.Predysfunc (8,897 cells) → T.CD8.Exhausted (1,454 cells)

The pre-dysfunctional population is the largest CD8 group - meaning the majority of cytotoxic T cells in these tumours are actively being pushed toward dysfunction, not yet exhausted but clearly losing their killing capacity.

The fully exhausted population co-expresses multiple immune checkpoint molecules simultaneously: CTLA4, LAG3, TIGIT, TOX, ENTPD1, and CXCL13. In contrast, cytotoxic T cells express effector molecules (GZMB, PRF1, IFNG) with minimal checkpoint expression. The dotplot makes this contrast striking and clear.

2. Regulatory T Cells Outnumber Cytotoxic T Cells ~4:1

Tregs (6,496 cells) vastly outnumber cytotoxic CD8+ T cells (1,688 cells). Tregs express high levels of CTLA4 and TOX, consistent with their role in actively suppressing anti-tumour immunity. This ratio alone explains a great deal about why these tumours resist immune attack.

3. Pro-Tumour Macrophages Dominate the Myeloid Compartment

Five macrophage subpopulations were identified:

Mac.SPP1 and Mac.SPP1.GPNMB (~7,054 cells combined) - pro-tumourigenic, SPP1/osteopontin-expressing macrophages associated with poor prognosis in multiple cancer types
Mac.SELENOP (3,082 cells) - anti-inflammatory, tissue-resident
Mac.CXCL9 (1,118 cells) - anti-tumour, recruits cytotoxic T cells

The pro-tumour to anti-tumour macrophage ratio is approximately 6:1. Combined with the T cell exhaustion data, this paints a picture of a TME that is actively - and efficiently - suppressing immune responses at multiple levels simultaneously.

4. The Paradox of Solid Tumours

Solid histologic subtype tumours had the highest immune infiltration (~75% immune cells vs ~63% in acinar/papillary). But solid subtype is also the most aggressive histology.

This counterintuitive finding is consistent with what's sometimes called immune suppression rather than immune exclusion: more immune cells arrive, but the suppressive environment is so potent that they can't function. High infiltration doesn't automatically mean effective anti-tumour immunity.

What This Means for Immunotherapy

The data points to a few specific vulnerabilities worth thinking about:

The large pre-dysfunctional T cell population is a potential therapeutic opportunity - these cells aren't fully exhausted yet and might respond to checkpoint blockade
The 6:1 pro-tumour macrophage ratio suggests that myeloid reprogramming strategies (not just T cell-targeting therapies) may be needed
The Treg suppression appears to work partly through CTLA4, which is why anti-CTLA4 therapy has shown activity in some NSCLC settings

What I Learned

Subsetting for focused analysis pays off. Rather than analysing all 38 cell states at once, isolating the T cell and macrophage compartments separately and running marker analysis on those subsets gave much cleaner, more interpretable results.
Using the authors' cell type annotations directly was the right call. For a dataset this complex, re-clustering from scratch would have introduced unnecessary uncertainty. Their labels are validated and biologically grounded.
With 38 cell states and multiple findings, deciding what to prioritise in the write-up took as long as some analysis steps. Storytelling is part of the work.

The Code

Everything is in a fully annotated Jupyter Notebook. Download the H5AD file from CZ CELLxGENE and place it in the data/ folder to run it. https://github.com/Farhan89082/nsclc-tumour-microenvironment

The TME is one of the most complex biological systems we can now study at single-cell resolution - and it's increasingly clear that understanding it is key to making immunotherapy work for more patients. Happy to discuss the analysis, the biology, or the pipeline in the comments.

How I Mapped Brain Cell Changes in Alzheimer's Disease Using Single-Cell RNA Sequencing

Sat, 04 Jul 2026 19:49:43 +0000

Alzheimer's disease affects over 55 million people worldwide, yet the precise molecular changes happening inside individual brain cells remain poorly understood. I wanted to dig into that question - not at the tissue level, but at single-cell resolution.

So I built a full scRNA-seq analysis pipeline in Python using Scanpy, working with a publicly available dataset of 63,608 nuclei from human prefrontal cortex tissue (sourced from CZ CELLxGENE). The donors spanned three Braak stages: 0 (cognitively normal), 2 (early Alzheimer's), and 6 (severe Alzheimer's).

Here's what I found and how I found it.

The Dataset

The data came from a study on the molecular characterisation of selectively vulnerable neurons in AD. It covers the superior frontal gyrus, a prefrontal region known to be hit hard by neurodegeneration - and includes seven major brain cell types:

Glutamatergic neurons
GABAergic neurons
Oligodendrocytes
OPCs (oligodendrocyte precursor cells)
Astrocytes
Microglia
Endothelial cells

31,997 genes. 63,608 cells. Three disease stages. A lot to work with.

The Pipeline

1. Quality Control

No dataset is clean out of the box. I filtered cells to keep only those with between 200 and 6,000 detected genes, and excluded anything with more than 20% mitochondrial gene content (high mitochondrial reads usually signal a dying or damaged cell). This removed around 2,809 low-quality cells.

2. Normalisation

Library sizes were normalised to 10,000 counts per cell, followed by log1p transformation, standard practice that makes cells comparable regardless of how deeply they were sequenced. I then identified 5,607 highly variable genes to focus the downstream analysis.

3. Dimensionality Reduction

PCA (50 components) → neighbourhood graph (10 neighbours, 20 PCs) → UMAP embedding.

The UMAP is where the biology starts to become visible. All seven cell types separated into distinct clusters, with clear separation between neuronal subtypes and glial populations.

4. Differential Expression

For the microglial analysis, I used a Wilcoxon rank-sum test comparing AD vs normal microglia, with Benjamini-Hochberg multiple testing correction to control the false discovery rate.

The Findings

Glutamatergic Neurons Are Selectively Depleted

One of the most striking results: glutamatergic (excitatory) neurons dropped from ~34% of cells in normal tissue to ~30% in AD tissue. This might sound like a small shift, but at the scale of 60,000+ cells it's biologically meaningful and it's consistent with what the literature already tells us about the selective vulnerability of excitatory neurons in AD.

Alzheimer's Leaves a Clear Signature in Microglia

Microglia are the brain's resident immune cells, and they showed the most dramatic transcriptomic shifts between AD and normal tissue. The differential expression analysis revealed:

Upregulated in AD microglia:

MALAT1 - a long non-coding RNA strongly linked to neuroinflammation
FTH1 - ferritin heavy chain, pointing to iron dysregulation
B2M - beta-2 microglobulin, a known AD biomarker reflecting immune activation
FOXP1 - a transcription factor tied to microglial activation states

Downregulated in AD microglia:

MT-CO3, MT-CO1, MT-ATP6, MT-ND2 - mitochondrial complex genes, suggesting impaired energy metabolism in AD-affected microglia

This pattern is consistent with what's described as disease-associated microglia (DAM) in the literature, a distinct activation state that emerges in neurodegeneration.

Disease Progression Captured Across Braak Stages

Cells from all three Braak stages were distributed across every cluster in the UMAP. This reflects that AD-associated transcriptomic changes are not confined to one cell type, they propagate across the whole cellular ecosystem as the disease progresses.

What I Learned

Memory management matters. 60K+ cells × 30K+ genes is a big matrix. Working with sparse AnnData objects and being deliberate about which steps you checkpoint to disk makes a real difference.
Cell type annotation is an art. The dataset came with pre-annotated cell types, but validating them against canonical marker genes (the dotplot step) is essential and satisfying when the biology confirms itself.
Volcano plots are still one of the most readable ways to communicate differential expression. They give you significance and fold change in one glance.

The Code

Everything is in a fully annotated Jupyter Notebook. If you want to reproduce the analysis, download the H5AD file from CZ CELLxGENE and drop it in the data/ folder. https://github.com/Farhan89082/alzheimers-scrna-analysis

If you're working with single-cell data or have questions about the pipeline, I'd love to hear from you in the comments. There's something fascinating about watching biology emerge from a matrix of gene counts.

How Excel is used in Real World Data Analysis

Wed, 17 Jun 2026 07:34:38 +0000

EXCEL

This is a spreadsheet software developed by Microsoft used to insert , read and manipulate data.

This is an example of how an excel page looks like. It is called a workbook

Basics of excel

Formula bar - Area above the grid where the contents or formula of the selected cell appear.
Column header - Is the label at the very top of a column in a spreadsheet, database, or web table.
Row header - Is the first column in a data table that identifies the specific information, category, or ID for the corresponding data to its right.
Selected cell/Active Cell - Is the currently selected cell in a spreadsheet where you can directly enter data or formulas.
Sheet tab - Tabs at the bottom (Sheet1, Sheet2) that switch between spreadsheets within a workbook.
Cell - A single box where a row and a column intersect eg D9.
Row - Labelled 1,2,3... down the left side horizontally .
Columns - Labelled A,B,C... across the top vertically.

Ways in which Excel is used in real world data analytics

1. Financial reporting

Most business uses excel to create financial reports, forecast revenue and analyze profitability. Excel can also help track expenses , manage cash flow and optimize financial decision making.

This is an example of a profit and loss statement

2. Healthcare & Research

Patients records, Lab results , clinical results data are stored in excel . Lab results can be used to predict future outbreaks of diseases

This is an example of a medical research KPI dashboard in excell

3. Marketing and Sales

Excel is used in sales tracking, inventory control,revenue targets, conversion rates and supply chain management. Businesses can use formulas to calculate stock levels, reorder points, and sales growth trends. It helps companies reduce stockouts, prevent overstocking, and improve operational efficiency.

This is an example of an excel sales forecasting

4. Business forecasting & Decision making

This is predicting what will happen in the future to be able to make decision based on the prediction.By using historical data and forecasting models, businesses can predict market trends,plan for future growth, assess risk and make data-driven decisions.

This is an example of a forecast sheet

5. Automation & Productivity

Excel automation features like templates and predefined functions help businesses save time and reduce manual errors. It allows companies to automate repetitive tasks such as data entry, invoicing, and payroll processing, increasing overall productivity.

This is an example of an automated Periods Table.

Formulas in Excel

In excel they are very many function that makes data analysis simple . Here a few of the main functions one should know.

1. Sum

Adds all the values in the said range
E.g in the below it add all values from B2,B3,...,B8

=SUM(B2:B8)

2. Average

Finds the average of values in the said range
E.g In the below cell it finds the average of all values between cell D2,D3,...,D22

=AVERAGE(D2:D22)

3. Max

Finds the max number in said range of values
E.g In the below cell it finds the maximum number in all cells from F3,F4,...,F10

=MAX(F3:F10)

4. Count

Count cells with numbers
E.g It returns the number of cells with numbers

=COUNT(H2:H10)

5. Filter

Extracts data from a table based on criteria you specify.
E.g Checks the data and instantly spill out a list of only the rows where sales (Column C) are greater than 5000.

=FILTER(A2:C100, C2:C100>5000, "No Sales")

6. Sort

Sorts the contents of a range or array automatically.
E.g Sorts the data range based on the 2nd column, in descending order (highest to lowest).

=SORT(A2:B100, 2, -1)

7. IF

It checks if a condition is true or false and gives you a different result depending on the outcome.
E.g If a student's grade in cell B2 is 70 or higher, Excel prints "Pass". If it's 69 or lower, it prints "Fail".

=IF(B2>=70, "Pass", "Fail")

8. Median

Returns the exact middle number in a given set of numbers.
E.g Finds the middle value of all numbers in that range.

=MEDIAN(C2:C50)

9. Mode

Returns the most frequently occurring or repetitive value in a dataset.
E.g It shows you which number is most frequently.

=MODE(D2:D100)

10. Proper

It capitalizes the first letter of every word in a text string and converts all other letters to lowercase.
E.g If cell A2 contains the messy text: "jOHN mCcLAIN" the formula will cleanly output: "John Mcclain"

=PROPER(A2)

11.COUNTIF

Returns the value in a given a range given one condition
E.g How many total deals did the North region close?

=COUNTIF(B2:B100, "North")

12. COUNTIFS

Returns the value in a given range with more than one condition
E.g Returns the number of deals in the North region are still 'Pending'

=COUNTIFS(B2:B100, "North", D2:D100, "Pending")

13.SUMIF

Adds the values in a range that meet a single, specified condition
E.g Adds the total revenue brought in by the sales rep 'Sarah'?

=SUMIF(A2:A100, "Sarah", C2:C100)

14.SUMIFS

Adds all numbers in a range that meet multiple conditions
E.g Adds total revenue brought in by 'Sarah' for deals larger than $10,000?

=SUMIFS(C2:C100, A2:A100, "Sarah", C2:C100, ">10000")

15. AVERAGEIF

Calculates the average of numbers in a range that meet a specific condition
E.g Calculate the average deal size in the South region?

=AVERAGEIF(B2:B100, "South", C2:C100)

16. AVERAGEIFS

Calculates the averages of numbers in a specific range based on more than one conditions.
E.g Calculates the average size of 'Closed' deals in the South region?

=AVERAGEIFS(C2:C100, D2:D100, "Closed", B2:B100, "South")

17. ROUND

Used to round a number to a specified number of digits
E.g A sales commission calculates out to $1,250.4567. You need to round it to standard currency (2 decimal places) for payroll.

=ROUND(1250.4567, 2) returns 1250.46

18. ROUNDUP

Used to round a number up
E.g You need to ship items to a client. A box holds 10 items, and they ordered 41 items ($41 / 10 = 4.1$). You can't ship 4.1 boxes; you must round up to ensure you have enough physical boxes.

=ROUNDUP(4.1, 0) returns 5.

19. INT

Rounds a number down to the nearest whole interger by removing its decimals
E.g An employee clocked 38.75 hours this week. You want to extract just the full, completed hours they worked.

=INT(38.75) returns 38

20.ABS

Returns the positive value of a number by removing its negative sign
E.g If a refund is logged as -$450.00, you want to know the absolute dollar volume of the transaction.

=ABS(-450) returns 450

21. TODAY()

Is a formula that updates the day automatically
E.g Finding out exactly what day it is today to calculate invoice aging.

=TODAY() returns 06/05/2026 (based on today's date)

22. NOW

Inserts the current date and time
E.g Stamping a report so stakeholders know exactly what minute the data was pulled from the server.

=NOW() returns 06/05/2026 15:02

23. YEAR

To extract year in a date
E.g You want to group sales by their calendar year.If cell A2 holds 06/05/2026.

=YEAR(A2) returns 2026

24.WEEKDAY

To extract the day of the week in excel
E.g Checking if a client submitted a payment on a weekend so you can account for bank processing delays.

=WEEKDAY(A2) returns 6 (which represents Friday)

25. WEEKNUM

Finds the week number of a specific date
E.g

Excel is a very important tool in Data analysis. We can say it the mother of all since it can do almost all the analysis. Excel needs to be one of the most important tools one has to learn to be able to understand data analysis.

My Data Journey

Wed, 17 Jun 2026 07:33:46 +0000

Background

As a young kid i realized i had a passion with Mathematics. I would really struggle understanding the other subjects but Math to me was so easy . When i finished highschool the big question was what i wanted to do . I already new that i wanted to do something associated with Math. I did my research and the course that was coming up more frequent was Acturial Science . I applied for the course later on while still doing more research Data science came up and i was curious about it . Looked it up and so that the career opportunities were what i would be intrested . I was later called for Acturial Science but did a course transfer to Data Science

Life in Data Science

Back to September 2023 and i began my degree in Data Science . At first i was happy with what i was learning and enjoyed . The only challenge i faced is i expected alot of practical work and i only recieved the theory part of most my units. I wanted to learn the theory but i didn't know where to start. I tried out different online contents but i was never satisfied with whatever i was learning . I still was eager to get my hands on real data.As this was the reason i joined Lux academy to get hands on skill on Data Science

All my best as i begin this journey

The Leash That Makes AI Polite

Tue, 09 Jun 2026 05:30:00 +0000

How a 1951 statistics formula quietly keeps your chatbot from going feral

Two ideas sit underneath every modern AI assistant you've ever used. One is a seventy-year-old equation from information theory. The other is the training trick that turned a raw text-prediction engine into something you'd actually want to talk to. They're usually explained separately — but the interesting part is where they meet.

First: a way to measure surprise

Start with the Kullback–Leibler divergence. The name is intimidating; the idea isn't. It measures how far apart two sets of expectations are.

Picture a weather forecaster. Reality is 70% sunny, 30% rain. But you wake up convinced it's a coin flip — 50/50 — and dress accordingly. KL divergence is the price you pay, on average, for believing the wrong thing. Get caught in a drizzle, get overdressed in the sun: small mismatches, small cost. If instead you'd believed it was always sunny and walked straight into a downpour, the cost would be enormous.

That's the whole concept. Two distributions — how things actually are, and what you assumed — and a single number for the gap between them.

Three things are worth keeping in your pocket. It's zero only when your belief is perfect. It can climb toward infinity when you're badly wrong, so it's not a tidy score that sits between 0 and 1. And it's lopsided — being too optimistic and being too pessimistic don't cost the same amount. It isn't a neutral ruler; it's a directed penalty for surprise. (If you want the math laid out gently, with code, the DataCamp tutorial below is a good next stop.)

Then: teaching a model some taste

Now the second idea. A raw language model is a spectacularly good guesser of the next word, trained on a firehose of internet text. What it is not is helpful, honest, or polite. Left alone, it'll happily continue a sentence in whatever direction the statistics pull it.

RLHF — Reinforcement Learning from Human Feedback — is what fixes that. The recipe, roughly: show humans two answers to the same question, ask which is better, collect thousands of those judgments, and train a second model to predict human taste. Then nudge the language model to chase higher scores from that taste-model. That's how it learns to be useful instead of merely plausible.

Where they meet

Here's the catch. Let a model chase a reward with no restraint, and it cheats. It discovers that flattery scores well, that vague hedging is rarely marked wrong, that "Great question!" pleases the crowd. Unchecked, it drifts into a confident, sycophantic mush — gaming the scoreboard while quietly forgetting how to talk.

So engineers attach a leash. At every step they ask: how far has the model drifted from its original, pretrained self? That distance is measured with KL divergence. Drift a little — fine. Drift too far, and the penalty pulls it back.

The polished, agreeable voice of your favorite chatbot is the product of exactly this tension: a reward tugging it toward what people like, and a 1951 formula tugging the other way, whispering don't go feral.

Go deeper

KL divergence, with intuition and code — DataCamp's tutorial: https://www.datacamp.com/tutorial/kl-divergence
RLHF, from the ground up — Nathan Lambert's Reinforcement Learning from Human Feedback (free online book): https://arxiv.org/abs/2504.12501 (web version at rlhfbook.com)

Lessons from a Kaggle Playground Competition

Sun, 07 Jun 2026 08:44:50 +0000

Just participated in this month's Kaggle Tabular Playground challenge focused on Stellar Classification

The goal was to classify astronomical objects into:

STAR
GALAXY
QSO (Quasar)

A few interesting findings from the exploration:

Redshift turned out to be one of the strongest predictive signals.
Color indices such as u-g, g-r, and r-i were often more informative than raw photometric measurements.
A simple ensemble of LightGBM, XGBoost, and CatBoost performed surprisingly well.
Despite a moderately imbalanced dataset (~65% Galaxy, ~20% QSO, ~14% Star), explicit class weighting actually reduced validation performance in my experiments.
Hyperparameter tuning with Optuna did not outperform a carefully engineered baseline model, reinforcing the importance of feature engineering and domain understanding.

To document the entire process, I created a beginner-friendly guided notebook covering:

• Dataset understanding
• Exploratory analysis
• Feature engineering
• Categorical encoding
• Stratified K-Fold validation
• LightGBM, XGBoost, and CatBoost
• Model ensembling
• Experimentation with class imbalance and hyperparameter tuning

If you're learning machine learning, working on tabular data, or interested in astronomy-related datasets, feel free to take a look.

I'm also happy to discuss alternative approaches, feature engineering ideas, or leaderboard strategies. Constructive feedback and questions are always welcome.

Notebook Link: Click Here

Data Sanitization is a Design Flaw: How Penta-V Prevents Data Pollution at the Hardware-Software

Mon, 18 May 2026 03:10:10 +0000

Go to GitHub, search for "data cleaning library," and you will find thousands of repositories. From Pandas to Pydantic, the entire software engineering industry has been conditioned to treat data sanitization as a reactive, post-facto chore. You ingest data, you realize it’s full of missing values, NaNs, or volatile floating-point noise (Inf), and then you write heavy, resource-expensive loops to "clean" it.

Here is the cold, hard truth that legacy architectures refuse to admit: If your system is spending CPU cycles cleaning data inside its execution core, your architecture has already failed. In high-frequency pipelines and autonomous AI environments, traditional cleaning introduces massive latency spikes, destroys CPU cache locality, and leads to Logic Drift—where the state of your system silently decays.

When we engineered the Penta-V Kernel (v0.4.0), we didn't build another data-cleaning library. We built src/processing/cleaner.rs (PentaCleaner) to prove a different architectural paradigm: We don't clean data; we structurally prevent data pollution from existing.

The Legacy Trap: Why Runtime Cleaning Kills High-Frequency Systems

In standard hybrid pipelines (Python bridging into Rust via PyO3), data passed from autonomous AI agents is inherently probabilistic and volatile.

When a naive system encounters corrupted data at runtime, it usually triggers one of two reactive paths:

The Allocation Nightmare: It instantiates dynamic buffers on the heap to reconstruct tables (e.g., df.fillna()), causing memory fragmentation and triggering garbage collection jitter.
The Unsafe Race: It forces raw parallel mutation via unsafe pointers, risking data races and undefined behavior (UB) under sudden systemic stress.

Penta-V eliminates this by transforming data purification from an operational step into an Adaptive Geometric Constraint.

1. Zero-Allocation Parallel Extraction (Eliminating Memory Debt)

Instead of allocating new memory frames to clean data, the PentaCleaner leverages a highly optimized, lock-free parallel extraction pipeline driven by safe Rust abstractions (Polars and Rayon).

Look at how the core execution loop handles a contaminated DataFrame:

// src/processing/cleaner.rs

pub fn geometric_sanitize(df: &mut DataFrame, state: &ProcessingState) -> PolarsResult<()> {
    let pressure = state.data_pressure;

    // SAFE & LOCK-FREE PARALLELISM:
    // We leverage low-level pointer duplication (shallow clones) to parallelize 
    // column purification across all available CPU cores without single-line heap allocations.
    let updated_columns: Vec<Series> = df
        .get_columns()
        .par_iter()
        .map(|series| {
            let mut clned = series.clone(); // Cheap shallow clone (O(1) pointer copy)
            Self::purify_column(&mut clned, pressure);
            clned
        })
        .collect();

    // Reconstruct the DataFrame in a single CPU cycle by consuming the safe vector
    *df = DataFrame::new(updated_columns)?;

    Ok(())
}

Why this is prevention, not cleaning:

By enforcing compile-time data aliasing rules, the Rust compiler guarantees that no two threads can cause a data race. The allocation cost is exactly zero because series.clone() in Polars is merely a shallow reference duplication. The system doesn't wait for data to become a problem; it enforces a strict parallel layout where invalid structures cannot propagate.

2. Pressure-Aware Structural Adaptation (The Anti-Fragile Gate)

A static data cleaner is blind. It executes the exact same parsing script whether your server is idling at 5% or burning at 99% capacity. Under extreme load, this blindness causes memory queues to back up, leading to catastrophic system throttles.

Penta-V introduces Pressure-Aware Sanitization. The kernel dynamically mutates its purification strategy based on the thermodynamic state of the infrastructure (data_pressure):

fn purify_column(series: &mut Series, pressure: f64) {
    // Dynamic Strategy Selection based on real-time hardware telemetry
    let strategy = if pressure > 0.8 {
        // High-Intensity: Forward fill maintains the continuous geometric sequence 
        // under extreme load without interrupting the ALU pipeline.
        FillNullStrategy::Forward(None)
    } else {
        // Standard Baseline: Zero-fill enforces an absolute mathematical baseline.
        FillNullStrategy::Zero
    };

    if let Ok(filled) = series.fill_null(strategy) {
        *series = filled;
    }
}

The Engineering Magic:

When system stress crosses the critical 0.8 barrier, the kernel automatically swaps the logic profile. It stops attempting heavy static recalculations and shifts to an Asymptotic Geometric Sequence (Forward Fill). This prevents the execution queue from stalling, absorbing the kinetic shock of incoming data anomalies in real-time.

3. The Structural Immunity Audit: Penta-V vs. The Market

To understand why this approach renders traditional cleaning obsolete, we must look at how data corruption behaves across different software layers:

Architectural Metric	Traditional Cleaners (`Pandas` / `Pydantic`)	Native Engines (`Polars` Alone)	Penta-V Kernel Substrate (`PentaCleaner`)
System Interaction	Reactive (Cleans after corruption infects the memory).	Pure Processing (Executes instructions blindly).	Sovereign Prevention (Sanitizes at the FFI boundary before core access).
Hardware Awareness	Completely Blind (Heavy CPU/Memory overhead).	Static Multi-threading (Fixed execution path).	Dynamic Telemetry (`Pressure-Aware` adaptation under load).
Memory Delta	High (Continuous heap allocations and fragmentation).	Optimized but subject to continuous runtime re-allocations.	0.0000 MB Dynamic Delta via stack-resident pointer manipulation.
Execution Latency	Milliseconds ($ms$) — Destroys high-frequency pipelines.	Microseconds ($\mu s$).	Sub-nanosecond (0.85ns flat) via bit-level vector alignment.

Conclusion: Stop Cleaning. Start Hardening.

Writing code to "clean" corrupted data is a tacit admission that your system boundaries are porous. In the era of autonomous AI codebases and ultra-low latency requirements, you cannot afford to let bad data enter your core logic and then figure out how to patch it.

The PentaCleaner module within the Penta-V Kernel shifts the paradigm. By combining the raw parallel power of Polars and Rayon with a pressure-sensitive, zero-allocation design, it treats data corruption not as an administrative problem to be logged, but as a kinetic deficit to be neutralized.

When your system boundaries are architecturally incorruptible, "data cleaning" ceases to be a maintenance cost—it disappears from existence.

Full Source & Docs:

GitHub: https://github.com/narukihto/Penta-V-Kernel/tree/Heartbeat

Crates.io: https://crates.io/crates/penta_v_kernel

PyPI: https://pypi.org/project/penta-v-kernel/

Penta-V Kernel (v0.4.0) is compiled, hardened, and testing green at sub-nanosecond intervals. Let’s argue about system architecture in the comments below! ️⚡

How to Publish and Share a Power BI Report Online

Thu, 14 May 2026 10:11:40 +0000

As we continue to explore Data Science and Analytics, this weeks deliverable was to work on an electronics sales dashboard in PowerBI and publish it online. This is a common request in data analytics so in this article we will walk through the full process of publishing a Power BI report to the web, from clicking "Publish" in Desktop to getting a shareable link that anyone can open in a browser.

The Project in Brief

Before jumping into publishing, here is a quick overview of what the report contains so you have context for the screenshots:

Dataset: 100 electronics retail transactions across 10 products, 9 cities (USA and Canada), and 3 sales representatives.

Data preparation: The raw Excel data was loaded into Power Query where it was cleaned, validated, and normalized. Canadian Dollar transactions were converted to USD at a rate of 0.74. A star schema was built with six dimension tables (Products, Customers, Cities, SalesReps, PaymentMethod, DateTable) linked to a central fact table.

DAX measures: Total Revenue, Total Profit, Total Orders, Average Profit Margin, Average Shipping Cost, and a calculated Date Table using the CALENDAR() function.

Dashboard pages:

Page 1 - Executive Dashboard: KPI cards, sales by product, revenue by category, sales rep distribution, payment method breakdown, and interactive slicers.

Page 2 - Product Analysis: Revenue and profit comparison by product, geographic maps, profit margin rankings.

Page 3 - Sales Reps and Customers: Customer rankings, sales rep performance combo chart, customer age distribution.

With the report complete in Power BI Desktop, the next step is getting it online.

Step 1: Publish from Power BI Desktop

Open your .pbix file in Power BI Desktop. In the top ribbon, go to Home and click the Publish button on the right side of the toolbar.

If you are not signed in, Power BI will prompt you to sign in with your Microsoft account. Use the account associated with your Power BI Service subscription.

Once signed in, a dialog box appears asking you to select a destination workspace.

You will see a list of available workspaces. If you have not created any custom workspaces, you will only see My workspace, which is the default personal workspace that comes with every Power BI account.

Select the workspace you want and click Select.

Power BI Desktop will upload the report. This usually takes a few seconds depending on the file size. When it finishes, you get a success confirmation.

Click the "Open in Power BI" link to jump directly to your report in the browser, or click Got it to stay in Desktop.

Step 2: Verify in Power BI Service

Head over to app.powerbi.com and navigate to the workspace where you published. You should see your report listed there.

Open the report and spend a few minutes checking everything:

Do all visuals render correctly?
Do the slicers filter across all charts as expected?
Do navigation buttons (if you added any) work?
Are the KPI numbers matching what you see in Desktop?

In the screenshot above, you can see the report is live in the Power BI Service. The left sidebar shows the Pages panel listing all four pages (Dashboard, Products, Sales Reps and Custom...). The report is fully interactive - slicers, tooltips, and filtering all work just like they do in Desktop.

Notice the toolbar at the top: File, Export, Share, Explore, Set alert, Monitor, and Edit. These are the options available to you as the report owner. The ones we care about for sharing are File, Export, and Share.

This is where it gets interesting, and also where you might run into some friction depending on your account type. There are several ways to share a Power BI report, each with different access requirements.

Publish to Web (Public Embed)

This is the simplest option for making your report accessible to anyone without requiring them to have a Power BI account.

Path: File > Embed report > Publish to web (public)

This generates two things:

A direct URL that opens the report in any browser
An iframe embed code that you can paste into a website, blog, or portfolio page

The direct link looks something like:

https://app.powerbi.com/view?r=eyJrIjoi...

The iframe code looks like:

<iframe
  title="Electronics Sales Dashboard"
  width="1140"
  height="541.25"
  src="https://app.powerbi.com/view?r=eyJrIjoi..."
  frameborder="0"
  allowFullScreen="true">
</iframe>

Anyone with the link can view and interact with the report. No sign-in required.

Step 4: Embed on a Website (If Publish to Web Worked)

If you got the embed code from "Publish to web", you can host your report on any website. Here are two common approaches:

Simple HTML Page

Create an index.html file:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Electronics Sales Dashboard</title>
    <style>
        body {
            margin: 0;
            font-family: Arial, sans-serif;
            background: #f5f5f5;
        }
        .header {
            background: #1B2A4A;
            color: white;
            padding: 20px 40px;
        }
        .header h1 {
            margin: 0;
            font-size: 24px;
        }
        .header p {
            margin: 8px 0 0;
            font-size: 14px;
            color: #B5D4F4;
        }
        .container {
            max-width: 1200px;
            margin: 24px auto;
            padding: 0 20px;
        }
        iframe {
            width: 100%;
            height: 600px;
            border: none;
            border-radius: 8px;
        }
    </style>
</head>
<body>
    <div class="header">
        <h1>Electronics Sales Dashboard</h1>
        <p>Power BI Report -- January 2026</p>
    </div>
    <div class="container">
        <iframe
          title="Electronics Sales Dashboard"
          src="YOUR_EMBED_URL_HERE"
          allowFullScreen="true">
        </iframe>
    </div>
</body>
</html>

Replace YOUR_EMBED_URL_HERE with the actual embed URL from Power BI.

Host on GitHub Pages

If your project is already in a GitHub repo, you can host the embed page for free using GitHub Pages:

Create a subfolder in your repo (e.g., electronics-sales/).
Put the index.html file inside it.
Push to GitHub.
Go to Settings > Pages in your repo.
Set the source to deploy from your main branch.
After a minute or two, your page will be live at https://yourusername.github.io/repo-name/electronics-sales/.

If your repo contains multiple projects, each subfolder with its own index.html becomes a separate page under the same base URL. You only need to enable GitHub Pages once per repo.

Summary

The publishing workflow in Power BI goes like this:

Build the report in Power BI Desktop.
Publish to a workspace in the Power BI Service.
Verify that everything works in the browser.
Share using one of the available methods (Publish to web, direct share, or export).
Embed on a website or GitHub Pages if you want a standalone hosted version.

I hope you enjoyed this guide. If you would like to learn more about the PowerBI project, the insights gained and the original data used, check out my GitHub repo that contains the project: Electronics Sale Dashboard

Python in data analytics: a beginners guide

Thu, 14 May 2026 10:10:22 +0000

Python started as a hobby project. Guido van Rossum spent Christmas break in 1989 writing an interpreter he described as "a little scripting language." He published it in 1991. That origin matters because it shows how much Python has developed to get to where it is today. Python has grown to be one of the most popular programming language. In data analytics specifically, Python overtook R, SAS, Excel, and other proprietary tools not because it was marketed well, but because analysts kept choosing it for each new problem they encountered.

What Python actually is

Python is a general-purpose, interpreted programming language. "Interpreted" means you run code directly without compiling it first, which makes it fast. "General-purpose" means the same language handles web servers, automation, scientific computing, and data analysis.

The syntax is minimal and indentation defines code blocks instead of brackets. You can often read a Python function and understand what it does even if you have never written Python before. Python is also free, open source, and runs on Windows, Mac, and Linux without modification. That combination was actually decisive in getting Python adopted inside companies where budget and IT policy often block paid software.

Why data analysts picked Python over the alternatives

R was excellent for statistics but hard to learn for people without a formal statistics background. SAS was comprehensive but expensive and proprietary. Excel worked for small datasets but fell apart past a few hundred thousand rows. Python was cheap, readable, and already had libraries capable of serious data work.

In 2008, Wes McKinney was a quantitative analyst at AQR Capital Management running financial calculations that he found painful to do in existing tools. He built pandas, a library that gave Python a spreadsheet-like structure for data manipulation. It became public in 2009. By 2012, analysts across finance, tech, and academia had adopted it.

Pandas gave Python something it was missing: a clean way to load a CSV, examine the data, filter rows, group by category, and aggregate columns in a few lines. Before pandas, doing any of that required more setup than most analysts would tolerate.

Once pandas existed, the rest Python adoption moved quickly. More people used Python for data work, which meant more contributors improving the libraries, which attracted more people. That cycle is still running.

The libraries worth knowing

Python itself does not do statistics or draw charts. The libraries do. Four of them appear in almost every data analytics project.

NumPy handles numerical computation. It provides arrays that behave like mathematical vectors and matrices, and operations on those arrays run fast because NumPy's internals are written in C. Most other data libraries sit on top of NumPy.

pandas provides the DataFrame, which is the structure data analysts actually work with. Think of it as a table with column names and row labels. You load a file, you get a DataFrame, you clean the data in that DataFrame, and you pass it to a model or a chart.

Matplotlib handles visualization. John Hunter created it in 2003 to generate figures in Python, drawing some inspiration from MATLAB. It is verbose by modern standards but configurable down to individual pixels when you need that level of control.

Seaborn wraps around Matplotlib and produces statistical charts with less code. If you want a box plot, a correlation heatmap, or a regression line on a scatter plot, Seaborn gets you there faster than raw Matplotlib would.

Cleaning data: where most of the work goes

Most data analysts spend most of their time on data cleaning or handling data that is broken in some way.

Missing values are one of the most common problems. A column might have null entries because a form field was optional or because records were merged from different systems. You cannot simply ignore them; they cause errors in calculations and bias in results.

import pandas as pd

df = pd.read_csv("sales_data.csv")

# Check which columns have missing values and how many
print(df.isnull().sum())

# Drop rows where any value is missing
df_clean = df.dropna()

# Or fill missing values with the column mean
df["revenue"] = df["revenue"].fillna(df["revenue"].mean())

Beyond missing values, you deal with duplicates, incorrect data types (a column that should be numbers stored as text), inconsistent category labels ("New York," "new york," "NY"), and dates stored in five different formats within the same column.

These problems are not glamorous. But fixing them before analysis is the difference between results you can trust and results that only look reasonable.

Analyzing data: from raw numbers to answers

Once the data is clean, Python makes it straightforward to answer questions about it. Say you have sales data for a retail chain across 200 stores and three years. You want to know which regions are growing and which are not.

# Group sales by region and year, then sum revenue
regional_sales = df.groupby(["region", "year"])["revenue"].sum().reset_index()

# Compute year-over-year growth rate
regional_sales["growth"] = regional_sales.groupby("region")["revenue"].pct_change()

print(regional_sales.sort_values("growth", ascending=False).head(10))

It runs and produces a sorted table showing the ten fastest-growing region-year combinations in your dataset.

Beyond grouping and aggregating, Python handles statistical tests, time series analysis, and correlation matrices. SciPy adds t-tests, chi-squared tests, and regression analysis for more rigorous statistical questions. Scikit-learn adds the full range of machine learning models when the goal shifts from describing what happened to predicting what will happen next.

Visualizing data: making the analysis usable

Numbers in a table and numbers in a chart communicate differently. A chart of monthly revenue over three years shows trends, seasonality, and anomalies in ways that scrolling through a spreadsheet does not.

import matplotlib.pyplot as plt
import seaborn as sns

plt.figure(figsize=(12, 5))
plt.plot(df["month"], df["revenue"])
plt.title("Monthly revenue, 2022-2024")
plt.xlabel("Month")
plt.ylabel("Revenue (USD)")
plt.tight_layout()
plt.savefig("revenue_trend.png")

Seaborn makes it easier to produce charts that carry statistical meaning. A box plot shows the full distribution of values including outliers. A heatmap shows correlations between many variables at once. A pair plot generates scatter plots for every combination of numerical columns in your dataset, which is often the fastest way to spot patterns worth investigating before you commit to a specific analysis.

Why beginners should start with Python

The first reason is practical: Python is what the job market wants. Data analyst job postings ask for Python more than any other programming language, by a wide margin in most tech-adjacent industries. The 2023 Kaggle Data Science Survey found that over 87% of respondents use Python regularly in their work.

The second reason is the feedback loop. Python is interpreted, so you see the result of each line immediately. You write a line, run it, see what happens, and adjust. That makes learning faster and less frustrating than compiled languages where you chase errors through a build process before seeing any output.

The third reason is the tooling. There is a tone of freely available browser-based Python environment with no local setup required, which means you can practice real data analysis without installing anything. Jupyter Notebooks let you mix code, output, and notes in a single document, which is how most analysts actually share their work.

This being said, it is important to mention that Python will not teach you statistics, though. You can use it to run a regression without understanding what a regression assumes about your data. The code might run, the output look authoritative, but the result is wrong because the data violated an assumption the analyst did not know to check for. Learning Python alongside statistics, is the right approach.

Finally, it worth noting that when you run into a problem in Python, someone has usually had it before and written about it. Stack Overflow alone has over 2.2 million Python questions and answers as of 2024. Reviewing other people's code is also a good way to learn and develop your own problem solving skills.

Start with a real dataset you care about. Load it with pandas. See what is broken. Fix it. Ask a question. Answer it. That is the whole Python loop.

How far are you in your Python learning journey? Let me know in the comments.

What 500 curated failure pairs actually fix: a breakdown across 3 seeds

Sat, 02 May 2026 11:57:46 +0000

In the previous post, I described the curation philosophy for IDFU's rejected-side dataset — why I avoid synthetic bug generation, why stub detection matters, why "honest failures" are hard to come by.

A few people asked the obvious question: does it work?

This post is the answer. Not as a marketing pitch, but as a breakdown. Aggregate scores hide more than they reveal, and I want to show what's actually changing under the hood.

The setup

Base model: Qwen2.5-Coder-3B-Instruct (trained on 92 programming languages)
Method: DPO via TRL with LoRA
Data: 500 preference pairs from the IDFU dataset
Eval: HumanEval, pass@1, three random seeds (42, 123, 7)
Hardware: RTX 4060, single-GPU, ~3-4 hours of training per seed

A note on the data shape, since I left this implicit in the previous post. Each preference pair has a chosen side and a rejected side, both produced inside IDFU. The rejected side is what Part 1 was about — honest failures, curated rather than synthesized. The chosen side comes from IDFU's internal certified pool — samples that passed all of IDFU's validation gates and are tracked separately from the commercially-released portion. (The gate composition itself is proprietary; this article describes only what the pool produces, not how it's filtered.)

So both sides of the pair are produced and verified by the same internal pipeline, against the same quality bar. The contrast that DPO trains against isn't "model output vs. ideal answer," it's "honest failure vs. honest success on the same task," with both sides held to the same internal standard. That's the configuration that produced the numbers below.

The hyperparameters were deliberately conservative. No sweep, no exotic tricks:

LoRA: r=16, alpha=32, dropout=0.05
Target modules: q/k/v/o/gate/up/down_proj (Qwen standard)
DPO: beta=0.1
Optimizer: learning_rate=5e-5
Batch: size=1, grad_accum=4 (effective batch 4)
Epochs: 3 (= 375 optimizer steps for 500 pairs)
Quantization: 4-bit NF4 + bf16 compute (bitsandbytes)
gradient_checkpointing=True
max_length=2048, max_prompt_length=512

Three seeds because one seed is a coincidence, two seeds is suggestive, and three seeds with consistent direction starts to look like signal. I'd run more if I could afford the GPU time, but training and data generation share the same card, and curation can't pause for a week.

The aggregate

Pass@1 across seeds:

seed=42:   84.1%   delta +3.66 pp
seed=123:  84.1%   delta +3.66 pp
seed=7:    83.5%   delta +3.05 pp
mean:      83.94 ± 0.35%   /   +3.46 ± 0.35 pp

Two seeds landed on the exact same delta, which I noticed and double-checked because it looked too clean. The HumanEval problem set is small (164 problems), failures move in integer counts, and ties at the same delta happen more often than people expect. Seed=7 was lower but in the same direction.

Not earth-shattering. But the headline number isn't really the point.

Where the improvement actually comes from

I categorized every HumanEval failure by exception type. Five categories cover everything:

Category	Base	DPO (mean ± std)	Δ	Relative
ASSERTION_FAIL	23	18.67 ± 0.58	-4.33	-19%
NAME_ERROR	6	3.67 ± 0.58	-2.33	-39%
OTHER_RUNTIME	2	2.67 ± 0.58	+0.67	+33% (n=2)
SYNTAX_ERROR	0	0.33 ± 0.58	+0.33	(1/3 seeds)
TYPE_ERROR	1	1.00 ± 0.00	0.00	0%
Total failures	32	26.34	-5.66	-18%

A few things stand out.

NAME_ERROR drops 39%. This is the cleanest signal. NAME_ERROR on HumanEval is typically the model using a stdlib name without importing it (e.g., math.ceil without import math, or re.match without import re), or referencing an undefined helper. One concrete case I traced was HumanEval/115 (max_fill), where the base model called math.ceil with no import; after DPO training, the model added the import inline. The IDFU rejected pool has a lot of these missing-import patterns, because they're a common failure mode when a model fakes confidence on partial knowledge. The transfer to HumanEval was the most direct prediction I had going in, and it's the one that landed hardest.

ASSERTION_FAIL drops 19%. This one I didn't expect, and the mechanism is interesting enough to be worth tracing.

I went through the 8 newly-passing problems on seed=42 by hand. Six of them were genuine algorithmic improvements — the base model's logic was wrong, the DPO model's logic was right. Standard stuff.

The other two (HumanEval/116 and /123) had identical algorithmic content between base and DPO outputs. The actual difference was that the base model wrote a self-test at the end of the function ("if this fails, raise"), and that self-test crashed at import time on the test harness, registering as ASSERTION_FAIL. The DPO model produced clean function-only output, no embedded self-tests, so import succeeded and the harness's actual tests ran and passed.

That's a behavioral artifact, not an algorithmic improvement. It reflects the chosen-side distribution: clean idiomatic Python without embedded test scaffolding. DPO learned to not tack on self-tests, and two of the eight ASSERTION_FAIL fixes are downstream of that.

So if I want to be honest about the breakdown: the real algorithmic improvement is roughly +2.4 pp out of the +3.46 pp aggregate. The rest is the model learning to output the kind of thing the chosen pool actually contained. That's still a real and useful effect — nobody wants self-tests injected into their library code — but it's not the same thing as "the model got smarter."

This is the kind of detail that aggregate scores erase. I think it matters.

TYPE_ERROR doesn't move at all. Three seeds, all returning exactly 1 failure. This is one specific HumanEval problem that the base model fails on and the DPO model also fails on, in the same way. DPO didn't touch it, didn't make it worse, didn't make it better. A clean null result.

OTHER_RUNTIME and SYNTAX_ERROR are noise. The absolute counts are 0-3 across seeds. A single failure shifting between categories produces large relative percentages on tiny denominators. I'd want a much bigger eval set to say anything about these.

What this is and isn't

What it isn't: a refutation of large-data RLHF. 500 samples is not going to compete with a 50,000-sample run done properly. The improvement is a few percentage points, not transformative.

What it is, I think: a signal that what you put in the preference pairs matters more than how much. AP2O reports up to ~+3% pass@k improvement on coding benchmarks (per the paper's abstract; see arXiv:2510.02393 for the full per-benchmark breakdown). This run got into the same ballpark with 500 pairs. The data efficiency ratio probably doesn't survive scaling — there's almost certainly diminishing returns where the curation work stops paying off and you just need volume. But at the small-data end, where most solo developers actually live, the curation seems to do real work.

The other thing I take from the breakdown is that the improvements aren't coming from gaming the eval. If 500 pairs of curated data caused the model to start failing in new ways — spike of SYNTAX_ERROR, models refusing to attempt — I'd be worried about distribution collapse. None of that shows up. The two categories that move (NAME_ERROR, ASSERTION_FAIL) move down. The categories that don't move stay put. That's roughly the shape I'd want.

What I'm doing next

Currently in production: a Tokenization/BPE-focused specialty pack, ETA a few days. The benchmark methodology used here — 3 seeds, full failure breakdown, manual inspection of the deltas — will be re-run on each new specialty as it ships. I'd rather publish slow and verified than fast and aggregate-only.

If anyone has run similar breakdowns on their own DPO experiments, I'd genuinely like to compare notes. The thing I can't tell from one run is whether the NAME_ERROR / ASSERTION_FAIL split is specific to IDFU's curation choices or a general property of curated DPO with this kind of chosen-pool composition. That's a question with real implications for what to put in the pairs, and I don't have enough data to answer it alone.

The dataset

A 100-row sample is up at huggingface.co/datasets/namakoo/idfu-verified-code if you want to look at what the curated dataset actually looks like. The full set is available there too.

If you're working on DPO or RLHF training and want to compare notes, find me on Twitter (@namakoo123) or reply here.

Curation continues. The next snapshot is being collected as you read this.

️ How to Detect Minerals (Au, Cu, Li, Fe, etc.) Using Multispectral & Hyperspectral Remote Sensing

Wed, 29 Apr 2026 10:39:31 +0000

Gérard Cubaka — Geospatial / AI / Earth Observation
You can start now detecting minerals using our plateform : GeoCongo AI

Introduction

Mineral exploration has been revolutionized by remote sensing. Instead of relying only on fieldwork, geologists can now detect mineralization zones from satellite imagery.

The key idea is simple:

Each mineral interacts with electromagnetic radiation differently — creating a unique spectral signature across Visible, NIR, SWIR, and TIR wavelengths.

This article explains:

How to detect key minerals (Au, Cu, Fe, Li, etc.) using multispectral data
The band ratios and indices used in practice
How hyperspectral data unlocks precise mineral identification

1. Core Principle: Spectral Signatures

Every mineral:

Reflects certain wavelengths
Absorbs others

For example:

Iron oxides → strong in visible (red)
Clays → absorption near 2.2 µm (SWIR)
Quartz → better detected in thermal infrared (TIR)

️ 2. Multispectral Mineral Detection

Multispectral sensors (e.g., ASTER, Landsat 8/9) use broad bands, so we rely on:

Band ratios
Indices
PCA (Principal Component Analysis)

2.1 Iron Oxides (Fe, Mn, Ti)

These are the easiest to detect.

Characteristics:

Absorb blue
Reflect red

Band Ratios:

Landsat 8:

B4 / B2

ASTER:

B2 / B1

Interpretation:

Bright areas → Hematite / Goethite (gossans)
Often indicate oxidized ore zones

2.2 Hydrothermal Alteration Minerals (Au, Cu, Ag, Zn, Ni, Co)

You rarely detect the metal directly. Instead, detect alteration halos:

Clays (Al-OH)
Carbonates (Mg-OH)
Sulfate alteration (Jarosite, Alunite)

Key Indices:

Clay Index (Al-OH)

ASTER:

B4 / B6

Landsat 8:

B6 / B7

Indicates:

Kaolinite, Illite
Strong indicator of gold (Au) and copper (Cu) systems

Carbonate / Chlorite Index (Mg-OH)

ASTER:

B8 / B9

Useful for:

Copper (Cu)
Zinc (Zn)

Jarosite Index (Gold Indicator)

ASTER:

B4 / B3

Indicates:

Acidic alteration → often linked to gold deposits

⚪ 2.3 Silica & Quartz (SiO₂)

⚠️ Problem:
Quartz has no strong SWIR absorption

Solution: Use Thermal Infrared (TIR)

Silica Index:

ASTER:

(B11 * B11) / (B10 * B12)

Detects:

Quartz veins
Silicification zones (important for Au)

2.4 Mafic & Ultramafic Rocks (Ni, Co, Diamond Indicators)

We detect the host rocks, not the minerals directly.

Indices:

Mafic Index:

ASTER:

B12 / B13

Ultramafic Index:

ASTER:

(B1 + B3) / B2

Indicates:

Kimberlites → Diamond exploration
Ultramafic intrusions → Nickel, Cobalt

2.5 Pegmatites (Li, Be, Nb)

No direct detection ❌

Strategy:

Detect associated minerals:
Micas (Lepidolite)
Feldspars

Use:

Clay/mica indices (Al-OH absorption ~2.2 µm)

⚫ 2.6 Special Cases

Diamond

Not directly detectable
Target: Kimberlite pipes
Look for:
Mg-rich signatures
Circular anomalies

☢️ Uranium (U)

Detect indirectly:
Clay alteration
Redox boundaries (Fe²⁺ / Fe³⁺)

Note:

Radiometric surveys are often better than optical

3. RGB Composite Strategy

For fast visual interpretation:

Example (Gold/Copper exploration):

Channel	Data
Red	Iron Oxide Ratio
Green	Clay Index
Blue	Carbonate Index

Result:

Bright composite zones = high exploration targets

4. Hyperspectral Data (The Game Changer)

Multispectral = ~10 bands
Hyperspectral = 100–300 narrow bands

This allows true mineral identification

4.1 Preprocessing

Atmospheric correction:
ATREM
QUAC
Noise filtering:
Savitzky-Golay

4.2 Endmember Extraction

Pixel Purity Index (PPI)
n-D Visualizer

Goal:
Extract pure mineral signatures

4.3 Spectral Matching Algorithms

SAM (Spectral Angle Mapper)

Measures similarity between spectra

SFF (Spectral Feature Fitting)

Matches absorption features precisely

⚖️ Linear Unmixing

Determines proportions:

Pixel = 30% Quartz + 70% Iron Oxide

4.4 Targeting Specific Minerals

Mineral	Strategy
Li, Be, Nb	Shift in mica absorption (~2200 nm)
U	Redox zones (Fe²⁺ vs Fe³⁺)
Diamond	Indicator minerals (pyrope, magnesite)
Cu	Malachite spectral signature
Au	Hydrothermal alteration zones

️ 4.5 Recommended Tools

ENVI → Industry standard
QGIS + EnMAP-Box → Open-source alternative
GeoCongo AI → Web plateform

Conclusion

Multispectral remote sensing is powerful for:

Mapping alteration zones
Identifying exploration targets

But:

Hyperspectral data enables direct mineral identification.

Key Takeaways

You rarely detect metals directly — focus on alteration minerals
SWIR (2.1–2.4 µm) is critical for clays and hydrothermal systems
Use band ratios to enhance signals
Combine indices into RGB composites
Use hyperspectral data for high-precision exploration

Before You Trust Linear Regression, Read This

Mon, 20 Apr 2026 14:12:38 +0000

Linear regression is often the first model we learn and the one we misuse the most.

Not because it’s complicated, but because it’s too easy to run.
A few lines of code, a decent score, and we move on.

But linear regression isn’t just a model.
It quietly assumes a lot about your data.

And if those assumptions don’t hold, your results might still look fine, just not reliable.

Before trusting a linear regression model, there are four things worth checking:

1. Linearity
Are you actually modeling a straight-line relationship?
If the data is curved and you force a line through it, the model will miss patterns.
A quick scatter plot or residual plot usually tells the story.

2. Independent Observations
Your data points shouldn’t influence each other.
This becomes important in time-based data, where yesterday affects today.
If errors are connected, your model is learning patterns it shouldn’t.

3. Normality of Errors
The residuals (errors) should roughly follow a normal distribution.
This matters if you care about interpreting the model, not just predicting.

4. Homoscedasticity
In simple terms, your model should be equally wrong across all values.
If errors grow or shrink in certain regions, the model becomes biased.

The tricky part is that none of these checks stop your model from running.

You’ll still get predictions.
You’ll still get metrics.

But without checking these, you’re not really using linear regression, you’re just fitting a line and hoping it works.

A small habit that changes everything:

Before tuning the model, spend a few minutes validating these assumptions.

When was the last time you actually checked them before trusting your model?

Grok in 2026: Powerful, Polarizing, and Hard to Ignore

Sun, 12 Apr 2026 13:15:18 +0000

Originally published at https://blog.akshatuniyal.com.

Technical progress, real-time power, and a controversy trail that still raises hard questions. Here’s where Grok actually stands.

There’s no AI story in 2026 quite like Grok’s.

On paper, it is one of the most ambitious AI products in the market. Strong benchmark scores, a real-time information advantage that very few rivals can match, serious computing infrastructure, and a release cadence that barely slows down. xAI has been moving fast — sometimes faster than its critics are comfortable with.

Off paper, the story has been far messier. Grok has been tied to a string of controversies: harmful outputs, questions around system-level moderation choices, and image-generation incidents that triggered regulatory scrutiny in multiple countries.

And yet — people keep using it. Developers keep benchmarking it. The US Department of Defense integrated it into select classified networks. xAI’s valuation climbed into the hundreds of billions. None of that happens if the model is just hype.

So what is Grok, really? A serious contender with distinctive strengths, or a product still carrying unresolved trust questions? At this point, probably both. Let’s dig in.

From Chatbot to Colossus: How Fast Grok Has Moved

Grok launched in November 2023 as a beta on X (formerly Twitter), accessible only to paid users. It was honest about what it was: an early product with two months of training behind it, designed to answer almost anything with a bit of wit and a rebellious streak.

That version feels like ancient history now.

By July 2025, xAI had released Grok 4 and Grok 4 Heavy, trained on the Colossus supercomputer cluster — at the time housing around 200,000 GPUs in Memphis, Tennessee. Grok 4 Heavy became the first model to achieve a near-passing score on Humanity’s Last Exam, widely regarded as the hardest multi-domain benchmark ever constructed. Musk claimed on the launch stream that the model “is smarter than almost all graduate students in all disciplines simultaneously.” That’s the kind of sentence that’s easy to dismiss as hype, except the benchmark results were genuinely hard to argue with.

Then came the 4.x series. Grok 4.1 in November 2025 cut hallucination rates from 12% down to around 4% — a 65% reduction that meaningfully changed the enterprise conversation around the model. Grok 4.20 Beta followed in February 2026 with improved instruction following, LaTeX rendering for scientific outputs, and a multi-agent architecture. By March 2026, Grok 4.20 Beta 2 was live with five further improvements (e.g., "including enhanced vision capabilities and multi-image rendering").

To put that in perspective: the pace of improvement from Grok 1 to Grok 4 Heavy is genuinely one of the more impressive model trajectories in AI right now. Very few labs have moved this fast on core capability benchmarks in such a short window.

"The pace of iteration is unusually fast, even by current frontier-model standards."

That speed comes with trade-offs, some of which we’ll get to. But from a pure capability trajectory, xAI’s progress over 18 months has been extraordinary.

Grok’s Clearest Edge: Real-Time Intelligence

If there is one thing that most clearly separates Grok from other frontier models, it is this: it is built around what is happening right now.

Most major AI assistants still depend on a training cutoff and then use search or retrieval layers to stay current. That can work well, but it usually feels like an added layer rather than the core product experience.

Grok is different in that respect. It is deeply integrated with X and can draw on a platform that produces hundreds of millions of posts each day. Breaking news, live reactions, market chatter, sports conversations, memes, and the texture of the internet in motion — this is where Grok feels unusually native.

For certain use cases, this is a meaningful advantage:

Journalists and researchers tracking breaking stories
Market analysts who need to know what people are saying about a stock, right now
Social media managers monitoring brand sentiment in real time
Anyone who needs to understand what’s actually trending vs. what was trending last month

Few major models offer this kind of live social-context access so natively. And it matters more than it may sound on paper. A lot of real-world information needs are time-sensitive. Being able to answer ‘what are people saying about this right now?’ is a meaningful product advantage, even if freshness does not always guarantee accuracy.

The Musk Ecosystem Play

One of the more underappreciated parts of Grok’s story is how it sits inside a much larger infrastructure.

xAI was brought together with SpaceX in February 2026, putting Grok inside a much larger ecosystem that also touches Tesla, Starlink, Neuralink, and X. That is not just a corporate footnote. It suggests access to a broader strategic stack:

Tesla’s fleet data — millions of miles of real-world video, feeding into vision and robotics training
Starlink’s satellite network — potentially bringing AI inference to places that have never had reliable internet
X’s social graph — the real-time pulse of global conversation
Optimus robot integration — xAI is already using Grok’s reasoning to power humanoid robots
US Department of Defense contracts — Grok was integrated into select classified and unclassified military networks in January 2026

The DoD integration is particularly notable. It represents a level of institutional trust that usually takes time to build. At the same time, it has drawn criticism from people who believe a model with Grok’s public controversy history warrants closer scrutiny before being embedded in government systems. Both realities can be true at once.

There’s also the financial picture: a pre-merger valuation of around $230 billion, now part of a combined SpaceX-xAI entity valued at over $1 trillion, with backing from Nvidia, AMD, Sequoia, a16z, BlackRock, and Fidelity. That’s not a scrappy startup anymore. That’s a serious institution with the resources to match.

"Very few AI companies have this kind of cross-industry data and distribution story. Whether that becomes a lasting moat or a governance headache is still an open question."

Where Grok Actually Performs Well

Enough big picture. What does Grok actually do well in practice?

Real-time research and news analysis
This is probably Grok’s clearest practical strength. If your question touches something that happened recently, Grok’s X integration can give it a real edge on freshness and signal detection. The output is not always clean — X is fast, not always reliable — but in terms of immediacy, Grok is unusually strong.

Coding and technical reasoning
Grok 4 Heavy benchmarks exceptionally well on coding tasks. The multi-agent architecture in the 4.20 series, where multiple AI agents collaborate on complex problems, has been particularly well received by developers working on larger codebases. The hallucination reduction in 4.1 also made a meaningful difference for technical use cases where wrong answers have real costs.

Internet culture and tone
This sounds minor but it’s genuinely useful in practice. Grok gets internet humour, meme references, and the texture of online conversation in a way that more formally trained models sometimes miss. That makes it particularly good for content creators, social media work, and anyone who needs writing that feels alive rather than polished-but-sterile.

Long-context tasks
Grok 4 supports very large context windows — in practice useful for things like feeding in entire codebases, long research papers, or extended document sets that would overwhelm smaller windows. This is becoming table stakes for frontier models, but Grok handles it well.

The Reality Check: Growth Pains & The Safety Evolution

Any fair assessment of Grok also has to account for the friction. xAI’s tendency to ship fast and iterate in public has come with some very visible growing pains.

Over the last 18 months, Grok has gone through a number of public incidents — from system prompt leaks tied to political misinformation concerns to the 2025 "MechaHitler" episode, and later the "digital undressing" controversy that drew regulatory scrutiny from the EU and UK.

By March 2026, the fallout had moved well beyond scrutiny. The UK ICO, Ireland's DPC, Canada's Privacy Commissioner, and Ofcom had all opened formal investigations into xAI over AI-generated harmful imagery. A Tennessee lawsuit alleging Grok had generated sexual images of minors added a legal dimension that no amount of product iteration can paper over. This is no longer just a safety story — it's an active legal exposure story.

What is also worth noting is that xAI has not treated these issues as background noise. It has tried to translate some of those lessons into product and architecture changes.

From Chaos to Context: The 4.1 update was more than a routine patch; it was a focused attempt to improve stability, and xAI said it reduced hallucination rates by roughly 65%.
The Multi-Agent Guardrail: The current 4.20 series moved toward a multi-agent setup intended to add more internal checks and balances around reasoning and safety.
Institutional Vetting: While regulators were asking questions, the US Department of Defense was also doing its own due diligence, eventually integrating Grok into select classified networks in early 2026. That suggests at least some institutions see the trust picture as improving, even if concerns remain.

The story of Grok is not just about a model that stumbled in public. It is also about a model being refined in one of the most visible real-world AI testing grounds. Is it perfect? No. But the pace at which xAI is trying to tighten capability and safety together is part of the story too.

Where Does Grok Sit in the Current AI Landscape?

By the numbers, Grok 4 Heavy is clearly one of the strongest models in the world. The Humanity’s Last Exam performance, the hallucination reduction, the LMArena visibility — these are not imaginary. The technical progress is real.

But the current AI landscape is crowded with genuinely strong models. GPT-4o remains the most versatile general-purpose assistant for most professional workflows. Claude has built a strong reputation for writing quality, long-context reasoning, and the kind of calm, deliberate approach to complex tasks that developers value. Gemini has deep Google ecosystem integration and strong multimodal performance. DeepSeek has raised questions about what’s possible at much lower cost.

Grok’s clearest advantages are real-time information access and the broader Musk ecosystem around it. Its clearest concerns are around guardrails, rollout discipline, and the trust questions that come with a documented history of controversial outputs.

Where Grok wins

Real-time research and social listening
Coding and technical tasks, especially complex multi-step workflows
Users deeply embedded in the X and Tesla ecosystems
Applications where cultural relevance and internet-native tone matter
High-stakes benchmark performance in controlled environments

Where the competition still leads

Enterprise deployments where reliability and trust matter more than raw performance
Long-form writing with consistent voice and quality
Workflows requiring deep Google or Microsoft ecosystem integration
Regulated industries where guardrail robustness is non-negotiable
Teams where the AI safety and controversy track record is a dealbreaker

What Comes Next

xAI has been open about its ambitions. Musk has publicly suggested a meaningful chance of reaching the world’s first AGI with upcoming models — which may prove visionary, promotional, or a bit of both. The Colossus supercomputer is reportedly continuing to scale. Grok Imagine, the video generation product, released an improved version in February 2026 with full text-to-video and video editing capabilities, positioning Grok as more than a chatbot.

The SpaceX tie-up also creates a bigger strategic story: an AI company with potential access to satellite infrastructure for global inference, automotive data from one of the world’s largest vehicle fleets, and robotics integration through Optimus. Whether that becomes a durable advantage or creates larger governance challenges is still unclear.

One signal worth noting: Grok 5 was publicly confirmed for Q1 2026 by Musk himself. That window has passed. xAI now points to Q2 2026, with the model reportedly carrying 6 trillion parameters and training on Colossus 2 — a 1-gigawatt supercluster in Memphis. For a company that prides itself on shipping fast, a missed self-imposed deadline is worth flagging. It doesn't change the capability story, but it's a useful data point on the gap between Musk's timelines and xAI's actual cadence.

What seems certain is that xAI will keep shipping. They’ve demonstrated that convincingly.

Final Thoughts

Grok is one of the most technically impressive and most debated AI stories of the moment.

The capabilities are real. The real-time intelligence advantage is real. The benchmark performance is real. The ecosystem play is real.

And the improvement arc is worth stating plainly: from a two-month-old beta in 2023 to near-passing on the hardest AI benchmark ever built in under two years. Whatever else you think about Grok, that trajectory is genuinely remarkable.

So are the controversies, the guardrail questions, and the trust gap that can emerge when a model advances this quickly in public.

If you need real-time intelligence, are building on X’s ecosystem, or are doing heavy technical work where raw model performance is the primary criterion — Grok deserves a serious look. It might be the best tool for your specific job.

If you are building for regulated industries, enterprise environments where reliability is non-negotiable, or any setting where harmful outputs would carry serious consequences, this history deserves careful weight.

"The model closest to the live internet is also the one with the most unresolved story. And that’s exactly what makes it worth watching."

— What’s your experience with Grok? Has it earned your trust yet, or are you still watching from the sidelines? Drop it in the comments below.

About the Author

Akshat Uniyal writes about Artificial Intelligence, engineering systems, and practical technology thinking.
Explore more articles at https://blog.akshatuniyal.com.

Pandas 3.0's PyArrow String Revolution: A Deep Dive into Memory and Performance

Wed, 08 Apr 2026 06:01:21 +0000

Introduction

Pandas 3.0 made a game-changing decision: PyArrow-backed strings are now the default. Instead of storing strings as Python objects (the old object dtype), pandas now uses Apache Arrow's columnar format with the new string[pyarrow] dtype.

But here's the question that matters: How much does this new string dtype actually improve performance and memory usage in real-world scenarios?

To find out, I ran comprehensive benchmarks across diverse datasets and common string operations. The results? 51.8% memory savings on average, with operations running 2-27x faster.

This isn't a theoretical improvement, it's a fundamental shift in how pandas handles string data.

The Results: Summary Dashboard

Let me start with the headline numbers, then we'll dive into how I got them.

The Four Key Metrics

1. 51.8% Memory Savings

Across all test datasets, the new PyArrow string dtype used half the memory of the old object dtype. This isn't a marginal improvement, it's transformative for memory-constrained environments.

2. 6.17x Average Operation Speedup

String operations aren't just more memory-efficient: they're dramatically faster. On average, operations like str.lower(), str.contains(), and str.len() run 6x faster with PyArrow strings.

Some operations are even more impressive:

str.len(): 27x faster
str.startswith(): 16x faster
str.endswith(): 15x faster

3. 889 MB Total Memory Saved

Across our test datasets (totaling 645 MB on disk), we saved nearly 1 GB of RAM in memory. For a real data pipeline processing dozens of datasets, this compounds quickly.

4. Memory Overhead: The Game Changer

The bottom chart reveals something crucial about how pandas handles strings:

Old string dtype (object):

CSV files on disk: 645 MB
Loaded into pandas: 1,714 MB
Memory overhead: 165.7% (more than doubles!)

New string dtype (PyArrow):

CSV files on disk: 645 MB
Loaded into pandas: 825 MB
Memory overhead: 27.9% (minimal overhead)

What does this mean?

When pandas reads a CSV file, it doesn't just store the raw bytes: it creates in-memory data structures for fast operations. The old object dtype was incredibly inefficient, essentially duplicating string data multiple times. The new PyArrow string dtype keeps overhead minimal with a smarter memory layout.

This is the difference between pandas 2's Python-object approach and pandas 3's columnar Arrow approach.

The Methodology: Why 5 Different Datasets?

Now that you've seen the results, let me explain how I tested this. Real-world data comes in many shapes and sizes. A single benchmark on one type of data wouldn't tell the whole story.

That's why I created 5 distinct datasets, each representing common patterns you'll encounter in production:

1. Low Cardinality Dataset (1M rows)

What it is: Repeated categorical values like product categories, status codes, regions, and priorities.

Why it matters: This is typical of business data - think order statuses, customer segments, or department codes. The same values repeat millions of times.

Example columns:

category: "Electronics", "Clothing", "Food" (10 unique values)
status: "pending", "completed", "failed" (4 unique values)

2. High Cardinality Dataset (1M rows)

What it is: Mostly unique strings like user IDs, email addresses, and session tokens.

Why it matters: When every row is different (like customer emails or transaction IDs), pandas can't use simple optimizations. This tests worst-case scenarios.

Example columns:

user_id: "USER_00000001", "USER_00000002"... (1M unique)
email: "user123@example45.com" (1M unique)

3. Mixed String Lengths Dataset (1M rows)

What it is: A combination of short codes (2-5 chars), medium names (20-50 chars), and long descriptions (100-300 chars).

Why it matters: Real data isn't uniform. You might have product codes next to customer addresses next to order notes. This tests how pandas handles variable-length strings.

4. Dataset With Nulls (1M rows)

What it is: Data with missing values (10-33% nulls in different columns).

Why it matters: Messy data is reality. How does pandas 3.0 handle missing string data compared to pandas 2?

5. Large Dataset (10M rows)

What it is: A scaled-up version to test performance at scale.

Why it matters: Memory savings that look good at 1M rows might behave differently at 10M rows. This validates the findings scale linearly.

Memory Savings by Dataset Type

The memory savings from PyArrow strings vary significantly by dataset characteristics:

Best Case: Low Cardinality Data (-71.6%)

When data has repeated values (like categories), PyArrow strings shine:

Object dtype: 219 MB
PyArrow string dtype: 62 MB
Savings: 71.6%

Worst Case: Mixed String Lengths (-30.6%)

Variable-length strings see smaller (but still significant) savings:

Object dtype: 383 MB
PyArrow string dtype: 266 MB
Savings: 30.6%

The Pattern

Notice how savings correlate with data characteristics:

Repeated values (low cardinality) → Best savings (64-72%)
Unique values (high cardinality) → Good savings (53-55%)
Variable length (mixed sizes) → Moderate savings (31%)

Takeaway: PyArrow strings help everywhere, but they're especially powerful for categorical-like data.

Performance: Operation-Specific Speedups

This heatmap shows how much faster PyArrow strings are compared to object dtype for common string operations (values > 1.0 mean PyArrow is faster).

The Fastest Operations

str.len(): 10-27x faster
str.startswith() and str.endswith(): 11-18x faster
str.contains(): 3-5x faster
str.split(): 1-8x faster

The Pattern

Read operations (like len(), startswith()) → Massive speedups (10-27x)

These operations just examine existing data without modification

Transform operations (like replace(), split()) → Good speedups (2-5x)

These operations create new data, which limits the performance gains

The Trade-off: CSV Loading Time

There's no such thing as a free lunch. While PyArrow strings save memory and run operations faster, loading CSV files is 9%-61% slower.

Why the Slowdown?

When pandas reads a CSV with PyArrow strings enabled:

It parses the text (same as before)
It converts strings to PyArrow's columnar format (extra step)
This conversion involves building dictionary encodings and optimized memory structures

Pandas is doing more work upfront to enable better performance downstream.

Real-world impact: On our 10M row dataset, the difference is 1.63s vs 2.02s, an extra 0.4 seconds for 10 million rows. For many data pipelines, this upfront cost might be negligible compared to the 2-27x speedup in subsequent operations.

Pros and Cons: Should You Adopt PyArrow Strings?

Benefits of PyArrow String Dtype

Massive Memory Savings (30-72%)
Dramatically Faster String Operations (2-27x)
Minimal Memory Overhead (28% vs 166%)
Modern Data Ecosystem Integration

Trade-offs to Consider

Slower CSV Loading (9-61% slower)
- Initial data ingestion takes longer
- May impact workflows that repeatedly load small files
- The trade-off: slower start, much faster operations
Behavioral Changes
- String dtype behaves differently from object dtype in edge cases
- Need to update code that explicitly checks for object dtype
- Testing required for migration

The Recommendation

For most data workflows, PyArrow strings are a clear win. The memory and performance benefits far outweigh the trade-offs.

Consider staying with object dtype if:

You rarely work with string columns
Your datasets easily fit in memory
Load time is critical and you rarely perform string operations
You have legacy code that's deeply coupled to object dtype behavior

Definitely adopt PyArrow strings if:

You process large datasets with text data
String operations are a significant part of your workflow
Memory is a constraint in your environment
You're building production data pipelines
You work with modern data tools (Parquet, Arrow, DuckDB, etc.)

Conclusion

Our comprehensive analysis across 5 diverse datasets and 15+ string operations conclusively shows that PyArrow-backed strings deliver transformative improvements:

51.8% average memory savings across all dataset types
6.17x average operation speedup for string operations
Minimal memory overhead (28% vs 166% with Python objects)

PyArrow strings aren't just an incremental improvement, they're a fundamental reimagining of how pandas handles text data. By adopting Apache Arrow's proven columnar format, pandas has joined the modern data ecosystem while delivering massive performance and memory improvements.

For most data practitioners working with text data, the question isn't "Should I use PyArrow strings?" but rather "How quickly can I migrate?"

Questions or feedback? Feel free to open an issue or contribute to this analysis! The code we used in this analysis has been uploaded to this repo.

How to Handle NULLs in Snowflake: A Complete Guide!

Tue, 07 Apr 2026 14:39:38 +0000

Dear Readers,

NULL in data is like a weed, it can mess up your analysis and calculations, and ultimately bleed the value out of your analytics work. In this article, we’ll dive into what NULL is, why it exists, when it appears, and how to deal with it in Snowflake ?

So, without further ado, let’s meet this culprit.

What NULL?

In SQL, NULL is basically the database equivalent of a shrug.It is a special marker that represents a missing or unknown value. It's not the same as an empty string ('') or the number zero (0). It simply means “I don’t know” or “there’s no value here at all.”

Why NULL Exists?

NULL is essential because sometimes your data just doesn’t exist yet, or never will. NULL allows the database to acknowledge the gap without faking a value.

Eg:

A customer doesn’t have a middle name.
A product doesn’t have a discount yet.
A delivery date is unknown at the time of record insertion.

When NULL Appears?

NULL occurs anytime there is a lack of value from the user.

Common reasons include:

Optional fields with no provided value.
Data not yet available (future delivery dates, pending approvals)
Inapplicable values (someone’s “date of divorce” who never married)

How to Deal with NULL in Snowflake?

Snowflake provides multiple functions and methods to deal with NULL dataset aka unknown data similar to other databases. I have listed some of them below:

Use IS NULL / IS NOT NULL to identify the NULL data.
Use COALESCE(col, default) or IFNULL(col, default) to replace them in select , joins, aggregates etc
Use explicit filter on WHERE clauses to include NULL rows as it can can drop NULL rows unexpectedly.
Use NVL, CASE WHEN, or similar functions to handle them in calculations or strings.
Comparisons with NULL return UNKNOWN in SQL’s three-valued logic. Use COALESCE or IS NULL checks to avoid surprises.

Eg:

==Using NULL/NOT NULL

-- Finds all employees without a manager
SELECT name FROM employees WHERE manager_id IS NULL;

-- Finds all employees with a manager
SELECT name FROM employees WHERE manager_id IS NOT NULL;

==Using COALESCE

-- Calculates total compensation, treating a NULL bonus as 0

SELECT salary + COALESCE(bonus, 0) AS total_compensation FROM employees

The Trouble NULL Causes:

1. Comparison Traps

NULL comparisons are a common pitfall because standard operators like = or != don’t work as expected. Any comparison involving NULL returns UNKNOWN, not TRUE or FALSE. Use IS NULL or IS NOT NULL to handle the traps.

Eg:

--Evaluates to UNKNOWN when NULL = NULL

SELECT *
FROM employees
WHERE manager_id = NULL;

-- The correct way to find employees with a NULL manager_id is:
SELECT *
FROM employees
WHERE manager_id IS NULL;

2. Math & Concatenation Failures:

NULL acts like a virus in arithmetic and string concatenation operations. Any operation involving NULL will result in NULL.

Eg:

--Return of NULL total+compensation when bonus is NULL:

SELECT salary + bonus AS total_compensation
FROM employees;

--The correct way to add bonus is use COALESCE :

SELECT salary + COALESCE(bonus, 0) AS total_compensation
FROM employees;

3. Aggregate Surprises:

Aggregate functions like SUM() and AVG() ignore NULL values. This can lead to misleading results if you’re not aware of it.

Eg:

--Excludes NULL rows

SELECT AVG(salary) AS avg_salary, SUM(salary) AS total_salary
FROM employees;

--Use COALESCE to treat NULL salaries as 0 to get more accurate company-wide average

SELECT AVG(COALESCE(salary, 0)) FROM employees;

4. Join Issues:

NULL values in join keys never match each other. If an employee has a NULL in manager_id, that row won’t link to any id in the employees table even if other employees also have NULL as manager_id.

Use a LEFT JOIN + COALESCE to tackle the NULL issue.In following demonstrated way, you keep all employees in the output, and NULL managers are clearly labeled without risking false matches.

Eg:

--This INNER JOIN excludes employees with no manager_id

SELECT

   E.name AS employee,
   M.name AS manager

FROM employees AS E
INNER JOIN
employees AS M
ON E.manager_id = M.id;

--Use COALESCE to include employees without a manager

SELECT
E.name AS employee,
COALESCE(M.name, 'No Manager') AS manager
FROM employees E
LEFT JOIN employees M
ON E.manager_id = M.id;

5.Logic Surprises

In SQL’s three-valued logic (TRUE, FALSE, UNKNOWN), an UNKNOWN result in a WHERE clause acts like FALSE and discards the row. However, UNKNOWN can have different effects in other clauses such as CASE.

Eg:

--Logic Mishandling for NULL data.

SELECT
CASE

WHEN bonus > 10000 THEN 'High Bonus'
WHEN bonus > 5000 THEN 'Mid Bonus'
ELSE 'Low or No Bonus'

END AS bonus_category
FROM employees;

-- Explicitly checks for NULL first

SELECT
CASE

WHEN bonus IS NULL THEN 'No Bonus'
WHEN bonus > 10000 THEN 'High Bonus'
WHEN bonus > 5000 THEN 'Mid Bonus'
ELSE 'Low Bonus'

END AS bonus_category
FROM employees;
Closing thoughts,

NULLs aren’t a bug, they’re part of SQL’s design.NULL is tricky. The safest approach is to scan your dataset for every attribute that could be NULL, then apply handling strategies suited to your business context. In Snowflake, a few well-placed COALESCE, IS NULL, and CASE statements can save you from major data quality headaches.

Hope you enjoyed the article! If you like my writing, feel free to connect with me on Linkedin and follow for more such interesting reads.

“Why Averages Lie: Understanding Mean, Median & Mode with Real Data”

Mon, 06 Apr 2026 11:16:25 +0000

???? Lecture 1 — Central Tendency of Data

Vyoma Data Science Initiative | Module 2: Statistics

???? Introduction

In any dataset, we often face a simple question:

Can we represent all this data using a single number?

Whether it’s exam scores, income levels, or age distributions — we try to summarize complexity into simplicity.

This idea leads us to central tendency.

???? What is Central Tendency?

Central tendency refers to the representative value of a dataset — the point around which the data tends to cluster.

It helps answer:

What is typical?
Where does the data concentrate?
What does this dataset “look like” overall?

???? Measures of Central Tendency

1️⃣ Mean (Average)

[ \text{Mean} = \frac{\sum x_i}{n} ]

Represents the balance point of the dataset
Highly sensitive to outliers

2️⃣ Median (Middle Value)

The middle value after sorting data
If even number of values → average of two middle values

???? More robust than mean in real-world datasets

3️⃣ Mode (Most Frequent Value)

The value that appears most often
Useful for both numerical and categorical data

⚠️ When Averages Mislead

Consider:

2, 3, 4, 5, 100

Mean = 22.8 ❌
Median = 4 ✅

???? The mean is distorted by an extreme value.

???? Insight

A single number cannot always capture reality.
The choice of measure defines the “truth” you see.

???? Practical Implementation (Python)

Using the Titanic dataset, we analyze the Age column.

???? Code

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

data = pd.read_csv(r"C:\Users\praso\vyomadatascience\module02\titanic.csv")

print(data.head())

print("Mean Age:", data["Age"].mean())
print("Mode Age:", data["Age"].mode())
print("Median Age:", data["Age"].median())

???? Optional Visualization

sns.histplot(data["Age"].dropna(), kde=True)
plt.title("Age Distribution (Titanic Dataset)")
plt.xlabel("Age")
plt.ylabel("Frequency")
plt.show()

???? Dataset Reference

You can download the Titanic dataset from:

Titanic Dataset
Official Kaggle link: https://www.kaggle.com/datasets/yasserh/titanic-dataset

(You can place it in your project folder: module02/titanic.csv)

???? GitHub Repo

Download the github repo of our course from here :

https://github.com/psjdeveloper/vyomadatascience

???? Final Reflection

Central tendency is not just about computing mean, median, or mode.

It is about understanding:

How data behaves
How summaries can distort reality
And how interpretation matters more than calculation

✍️ Closing Line

Data does not speak for itself.
The way we summarize it decides what it says.

Mastering Pandas — Part 4: Data Visualization with Matplotlib & Seaborn

Thu, 02 Apr 2026 19:32:22 +0000

From Clean Data to Clear Insight

In Part 3, you learned how to clean messy data and combine multiple sources into one unified DataFrame. Now that your data is ready, the next step is to communicate what's inside it — and nothing communicates data faster or more clearly than a chart.

This article covers the two most important visualization libraries in Python: Matplotlib, the core engine that powers all plotting, and Seaborn, a higher-level library built on top of it that produces beautiful statistical charts with minimal code. By the end, you'll know which tool to reach for depending on what you need to show.

We'll use this sample dataset throughout:

import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns

data = {
    'Country':         ['China', 'India', 'USA', 'Brazil', 'UK'],
    'Continent':       ['Asia', 'Asia', 'North America', 'South America', 'Europe'],
    '2022 Population': [1412000000, 1380000000, 331000000, 214000000, 67000000],
    'Area (km2)':      [9597000, 3287000, 9834000, 8516000, 243000]
}
df = pd.DataFrame(data)

Part 1 — Matplotlib: The Engine

What is Matplotlib?

Matplotlib is the foundational plotting library in Python. Every chart you create in Python — whether through Pandas, Seaborn, or directly — eventually goes through Matplotlib to render the final output. Think of it as the engine: it handles the low-level work of drawing lines, shapes, colors, axes, and text.

The submodule you'll use in practice is matplotlib.pyplot, which gives you a clean, high-level interface without managing every visual detail manually:

import matplotlib.pyplot as plt

Matplotlib works across three styles, each suited to a different situation:

Style	When to use
`df.plot(kind="bar", y="col")`	Your data is already in a DataFrame — quickest way
`plt.plot(xs, ys)`	Working with raw lists, arrays, or math functions
`fig, ax = plt.subplots()`	Multiple subplots or full layout control

All three ultimately use Matplotlib to render the chart. plt.show() is always called at the end to display it.

Global Settings with `plt.rcParams`

plt.rcParams is a global settings dictionary. Any value you set here applies to every chart created afterward in the session — so you set it once at the top of your notebook and never repeat it.

plt.rcParams['figure.figsize'] = (12, 5)   # default width and height
plt.rcParams['font.size']      = 13         # default font size
plt.rcParams['figure.dpi']     = 100        # image sharpness
plt.rcParams['lines.linewidth'] = 2         # default line thickness
plt.rcParams['axes.grid']      = True       # show grid on all charts

Setting a Visual Theme with `plt.style.use()`

One line changes the entire look of your charts — colors, background, grid style, and fonts. Call it before your plot, not after.

plt.style.use("ggplot")
df.plot(kind="bar", y="2022 Population")
plt.show()

Style	Look
`"ggplot"`	Gray background, colored lines — popular R-like style
`"seaborn-v0_8-whitegrid"`	Clean, modern, white with grid
`"fivethirtyeight"`	Bold, thick lines — news article style
`"dark_background"`	Black background
`"bmh"`	Soft colors, Bayesian style
`"grayscale"`	Shades of gray only
`"tableau-colorblind10"`	Colorblind-friendly palette

Run print(plt.style.available) to see every available option.

Plotting from a DataFrame with `.plot()`

Since Pandas is built on Matplotlib, every DataFrame and Series has a .plot() method that creates a chart in one line.

df.plot(kind="bar", x="Country", y="2022 Population", figsize=(10, 5), title="Population by Country")
plt.show()

The kind= parameter selects the chart type. All common types are supported:

`kind=`	Chart
`"bar"`	Vertical bars
`"barh"`	Horizontal bars
`"line"`	Line chart (default)
`"pie"`	Pie chart
`"hist"`	Histogram
`"scatter"`	Scatter plot
`"box"`	Box plot

You can also call the chart type directly as a method — both styles produce identical output:

# These two are exactly the same:
df.plot(kind="bar", x="Country", y="2022 Population")
df.plot.bar(x="Country", y="2022 Population")

The direct method style (df.plot.bar()) is shorter and more common in practice.

Key parameters you'll use on almost every chart:

Parameter	What it does	Example
`kind`	Chart type	`"bar"`, `"line"`, `"scatter"`
`x` / `y`	Columns for axes	`x="Country"`, `y="2022 Population"`
`figsize`	Width and height in inches	`figsize=(10, 5)`
`title`	Chart title	`title="Population by Country"`
`color`	Bar/line color	`color="steelblue"`
`legend`	Show or hide legend	`legend=True`
`xlabel` / `ylabel`	Axis labels	`xlabel="Country"`
`rot`	Rotation of tick labels	`rot=45`
`grid`	Show background grid	`grid=True`
`alpha`	Transparency	`alpha=0.8`
`bins`	Bin count (histogram only)	`bins=20`

df.plot(
    kind="bar",
    x="Country",
    y="2022 Population",
    figsize=(10, 5),
    title="2022 World Population",
    color="steelblue",
    xlabel="Country",
    ylabel="Population",
    rot=0,
    grid=True,
    alpha=0.85
)
plt.show()

Plotting Directly with `plt.plot()`

When your data is not in a DataFrame — raw lists, NumPy arrays, or a mathematical function — use plt.plot() directly.

import numpy as np

xs = np.arange(-5, 5, 0.25)
ys = xs ** 2

plt.figure(figsize=(8, 5))
plt.plot(xs, ys, color="steelblue", linewidth=2, linestyle="--")
plt.title("Quadratic Function")
plt.xlabel("X")
plt.ylabel("Y")
plt.grid(True)
plt.show()

Matplotlib doesn't know your data is a curve — it connects the dots you give it with straight lines. When points are closely spaced (as with np.arange step 0.25), hundreds of short segments visually blend into a smooth curve.

Customizing plt.plot():

# Color
plt.plot(xs, ys, color='red')

# Line style
plt.plot(xs, ys, linestyle='--')    # dashed
plt.plot(xs, ys, linestyle=':')     # dotted
plt.plot(xs, ys, linestyle='-.')    # dash-dot

# Markers at each point
plt.plot(xs, ys, marker='o')        # circles
plt.plot(xs, ys, marker='s')        # squares
plt.plot(xs, ys, marker='^')        # triangles

# Everything combined
plt.plot(xs, ys, color='green', linestyle='--', linewidth=2, marker='o')
plt.show()

If you want dots without a line, use plt.scatter():

plt.scatter(xs, ys)
plt.show()

Decorating Any Chart

These functions work on any chart — whether drawn through Pandas or directly:

plt.title("Chart Title")
plt.xlabel("X Axis Label")
plt.ylabel("Y Axis Label")
plt.grid(True)
plt.legend()      # shows labels defined with label= on each line
plt.show()

When plotting multiple lines, use label= with each plt.plot() call, then plt.legend() to display them:

ys1 = xs ** 2
ys2 = xs ** 3

plt.plot(xs, ys1, label="f(x) = x²")
plt.plot(xs, ys2, label="f(x) = x³", linestyle="--")
plt.legend()
plt.show()

Multiple Subplots with Object-Oriented Style

When you need more than one chart in the same figure, use the object-oriented style. It gives you full control over each subplot independently.

# 1 row, 2 columns — side by side
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(14, 5))

ax1.bar(df["Country"], df["2022 Population"], color="steelblue")
ax1.set_title("Population by Country")
ax1.set_xlabel("Country")
ax1.set_ylabel("Population")

ax2.scatter(df["Area (km2)"], df["2022 Population"], color="tomato")
ax2.set_title("Area vs Population")
ax2.set_xlabel("Area (km²)")
ax2.set_ylabel("Population")

plt.tight_layout()
plt.show()

fig is the whole figure container. ax1 and ax2 are the individual plotting areas. The pattern is the same as plt.* calls, just moved to ax.set_*() methods.

Part 2 — Seaborn: Statistical Visualization

What is Seaborn?

Seaborn is a library built on top of Matplotlib, designed for statistical data visualization. It produces polished, publication-quality charts with far less code than raw Matplotlib, and works natively with Pandas DataFrames.

import seaborn as sns

Every Seaborn chart follows the same basic structure:

sns.chart_type(data=df, x='column_name', y='column_name')
plt.show()

The hue= parameter is Seaborn's most powerful feature — it automatically splits and colors your data by a category column, adding a legend with no extra work:

sns.scatterplot(data=df, x='Area (km2)', y='2022 Population', hue='Continent')
plt.show()

Global Styling with `set_theme()`

Call this once at the top. Every chart afterward follows these settings automatically.

sns.set_theme(style='whitegrid', palette='deep', font_scale=1.2)

style controls the background:

sns.set_theme(style='whitegrid')   # white background with grid — most common
sns.set_theme(style='darkgrid')    # dark background with grid
sns.set_theme(style='white')       # white background, no grid
sns.set_theme(style='ticks')       # minimal, axis ticks only

palette controls the colors of chart elements:

sns.set_theme(palette='deep')        # default, rich colors
sns.set_theme(palette='muted')       # softer, less saturated
sns.set_theme(palette='pastel')      # light pastel colors
sns.set_theme(palette='colorblind')  # accessible for colorblind readers

font_scale scales all text up or down:

sns.set_theme(font_scale=1.5)    # larger text
sns.set_theme(font_scale=0.8)    # smaller text

To apply a style to one chart only without changing global settings:

with sns.axes_style('whitegrid'):
    sns.barplot(data=df, x='Continent', y='2022 Population')
    plt.show()

To reset everything back to Matplotlib defaults: sns.reset_defaults()

Relational Charts — Showing Relationships Between Numbers

Use these when you want to see how two numeric columns relate to each other.

scatterplot() — each row becomes one dot on the chart:

# Basic
sns.scatterplot(data=df, x='Area (km2)', y='2022 Population')

# Colored by continent
sns.scatterplot(data=df, x='Area (km2)', y='2022 Population', hue='Continent')
plt.title("Area vs Population")
plt.show()

lineplot() — connects points with a line. Best for time series or ordered data:

sns.lineplot(data=df, x='Country', y='2022 Population', hue='Continent')
plt.show()

relplot() — a single figure-level function for both scatter and line. Use kind= to switch:

sns.relplot(data=df, x='Area (km2)', y='2022 Population', kind='scatter', hue='Continent')
sns.relplot(data=df, x='Country', y='2022 Population', kind='line')
plt.show()

Distribution Charts — Showing How Values Are Spread

Use these to understand the shape and spread of a numeric column.

histplot() — bars showing how many values fall within each range:

# Basic histogram
sns.histplot(data=df, x='2022 Population')

# With smooth density curve on top
sns.histplot(data=df, x='2022 Population', kde=True)

# Split by category
sns.histplot(data=df, x='2022 Population', hue='Continent')
plt.show()

kdeplot() — a smooth density curve, more refined than a histogram:

sns.kdeplot(data=df, x='2022 Population', hue='Continent')
plt.show()

For a 2D density plot showing where two numeric columns overlap:

sns.kdeplot(data=df, x='Area (km2)', y='2022 Population')
plt.show()

rugplot() — adds small tick marks along an axis showing where individual data points are. Typically layered on top of a kdeplot:

sns.kdeplot(data=df, x='2022 Population')
sns.rugplot(data=df, x='2022 Population')
plt.show()

ecdfplot() — shows the cumulative distribution: what percentage of values fall below each point:

sns.ecdfplot(data=df, x='2022 Population')
plt.show()

displot() — one function to rule all distribution charts. Use kind= to switch:

sns.displot(data=df, x='2022 Population', kind='hist', hue='Continent')
sns.displot(data=df, x='2022 Population', kind='kde')
sns.displot(data=df, x='2022 Population', kind='ecdf')
plt.show()

Categorical Charts — Comparing Groups

Use these when one axis is a category and the other is numeric.

barplot() — shows the mean value per category, with confidence interval bars:

sns.barplot(data=df, x='Continent', y='2022 Population', hue='Continent')
plt.title("Average Population by Continent")
plt.show()

Note: barplot() shows the mean, not the raw values. If you want raw counts by category, use countplot().

countplot() — counts how many rows belong to each category:

sns.countplot(data=df, x='Continent')
plt.show()

boxplot() — shows median, quartiles, and outliers. A fast way to spot unusual values:

sns.boxplot(data=df, x='Continent', y='2022 Population')
plt.show()

violinplot() — like a boxplot, but also shows the full distribution shape on both sides. More informative for larger datasets:

sns.violinplot(data=df, x='Continent', y='2022 Population')
plt.show()

stripplot() — shows every individual data point as a dot per category. Useful when you want to see actual values rather than a summary:

sns.stripplot(data=df, x='Continent', y='2022 Population', hue='Continent')
plt.show()

swarmplot() — same as stripplot but repositions overlapping points so each one is visible:

sns.swarmplot(data=df, x='Continent', y='2022 Population')
plt.show()

pointplot() — shows the mean per category as a dot, connected by lines across categories. Good for spotting trends:

sns.pointplot(data=df, x='Continent', y='2022 Population', hue='Continent')
plt.show()

catplot() — one function for all categorical charts. Use kind= to switch between them:

sns.catplot(data=df, x='Continent', y='2022 Population', kind='bar')
sns.catplot(data=df, x='Continent', y='2022 Population', kind='box')
sns.catplot(data=df, x='Continent', y='2022 Population', kind='violin')
sns.catplot(data=df, x='Continent', y='2022 Population', kind='strip')
sns.catplot(data=df, x='Continent', y='2022 Population', kind='swarm')
sns.catplot(data=df, x='Continent', y='2022 Population', kind='point')
plt.show()

Matrix & Regression Charts — Correlations and Trends

corr() + heatmap() — the most common combination for exploring how numeric columns relate to each other:

corr = df.select_dtypes(include='number').corr()
sns.heatmap(corr, annot=True, cmap='coolwarm')
plt.title("Correlation Matrix")
plt.show()

The values range from -1.0 (perfect negative relationship) to 1.0 (perfect positive relationship). annot=True displays the number inside each cell. cmap='coolwarm' colors high values red and low values blue.

clustermap() — same as heatmap but automatically reorders rows and columns by similarity using clustering:

sns.clustermap(corr, annot=True, cmap='coolwarm')
plt.show()

pairplot() — creates scatter plots for every combination of numeric columns at once, with distribution plots along the diagonal. The fastest way to get a complete overview of your data:

# All numeric columns
sns.pairplot(df.select_dtypes(include='number'))

# Colored by a category
sns.pairplot(df, hue='Continent')
plt.show()

jointplot() — shows a scatter plot of two columns, plus the distribution of each column on the margins:

sns.jointplot(data=df, x='Area (km2)', y='2022 Population')

# With a regression line
sns.jointplot(data=df, x='Area (km2)', y='2022 Population', kind='reg')
plt.show()

lmplot() — scatter plot with a regression line fitted through the data. Supports separate lines per group:

sns.lmplot(data=df, x='Area (km2)', y='2022 Population')

# Separate regression line per continent
sns.lmplot(data=df, x='Area (km2)', y='2022 Population', hue='Continent')
plt.show()

regplot() — the axes-level version of lmplot(). Does the same thing but doesn't support grouping. Use when you want to embed a regression plot inside a larger figure with subplots:

sns.regplot(data=df, x='Area (km2)', y='2022 Population')
plt.show()

residplot() — shows the residuals of a regression: how far each actual point is from the predicted line. A flat, random scatter around zero means the regression is a good fit:

sns.residplot(data=df, x='Area (km2)', y='2022 Population')
plt.show()

Combining Seaborn with Matplotlib

Seaborn creates the chart. Matplotlib customizes it. Both work together in the same block — you never have to choose one or the other:

sns.boxplot(data=df, x='Continent', y='2022 Population', palette='pastel')

plt.title('Population Distribution by Continent')
plt.xlabel('Continent')
plt.ylabel('Population')
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()

This is the standard workflow: Seaborn for the chart, Matplotlib for the title, labels, rotation, and layout.

Choosing the Right Tool

Use Matplotlib when:

Your data is in raw lists or NumPy arrays, not a DataFrame
You need multiple subplots in a specific layout
You want pixel-level control over every visual element

Use Seaborn when:

Your data is in a DataFrame and ready to visualize
You want a beautiful chart quickly with minimal code
You're doing statistical analysis — distributions, correlations, regressions, group comparisons

In most real projects, you'll use both: Seaborn to draw the chart, Matplotlib to label and polish it.

References

GitHub Repo: https://github.com/Hu8MA/Mastering-Pandas-Reference

Matplotlib Documentation: https://matplotlib.org/

Seaborn Documentation: https://seaborn.pydata.org/

Pandas Visualization Guide: https://pandas.pydata.org/docs/user_guide/visualization.html

Build a Data Science Query Language in Python using Lark

Sun, 29 Mar 2026 06:57:34 +0000

Build a Data Science Query Language in Python using Lark

What if you could write something like this:

DATA [1, 2, 3, 4, 5]
SUM
MEAN
STD


…and have it behave like a mini data science engine?

In this tutorial, we’ll build a **Domain-Specific Language (DSL)** for data analysis using:

- Python   
- Lark (parser library)   
- NumPy   

---

#  What Are We Building?

We are creating a **custom query language** that:

- Accepts a dataset
- Runs statistical commands
- Prints results

---

#  Step 1: Install Dependencies

```bash
pip install lark numpy

Step 2: Define the Grammar

The grammar defines how our language looks.

from lark import Lark, Transformer
import numpy as np

grammar = """
start: data command+

data: "DATA" list

command: "SUM" -> sum
       | "MEAN" -> mean
       | "STD" -> std
       | "MAX" -> max
       | "MIN" -> min

list: "[" NUMBER ("," NUMBER)* "]"

%import common.NUMBER
%import common.WS
%ignore WS
"""

Explanation

`start: data command+`

Program must start with DATA
Followed by one or more commands

`data: "DATA" list`

Defines dataset input
Example:
```
DATA [1, 2, 3]
```

Commands

SUM → sum
MEAN → mean
STD → std
MAX → max
MIN → min

These map text → function names
-> sum means call sum() in Transformer

List Rule

list: "[" NUMBER ("," NUMBER)* "]"

Accepts:
- [1]
- [1, 2, 3]
(, NUMBER)* means repeat

Ignore Spaces

%ignore WS

Allows flexible formatting

⚙️ Step 3: Build the Interpreter

Now we convert parsed text into execution.

class DLangInterpreter(Transformer):

    def data(self, items):
        self.data = np.array([float(x) for x in items[0]])
        return self.data

Explanation

items[0] → list of numbers
Convert to NumPy array
Store in self.data for reuse

Step 4: Add Operations

SUM

def sum(self, _):
    print(np.sum(self.data))

MEAN

def mean(self, _):
    print(np.mean(self.data))

STD

def std(self, _):
    print(np.std(self.data))

MAX

def max(self, _):
    print(np.max(self.data))

MIN

def min(self, _):
    print(np.min(self.data))

Explanation

Each function matches grammar rule
_ = unused input
Uses NumPy for computation
Prints result immediately

Step 5: Parse List

def list(self, items):
    return items

Explanation

Returns list of numbers
Passed to data() method

Step 6: Create the Parser

parser = Lark(grammar, parser="lalr", transformer=DLangInterpreter())

Explanation

lalr → fast parsing algorithm
transformer → auto-executes logic

Step 7: Read Input File

with open("example.dl") as f:
    code = f.read()

parser.parse(code)

Example `example.dl`

DATA [10, 20, 30, 40]
SUM
MEAN
MAX

✅ Output

100
25.0
40

How It Works (Flow)

Text Input
   ↓
Parser (Lark)
   ↓
Grammar Rules Match
   ↓
Transformer Methods Trigger
   ↓
NumPy Executes
   ↓
Output Printed

✨ Why This Is Powerful

You built a mini programming language
Clean separation of:
- Syntax (grammar)
- Execution (Transformer)
Easily extensible

Next Features You Can Add

1. Filtering

FILTER > 10

2. Sorting

SORT ASC

3. CSV Support

DATA file.csv

4. Chaining

DATA [1,2,3,4]
FILTER > 2
MEAN

Final Thought

This is how real systems like:

SQL
Pandas query engine
Spark

…start at a basic level.

You just built the foundation of a data query engine

If You Liked This

Drop a like ❤️
Follow for more AI + Systems content
And try extending this DSL yourself!

Mastering Pandas — Part 3: Data Cleaning, Merging & Joining

Wed, 18 Mar 2026 23:24:55 +0000

Real Data Is Never Clean

In Part 2, you learned how to group and slice a DataFrame with precision. But in real projects, the data you receive rarely arrives ready to analyze. Column names have spaces. Ages are negative. Phone numbers mix letters with digits. Duplicate rows sneak in. And your data often lives across multiple files that need to be combined before you can do anything useful.

This article covers the two skills that bridge raw data and real analysis:

Data Cleaning — detecting and fixing everything wrong with your data before it corrupts your results
Merging & Joining — combining separate DataFrames into a single, unified dataset

These topics flow naturally into each other. You clean each source first, then you combine them. That's the order every data analyst follows in practice — and it's the order we'll follow here.

We'll use this sample dataset throughout the cleaning section:

import pandas as pd
import numpy as np

data = {
    "Name":             ["Alice", "bob", "Carol", "Alice", "  dave  ", "Eve"],
    "Age":              [25, -5, 200, 25, 30, None],
    "Phone":            ["055-123-4567", None, "55123x", "055-123-4567", "055-987-0000", "055abc"],
    "Email":            ["*Emails are not allowed*", "*Emails are not allowed*", None, "*Emails are not allowed*", "*Emails are not allowed*", "not-an-email"],
    "Paying_Customer":  ["Y", "N", "Yes", "Y", "N", "y"],
    "Score":            [85, 110, 72, 85, -10, 90],
}
df = pd.DataFrame(data)

Step 1 — Inspect Before You Touch Anything

The most important rule in data cleaning: look before you act. Jumping straight into fixes without understanding the full picture leads to mistakes you won't catch until much later.

These are the inspection tools you should run on every new dataset:

df.shape          # (rows, columns) — how big is this?
df.info()         # column names, data types, non-null counts
df.head()         # first 5 rows
df.tail()         # last 5 rows
df.describe()     # min, max, mean, std — spot impossible values here

For missing values specifically:

df.isna().sum()           # count NaN per column
df.isnull().sum()         # same as isna() — both work
df[df["Phone"].notna()]   # rows that actually have a phone number

For duplicates and inconsistent categories:

df.duplicated()                         # True/False per row
df[df.duplicated()]                     # show only the duplicate rows
df["Paying_Customer"].value_counts()    # reveals: Y, N, Yes, y — all meaning the same thing

Always run describe() on numeric columns — the min and max alone will tell you immediately if something is wrong. A minimum age of -5 or a score of 110 out of 100 is a red flag you can't miss.

Step 2 — Fix Column Names First

Messy column names make everything harder. Clean them before anything else, so every subsequent operation uses predictable, consistent names.

Rename specific columns:

df.rename(columns={"Old Name": "New_Name", "phone number": "Phone"})

Clean all column names at once — the standard one-liner:

df.columns = df.columns.str.strip().str.lower().str.replace(" ", "_")

This strips surrounding spaces, converts to lowercase, and replaces spaces with underscores in a single chain. After this, "First Name" becomes "first_name", " Age " becomes "age", and so on.

Before:              After:
"First Name"    →    "first_name"
"  Age "        →    "age"
"Phone Number"  →    "phone_number"
"ZIP Code"      →    "zip_code"

Step 3 — Handle Missing Values

Missing values appear as NaN in Pandas. Before deciding what to do with them, you need to understand why they're missing — because the right fix depends on the cause.

Fill missing values with a fixed value:

df["Phone"].fillna("000-000-0000")    # placeholder for missing phone
df["Age"].fillna(0)                   # fill numeric with 0
df["Name"].fillna("Unknown")          # fill text with label
df["Score"].fillna(df["Score"].mean()) # fill with column average

Fill missing numeric values by estimating from neighbors:

df["Sales"] = df["Sales"].interpolate()

interpolate() fills NaN by calculating the midpoint between the values above and below it. This is ideal for time series or sensor data where values change gradually — it would be wrong to use for random datasets.

Before:     After interpolate():
100         100
NaN    →    150   ← estimated midpoint
200         200
NaN    →    225   ← estimated midpoint
250         250

Remove rows with missing values:

df.dropna()                            # drop any row with at least one NaN
df.dropna(subset=["Phone"])            # drop only if Phone is NaN
df.dropna(thresh=3)                    # drop rows that have fewer than 3 non-NaN values

inplace rule: use either inplace=True or variable assignment — never both. They do the same thing, and combining them produces a bug where df becomes None.
df.dropna(subset=["Phone"], inplace=True)   # ✅ modifies df directly
df = df.dropna(subset=["Phone"])            # ✅ returns new df
df = df.dropna(subset=["Phone"], inplace=True)  # ❌ df becomes None

Step 4 — Remove Duplicates

Duplicate rows silently inflate counts, averages, and totals. Always check for them.

df.drop_duplicates()                         # remove fully identical rows
df.drop_duplicates(subset=["Name"])          # remove rows with duplicate Name only
df.drop_duplicates(subset=["Name", "Email"]) # remove rows where both columns match
df.drop_duplicates(keep="last")              # keep the last occurrence instead of the first

Before:                        After drop_duplicates():
Name   Email                   Name   Email
Alice  *Emails are not allowed*          Alice  *Emails are not allowed*
Bob    *Emails are not allowed*      →     Bob    *Emails are not allowed*
Alice  *Emails are not allowed* ← dup    Carol  *Emails are not allowed*
Carol  *Emails are not allowed*

Step 5 — Fix Data Types

Pandas sometimes reads numeric columns as strings, or dates as plain text. Operations on wrong types either crash or silently produce wrong results.

Change column type directly:

df["Zip_Code"] = df["Zip_Code"].astype(str)     # treat as text, not a number
df["Age"] = df["Age"].astype(int)               # convert to integer
df["Price"] = df["Price"].astype(float)         # convert to decimal

Handle messy numbers that contain symbols or text:

df["Price"] = pd.to_numeric(df["Price"], errors="coerce")

errors="coerce" converts anything that can't be parsed into NaN instead of crashing. This is the safe way to clean columns like "$1,200" or "N/A" that can't be directly cast with astype().

Before:           After pd.to_numeric(..., errors="coerce"):
"1200"    →       1200.0
"$850"    →       NaN     ← couldn't parse
"700.5"   →       700.5
"N/A"     →       NaN     ← couldn't parse

Convert to datetime:

df["Date"] = pd.to_datetime(df["Date"])
df["Date"] = pd.to_datetime(df["Date"], format="%d/%m/%Y")  # specify format if needed

Once a column is proper datetime, you unlock sorting by date, filtering by year/month, calculating time differences, and much more.

Step 6 — Clean Strings

String columns are almost always the messiest part of a real dataset. Pandas provides a full .str accessor with everything you need.

Strip unwanted characters from edges:

df["Name"] = df["Name"].str.strip()           # remove spaces from both sides
df["Name"] = df["Name"].str.strip(" ./-")     # remove spaces, dots, slashes, dashes
df["Name"] = df["Name"].str.lstrip()          # left side only
df["Name"] = df["Name"].str.rstrip()          # right side only

Replace characters anywhere in the string:

df["Phone"] = df["Phone"].str.replace(r"[a-zA-Z]", "", regex=True)   # remove all letters
df["Phone"] = df["Phone"].str.replace(r"\D", "", regex=True)          # keep digits only
df["Name"]  = df["Name"].str.replace(r"[./]", "", regex=True)         # remove dots and slashes

Standardize text case:

df["Name"] = df["Name"].str.lower()     # all lowercase → "alice"
df["Name"] = df["Name"].str.upper()     # all uppercase → "ALICE"
df["Name"] = df["Name"].str.title()     # title case    → "Alice"

Check text content:

df[df["Email"].str.contains("@")]           # filter valid-looking emails
df[df["Phone"].str.len() != 12]             # find rows where phone length is wrong
df[df["Name"].str.startswith("A")]          # names starting with A
df[df["Name"].str.endswith("e")]            # names ending with e

Extract a pattern using regex:

df["Zip"] = df["Address"].str.extract(r"(\d{5})")        # pull 5-digit zip code
df["Area_Code"] = df["Phone"].str.extract(r"^(\d{3})")   # pull first 3 digits

Split one column into multiple columns:

df[["Street", "State", "Zip"]] = (
    df["Address"]
    .str.split(",", n=2, expand=True)
)

n=2 limits the split to 2 times, producing exactly 3 parts regardless of how many commas the string contains. expand=True spreads those parts into separate columns.

Before:                              After:
"123 Main St, Texas, 75001"   →   Street="123 Main St"  State="Texas"  Zip="75001"

Step 7 — Replace and Standardize Values

Raw data often has the same value written multiple ways: "Y", "Yes", "yes", "y" — all meaning the same thing. Standardize them.

Replace whole cell values using a dictionary:

df["Status"] = df["Status"].replace({"Y": "Yes", "N": "No", "y": "Yes"})

# Apply to multiple columns at once
df[["Paying_Customer", "Do_Not_Contact"]] = (
    df[["Paying_Customer", "Do_Not_Contact"]]
    .replace({"Y": "Yes", "N": "No", "y": "Yes", "n": "No"})
)

Overwrite invalid values using loc[]:

df.loc[df["Age"] < 0,   "Age"] = None     # negative age → invalid
df.loc[df["Age"] > 120, "Age"] = None     # age over 120 → invalid
df.loc[df["Price"] < 0, "Price"] = None   # negative price → invalid

Keep valid values, replace everything else with NaN:

df["Salary"] = df["Salary"].where(df["Salary"] > 0)
# Rows where Salary <= 0 become NaN; the rest are untouched

Now Combine — Merging & Joining DataFrames

Once each source is clean, the next challenge is combining them. In the real world, data almost never lives in a single file. You might have orders in one table, customers in another, and products in a third. Merging is how you bring those pieces together.

Pandas provides three main tools for this:

Tool	What it does
`merge()`	Joins two DataFrames by matching on a shared column
`concat()`	Stacks DataFrames vertically or side by side
`join()`	Joins on the index instead of a column

We'll use these two DataFrames for all the examples:

customers = pd.DataFrame({
    "ID":        [1, 2, 3, 4],
    "Name":      ["Alice", "Bob", "Carol", "Dave"],
    "Country":   ["USA", "UK", "India", "USA"],
})

orders = pd.DataFrame({
    "ID":        [1, 2, 3, 5],
    "Product":   ["Laptop", "Phone", "Tablet", "Monitor"],
    "Price":     [1200, 800, 600, 400],
})

Notice that ID=4 (Dave) exists in customers but not in orders, and ID=5 exists in orders but not in customers. This is exactly the kind of mismatch you encounter in practice — and why the how= parameter matters so much.

1️⃣ `merge()` — The Main Tool

merge() works like a SQL JOIN. It aligns two DataFrames by finding rows where a shared column has the same value.

Basic syntax:

customers.merge(orders, how="inner", on="ID")

The `how=` Parameter — All Four Types

# INNER → only rows that match in both DataFrames (default)
customers.merge(orders, how="inner", on="ID")
# Result: IDs 1, 2, 3 only — Dave (4) and Monitor (5) are excluded

# LEFT → all rows from the left + matched rows from the right
customers.merge(orders, how="left", on="ID")
# Result: all 4 customers — Dave gets NaN for Product and Price

# RIGHT → all rows from the right + matched rows from the left
customers.merge(orders, how="right", on="ID")
# Result: all 4 orders — Monitor (ID=5) gets NaN for Name and Country

# OUTER → all rows from both, NaN where no match
customers.merge(orders, how="outer", on="ID")
# Result: all 5 unique IDs — gaps filled with NaN on both sides

`how=`	Keeps from Left	Keeps from Right	NaN Possible
`inner`	Matched only	Matched only	No
`left`	All rows	Matched only	Right side
`right`	Matched only	All rows	Left side
`outer`	All rows	All rows	Both sides

CROSS merge — every row with every row:

customers.merge(orders, how="cross")
# Produces 4 × 4 = 16 rows — every customer paired with every order
# No on= parameter — it will raise an error if you include it

The `on=` Parameter — What to Match On

# Match on a single column
customers.merge(orders, how="inner", on="ID")

# Match on multiple columns — both must match simultaneously
customers.merge(orders, how="inner", on=["ID", "Country"])

# Auto-detect — Pandas finds all shared column names automatically
customers.merge(orders)

Be careful with auto-detect. If both DataFrames happen to share a column that you don't intend to join on (like a generic "Name" column), Pandas will include it in the match condition and produce unexpected results. Explicit is safer.

Handling Shared Column Names — `_x` and `_y`

If both DataFrames have a column with the same name, but you only join on ID, Pandas keeps both copies and renames them automatically:

# Both DataFrames have a "Name" column, but you only merge on "ID"
customers.merge(orders, how="inner", on="ID")
# Result: Name_x (from customers), Name_y (from orders)

Three ways to fix this:

# Option 1 — merge on all shared columns so no duplicates appear
customers.merge(orders, how="inner", on=["ID", "Name"])

# Option 2 — use custom suffixes instead of _x and _y
customers.merge(orders, how="inner", on="ID", suffixes=("_customer", "_order"))

# Option 3 — drop the shared column from one DataFrame before merging
orders_clean = orders.drop(columns=["Name"])
customers.merge(orders_clean, how="inner", on="ID")

Other Useful Parameters

# left_on / right_on — when the key column has different names in each DataFrame
customers.merge(orders, how="inner", left_on="CustomerID", right_on="OrderID")

# indicator — adds a column showing where each row came from
customers.merge(orders, how="outer", on="ID", indicator=True)
# _merge column values: "left_only", "right_only", "both"

The indicator=True parameter is especially useful for auditing: after a merge, you can immediately see which rows matched, which came only from the left, and which came only from the right.

2️⃣ `concat()` — Stacking DataFrames

While merge() combines DataFrames horizontally by matching values, concat() simply stacks them — either rows on top of rows, or columns side by side.

pd.concat([df1, df2])        # stack vertically (default)
pd.concat([df1, df2], axis=1)  # stack horizontally

`axis=` Parameter

# axis=0 → stack rows vertically (append one DataFrame below another)
monthly_data = pd.concat([january, february, march], axis=0)

# axis=1 → stack columns side by side
combined = pd.concat([names_df, scores_df], axis=1)

`join=` Parameter

# outer (default) → keep all columns, NaN where a column doesn't exist in one source
pd.concat([df1, df2], join="outer")

# inner → keep only columns that exist in all DataFrames
pd.concat([df1, df2], join="inner")

`ignore_index=` Parameter

# Without ignore_index — original indices are preserved, which can repeat
pd.concat([df1, df2])
# Index: 0, 1, 2, 0, 1, 2  ← 0, 1, 2 repeats

# With ignore_index=True — fresh sequential index
pd.concat([df1, df2], ignore_index=True)
# Index: 0, 1, 2, 3, 4, 5  ← clean

Always use ignore_index=True after stacking unless you specifically need the original indices.

`keys=` Parameter

# Label each source so you can identify where each row came from
combined = pd.concat([df1, df2], keys=["source_a", "source_b"])

This creates a MultiIndex where the first level is your label and the second is the original row index — useful when you need to trace data back to its source after combining.

`verify_integrity=` Parameter

# Raises an error if any duplicate index values are found after concatenation
pd.concat([df1, df2], verify_integrity=True)

`merge()` vs `concat()` — When to Use Which

	`merge()`	`concat()`
Direction	Horizontal (by matching)	Vertical or horizontal (by stacking)
Needs a shared key column	Yes	No
Best for	Joining related tables	Combining same-structure data
Equivalent to	SQL JOIN	SQL UNION

A simple rule: if you're combining columns from different tables, use merge(). If you're stacking more rows of the same structure, use concat().

3️⃣ `join()` — Index-Based Joining

join() works like merge(), but it joins on the index instead of a regular column. You need to set the index first.

customers_indexed = customers.set_index("ID")
orders_indexed    = orders.set_index("ID")

customers_indexed.join(orders_indexed, how="inner")

The how= options are the same as merge(): inner, left, right, outer.

Why `merge()` Is Almost Always Better

	`merge()`	`join()`
Join on any column	✅ Yes	❌ No — index only
Join on index	✅ Yes	✅ Yes
Requires `set_index()` first	❌ No	✅ Yes
Simpler syntax	✅ Yes	⚠️ Extra steps

> Use `join()` only when your data is already indexed — for example, after a `groupby()` that produced a meaningful index. Otherwise, `merge()` is cleaner and more flexible.

This is Part 3 of the Pandas for Data Science series. Next up: Part 4 — Data Visualization with Matplotlib & Seaborn

To access the full article, please visit the link below or directly access the repository via the links below.

Refrences

GitHub Repo : https://github.com/Hu8MA/Mastering-Pandas-Reference
Website : https://pandas.pydata.org/
GitHub Library : https://github.com/pandas-dev/pandas
Course : https://youtu.be/Mdq1WWSdUtw
Full article : https://hussein16mahdi.medium.com/mastering-pandas-part-3-data-cleaning-merging-joining-817a070147ca

Lecture 3 — The Art of Collecting Data

Mon, 09 Mar 2026 11:09:38 +0000

Lecture 3 — The Art of Collecting Data

By Vyoma Youth Society

In this lecture, we will discuss how data is collected and perform a small practical example using Python.

In the previous lectures, we discussed:

what data really is
why uncertainty exists in data

But before moving deeper into statistics and machine learning, we must understand something very basic:

Where does data come from?

Data does not appear magically.
Collecting data is a skill and a process.

So the first question we should ask is:

Where can we collect data from?

Before continuing, pause for a moment and write down a few places where you think data can be collected.

Collection of Data

Before we start working with statistics or building models, we must first have data to work with.

Data collection is the foundation of data science.
Without data, analysis is impossible.

When we think about collecting data, there are generally two main types of sources.

Types of Data Sources

1️⃣ Primary Sources

Primary data is the data that you collect yourself.

This usually involves first-hand methods such as:

surveys
interviews
experiments
observations

In primary data collection, you design the method and gather the data directly.

Example:
If you conduct a survey asking students about their study hours, the data you collect is primary data.

2️⃣ Secondary Sources

Secondary data is data that has already been collected by someone else.

Instead of collecting it yourself, you use existing datasets.

Examples include:

public datasets
research papers
government statistics
online databases

Using secondary data is very common in data science because many organizations publish large datasets.

For now, we will only introduce these concepts.
Later in this course, we will have a full module dedicated to data collection, where we will study these methods in much greater detail.

Practical Section

In the last two lectures we focused mainly on theory.
So today we will do a small practical exercise.

Our goal is simple:

Compare the manual calculation of the mean with the mean calculated by a Python library.

This helps us understand that libraries are simply implementations of mathematical formulas.

Python Practical

First, create a Python file and write the following code.

import numpy as np

a = np.array([i for i in range(10)])

# 1. Manual calculation
manual_mean = sum(a) / len(a)

# 2. Using NumPy
numpy_mean = np.mean(a)

print(f"Manual mean: {manual_mean}")
print(f"Numpy mean: {numpy_mean}")

Understanding What Happened

Before understanding the code, we should understand what mean really is.

The mean is simply the average value of a dataset.

The formula is:

[ Mean = \frac{\text{Sum of all values}}{\text{Number of values}} ]

In the program:

sum(a) adds all numbers in the dataset
len(a) gives the number of elements
sum(a) / len(a) calculates the mean manually

Then we use the NumPy function:

np.mean(a)

This function performs the same calculation internally.

When you run the code, both methods give the same result:

4.5

This shows that library functions are simply automated versions of mathematical formulas.

What This Practical Teaches Us

1️⃣ Data must first be collected before analysis.
2️⃣ Statistics is built on simple mathematical ideas.
3️⃣ Programming libraries automate these calculations.

Understanding the mathematics behind the libraries is an important goal of this initiative.

Our open source repository :-

https://github.com/psjdeveloper/vyomadatascience

Forecast Kebutuhan Bahan & Produksi Konveksi dengan Python (Praktis + Template)

Sun, 08 Mar 2026 17:00:00 +0000

Di konveksi, “ramai order” bukan selalu kabar baik—kalau stok kain meleset, jadwal jahit jadi chaos, dan vendor bahan mulai susah diajak senyum. Industri manufaktur (termasuk apparel) makin banyak mengadopsi smart production planning untuk menekan biaya dan menaikkan akurasi keputusan, seperti dibahas dalam artikel tentang perencanaan produksi cerdas di FMIPA Unpad. Kalau Anda pernah “kehabisan kain 2 roll” atau “kebanyakan stok warna yang salah”, artikel ini akan jadi playbook yang bisa dipraktikkan—dan kita akan menutup paragraf ini dengan satu frasa yang akan jadi kompas SEO sekaligus kompas kerja: forecast kebutuhan bahan python.

Dari sisi ilmiah, praktik peramalan permintaan (demand forecasting) yang memadukan data historis dan fitur kontekstual sudah jadi fondasi kuat di supply chain modern; salah satu rujukan riset yang relevan bisa Anda jadikan landasan adalah artikel di ScienceDirect tentang demand forecasting. Kita mengangkat tema ini karena banyak developer (dan founder) membangun sistem order tapi lupa “otak belakangnya”: perencanaan bahan, capacity planning, dan jadwal produksi yang data-driven—padahal di lapangan, itu yang menentukan order selesai tepat waktu.

“Kalau data order sudah rapi, stok bahan seharusnya bisa diprediksi. Kalau stok bisa diprediksi, jadwal produksi bisa diatur. Kalau jadwal teratur, reputasi naik.”
Ringkasnya: kita tidak sedang membahas “sekadar analitik”, tapi cara membuat operasional konveksi terasa seperti product engineering—ada metrik, ada logika, ada iterasi.

1. Kenapa Konveksi Perlu Forecast, Bukan Tebak-Tebakan

Di banyak bisnis konveksi, pengambilan keputusan bahan masih mengandalkan intuisi: “bulan lalu laku, bulan ini pasti laku.” Masalahnya, pola permintaan seragam kerja, jersey event, sampai pakaian muslim sering dipengaruhi kalender (lebaran, wisuda, agenda HR, event olahraga), tren desain, bahkan lead time vendor. Di sinilah forecast kebutuhan bahan python jadi alat bantu: bukan untuk meramal masa depan secara mistis, tapi untuk mengurangi variance keputusan.

Apa yang Biasanya “Bocor” di Alur Produksi

Overstock: kain numpuk, cashflow ketahan.
Stockout: produksi berhenti, deadline kacau.
Mismatch spesifikasi: warna/gramasi salah karena reorder buru-buru.
Bottleneck jahit/finishing: bahan ada, kapasitas tidak siap.

Sinyal Data yang Sering Diabaikan

Tanggal PO vs tanggal delivery (indikator lead time real).
Ukuran dominan (S/M/L/XL) per klien/per industri.
Rasio reject/QC (pengaruh ke kebutuhan kain ekstra).

2. Data Minimal yang Anda Butuhkan (Dan Cara Menyusunnya)

Sebelum ngomong model, rapikan dulu datanya. Kabar baiknya: Anda tidak butuh data warehouse mewah. Cukup satu tabel order yang konsisten.

Struktur Tabel Data Order

Kolom	Contoh	Kenapa penting	Catatan praktis
order_id	ORD-2026-0021	identitas transaksi	unik dan konsisten
tanggal_order	2026-02-10	time series	gunakan ISO date
kategori	seragam kerja	segmentasi permintaan	bisa: seragam/jersey/muslim/merch
sku/desain	SPV-KIIC-01	pola demand per desain	opsional tapi bagus
qty	120	target produksi	integer
ukuran_mix	M:60,L:40,XL:20	estimasi konsumsi kain	bisa JSON
bahan	oxford 200gsm	konsumsi & substitusi	normalisasi nama
lebar_kain_cm	150	konversi meter	penting untuk yield
allowance_qc	0.03	buffer reject	3%–7% sesuai historis
tanggal_kirim	2026-02-25	SLA	untuk evaluasi jadwal

Data Konsumsi Kain per Item

Buat tabel kecil lain (bisa spreadsheet) untuk “konsumsi standar” per kategori/ukuran.

kategori	ukuran	konsumsi_meter	catatan
seragam kerja	M	1.6	termasuk kampuh
seragam kerja	L	1.7
jersey	M	1.3	tergantung pola
gamis	L	2.6	termasuk rok

Di titik ini, Anda sudah siap menjalankan forecast kebutuhan bahan python dengan cara yang realistis: mulai dari demand (qty) → convert ke meter kain → tambahkan buffer QC → hasilkan rencana pembelian.

3. HowTo Scheme: Dari Data Order ke Rencana Beli Kain

Bab ini sengaja dibuat seperti engineering playbook—biar bisa Anda copy ke repo internal.

Step-by-step (Skema Praktis)

Ambil data order per hari/minggu dan segmentasi per kategori atau bahan.
Konversi qty → kebutuhan meter kain (berdasarkan ukuran mix + konsumsi standar).
Buat time series kebutuhan kain (meter) per bahan.
Forecast 4–8 minggu ke depan (pilih metode sederhana dulu).
Hitung reorder point = forecast demand selama lead time + safety stock.
Generate purchase plan: tanggal pesan, jumlah meter, prioritas supplier.

Template Python (Ringkas Tapi Siap Jalan)

Fokus contoh ini: forecasting mingguan untuk satu bahan. Anda bisa loop untuk semua bahan.

import pandas as pd
import numpy as np

# 1) Load data order yang sudah dikonversi ke kebutuhan meter per bahan
# kolom minimal: tanggal_order, bahan, kebutuhan_meter

df = pd.read_csv("order_kebutuhan_meter.csv")
df["tanggal_order"] = pd.to_datetime(df["tanggal_order"])

bahan_target = "oxford 200gsm"
ts = (
    df[df["bahan"] == bahan_target]
    .set_index("tanggal_order")["kebutuhan_meter"]
    .resample("W")
    .sum()
    .asfreq("W", fill_value=0)
)

# 2) Baseline forecast: moving average 4 minggu (simple & stable)
window = 4
forecast_horizon = 8

ma = ts.rolling(window=window, min_periods=1).mean()
last_ma = float(ma.iloc[-1])
forecast = pd.Series(
    [last_ma] * forecast_horizon,
    index=pd.date_range(ts.index[-1] + pd.Timedelta(weeks=1), periods=forecast_horizon, freq="W")
)

# 3) Reorder point (ROP)
lead_time_weeks = 2
service_factor = 1.65  # ~95% service level

hist = ts.tail(12)  # 3 bulan terakhir
sigma = float(hist.std(ddof=0))

expected_demand_lt = float(forecast.head(lead_time_weeks).sum())
safety_stock = service_factor * sigma * np.sqrt(lead_time_weeks)
rop = expected_demand_lt + safety_stock

print("Bahan:", bahan_target)
print("Forecast 8 minggu (meter/minggu):")
print(forecast.round(2))
print("\nROP (meter):", round(rop, 2))

Kenapa baseline ini cukup masuk akal? Karena di operasional, forecast yang “sedikit salah tapi stabil” sering lebih berguna daripada model kompleks yang susah dipelihara. Setelah baseline, baru naik level: feature engineering (musiman, event), atau model time series lain.

Tip: Di CoderLegion ada contoh yang relevan untuk memahami pola forecasting dari data historis—lihat Weather Service Project (Part 3): Predicting the Future with AI and OpenWeatherMap untuk gambaran alur “data → model → keputusan”.

Dalam praktik, Anda bisa menyebut pipeline ini sebagai forecast kebutuhan bahan python versi MVP.

4. Metrik yang Harus Anda Pantau (Biar Forecast Tidak Jadi Pajangan)

Forecast yang bagus itu bukan yang terlihat keren di notebook, tapi yang memperbaiki keputusan pembelian dan produksi.

KPI Operasional yang Nyambung

Fill rate: % order yang dipenuhi tanpa stockout.
Lead time aktual: dari PO ke bahan datang (per supplier).
Waste rate: sisa kain/lembar cutting.
On-time delivery: ketepatan kirim.

Error Metric (Untuk Model)

MAE (Mean Absolute Error): mudah dipahami tim ops.
MAPE: hati-hati kalau demand sering nol.
Bias: apakah forecast cenderung over/under.

Kalau Anda baru mulai, cukup pantau: MAE + bias. Jalankan forecast kebutuhan bahan python mingguan, lalu review tiap akhir minggu.

5. Studi Kasus Mini: Seragam Industri dengan Size Mix

Misal ada 3 klien industri minggu ini, total 420 pcs. Komposisi ukuran dan konsumsi standar memengaruhi kebutuhan kain.

Contoh Perhitungan Cepat

Ukuran mix: M 50%, L 35%, XL 15%
Konsumsi (meter): M 1.6, L 1.7, XL 1.85
Allowance QC: 4%

Kebutuhan meter (tanpa QC):

M: 210 × 1.6 = 336
L: 147 × 1.7 = 249.9
XL: 63 × 1.85 = 116.55
Total = 702.45 m

Dengan QC 4%: 702.45 × 1.04 = 730.55 m

Kalau Anda merekam perhitungan ini per minggu, Anda akan punya time series yang siap untuk forecast kebutuhan bahan python tanpa ribet.

6. FAQ (Yang Sering Ditanya Developer & Owner Konveksi)

Apakah harus pakai model machine learning?

Tidak. Mulai dari baseline (moving average / exponential smoothing) dulu. ML masuk kalau Anda sudah punya data stabil dan ingin memanfaatkan fitur kontekstual.

Data saya bolong-bolong, bisa?

Bisa, asal Anda konsisten “mengisi nol” untuk periode tanpa order. Time series butuh kontinuitas.

Berapa horizon forecast yang realistis?

Untuk konveksi: 4–8 minggu biasanya paling actionable. Di atas itu, ketidakpastian naik (event/seasonality/negosiasi klien).

Bagaimana kalau ada order mendadak besar?

Tambahkan mekanisme override: forecast baseline + “known future orders”. Ini lebih jujur daripada memaksa model mengerti kejutan.

Apa kaitannya dengan produksi (bukan cuma beli kain)?

Forecast bahan bisa diturunkan ke capacity planning: estimasi jam jahit, kebutuhan operator, urutan produksi per deadline.

7. Closing yang Lebih “Engineering” dari Sekadar Operasional

Sebagai penutup, menarik mengingat analogi dari Andrew Ng: “AI is the new electricity.” Artinya: AI (dan praktik data-driven) akan menjadi utility yang meresap ke banyak industri—bukan cuma tech company. Dalam konteks konveksi, “listrik” itu bisa berupa pipeline sederhana: data order → metrik → keputusan pembelian → jadwal produksi. Anda tidak harus mulai dari model canggih; yang penting adalah kebiasaan membuat keputusan berdasarkan data, lalu mengiterasinya.

Kalau Anda ingin melihat implementasi nyata di lapangan (seragam kerja, jersey, pakaian muslim, hingga merchandise) dan bagaimana kebutuhan produksi diterjemahkan jadi eksekusi yang rapi, Anda bisa cek profil layanan kami di akhir tulisan ini—dan jadikan artikel ini sebagai fondasi forecast kebutuhan bahan python yang dapat Anda adaptasi ke proyek internal.

Backlink (narasi yang wajar): Jika Anda sedang mencari mitra produksi seragam/konveksi yang siap diajak kolaborasi berbasis data—mulai dari estimasi kebutuhan bahan sampai penjadwalan produksi—kunjungi Konveksi Karawang — CV Mitra Mandiri Design.

{
  "@context": "https://schema.org",
  "@type": "Article",
  "headline": "Forecast Kebutuhan Bahan & Produksi Konveksi dengan Python (Praktis + Template)",
  "description": "Panduan praktis forecasting kebutuhan bahan dan perencanaan produksi konveksi dengan Python: dari data order, konversi konsumsi kain, baseline forecast, hingga reorder point.",
  "inLanguage": "id-ID",
  "mainEntityOfPage": {
    "@type": "WebPage",
    "@id": "https://coderlegion.com/"
  },
  "author": {
    "@type": "Organization",
    "name": "CV Mitra Mandiri Design",
    "url": "https://www.konveksikarawang.co.id/"
  },
  "publisher": {
    "@type": "Organization",
    "name": "CV Mitra Mandiri Design",
    "url": "https://www.konveksikarawang.co.id/",
    "logo": {
      "@type": "ImageObject",
      "url": "https://www.konveksikarawang.co.id/wp-content/uploads/"
    }
  },
  "about": [
    {"@type": "Thing", "name": "forecast kebutuhan bahan python"},
    {"@type": "Thing", "name": "demand forecasting"},
    {"@type": "Thing", "name": "production planning"},
    {"@type": "Thing", "name": "python"},
    {"@type": "Thing", "name": "konveksi"}
  ],
  "isAccessibleForFree": true
}

Uncertainty & Randomness — Why Data Is Never Perfec

Sat, 07 Mar 2026 07:46:37 +0000

Uncertainty & Randomness — Why Data Is Never Perfect

By Vyoma Youth Society

If we collect data carefully, why are we still uncertain?

In the first lecture, we learned that data is not reality — it is a representation of reality.
Today, we take one step further and ask:

Why can’t data ever be perfectly certain?

Why do two measurements of the same thing give different results?
Why can’t we predict the future exactly?
Why does error always exist?

This brings us to two ideas:
uncertainty and randomness.

1️⃣ Every Measurement Has Error

No measurement is perfect.

A thermometer does not show the exact temperature of the room.
A survey does not show the exact opinion of society.
A step counter does not count every movement of your body.

Between reality and numbers, something is always lost.

So uncertainty is not a mistake.
It is unavoidable.

Uncertainty is part of measurement itself.

2️⃣ Randomness vs Determinism

Some systems are predictable:

a calculator
a clock
simple machines

These are called deterministic systems.

Some systems are unpredictable:

coin toss
weather
stock market
human behavior

These are called random systems.

But here is a deep question:

Is randomness part of nature, or part of our ignorance?

Sometimes things look random because we do not know all the causes.

3️⃣ Random Variable: Turning Randomness into Numbers

In data science, we turn uncertain events into numbers.
These are called random variables.

Examples:

Coin toss → 0 or 1
Dice roll → 1 to 6
Daily temperature → a number

We cannot predict the exact value.
But we can study how often values appear.

So instead of predicting one number,
we study a distribution of numbers.

4️⃣ Order from Chaos: The Average

If we toss a coin once, the result is unpredictable.
If we toss it many times, something interesting happens.

The average slowly becomes stable near 0.5.

This idea is called the law of large numbers.

It teaches us:

individual events are uncertain
large patterns are stable

Chaos at small scale, order at large scale.

5️⃣ Variance: Measuring Uncertainty

Two datasets can have the same average but different uncertainty.

Example:

Dataset A: values are close together
Dataset B: values are spread out

Both can have the same mean, but:

one is reliable
one is risky

Variance measures how spread out data is.

Mean tells us the center.
Variance tells us the uncertainty.

6️⃣ Why This Matters

Uncertainty affects decisions:

weather prediction
medical diagnosis
economic policy
social planning

A prediction without uncertainty is dangerous.

Good data science does not remove uncertainty.
It understands and measures it.

Reflection Challenges

Challenge 1:
Observe one uncertain thing in your daily life (weather, mood, traffic, steps).
Write:

what changes
what seems stable

Challenge 2:
Answer in one paragraph:

Is randomness in the world, or in our knowledge?

Connection to Previous Lecture

Lecture 1:
Data is a representation of reality.

Lecture 2:
That representation is uncertain.

Next Lecture will ask:

What is a model, and why do we need it?

Vyoma Data Science Initiative
Awareness • Mathematics • Open Knowledge

What Is Data, Really? - Module 1 vyoma data science initiative

Wed, 04 Mar 2026 12:40:25 +0000

What Is Data, Really?

By Vyoma Youth Society

“Can numbers represent truth?”

Before moving forward, let us pause and ask:
What is data, really?

And more importantly,
Why has the world started saying that data is the new power?

Before reading further, try this simple exercise:
Write down on a piece of paper what you understand by the word data.
At the end of this post, look again at what you wrote and compare it with what you understand now.

Data as Representation

Data is not reality.
It can never be reality.

We cannot directly compare data with reality, because data is only a measurement of reality.

This may sound confusing at first, so let us observe something simple around us.

Consider your phone charging.

Your phone is part of reality.
When you look at your phone and note its battery percentage — 20%, 40%, 75% — you are creating data about your phone.

Notice something important:
You did not capture the phone itself.
You captured a numerical description of one aspect of the phone.

This shows us that:

Data is reality expressed in numerical form.

In other words,
data is a representation of reality, not reality itself.

The Process of Collecting Data

Let us return to the phone charging example and break it into steps.

What was the first thing we did?

We observed the phone.

So the first step of collecting data is:

1. Observation

Next, we converted that observation into a number (battery percentage).

2. Measurement

Finally, we wrote it down.

3. Recording

So the process of collecting data looks like this:

Observation → Measurement → Writing down

This is not just a practical process.
It also works like a mathematical function.

Data as a Mathematical Mapping

We can describe this process mathematically as:

[ f: Reality \rightarrow Numbers ]

This means:
A function takes something from reality and converts it into a number.

In simple symbolic form:

[ \lambda(r) \rightarrow n ]

Where:

r = reality
n = number

So whenever we collect data, we are applying a function that transforms reality into numbers.

We will study this idea more deeply later when we discuss lambda calculus and mathematical modeling.
For now, just observe this transformation carefully.

Reflection Exercise

Before the next session, do this exercise:

Choose one thing from your daily life (for example: sleep hours, steps walked, water intake, or phone usage).
Write down:
- What exactly you are observing
- How you are measuring it
- What number you are recording

Then think:

What part of reality did you ignore?
What assumptions did you make?

We will continue this discussion in the next class.

For now, your task is not to calculate, but to think:

Is data closer to truth, or only a structured description of truth?

vyoma data science iniitiative by vyoma youth society

Launching the Vyoma Data Science Initiative

Mon, 02 Mar 2026 06:03:29 +0000

Launching the Vyoma Data Science Initiative

A 12-Month Public Journey into Mathematical Data Science

Today, I’m starting something long-term.

Not a tutorial series.
Not a “learn pandas in 10 minutes” guide.
Not another machine learning crash course.

This is a structured, 12-month public journey into mathematical data science, built under the vision of Vyoma Youth Society.

Why This Initiative Exists

We live in a world where:

Data influences policy.
Algorithms influence perception.
Statistics influence belief.
Machine learning influences markets.

But very few people deeply understand how these systems actually work.

Most learn:

Libraries without theory.
Models without mathematics.
Predictions without philosophy.
Tools without ethics.

That gap is dangerous.

So this initiative is built on a simple idea:

Data science is not just coding.
It is a way of thinking about uncertainty, structure, and truth.

What This Series Will Be

Over the next 12 months, I will publish:

1 structured module per month
Mathematical derivations
Python implementations from scratch
Research-style mini projects
Ethical discussions around data

This will be a progressive roadmap from:

Foundations of data thinking
Probability and statistics
Linear algebra for ML
Optimization theory
Machine learning from scratch
Economic market simulations
Data science for awareness and social impact

Every module will contain:

Theory
Mathematical explanation
Clean Python implementations
A practical project
A research challenge

All code will be open-source in a public repository.

The Core Philosophy

This initiative is built around three principles:

1️⃣ First Principles Thinking

We won’t just “use” algorithms.
We will derive them.

2️⃣ Mathematical Clarity

Understanding why something works matters more than memorizing how to use it.

3️⃣ Ethical Awareness

Data science is powerful.
Power requires responsibility.

Under Vyoma Youth Society, this connects to a broader idea:

Awareness before influence.
Understanding before optimization.

What Makes This Different?

This is not just a machine learning path.

Later modules will include:

Simulating market behavior using demand–supply functions
Modeling policy shocks
Predicting equilibrium using ML
Studying bias and misinformation patterns
Building awareness-oriented analytical tools

Data science will be treated as a discipline — not just a career skill.

Open Repository Structure

The full project will be organized month by month:

vyoma-data-science/
│
├── month01_foundations/
├── month02_python_math/
├── month03_probability/
...
├── month12_capstone/

Each month = one conceptual layer.

Who This Is For

This series is for:

Students who want depth, not shortcuts
Developers who want mathematical clarity
Thinkers who care about ethics
Anyone curious about uncertainty and structure

You don’t need to know everything already.

You just need curiosity and discipline.

Publishing Schedule

One module per month.
Four structured posts per module.

This is a long-term commitment.

Not fast content.
Structured growth.

A Personal Note

I am not positioning myself as an “expert”.

This is a documented journey into deeper mathematical data science.
Everything will be transparent — mistakes included.

The goal is growth, clarity, and contribution.

If you want to walk this path — follow the series.

Month 1 begins soon:
Foundations of Data Thinking.

Let’s build this properly.

—
Vyoma Youth Society
Awareness • Mathematics • Open Knowledge

AWS Sagemaker Notebook Jobs for Accelerating Data Science Experimentation Workflows

Mon, 23 Feb 2026 06:15:54 +0000

This post walks through a solution that combines Amazon SageMaker, MLflow, and Optuna to create an automated, scalable hyperparameter optimization pipeline.

MS Excel for Data Analytics: A Beginner-Friendly Introduction

Fri, 23 Jan 2026 20:51:23 +0000

Howdy! Ever heard of Microsoft Excel? Ever wondered what its use is? Worry not, let me simplify it for you. Microsoft Excel is one of the most poowerful and beginner-friendly tools for data analytics. Before people move to advanced tools like Python or SQL, they often start with Excel, and for good reason. It helps you organize data, clean it, analyze it, and visualize it all in one place. How cool is that?

Whether you are tracking sales, survey responses, expenses, or social media performance, Excel can turn your raw numbers (data) into meaningful insights.

What You Will Learn from This Article

By the end of this article, you will be able to:

Understand what Microsoft Excel is and how it is used in data analytics
Identify rows, columns, and cells in a worksheet
Enter and organize data in table format
Use basic Excel functions like SUM, AVERAGE, COUNT, and IF
Clean data by sorting, filtering, and removing duplicates
Create Pivot Tables to summarize large amounts of data
Build simple charts to visualize trends and comparisons
Understand how Excel can be used to create basic dashboards for reporting
Gain a strong foundation to start working with data confidently, even as a beginner.

Excel Terms You Will Encounter (Excel & Data Analytics)

Here are simple explanations of key terms used in this article:

Cell - A single box in Excel where a row and column meet (example: A1).
Column - A vertical group of cells labeled with letters (A, B, C…).
Row - A horizontal group of cells labeled with numbers (1, 2, 3…).
Worksheet - One page inside an Excel file where you enter and analyze data.
Workbook - An Excel file that contains one or more worksheets.
Dataset - A collection of related data arranged in rows and columns.
Formula - An equation used in Excel to perform calculations (example: =A1+B1).
Function - A built-in formula in Excel that performs a specific task (example: SUM, AVERAGE).
Range - A group of selected cells (example: A1:A10).
Filter - A tool used to display only the data that meets certain conditions.
Sort - Arranging data in a specific order (A–Z, smallest to largest, etc.).
Pivot Table - A tool that summarizes and analyzes large datasets by grouping and calculating data automatically.
Chart - A visual representation of data, such as a bar chart or line graph.
Dashboard - A single page that shows key data insights using charts, numbers, and summaries.
Data Cleaning - The process of fixing or organizing data so it is accurate and ready for analysis.

All right, let’s get into understanding MS Excel.

1. Introduction to Excel

Definition

Microsoft Excel is a spreadsheet software developed by Microsoft that helps people store, organize, calculate, and analyze data using tables made of rows and columns.
It allows users to:

Perform calculations with formulas and functions
Sort and filter information
Create charts and graphs
Summarize data using tools like Pivot Tables

Simply put, Excel turns raw numbers into useful information that helps with decision-making.

Excel is a spreadsheet program made up of:

Rows – horizontal lines (1, 2, 3…)
Columns – vertical lines (A, B, C…)
Cells – where rows and columns meet (like A1, B2)

Each cell can contain:

Text (Names, Categories)
Numbers (Sales, Scores)
Dates (1/1/2026, 02-11-2026)
Formulas (calculations), among many others.

Example Dataset

Let’s say we have this simple sales dataset:

Date	Product	Quantity	Price	Total Sales
1/1/2026	Pen	10	15	150
1/2/2026	Book	5	70	350
2/3/2026	Pencil	7	25	175
1/4/2026	Rubber	3	10	30

*NOTE: In this table, the Total Sales column is calculated by multiplying Quantity × Price.*

This kind of table is the starting point for data analysis.

2. Excel Functions (Basic Calculations)

Functions are built-in formulas that help you perform calculations easily.

1. SUM – Add Numbers

Adds a range of numbers.

=SUM(C2:C5)

This adds all values from cell C2 to C5.

*Use case: Total sales, total expenses, total quantity sold.*

AVERAGE – Find the Mean

Calculates the average of numbers.

=AVERAGE(D2:D5)

*Use case: Average score, average daily sales.*

COUNT – Count Numbers

Counts how many cells contain numbers.

=COUNT(A2:A5)

*Use case: Number of transactions recorded.*

COUNTA – Count Non-Empty Cells

Counts cells that are not empty (text or numbers).

=COUNTA(B2:B5)

*Use case: Number of customers or products listed.*

IF – Logical Function

Performs a test and returns different results depending on the outcome.

=IF(C2>10, "High", "Low")

This checks if the value in C2 is greater than 10.

* Use case Categorizing sales as High/Low, Pass/Fail.*

3. Data Cleaning in Excel

Before analysis, data often needs cleaning.
Common cleaning tasks:

Removing duplicates
Data -> Remove Duplicates
Sorting data
Data -> Sort A to Z
Filtering data
Data -> Filter

Filtering lets you view only certain data, like:

Only sales for “Pen.”
Only records from January

4. Pivot Tables (Powerful Data Summaries)

Pivot Tables help you summarize large data quickly without complex formulas.

You can answer questions like:

Total sales per product
Sales per month
Quantity sold by category

Steps to create a Pivot Table:

Select your data
Go to Insert -> PivotTable
Choose where to place it
Drag fields into Rows, Columns, and Values

Example:

Rows -> Product
Values -> Sum of Total Sales

Excel will automatically calculate total sales for each product.

5. Charts in Excel

Charts help turn numbers into visuals that are easier to understand.

Common chart types:

Column Chart – Compares categories
Line Chart – Shows trends over time
Pie Chart – Shows proportions

How to insert a chart:

Select your data
Click Insert
Choose a chart type

*Example: Sales by product displayed as a column chart.*

6. Dashboards in Excel

A dashboard is a single page that shows key insights using charts, numbers, and summaries.

A simple Excel dashboard might include:

Total Sales (big number)
Sales by Product (chart)
Sales by Month (line chart)
Top Product (text or card)

Dashboards help decision-makers understand performance quickly.

Tools used in dashboards:

Pivot Tables
Charts
Slicers (interactive filters)

Conclusion

Excel is a powerful starting tool for data analytics because it helps you:

Organize data in tables
Perform calculations using functions
Clean and filter messy data
Summarize information with Pivot Tables
Visualize trends using charts
Build dashboards for reporting

Mastering Excel gives you a strong foundation before moving into advanced analytics tools like Power BI, Python, or SQL.

ALL THE BEST IN YOUR ANALYTICS JOURNEY!

Coder Legion - Recent posts tagged datascience

Advanced SQL Analytics for Data Analysis.

Advanced SQL analytics for running a hospital database.

Introduction

1. SQL Window Functions

OVER, PARTITION BY, ORDER BY, and Window Frames

Ranking Window Functions

Value & Navigation Window Function

Aggregate Window Functions

2. SQL Joins

INNER JOIN

LEFT JOIN

RIGHT JOIN

FULL OUTER JOIN

CROSS JOIN

SELF JOIN

Join Summary

3. Common Table Expressions (CTEs)

Simple CTEs

Multiple Chained CTEs

Recursive CTEs

CTEs vs. Subqueries

4. SQL Subqueries

Subquery Types

Subqueries in SELECT Clauses

Using Subqueries in FROM Clauses (Derived Table)

Subqueries in WHERE Clause

Conclusion

GitHub

A Thousand Cleared AI Devices: What the FDA's Own List Says About How AI Gets Approved

The three claims, computed

Cleared by analogy

The sharpest object is not in the file

The row is a snapshot of something licensed to move

The gate and the outcomes

What to do with this

Sources

Where the lineage would live

Introducing Native UUIDs in ThingsDB

How ThingsDB Handles UUIDs

Building a User Management System with UUIDs

1. Define the User Type

2. Set Up Collection Storage

3. Implement Procedures for CRUD Operations

Adding Users (NewUser)

Fetching Users (GetUserById)

Calling Procedures from Your Application

Using Native SDKs (Python, Go, C#, Java, JavaScript etc.)

Using the REST API

Conclusion

# Power BI Data Modelling: A Great Path to Great Analysis

Data Modeling in Power BI

Introduction

Why Data Modeling

Components of Power BI Data Model

Designing Source Tables

Loading Data

Cardinality

Cross Filter Directions

Creating a Relationship

The Model View Interface

Auto-Detect Relationships

Creating Relationships Manually

Step by Step for Creating Relationships for our Toy Store E-Commerce Database

Conclusion

The Forgetting Curve Everyone Cites Is Ebbinghaus's Own Column, Relabeled. The 90% Is Not in It.

What he measured

The table

The shape

Three people who ran it again

What a saving is

What decades look like

Our April sentence

Measuring your own

Keep the trail back to where the number was measured.

Building a Real-Time Social Listening & Brand Sentiment Engine with Python, Gemini 1.5, and DuckDB

1. The Architecture

Core Pillars:

2. Ingestion & Deduplication via DuckDB

3. Structured Gemini 1.5 Flash Evaluation

Adding Users (`NewUser`)

Fetching Users (`GetUserById`)