Coder Legion - Recent posts tagged csv

Reading a rental-yield CSV without inventing missing data

Thu, 01 Oct 2026 06:55:49 +0000

A small CSV can be easy to load and surprisingly easy to misinterpret. This walkthrough uses dataHabibi's owned June 2026 Dubai community rental-yield snapshot to show how to preserve units, missing values and the distinction between a community and a city aggregate.

Disclosure: I am Haron Merzaie, co-founder of dataHabibi alongside Ibrahim Qorraj. The source platform is https://datahabibi.ae/. This is a historical data-reading example, not a live feed or an investment recommendation.

Start with the file's actual schema

The CSV has six columns: Community, Gross rental yield percent, Buildings tracked, Yield sample size, Snapshot, and Source. There are twelve named community rows and one row labelled Dubai city average.

The dataset and its field documentation are available in the public repository datahabibi-bot/dubai-community-yields-june-2026 on GitHub. Download dubai-community-yields-june-2026.csv from that repository and put it beside your Python script.

For example, the International City row records 9.98 in the yield field, 358 buildings tracked and a yield sample size of 220. The city-average row records 8.16 and 2,561 buildings, but has a blank sample-size field. These are values in this particular snapshot; they do not describe today's market.

Preserve percentages and missing values

The value 9.98 means 9.98 percent. If a later calculation needs a decimal fraction, convert it explicitly by dividing by 100. Do not silently switch between percentage points and fractions.

The blank city-average sample size means the source did not supply it. Converting it to zero would change the meaning of the file. Represent it as None instead.

This standard-library example preserves those distinctions:

import csv
from decimal import Decimal

with open("dubai-community-yields-june-2026.csv", newline="", encoding="utf-8") as source:
    rows = list(csv.DictReader(source))

records = []
for row in rows:
    sample_text = row["Yield sample size"].strip()
    records.append({
        "community": row["Community"],
        "yield_percent": Decimal(row["Gross rental yield percent"]),
        "buildings_tracked": int(row["Buildings tracked"]),
        "yield_sample_size": int(sample_text) if sample_text else None,
        "snapshot": row["Snapshot"],
    })

communities = [r for r in records if r["community"] != "Dubai city average"]
city_rows = [r for r in records if r["community"] == "Dubai city average"]

for record in communities:
    print(record["community"], record["yield_percent"], record["yield_sample_size"])

Decimal keeps the recorded decimal representation explicit. The original CSV remains unchanged, so another reader can distinguish source values from transformations in your code.

Do not manufacture an aggregation method

Separating city_rows prevents the city aggregate from being treated as a thirteenth community. It also avoids accidentally including the aggregate in an average of community rows.

The snapshot does not specify the sample unit, weights or aggregation method. Buildings tracked therefore cannot be assumed to be the yield denominator. The displayed community rows cannot independently reconstruct the city-average yield. A chart can show recorded values, but its caption should name the June 2026 period and explain that limitation.

The CSV also does not include the underlying rents and property prices needed to recompute yields. Gross rental yield is different from net yield or total return. Service charges, vacancy, maintenance, financing and transaction costs are not fields in this file. It contains no price forecast or guaranteed future return.

Make reuse traceable

The owned dataset is released under CC BY 4.0. When sharing an adaptation, credit dataHabibi, identify the June 2026 snapshot, include the licence and indicate changes. The dataset release does not include rights to the logo or commercial platform source code.

The practical lesson is small: retain the source file, describe its units, preserve blanks, separate aggregate rows and state what the data cannot establish. Those choices make a simple CSV analysis easier for another developer to inspect and reproduce.

Standardizing the Ingestion Interface: OpenBrokerCSV and the MIT Importer

Fri, 29 May 2026 08:00:00 +0000

Standardizing the Ingestion Interface: OpenBrokerCSV and the MIT Importer

We open-source a financial interaction workflow — not a bag of UI widgets. The concrete artifact is @pocket-portfolio/importer (packages/importer, MIT on npm). It normalises broker-specific chaos into a Trade model the app and SDK consumers can reason about.

Package reality

From packages/importer/package.json:

Name: @pocket-portfolio/importer
License: MIT
Surface: TypeScript adapters (Robinhood, Fidelity, eToro, IBKR Flex, Koinly, TurboTax, Ghostfolio, Sharesight, and more) + universal inference for unknown CSV shapes.

Schema SSOT in-repo: packages/importer/README.md, packages/importer/src/universal/types.ts, adapter registry in src/registry.ts. A dedicated SCHEMA.md is on the roadmap; until then, cite TypeScript types and README — do not invent a phantom file path in external copy.

Client-side parse by design

Broker CSV parsing runs in the browser for the product architecture we ship — the file does not need to land on an app-server disk for transformation. That supports:

Inspectable transforms (clone the repo, read the adapter),
Reduced central warehouse temptation for raw exports,
Adversarial testing in src/universal/adversarial.ts without multi-tenant DB blast radius.

Universal mapping: truncated preview only

For unknown columns, docs/IP-TECHNICAL-MECHANISMS.md documents truncated payload for mapping assistance: headers + sample rows — full file stays client-side. This mirrors the Ask AI instinct (Part 2): send the minimum structure needed for the next step.

OpenBrokerCSV narrative

Open Portfolio surfaces OpenBrokerCSV as the procurement-facing name for the normalised interchange (app/open/openbrokercsv/page.tsx). Pocket exercises the same package under retail UX pressure — that is the harness relationship (Part 6).

What we do not promise

A plugin marketplace runtime in app/ — not shipped.
“Zero server” after import for signed-in users — Firebase still holds authoritative trades (Part 4).

Do point engineers to GitHub, Discussions, and npm — the real contributor loop (README-OSS-WORKFLOW.md).

Part 5 of Sovereign Ingestion & Stateless Inference.

Read the full Sovereign Intelligence book, explore Open Portfolio, or try Pocket Portfolio.

Introducing csv-faker-generator: Generate Realistic Test Data in Seconds

Fri, 28 Nov 2025 22:22:04 +0000

Photo courtesy of Pexels

TL;DR: I just published csv-faker-generator, a TypeScript CLI tool that generates CSV files with realistic fake data. Perfect for testing, prototyping, and development. Install it with npm install csv-faker-generator.

The Problem

How many times have you needed test data for your application? Whether you're building a data pipeline, testing import functionality, or prototyping a feature, you inevitably need realistic-looking CSV files with hundreds or thousands of rows.
You have a few options:

Manually create sample data — tedious and limited
Use existing faker libraries — great, but requires writing boilerplate code
Export production data — risky and time-consuming
Use a generic CSV generator — inflexible and often outdated

None of these felt ideal. I found myself repeatedly writing the same wrapper code around @faker-js/faker and csv-writer just to generate test datasets. That's when I decided to build csv-faker-generator.

The Solution

csv-faker-generator is a lightweight, open-source npm package that makes generating fake CSV data as simple as running a single command.

Features

19+ data types — UUID, names, emails, addresses, phone numbers, dates, and more
CLI & Programmatic API — Use it from the command line or integrate it into your Node.js/TypeScript projects
Full TypeScript support — Complete type definitions included
Customizable output — Define your own columns and data types
Fast & efficient — Generate thousands of rows in milliseconds

Getting Started

Installation

npm install csv-faker-generator
or use it globally
npx csv-faker-generator

CLI Usage

The simplest way to generate a CSV file is through the command line:

Generate 100 user records
npx csv-faker -c id:uuid,name:name,email:email -r 100 -o users.csv

This creates a file with 100 rows containing random UUIDs, names, and emails.

More Examples

Generate customer data:

npx csv-faker --columns "userId:uuid,firstName:firstName,lastName:lastName,email:email,phone:phone" --rows 50 --output customers.csv

Generate company data:

npx csv-faker -c id:uuid,company:company,city:city,country:country -r 20

Generate employee records with salary data:

npx csv-faker -c id:uuid,name:name,jobTitle:jobTitle,salary:number -r 200 --output employees.csv

Programmatic Usage

Use it directly in your Node.js or TypeScript projects:

import { generateCSV } from 'csv-faker-generator';

// Generate user test data
await generateCSV({
  columns: [
    { header: 'id', type: 'uuid' },
    { header: 'name', type: 'name' },
    { header: 'email', type: 'email' },
    { header: 'address', type: 'address' },
  ],
  rows: 100,
  output: 'users.csv',
});

Real-World Use Cases

Testing Data Import Features

When building a feature that imports CSV files, you need realistic test data:

npx csv-faker -c orderId:uuid,customerName:name,email:email,amount:number -r 1000 -o orders.csv

Now you have 1000 realistic order records to test your import logic.

Load Testing & Performance Testing

Generate large datasets quickly to test your application's performance:

npx csv-faker -c id:uuid,data:username,value:number -r 100000 -o large-dataset.csv

Prototyping & Demos

Show stakeholders what your data pipeline looks like with real-looking sample data instead of [SAMPLE DATA] placeholders.

Database Seeding

Generate initial data for your development environment:

import { generateCSV } from 'csv-faker-generator';
import { parseCSV } from './your-parser';

// Generate test data
await generateCSV({
  columns: [
    { header: 'id', type: 'uuid' },
    { header: 'username', type: 'username' },
    { header: 'email', type: 'email' },
  ],
  rows: 500,
  output: 'seed-data.csv',
});

// Then seed your database
const data = await parseCSV('seed-data.csv');
await database.insert(data);

Why I Built This

As an Engineer working heavily with data pipelines and API testing, I found myself repeatedly needing the same thing: quick, realistic test data. The existing solutions either required a lot of boilerplate code or were overly complex for simple use cases.
I wanted something that was:

Dead simple to use (one command)
Flexible for both CLI and programmatic use
Well-typed (since I work with TypeScript)
Focused on the specific problem of CSV generation

Open Source & Contributions

This package is completely open source (MIT licensed) and hosted on GitHub. If you find it useful, have feature requests, or want to contribute, I'd love to hear from you!

Repository: https://github.com/manojkken/csv-faker-generator

npm: https://www.npmjs.com/package/csv-faker-generator

Conclusion

Whether you're testing a data pipeline, prototyping an application, or just need realistic sample data, csv-faker-generator is here to save you time. Give it a try and let me know what you think!

npm install csv-faker-generator

Questions or feedback? Feel free to open an issue on GitHub or reach out to me on Twitter.

Guide to Python's CSV Module

Wed, 23 Oct 2024 20:26:28 +0000

Working with data is an inevitable part of programming, and as someone who often finds themselves knee-deep in various file formats, I’ve always appreciated how Python simplifies the whole process.

One such file format that comes up regularly, particularly in data analysis, is the CSV file.

The CSV, or Comma-Separated Values, is a popular data exchange format due to its simplicity.

Luckily, Python comes with a built-in module called csv, which makes working with these files remarkably efficient.

In this article, I’ll break down how the csv module works in Python, from basic usage to more advanced techniques that can save you tons of time when processing data.

What Is a CSV File?

Before diving into the csv module, let’s start with a basic understanding of what a CSV file is.

A CSV file is essentially a plain text file where each line represents a row of data, and each value is separated by a comma (or sometimes other delimiters like tabs).

Here's a quick example of what it might look like:

Name,Age,Occupation
Alice,30,Engineer
Bob,25,Data Scientist
Charlie,35,Teacher

Why the csv Module?

You might wonder why you'd need the csv module when CSV files are just text files that could theoretically be read using Python's standard file handling methods.

While this is true, CSV files can have complexities—like embedded commas, line breaks within cells, and different delimiters—that are tricky to handle manually.

The csv module abstracts all of this, letting you focus on your data.

Reading CSV Files

Let’s jump into the code.

The most common operation you'll perform on a CSV file is reading its contents.

The csv.reader() function in the module is an easy-to-use tool for that.

Here's a step-by-step guide on how to do it.

Basic CSV Reading

import csv

# Open a CSV file
with open('example.csv', 'r') as file:
    reader = csv.reader(file)

    # Iterate over the rows
    for row in reader:
        print(row)

This is the simplest way to read a CSV file.

The csv.reader() returns an iterable, where each iteration gives you a list representing a row of the file.

Handling Headers

Most CSV files come with headers in the first row, like column names.

If you don’t need these headers, you can simply skip the first row when iterating:

import csv

with open('example.csv', 'r') as file:
    reader = csv.reader(file)

    # Skip header
    next(reader)

    for row in reader:
        print(row)

Sometimes, I’m working with files that contain a mix of useful and irrelevant data, and I find myself skipping rows based on more than just the header.

You can do this easily within the for loop.

DictReader: A More Intuitive Way to Read CSV Files

If your CSV file has headers, the csv.DictReader() is another fantastic option that reads each row as a dictionary, with the keys being the column names:

import csv

with open('example.csv', 'r') as file:
    reader = csv.DictReader(file)

    for row in reader:
        print(row)

This approach can make your code more readable and intuitive, especially when working with large datasets.

For example, accessing row['Name'] feels much clearer than dealing with index-based access like row[0].

Writing to CSV Files

Once you’ve read and processed your data, chances are you'll want to save or export it.

The csv.writer() function is your go-to tool for writing to CSV files.

Basic CSV Writing

import csv

# Data to be written
data = [
    ['Name', 'Age', 'Occupation'],
    ['Alice', 30, 'Engineer'],
    ['Bob', 25, 'Data Scientist'],
    ['Charlie', 35, 'Teacher']
]

# Open a file in write mode
with open('output.csv', 'w', newline='') as file:
    writer = csv.writer(file)

    # Write data to the file
    writer.writerows(data)

The writer.writerows() function takes a list of lists and writes them to the CSV file, where each inner list represents a row of data.

DictWriter: A Cleaner Way to Write CSV Files

Just as we have DictReader for reading CSV files into dictionaries, we have DictWriter for writing dictionaries to a CSV.

This method can be particularly handy when you want to specify your column names explicitly.

import csv

# Data as list of dictionaries
data = [
    {'Name': 'Alice', 'Age': 30, 'Occupation': 'Engineer'},
    {'Name': 'Bob', 'Age': 25, 'Occupation': 'Data Scientist'},
    {'Name': 'Charlie', 'Age': 35, 'Occupation': 'Teacher'}
]

# Open file for writing
with open('output.csv', 'w', newline='') as file:
    fieldnames = ['Name', 'Age', 'Occupation']
    writer = csv.DictWriter(file, fieldnames=fieldnames)

    # Write the header
    writer.writeheader()

    # Write the data
    writer.writerows(data)

Using DictWriter, you get a nice, clean interface to write dictionaries to CSV while keeping your code readable and concise.

Customizing Delimiters

By default, the CSV module uses commas to separate values, but sometimes you might be working with files that use other delimiters, such as tabs or semicolons.

The csv module provides an easy way to handle these cases by specifying the delimiter argument.

import csv

with open('example_tab.csv', 'r') as file:
    reader = csv.reader(file, delimiter='\t')

    for row in reader:
        print(row)

I’ve come across CSV files that use semicolons instead of commas—usually from European sources—and it’s comforting to know that Python’s csv module handles this with ease.

Whether it's commas, tabs, or any other delimiter, the csv module has got you covered.

Handling Complex Data

What if your data contains commas within fields, quotes, or even line breaks?

The CSV module automatically handles such cases by using quoting mechanisms.

You can also control how quoting works using the quoting parameter.

import csv

data = [
    ['Name', 'Occupation', 'Description'],
    ['Alice', 'Engineer', 'Works on, "cutting-edge" technology'],
    ['Bob', 'Data Scientist', 'Loves analyzing data.']
]

with open('complex.csv', 'w', newline='') as file:
    writer = csv.writer(file, quoting=csv.QUOTE_ALL)
    writer.writerows(data)

In this example, QUOTE_ALL ensures that every field is wrapped in quotes.

Other quoting options include csv.QUOTE_MINIMAL, csv.QUOTE_NONNUMERIC, and csv.QUOTE_NONE, giving you full control over how your CSV data is formatted.

Conclusion

Over the years, I’ve come to rely on the CSV format as a lightweight, efficient way to move data around, and Python’s csv module has been a trusty companion in that journey.

Whether you’re dealing with simple spreadsheets or complex, multi-line data fields, this module makes the process feel intuitive and effortless.

While working with CSVs may seem like a mundane task at first, it’s a gateway to mastering data manipulation.

In my experience, once you’ve conquered CSVs, you'll find yourself confidently tackling larger, more complex formats like JSON or SQL databases. After all, everything starts with the basics.

Meet the .csv datasets and understand how they works on creating expert system Artificial Intelligence models

Sun, 20 Oct 2024 02:03:52 +0000

Introduction

I’m sure you have listened about CSV datasets. If you are here, it's because you're interested in Artificial Intelligence (AI). AI development is often associated with Machine Learning and huge datasets with big databases. Maybe you have a big idea in mind; you search on YouTube or Google, and the first thing you find out is that if your project doesn't require a large amount of data, a .csv dataset is suitable for this. That is true. It is also true that CSV datasets are very useful for data analysis and are used by enterprises to store and save their data.

I feel that when we talk about CSV datasets, not all is clear. There is much information, but it is not organized in a single place with simple words to allow beginners to understand and use them effectively. The goal of this post is to understand what CSV datasets are, the differences between Machine Learning and Expert Systems, how traditional Expert Systems allow us to automate many tasks, when and how to use .csv datasets, and what the main uses of CSV datasets are.

What this post will not teach you

In this post, I will not explain how to write code that makes CSV files work. This post is centered on CSV files/datasets. Perhaps next week, in another post, I will explain how to create a basic chatbot, assistant, or enterprise bot.

Csv dataset definition

I could use a technical concept from external sources to give a definition of a CSV file, also called CSV dataset or Comma Separated values, but it’s not necessary. Instead, I'll define it and explain it from my understanding and with my own words. A CSV dataset is a collection of structured/organized data often used for creating small artificial intelligence models that do not require much data. They are organized in variables. For example:

# Variable 
question, answer
# Information
What is a csv dataset?, is a collection of data saved very well organized.
How can i create a csv dataset?, at the end of the file add the extension .scv

# As you can see in the example the variables are separated with a comma

Tip: In the example above the part before the comma is the question and the next part is the answer, the order have to match with the variables names defined on the first line.

In this screenshot, I have a dataset with three variables: image_path refers to the source of the image, image_name is the name of the image, and image_description is, as you can imagine, the description given to the image. In this particular case, I'm working with image recognition. However, if you need to work only with messages and answers, the pattern will be the same but with two variables: question and answer, as shown in the next example.

    image_path, image_name, image_description

    static/images/tucan.jpg,Tucán,Un ave colorida con un gran pico, nativa de las selvas tropicales.
    static/images/jaguar.jpeg,Jaguar,Un felino grande y poderoso que habita en América Latina.
    static/images/tapir.jpeg,Tapir,Un mamífero grande y herbívoro que se encuentra en las selvas tropicales.
    static/images/mango.jpeg,Mango,Un fruto tropical dulce y jugoso, muy popular en América Latina.
    static/images/cafe.jpeg,Café,Granos que se utilizan para hacer una bebida estimulante conocida mundialmente.
    static/images/guayaba.jpeg,Guayaba,Una fruta tropical rica en vitamina C, con un sabor dulce y ácido.

Tools to create a CSV file

It depends on your goals or the type of data you’re working with. CSV files can be created using Notepad++, Microsoft Excel, or a code editor like Visual Studio or PyCharm.

Creating a CSV dataset

To create a CSV file using Excel, select the type CSV (comma delimited file, using a text or code editor, name your file and add the extension .csv, for example, dataset.csv you can use the file for creating a dataset on external services like Oracle, or any other cloud service provider.

Reading a CSV dataset

If you need to open and edit your CSV dataset, simply go to your files, locate the CSV dataset, right-click on it, and select "Open with." You can open it with Notepad or a code editor like Visual Studio Code. Once opened, you will be able to read, edit, and save your changes.

Differences between Machine Learning and Expert Systems

First of all, why am I talking about Machine Learning here? When somebody talks about artificial intelligence, we often associate it with Machine Learning or Natural Language Processing or think that it is something different apart from AI. But the reality is that Machine Learning, Natural Language Processing, Computer Vision, Expert Systems, and many others are types of Artificial Intelligence.

The traditional Expert System is a method of artificial intelligence that uses CSV files in their datasets. In this AI model, the developer defines the possible messages or characteristics and provides specific answers if one of those possible messages or characteristics is present.

In a Machine Learning model, you provide a database or a large collection of data, and the model itself uses this data to compare the content of the message with the existing information in the database. If this data coincides with one element in the database, the Machine Learning model will give an answer based on this data.

Both methods have their advantages and disadvantages. Machine Learning models are very useful when working with large language models and other projects with high level of unknowns, however the answers provided by these models can sometimes be inaccurate. So, when we need a bot that provides very accurate, precise answers, we use Expert System models trained with a CSV dataset.

How do traditional Expert Systems allow us to automate tasks?

Let me use an analogy so that we can understand very well how to use CSV datasets with a traditional Expert System AI model to automate repetitive tasks. Imagine that you are a small, medium, or large business that sells products, cellphones, let's say. To attract more clients, you provide support via WhatsApp or on your website. Those clients will ask specific questions about the prices of specific models. They will ask about promotions, available models, schedules of attention, and so on. All those tasks are repetitive and time-consuming. A good amount of those clients will ask questions and not buy, so you're losing your valuable time. What if we create a bot that can answer each question from each client automatically, specifically, and instantly?

Problems like this can be solved using a CSV dataset in which you define the answer for every possible question or message. Of course, you'll need to develop a simple model capable of reading the file, receiving the message sent to the server, and responding with the specific answer. Now, when the client asks something, the bot will respond instantly, reducing time, work, and providing a good experience for the user.

Uses of .csv datasets?

Use a .csv dataset anytime you need to develop a product (AI model) whose first requisite is very accurate and specific answers, as we discussed earlier. In simple terms, this kind of dataset is used for creating Expert System models, but a CSV dataset can also be used to store data in a structured or organized way. Data that can later be used for data analysis and facilitate decision-making for administrative personnel.

Main uses of CSV datasets explained

Chatbots: CSV datasets can be used to create assistant bots for small, medium, and large businesses or enterprises. These bots can be integrated with Twilio, WhatsApp, Telegram, or a website. Next week, I'll publish a tutorial to explain step-by-step how to create a chatbot.
Data store: Save data in an organized manner. You can also use other tools that interact with the data you want to save to write a CSV file or update it automatically.
Data analysis tools: CSV datasets can be easily manipulated with Python libraries like Pandas and Matplotlib for data analysis and visualization.
Image recognition (integrated with other tools): As shown in the earlier screenshot, you can create a CSV dataset with a specific structure to integrate with other tools for image recognition systems.
Other uses

How to use?

The answer to this question depends on the goal or what you’re using it for. Let's say that if you want to create an assistant chatbot, the first step is to create your CSV file with the specific possible questions and the respective answers for each question. But that is not all; you will also need a programming language to train the model (the Expert System I explained earlier). Additionally, this model needs to be integrated into a website or with services like WhatsApp, Twilio, or others. The "how-to" is not covered in this post because it needs more space to cover the step-by-step integration. In this post, as I explained in the introduction, my goal is to understand what CSV datasets are and how we can use them to make our lives easier.

Advantages of a CSV datasets

Accurate and specific answers: When used in assistants or chatbots, you can specify the answer for the possible questions or messages.
Personalization: You can create or update your data according to your needs.
Organization: You can save or write your data from other programs organized in variables according to your needs, which will improve readability and allow this data to be integrated with other programs, libraries, or tools in general to analyze its information.

Conclusion

A CSV dataset is a collection of organized data. These files can be used in combination with the traditional artificial intelligence methodology called Expert System to create automated bots, assistants, data analysis tools, and many other automation tools for small, medium, and large businesses interested in saving time, money, and work, while providing good, personalized, and instant assistance to their clients. They are also used to store data in a well-organized and structured way to be later used in data analysis, facilitating decision-making.

Mastering Reading and Writing CSV Files in Python

Wed, 27 Mar 2024 10:16:53 +0000

CSV stands for Comma-separated Values, a popular format for storing tabular data in a spreadsheet or database. Most of the time, people use databases for large amounts of data and spreadsheets for small amounts. But CSVs still have their places. They are simple and convenient. No drivers or special APIs are needed to use them. Python makes them even simpler with the CSV module. So, in this tutorial, I will be talking about the basic understanding of CSV files and how to read, parse, and write CSV files, with some advanced concepts of dealing with CSV efficiently. So, let's get started!

Understanding CSV Files

Overview of CSV File Format

A CSV is a text file that contains the data. Oftentimes, the first row in the file is a header, letting you know what the values represent, and the remaining lines contain the data. Here, each row represents a record in the database. Its plain text format makes it easy and nifty for the website developers to create it. Further, CSV files are also easy to import, and you can import the CSV file from a spreadsheet or a database.

CSV Format: Delimiters, Line Break, and Data Separation

In each row, the pieces of data are separated by commas known as delimiters. Also, you can use some other delimiters too based on the requirement, such as semicolon ';' tabs '\t' and pipe '|'. Alternatively, to separate the row or record, we need breaks or new lines, and this can be done by using the '\n' delimiter. Furthermore, there is no specific datatype, but you may mentally interpret the data as strings, dates, and numbers, and everything is represented as a string. Also, whenever we see two commas in a row, that means a piece of data is simply missing.

Note: Remember, data is separated by commas because this is a text file, and no datatype is defined with the values.

Different Variations of CSV File

CSV is a plain text file that is widely used for storing and exchanging data, but it has multiple variations based on the distinct delimiters used in the data. The popularity of each variation depends on the needs of the software application involved and the specific use case of the user preference. Let's discuss some of them:

1. TSV known as Tab-Separated value as the name defines is like a CSV file, but it uses tabs to separate the data instead of commas. It is mostly preferred to use when the data itself contains commas, therefore tabs are used to reduce conflict.

EMPLOYEE_ID \t FIRST_NAME \t LAST_NAME \t EMAIL \t PHONE_NUMBER
198 \t Donald \t OConnell \t DOCONNEL \t 650.507.9833

2. DSV known as Delimiter-separated values used to define the custom delimiters other than commas and tabs. It allows the developer to create its own delimiters like pipe '|' or semicolon ';' for separating the data. Furthermore, it is preferred in the case where commas and tabs are not suitable.

EMPLOYEE_ID | FIRST_NAME | LAST_NAME | EMAIL | PHONE_NUMBER 
198 | Donald | OConnell | DOCONNEL | 650.507.9833

Reading CSV Files

Overview of Python's Built-in 'CSV' Module

We can parse the CSV data using the list comprehension, but sometimes data contains multiple values for example, in books CSV files, there can be multiple authors' names separated by commas, so splitting on commas would split those authors' names. Therefore, handling problems like this is one of the many reasons why you should use the CSV module.

Reading Data from a CSV File: Using the 'csv.reader()' Function.

1. Import CSV library First, import the CSV module that provides the functionality of reading and writing the CSV file.

import csv

2. Open the CSV file: Use the open() function to open the file. Then, provide the path of the file with the mode of reading 'r' as we are performing reading operations.

3. Create the CSV Reader Function: After opening the file, use a csv.reader() function to parse the CSV data from the file by passing the file as an argument. Then, save the data into the object name as csv_content.

4. Iterate over Record: Use for loop for performing iteration over all the csv records and then print each record separately.

import csv #importing the csv module
file_path = "Sample-CSV.csv" #provide location of file in the same directory        
with open(file_path, 'r') as file: #here, the file open operation is performed with read mode    
    csv_content = csv.reader(file) #read the file using csvreader function and save it into variable
    for record in csv_content: #iterate through the rows
        print(record) #print the record

Output

Read CSV Files in Python: Using '.readlines()'

The second way to read the CSV file is using the .readlines() function of list comprehension. By using this way, you must strip the string data by removing all the white spaces and then this string will be split as a list as a substring where commas are used and generate the string of the list.

import csv #importing the csv module 
file_path = "Sample-CSV.csv" #provide location of file in the same directory          
with open(file_path, 'r') as file: #here, the file open operation is performed with read mode    
    rows_csv = file.readlines() 
    for row in rows_csv: #iterate through the lines 
        lines = row.strip().split(',') #this will remove the white space and then convert the strings into list 
        print(lines)

Output

Read CSV Files in Python: Using Pandas

Another way of reading the CSV file is using the panda's library. The function read_csv() is used for reading the record from the memory and returns it as a pandas data frame. But first, we have to import the pandas library.

import pandas as pd #importing the pandas library 
file_path = "Sample-CSV.csv" #provide location of file in the same directory        
result = pd.read_csv(file_path) #read the file and return as pandas data frame 
print(result)

Tip: If you are dealing with large CSV file then use Pandas library instead of csv module.

Output

Read CSV Files in Python: Using 'csv.DictReader()'

Now, if you want to access or read the data in the form of a dictionary, then you must import the csv module. The csv module provides various functionalities for reading the data and one of them is the Dictreader() function used to read the data in the form of a dictionary.

import csv #importing the csv module
file_path = "Sample-CSV.csv" #provide location of file in the same directory       
with open(file_path, 'r') as file: #here, the file open operation is performed with read mode   
    record_csv = csv.DictReader(file)
    for row in record_csv:
        print(row)

Output

Customizing CSV Reading Behavior

You can also customize the data of your CSV file by specifying the delimiters, quoting characters, and handling headers. This is done by using the csv module with csv.reader() function. Alternatively, you can also write the delimiter that you want to replace and some quoting characters as well. Also, using the next keyword, you can skip the header while reading the file.

import csv #importing the csv module
file_path = "Sample-CSV.csv" #provide location of file in the same directory        
with open(file_path, 'r', newline = '') as file: #here, the file open operation is performed with read mode   
    record_csv = csv.reader(file, delimiter=';', quotechar='"') #this will create the data with custom settings
    first_row = next(record_csv) #skip the header of the csv file
    for rows in record_csv:
        print(rows)

Output

Writing CSV Files

Writing Data to a CSV file: Using the 'csv.writer()' Function

1. Import CSV library: First, import the CSV module that provides the functionality of reading and writing the CSV file.

import csv

2. Open the CSV File: Use the open() function to open the file. Then, provide the path of the file with the mode of writing 'w' as we are performing writing operations.

3. Create the CSV Writer Function: After opening the file, use a csv.writer() function to parse the CSV data from the file by passing the file as an argument. Then, save the data into the object name as csv_content.

4. Write data to the file: Use the writerow() function of the record_csv object to write the data of each row. Here, the data is as a list or tuples containing the column values of each row. Always specify the header row first, before the other rows using the same function.

import csv #importing the csv module
file_path = "Sample-CSV.csv" #provide location of file in the same directory             
with open(file_path, 'w', newline='') as file: #here, the file open operation is performed with read mode   
    record_csv = csv.writer(file) #this will create the data with custom settings        record_csv.writerow(['EMPLOYEE_ID','FIRST_NAME','LAST_NAME','EMAIL','PHONE_NUMBER','JOB_ID','SALARY'])#write header        record_csv.writerow(['03','David','Paul,*Emails are not allowed*','88886433333','14','15000$'])    
with open(file_path, 'r', newline = '') as file: #here, the file open operation is performed with read mode   
    record_csv = csv.reader(file)
    for rows in record_csv: #iterate to read the data
        print(rows)

Note: Always consider including the header in your CSV file for understanding the data better.

Output

Appending Data to an Existing CSV File

When you use the ‘w’ write mode, all the current data from the file will be truncated. But if you want to add the rows in the existing CSV file without truncating the data, you can attain it by defining the file opening mode as append ‘a’. This way the indicator will be positioned at the end of the file.

import csv #importing the csv module
file_path = "Sample-CSV.csv" #provide location of file in the same directory             
with open(file_path, 'a', newline='') as file: #here, the file open operation is performed with append mode   
    record_csv = csv.writer(file) #this will use the writer function
    record_csv.writerow(['04','Henry','Ferned','*Emails are not allowed*','88898763333','15','10000$'])   
with open(file_path, 'r', newline = '') as file: #here, the file open operation is performed with read mode   
    record_csv = csv.reader(file)
    for rows in record_csv: #iterate to read the data
        print(rows)

Caution: Always back up the CSV file when dealing with the operation to avoid accidentally removing the entire file, so use append mode instead of write mode.

Output

Handling Unicode Data

If your CSV file contains a non-ASCII character, use the UTF-8 encoding scheme while opening the file. It will ensure the proper encoding and formatting of the data within the file.

import csv #importing the csv module
file_path = "Sample-CSV.csv" #provide location of file in the same directory       
with open(file_path, 'a', newline='', encoding='utf-8') as file: #here, the file open operation is performed with append mode   
    record_csv = csv.writer(file) #this will use the writer function
    record_csv.writerow(['06','Renée','Jürgen','jü*Emails are not allowed*','88898763333','15','10000$'])  
with open(file_path, 'r', newline = '', encoding='utf-8') as file: #here, the file open operation is performed with read mode   
    record_csv = csv.reader(file)
    for rows in record_csv: #iterate to read the data
        print(rows)

Output

CSV File Operations and Data Manipulation

Filtering and Transforming CSV data

Filtering involves checking the data based on the specific condition and selecting the rows. Whereas, transforming means changing the values of one or more columns. You can perform both these operations by using the for loop and some conditional statements to access the specific row. Let's say we have a CSV file with employee data, and we want to select only rows where the employee age is above 40 but before that, we want to transform the ages by adding 5 in each employee age.

import csv #importing the csv module
file_path = "Employee_csv.csv" #provide location of file in the same directory       
with open(file_path, 'r', newline='') as file: #here, the file open operation is performed with read mode   
    record_csv = csv.reader(file)
    header = next(record_csv) #skip the header
    age_index = header.index('Age') #get the age column index
    for rows in record_csv: #iterate to read the data
        age = int(rows[age_index]) # convert the string data of age into int
        tform_age = age + 5 # this will transform the age by adding 5 to employee age
        if age > 40: #check the condition for filtering
            print(rows)

Output

Aggregating and Summarizing CSV data

For advanced data manipulation tasks, pandas is the powerful library that provides the leverage to work with structured data like CSV files. To aggregate and summarize the CSV data across multiple rows, such as calculating the average, mean, etc. First, you need to import the pandas library. It has many built-in functions like count(), max(), min(), and mean() for summarizing the data.

import pandas as pd #importing the panda module
file_path = "Employee_csv.csv" #provide location of file in the same directory       
dataframe = pd.read_csv(file_path)
#aggregating techniques
total_age = dataframe['Age'].count()
print("Total Age count:", total_age)
#performing summarizing techniques
age_min = dataframe['Age'].min()
age_max = dataframe['Age'].max()
age_mean = dataframe['Age'].mean()
print("Maximum Age:",age_max)
print("Minimum Age", age_min,)
print("Mean of all Ages:", age_mean)

Output

FAQs Q: How can I create and write data to a file?
A: When you define the mode as 'w' while opening the file, it will create the file if it does not exist and then write the data in it. Q: How do I write to an existing CSV file in Python?
A: You can write to an existing CSV file by using the append mode while opening the file. Q: Can I read and write to a CSV at the same time?
A: Yes, You can perform both operations at the same time in Python by using the 'r+' mode while opening the file.

Wrapping Up

In conclusion, we discussed in detail the working of CSV files, how CSV file is a popular format in the world of programming, and data analysis for exchanging data between different applications. Throughout the article, we saw the valuable function of csv.reader() and csv.writer(), by which you can perform reading and writing operations in CSV files. Furthermore, we looked at some other ways of reading the files. However, these functions show how Python is the best choice for handling the csv file, whether you are performing data manipulation or integration with another system. I hope this guide is helpful for you. If you have any concerns or questions, feel free to give feedback. Thank you for reading this guide. Happy Coding!

Reference

CSV File in Python