Coder Legion - Recent posts tagged logging

How to Add Logging Without Cluttering Your Code

Thu, 27 Aug 2026 06:55:42 +0000

In one of my Python projects, I needed to create fairly detailed logs describing the program's execution. Most of the project is glue code that uses numerous external services and automates processes that were previously performed manually. I do not have full control over them, and my knowledge of how they work is very limited. Because different processes often use the same services, I wrapped those services in classes. The code implementing one of these automations looks roughly like this:

from clients.invoices import InvoiceClient
from clients.orders import OrderClient
from clients.partner import PartnerClient


def export_order(
    order_id: str,
    orders: OrderClient,
    invoices: InvoiceClient,
    partner: PartnerClient,
) -> None:
    order = orders.get_order(order_id)
    attachments = orders.get_attachments(order_id)

    if order.requires_invoice:
        invoice = invoices.create(
            order_id=order_id,
            positions=order.costs,
            payment_term=14,
        )
        attachments.append(invoice)

    partner.send_order(order, attachments)
    orders.set_status(order_id, "exported")

At first, I added logging calls directly to the code, like this:

def export_order(
    order_id: str,
    orders: OrderClient,
    invoices: InvoiceClient,
    partner: PartnerClient,
) -> None:
    logger.info(f"Exporting order {order_id}")
    order = orders.get_order(order_id)
    logger.info(f"Got order {order_id}: {order}")

    logger.info(f"Getting attachments for order {order_id}")
    attachments = orders.get_attachments(order_id)

    if order.requires_invoice:
        logger.info(f"Creating invoice for order {order_id}")
        invoice = invoices.create(
            order_id=order_id,
            positions=order.costs,
            payment_term=14,
        )
        logger.info(f"Created invoice for order {order_id}: {invoice}")
        attachments.append(invoice)

    logger.info(f"Sending order {order_id} to partner")
    partner.send_order(order, attachments)
    logger.info(f"Setting status of order {order_id} to 'exported'")
    orders.set_status(order_id, "exported")

It is easy to see how cluttered the code has become. Add dozens of similar functions throughout the project, and the code becomes difficult to read and maintain.

My first step toward cleaning it up was to extract the repeated pattern into a separate context manager. After all, every stage of the process looked similar: record when it starts, perform the operation, and then record either its success or failure.

from collections.abc import Iterator
from contextlib import contextmanager


@contextmanager
def log_step(message: str) -> Iterator[None]:
    try:
        logger.info(f"Starting: {message}")
        yield
    except Exception:
        logger.exception(f"Error during: {message}")
        raise
    else:
        logger.info(f"Finished: {message}")

Everything before yield runs when execution enters the with block. If the block completes successfully, the else branch runs. If an exception occurs, the context manager records the failure and raises the exception again, so logging does not change the application's behavior.

I could now write the automation code like this:

def export_order(
    order_id: str,
    orders: OrderClient,
    invoices: InvoiceClient,
    partner: PartnerClient,
) -> None:
    with log_step(f"Getting order {order_id}"):
        order = orders.get_order(order_id)
        attachments = orders.get_attachments(order_id)

    if order.requires_invoice:
        with log_step(f"Creating invoice for order {order_id}"):
            invoice = invoices.create(
                order_id=order_id,
                positions=order.costs,
                payment_term=14,
            )
            attachments.append(invoice)

    with log_step(f"Sending order {order_id} to partner"):
        partner.send_order(order, attachments)

    with log_step(f"Setting status of order {order_id} to 'exported'"):
        orders.set_status(order_id, "exported")

This way, I still get an entry for the start, completion, or failure of every stage, but the business logic is no longer interleaved with pairs of nearly identical logger.info calls. The context manager also ensures that every event is recorded in a consistent format.

There is admittedly a little less code, but the result still does not look ideal. In more complex processes, successive with blocks introduce many additional levels of indentation, which can be cumbersome and significantly reduce readability.

At this point, I decided to use decorators for logging and apply them to methods of classes responsible for communicating with external services. The decorator did not need to implement the entire mechanism again—it only needed to execute the function inside the context manager I had already prepared:

from functools import wraps
from typing import Callable, ParamSpec, TypeVar


P = ParamSpec("P")
R = TypeVar("R")


def log_function(message: str):
    def decorator(func: Callable[P, R]) -> Callable[P, R]:
        @wraps(func)
        def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
            with log_step(message):
                return func(*args, **kwargs)

        return wrapper

    return decorator

This decorator also works with instance methods. The wrapped function remains a descriptor, so Python automatically passes the instance as the first argument, self. Meanwhile, @wraps preserves the original method's name, documentation, and other metadata.

I could therefore move responsibility for logging into the client classes:

class OrderClient:
    @log_function("Getting order")
    def get_order(self, order_id: str) -> Order:
        return self._api.get_order(order_id)

    @log_function("Getting order attachments")
    def get_attachments(self, order_id: str) -> list[Attachment]:
        return self._api.get_attachments(order_id)

    @log_function("Setting order status")
    def set_status(self, order_id: str, status: str) -> None:
        self._api.set_status(order_id, status)


class InvoiceClient:
    @log_function("Creating invoice")
    def create(
        self,
        order_id: str,
        positions: list[Cost],
        payment_term: int,
    ) -> Invoice:
        return self._api.create_invoice(
            order_id=order_id,
            positions=positions,
            payment_term=payment_term,
        )


class PartnerClient:
    @log_function("Sending order to partner")
    def send_order(
        self,
        order: Order,
        attachments: list[Attachment],
    ) -> None:
        self._api.send_order(order, attachments)

As a result, the process code returned almost to its original form:

def export_order(
    order_id: str,
    orders: OrderClient,
    invoices: InvoiceClient,
    partner: PartnerClient,
) -> None:
    order = orders.get_order(order_id)
    attachments = orders.get_attachments(order_id)

    if order.requires_invoice:
        invoice = invoices.create(
            order_id=order_id,
            positions=order.costs,
            payment_term=14,
        )
        attachments.append(invoice)

    partner.send_order(order, attachments)
    orders.set_status(order_id, "exported")

Logging now lives at the boundary where the application communicates with external systems, while the function describing the process once again focuses on its logic. Every call to a decorated method still generates an entry for its start, completion, or failure—regardless of how many automations use a given client.

This solution has an important limitation, however: the message passed to the decorator is static. Every order therefore produces exactly the same Getting order entry. While defining the class, I cannot use f"Getting order {order_id}" because the value of order_id does not exist until the method is called.

I wanted to retain the decorator's simple syntax while allowing the message to refer to function arguments:

class OrderClient:
    @log_function("Getting order {order_id}")
    def get_order(self, order_id: str) -> Order:
        return self._api.get_order(order_id)

Simply calling message.format(**kwargs) is not enough. After all, the method can be called as either get_order(order_id="123") or get_order("123"). In the second case, the kwargs dictionary is empty, while args contains the client instance and the order identifier.

Python provides the appropriate tools for this in the inspect module. The signature() function reads the decorated function's signature, while Signature.bind() maps the supplied positional and keyword arguments to its parameter names. The result is a BoundArguments object containing a ready-to-use mapping such as {"self": client, "order_id": "123"}.

The extended version of the decorator looks like this:

from functools import wraps
from inspect import signature
from typing import Callable, ParamSpec, TypeVar


P = ParamSpec("P")
R = TypeVar("R")


def log_function(
    message_template: str,
) -> Callable[[Callable[P, R]], Callable[P, R]]:
    def decorator(func: Callable[P, R]) -> Callable[P, R]:
        func_signature = signature(func)

        @wraps(func)
        def wrapper(*args: P.args, **kwargs: P.kwargs) -> R:
            bound_args = func_signature.bind(*args, **kwargs)
            bound_args.apply_defaults()
            log_context = {
                name: value
                for name, value in bound_args.arguments.items()
                if name not in {"self", "cls"}
            }
            message = message_template.format_map(log_context)

            with log_step(message):
                return func(*args, **kwargs)

        return wrapper

    return decorator

The signature is read only once, when the function is decorated. On every call, bind() builds a mapping of the supplied values, and apply_defaults() fills in the default values of omitted arguments. I remove self and cls from that mapping to prevent the representation of an entire instance from accidentally ending up in the message. Finally, format_map() substitutes the selected values into the template.

The decorators can now use method arguments regardless of how those arguments are passed:

class OrderClient:
    @log_function("Getting order {order_id}")
    def get_order(self, order_id: str) -> Order:
        return self._api.get_order(order_id)

    @log_function("Getting attachments for order {order_id}")
    def get_attachments(self, order_id: str) -> list[Attachment]:
        return self._api.get_attachments(order_id)

    @log_function("Setting order {order_id} status to {status}")
    def set_status(self, order_id: str, status: str) -> None:
        self._api.set_status(order_id, status)


class InvoiceClient:
    @log_function(
        "Creating invoice for order {order_id} "
        "with a {payment_term}-day payment term"
    )
    def create(
        self,
        order_id: str,
        positions: list[Cost],
        payment_term: int,
    ) -> Invoice:
        return self._api.create_invoice(
            order_id=order_id,
            positions=positions,
            payment_term=payment_term,
        )


class PartnerClient:
    @log_function("Sending order {order.id} to partner")
    def send_order(
        self,
        order: Order,
        attachments: list[Attachment],
    ) -> None:
        self._api.send_order(order, attachments)

Format string syntax supports references not only to arguments themselves but also to their attributes. Therefore, {order.id} retrieves only the identifier from an Order object. The resulting logs become much more useful:

Starting: Getting order 123
Finished: Getting order 123
Starting: Creating invoice for order 123 with a 14-day payment term
Finished: Creating invoice for order 123 with a 14-day payment term

I chose not to append every argument automatically. A list of costs, attachments, or the entire order object could generate enormous entries and, worse, expose personal data, tokens, or passwords. The template forces a deliberate choice of the information needed to diagnose a problem—usually an order identifier, the operation name, and the new status. It is also important to remember that a typo in a template field name raises a KeyError, so decorators like this should be covered by tests.

Ultimately, a small abstraction allowed me to retain detailed, consistent logs without letting them obscure the program's actual logic. It does not solve every logging problem, but it has worked very well in code that integrates numerous external services.

What does logging look like in your projects? Do you prefer explicit logger calls, decorators, or another solution? Where do you draw the line between useful context and too much information in your logs?

Effective Logging in Go: Best Practices and Implementation Guide

Wed, 08 Jul 2026 15:58:17 +0000

Introduction

Logging is an essential part of software development, allowing developers to debug applications, monitor performance, and trace issues efficiently. In Go, logging plays a crucial role in building scalable and maintainable applications.

In this blog, you’ll learn about:

The importance of logging in Go
How developers use it for debugging, monitoring, and auditing
Different ways to implement logging in Go
Performance considerations and best practices

By the end, you'll have a clear understanding of how to set up an efficient logging system in your Go applications.

Why Logging Matters in Go?

Logging is more than just printing messages to the console. It serves multiple purposes:

1. Debugging & Troubleshooting

Logs help identify bugs, errors, and unexpected behavior in an application.
Example: If an API request fails, logs can show request parameters, error codes, and stack traces for debugging.

2. Application Monitoring & Performance Tracking

Logs provide insights into execution time, memory usage, and request handling.
Helps in profiling slow functions and optimizing performance.

3. Security & Audit Logs

Tracks user actions (e.g., login attempts, database changes).
Helps detect unauthorized access or suspicious activities.

4. Distributed Systems & Microservices Logging

In microservices, logs are crucial for tracking requests across services.
Helps in tracing failures in distributed systems.

5. Compliance & Legal Requirements

Many industries require logging for compliance with GDPR, HIPAA, or ISO standards.

Logging in Go: Different Approaches

Go provides multiple ways to implement logging, ranging from built-in packages to third-party logging libraries.

1. Using the Standard log Package

Go’s built-in log package is simple but lacks advanced features like structured logging and log levels.

Basic Example:

package main

import (
	"log"
)

func main() {
	log.Println("This is a simple log message")
	log.Fatal("This is a fatal error") // Terminates the program
}

log.Println() → Prints a log message
log.Fatal() → Logs message and exits the program

Limitations:

No log levels (e.g., DEBUG, INFO, WARN, ERROR)
No structured logging
Not ideal for large applications

2. Using log.Logger for More Control

The log.Logger type allows custom configurations like output destinations.

Example: Custom Logger with File Output

package main

import (
	"log"
	"os"
)

func main() {
	file, err := os.OpenFile("app.log", os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
	if err != nil {
		log.Fatal(err)
	}
	logger := log.New(file, "INFO: ", log.Ldate|log.Ltime|log.Lshortfile)
	logger.Println("Application started successfully")
}

Advantages:

✅ Logs to a file instead of the console
✅ Customizable log prefix and timestamp

3.Using log/slog for Structured Logging (Recommended for Modern Go Apps)

Go 1.21 introduced slog, a structured logging package that enhances readability and performance.

Example: Using slog for JSON Logs

package main

import (
	"log/slog"
	"os"
)

func main() {
	logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
	logger.Info("Application started", slog.String("env", "production"), slog.Int("port", 8080))
}

Why slog?

✅ Structured JSON logs for better processing
✅ Log levels (Info, Warn, Error, Debug)
✅ Custom attributes for better context

4. Using Third-Party Logging Libraries (logrus, zap, zerolog)

For production-grade applications, third-party libraries offer performance optimizations and better flexibility.

Example: Logging with logrus

package main

import (
	"github.com/sirupsen/logrus"
)

func main() {
	log := logrus.New()
	log.SetFormatter(&logrus.JSONFormatter{})
	log.WithFields(logrus.Fields{
		"module": "main",
		"status": "running",
	}).Info("Application started")
}

Why logrus?

✅ Formatted logs (JSON, text)
✅ Log levels & hooks for integrations
✅ Performance optimized

Example: Fast Logging with zap (High Performance)

package main

import (
	"go.uber.org/zap"
)

func main() {
	logger, _ := zap.NewProduction()
	defer logger.Sync()

	logger.Info("Application started", zap.String("env", "production"), zap.Int("port", 8080))
}

Why zap?

✅ Extremely fast (zero-allocation logger)
✅ Ideal for high-performance applications

Best Practices for Efficient Logging in Go

1️⃣ Use Log Levels: Always categorize logs into DEBUG, INFO, WARN, ERROR, FATAL.
2️⃣ Avoid Logging Sensitive Data: Exclude passwords, tokens, and personal data.
3️⃣ Use Structured Logging: JSON logs are machine-readable and easy to process.
4️⃣ Rotate Logs: Use tools like logrotate or Lumberjack to prevent log files from growing too large.
5️⃣ Use Context in Logs: Add request IDs, timestamps, and service names for debugging distributed systems.
6️⃣ Optimize for Performance: Use asynchronous logging if high throughput is required.

Additional Tip: Best Practice for Passing a Logger as a Parameter

When passing a logger into a function, it’s best to place it as the last parameter. This makes it optional when possible and ensures that the most critical parameters come first.

For example, instead of this:
func ProcessData(logger *log.Logger, data string) { ... }
Do this:
func ProcessData(data string, logger *log.Logger) { ... }

Conclusion:

Logging is an essential part of maintaining, debugging, and scaling Go applications. Whether you use the built-in log package or a structured logging library like slog, logrus, or zap, choosing the right approach can greatly impact your application's performance and maintainability.

What logging practices do you follow in your Go projects? Do you have a preferred logging library? Drop a comment below—I’d love to hear your thoughts!

How I Design a Logging Strategy for My Application (That Actually Works in Production)

Thu, 26 Mar 2026 15:22:50 +0000

Let’s be honest most developers don’t think about logging until something breaks.

You deploy your app, everything looks fine… until users start complaining. Suddenly, you're SSH-ing into servers, adding random console.log() statements, redeploying, and hoping to catch the issue in real time.

I’ve been there. It’s stressful, inefficient, and completely avoidable.

Over time, I’ve learned that a solid logging strategy isn’t just a “nice to have” it’s a core part of system design. Done right, logging becomes your eyes and ears in production.

In this post, I’ll walk you through how I design a logging strategy for my applications, whether it's a simple API or a distributed system.

Why Logging Matters More Than You Think

Before we dive into the “how,” let’s align on the “why.”

A good logging strategy helps you:

Debug issues faster
Monitor system behavior in real time
Detect security threats
Understand user interactions
Improve system performance

Without proper logs, you're essentially flying blind in production.

Note: Logs are not just for debugging they’re for observability, auditing, and decision making.

Step 1: Define What You Actually Need to Log

One of the biggest mistakes developers make is logging everything or nothing useful.

I start by asking:

What events are critical to my system?
What failures would I need to investigate later?
What actions should be auditable?

Core Categories I Always Log:

1. Application Events

Server start/stop
Background jobs
Scheduled tasks

2. User Actions

Logins/logouts
Transactions
Key interactions

3. Errors & Exceptions

Stack traces
Failed API calls
Validation errors

4. Security Events

Failed authentication attempts
Permission denials
Suspicious activity

Note: If it affects **system behavior, security, or user experience**, it should probably be logged.

Step 2: Use Log Levels Properly (Most People Don’t)

Not all logs are equal. That’s where log levels come in.

Here’s how I structure mine:

DEBUG → Deep technical details (used in development)
INFO → Normal system operations
WARN → Something unexpected, but not breaking
ERROR → Failures that affect functionality
FATAL → Critical issues that may crash the system

Real World Example:

Instead of:

"User login failed"

Do this:

{
  "level": "WARN",
  "message": "User login failed",
  "userId": "12345",
  "reason": "Invalid password",
  "ip": "192.168.1.1"
}

Note: Structured, leveled logs = faster debugging and better filtering.

Step 3: Always Use Structured Logging (Not Plain Text)

If you’re still logging plain strings, you’re making life harder for yourself.

I use structured logging (JSON format) because it allows logs to be:

Searchable
Filterable
Machine readable

Example:

{
  "timestamp": "2026-03-26T12:00:00Z",
  "level": "ERROR",
  "service": "payment-service",
  "message": "Transaction failed",
  "userId": "abc123",
  "transactionId": "txn_456",
  "error": "Insufficient funds"
}

This becomes incredibly powerful when integrated with tools like:

ELK Stack (Elasticsearch, Logstash, Kibana)
Grafana + Loki
Datadog

Note: Think of logs as data, not text.

Step 4: Add Context Logs Without Context Are Useless

A log without context is like a stack trace without a file name.

Every log I write answers:

Where did this happen? (service/module)
Who triggered it? (user ID, session ID)
What request caused it? (request ID, endpoint)

Must Have Context Fields:

requestId (for tracing across services)
userId (if applicable)
serviceName
environment (dev, staging, prod)

Note: Context turns logs into **stories you can follow**.

Step 5: Implement Centralized Logging Early

If your logs live only on your server, you’re already in trouble.

I always centralize logs using tools like:

ELK Stack
Grafana Loki
Cloud logging (AWS CloudWatch, GCP Logging)

Why this matters:

Logs are searchable in one place
You can correlate events across services
You don’t lose logs if a server crashes

Note: In distributed systems (like microservices), centralized logging is non-negotiable.

Step 6: Don’t Log Sensitive Data (Seriously)

This is where many teams make dangerous mistakes.

Never log:

Passwords
API keys
Tokens
Credit card details
Personally identifiable information (PII)

Instead:

Mask sensitive fields
Use hashing where needed
Log references, not raw data

Note: Logs should help you debug not create a security breach.

Step 7: Optimize Log Volume (Avoid Noise)

Too many logs can be just as bad as too few.

I’ve seen systems where:

Logs cost more than infrastructure
Important signals get buried in noise

My approach:

Use DEBUG only in development
Reduce repetitive logs
Sample high frequency events

Tip: Log **what matters**, not everything..

Step 8: Set Up Alerts Based on Logs

Logging without alerting is passive.

I connect logs to alerts so I know when something goes wrong before users complain.

Examples:

Spike in ERROR logs → trigger alert
Multiple failed logins → security alert
Service downtime → immediate notification

Tools:

Grafana Alerts
Datadog Monitors
PagerDuty

Word: Your logs should *talk to you* when something breaks.

Step 9: Correlate Logs with Metrics and Traces

Logs alone are powerful but combined with metrics and tracing, they become unstoppable.

This is called observability.

The Trio:

Logs → What happened
Metrics → How often it happened
Traces → Where it happened

Word: This is how you debug complex systems in minutes instead of hours.

Step 10: Treat Logging as a First Class Feature

This is the mindset shift that changed everything for me.

Logging is not:

An afterthought
A debugging hack
A “we’ll add it later” task

It’s part of your system design.

Every feature I build includes:

Logging points
Error tracking
Context propagation

A Simple Logging Checklist I Use

Before shipping any feature, I ask:

Are critical actions logged?
Are errors descriptive and structured?
Is sensitive data excluded?
Can I trace a request end-to-end?
Are logs centralized and searchable?
Do alerts exist for critical failures?

If not, it’s not ready for production.

Final Thoughts: Logging Is Your Production Superpower

When your system scales, bugs become harder to reproduce, and users become less forgiving.

That’s when your logging strategy either:

Saves you hours…
Or costs you days

Word: A well designed logging system gives you **confidence, visibility, and control**.

Call to Action

If this helped you rethink your logging strategy:

Share it with your team (especially before your next production incident )
Bookmark it as your logging checklist
Drop a comment: What’s the worst debugging experience you’ve had due to poor logging?

And if you're building serious systems, start treating logging like what it is a core engineering discipline, not an afterthought.

From println to Production Logging: Internals and Performance Across Languages and the OS

Sun, 22 Feb 2026 16:05:58 +0000

TL;DR

We all assume a disabled log call costs nothing. It doesn't — the level check is cheap, but any string you constructed before passing it to the logger is already gone, whether the log fires or not.
Every time you see a class name and line number in a log output, something paid for that. In Java, when caller info is enabled, it's a runtime stack walk. In C and Rust, it was resolved at compile time and costs nothing at runtime. Most engineers have never had reason to think about the difference.
logger.info("User {}", user) is not just cleaner syntax. It's a different evaluation model — the string is only built if the log actually fires. "User " + user is evaluated before the logger even sees it.
Async logging feels like a free upgrade. It isn't. It changes what you can trust about your logs when something crashes — and the logs you lose are exactly the ones you needed.
In Rust and C/C++, a disabled log call can be removed from the binary entirely at compile time. In Java and Python, it always exists at runtime, even if it does nothing. The language made this choice.
Go and C logging stacks sit closer to the OS than JVM-based logging stacks. There are fewer layers between the log call and the syscall. That distance has a cost, and it compounds under load.

Thanks to LLMs I could create this: https://github.com/rajkundalia/logger-internals-java

Why Logging Is Not Just Printing

Most of us haven't considered how much happens between our code calling logger.info(...) and that string reaching disk: a level check, a formatter, a handler with its own buffering strategy, a lock or queue depending on sync versus async mode, a syscall into the kernel, and sometimes a second system — syslog, journald — that takes over from there. At scale, that pipeline has real cost. String formatting allocates. Synchronous file writes add latency to every thread that logs. A slow disk creates backpressure that stalls application threads. And in a distributed system where logs are your only audit trail, how that pipeline behaves during a crash is not an edge case — it is a design constraint you either chose or inherited without knowing it. None of that is obvious from a println.

The Pipeline: What a Logger Actually Does

Before pulling any of this apart, it helps to see the whole shape at once:

Application
    ↓
Logger
    ↓
Level Filter
    ↓
Formatter
    ↓
Appender / Handler
    ↓
Operating System
    ↓
Disk / Stream

The application emits a log event with a level, message, and arguments. The logger checks whether the configured threshold allows the event through. If it passes, the formatter constructs the final string — interpolating placeholders, appending timestamps, resolving caller location. The appender or handler takes that string and writes it somewhere: a file, stdout, a socket, a rolling buffer. That write becomes a system call, handing control to the OS, which manages buffering and flush behavior before data actually hits disk. Each stage has cost. Each stage is a place where things can go wrong or get optimized. The rest of this post is about what happens at each one.

Log Level Filtering Internals

Here's something that seems obvious until you think about it: a DEBUG log call in a hot loop, in a production service configured at INFO, runs on every single iteration. It doesn't log anything — but it doesn't disappear either.

The level check itself is cheap. Each level maps to an integer, and the check is a comparison — INFO against whatever the event's level is, early return if it doesn't pass. No formatting, no allocation, no appender invocation. In Logback, higher integers map to higher severity — TRACE is 5000, ERROR is 40000. java.util.logging follows the same direction but uses a different numeric scale and different level names: FINE is 500, SEVERE is 1000. The ordering is not inverted — the scales and names just don't align. Either way, the comparison is fast.

What I found more interesting is where in the pipeline the check actually happens. I assumed there was one gate. There are often several. In Java's SLF4J backed by Logback, the logger checks first — that's the fast path. But appenders can have their own filter chains, meaning an event can clear the logger-level check and still be dropped downstream. This is deliberate and useful: you can send WARN and above to a file, ERROR and above to an alert sink, and everything to stdout, all from the same pipeline. But it means filtering is not a single decision — it's a sequence of decisions, each adding a small amount of overhead to events that reach it.

The real cost isn't the check. It's everything you did before the call site. If you constructed a string before passing it to the logger, that work happened regardless of whether the log fires. Which is exactly why placeholder syntax exists, and why it's not just a style preference.

How a Logger Knows Where It Came From

You've probably never thought about how a log line knows it came from UserService.java:142. It just appears. What's actually happening underneath varies so much across languages that it's worth making explicit — because the cost difference is not small.

In Java, two approaches exist. The older one constructs a Throwable and extracts the stack trace — the JVM walks the call stack and allocates an array of frame objects. The newer approach, StackWalker introduced in Java 9, is lazy and stream-based: you only materialize the frames you actually need. Both are runtime operations with real cost, which is why caller location logging is configurable in most Java frameworks and off by default in many Logback configurations. You can see how this plays out in the reference implementation at https://github.com/rajkundalia/logger-internals-java.

Python captures caller information as part of LogRecord creation, inside _log(), which is only reached after the level check passes. The depth of that inspection — whether stack info is captured, whether additional frame walking occurs — depends on configuration and what the formatter requests. The cost is not paid on every call, but it is paid at record creation time, not at formatting time.

Go makes this explicit. runtime.Caller(skip int) returns the file, line, and function name when you ask for it. It's a runtime operation, but controlled — you call it when you need it, rather than it being woven into every log record automatically.

C and C++ sidestep runtime cost entirely. __FILE__ and __LINE__ are preprocessor macros, expanded at compile time. By the time the binary runs, those values are string literals and integers baked into the executable. No stack walking, no frame introspection, nothing.

Rust takes the same approach through the log crate's macro system. log::info!("...") expands at compile time to include the module path and line number as constants. The binary contains no machinery for discovering caller location — it was resolved before the program ran.

The gap between compile-time resolution and runtime stack walking is the kind of thing that's invisible until you're logging at high volume. C/C++ and Rust pay nothing. Java pays on every logged event where caller info is enabled. Go pays when you ask. Most engineers pick a logging framework without knowing which of these models they've signed up for.

Placeholders vs String Concatenation

These two lines look similar. They are not:

// Eager: string is built before the logger is invoked
logger.info("Connected user: " + user.toString());

// Lazy: string is only built if the level check passes
logger.info("Connected user: {}", user);

In the first version, the JVM evaluates user.toString() and concatenates the string before the logger receives anything. If the level check drops the event — which it will, for any DEBUG or TRACE call in a production service configured at INFO — that allocation and work was wasted. At low log volumes this is invisible. Scattered through hot paths at high throughput, it accumulates.

In the second version, user is passed as an object reference. The logger receives the raw argument. Only if the event clears the level filter does the formatter resolve the placeholder and build the final string. toString() is never called otherwise, and no intermediate string is allocated.

This only matters because of how filtering works — specifically the early return discussed in the filtering section. The two design choices reinforce each other: a cheap level check creates the condition under which deferred string construction delivers its benefit. If logging were unconditional, the distinction wouldn't save anything.

OS Interaction: Where Language Logging Ends and the OS Begins

There's a boundary in every logging pipeline that most application engineers have never had reason to think about: the point where your code hands a string to the OS and stops being in control of what happens next.

When an appender writes to a file, it eventually calls write() — a system call. Everything above that boundary is the language runtime: string formatting, in-memory buffering, lock acquisition. Everything below it is the kernel: its own buffers, filesystem cache, eventual persistence to disk. Crossing that boundary involves a context switch from user space to kernel space. It's not free, and it happens on every unbuffered write.

This is why buffered I/O matters. Rather than one write() per log line, most production logging configurations accumulate output in memory and flush periodically or when the buffer is full. Fewer syscalls, higher throughput. The trade-off: a crash can lose whatever is buffered and not yet flushed. You are always choosing between durability and throughput at that boundary, whether you know it or not.

The OS also offers its own logging infrastructure — syslog on POSIX systems, journald on Linux. These are daemons that accept log messages via a socket and handle buffering, rotation, and persistence outside your application entirely. The boundary shifts: your application writes to a socket, and the daemon takes responsibility for the rest. Structured fields are first-class in journald. Log rotation is not your problem. The cost is IPC (Inter-process communication) overhead — a socket write instead of a local file write.

Go and C-adjacent logging stacks sit naturally close to this boundary. Go's os.File.Write is a thin wrapper over write() with minimal overhead between your code and the syscall. JVM logging absolutely works at scale — but it involves more layers: GC-managed heap allocations, object creation for log events, the JVM's own I/O abstraction. Those layers add up under load.

Synchronous vs Asynchronous Logging

At some point, most engineers configure async logging and move on. Throughput goes up, latency on application threads drops, and nothing seems worse. It feels like a free upgrade.

Here's what actually changed: you no longer have a guarantee that a log line you wrote ever reached disk.

Synchronous logging blocks the calling thread until the write completes. The appender acquires a lock, formats the string, calls write(), releases the lock. Every log call has latency. Under high write volume to a slow disk, this becomes a bottleneck that shows up on every application thread that logs.

Async logging breaks this coupling. Your thread drops an event into a queue and returns immediately. A dedicated logging thread drains the queue, formats events, and writes to the appender. Throughput increases because writes get batched. Thread latency drops to the cost of a queue insertion. This sounds like a strict improvement. It is not.

The queue is bounded. Under sustained high load it fills up. At that point the framework has a decision to make: block the calling thread, drop the event, or expand the queue. Many async logging implementations are configured to drop lower-severity events under pressure unless explicitly set to block — Logback's AsyncAppender, for instance, starts discarding TRACE, DEBUG, and INFO events when the queue reaches 80% capacity by default, while WARN and ERROR are retained. Which means under the conditions where your system is most stressed, in the moments just before something breaks, you may be losing the exact log lines that would have told you why.

The crash case is worse. Events sitting in the queue when the application crashes never reach the appender. Your crash logs — the ones you needed most — may not exist.

Async logging is worth using. It is the right choice in many high-throughput systems. But it is an architectural decision about what you are willing to lose and when. Using it without understanding the failure contract means you have made that trade without knowing it.

Compile-Time vs Runtime Filtering

Something I hadn't considered when I started this: in Java, Python, and Go, a disabled log call still exists in the binary. In Java and Python this is unambiguous — the level check runs on every call. Go's compiler is more aggressive about in-lining and dead code elimination, so the picture is less clear-cut and depends on the logging library and how it's implemented. But in none of these languages can the call be eliminated entirely at compile time the way it can in Rust or C/C++.

Take a TRACE call inside a hot loop in a Java service configured at INFO. On every iteration, the JVM executes an integer comparison and branches. The call is suppressed, but it was visited. At high enough frequency, that cost appears.

In Rust and C/C++, this can be eliminated entirely. A trace!() macro in Rust, conditioned on a compile-time feature flag, is removed by the compiler if tracing is disabled at build time. The instruction does not exist in the binary. There is no branch, no comparison, no overhead of any kind. The code was removed before the program ran.

The trade-off is operational flexibility. A Java application can change its log level at runtime — attach to a running JVM, set the Logback threshold to TRACE, watch debug output appear without a restart. A C binary compiled with TRACE disabled cannot do this. The capability is gone. You traded dynamic observability for zero runtime cost.

Which is right depends on context. A long-running service that needs live level adjustment values the runtime flexibility. A systems program where every cycle matters may prefer compile-time elimination. Most languages make this choice implicitly, as part of how their logging ecosystem is designed. It is worth knowing which choice your language made for you.

Cross-Language Comparison Table

Language	Caller Detection	Filter Type	Async Ecosystem	Compile-time Elimination
Java	StackWalker / Throwable	Runtime	Logback AsyncAppender	No
Go	runtime.Caller	Runtime	zap, zerolog (non-block)	No
Python	currentframe / LogRecord	Runtime	QueueHandler	No
C/C++	FILE, LINE macros	Runtime / Compile	spdlog async mode	Yes (preprocessor)
Rust	Compile-time macro expansion	Runtime / Compile	tracing crate	Yes (feature flags)

Markers in Java/SLF4J — A Brief Callout

Log levels give you one axis for filtering: severity. But severity alone can't answer a question like "show me all security-related events, regardless of level." That's what Markers solve. In SLF4J, a Marker is a named tag attached to a log event — SECURITY, AUDIT, BILLING — that appenders can filter on independently of level. You can route all AUDIT-marked events to a dedicated file while dropping untagged DEBUG events entirely. It's multi-dimensional filtering: level is one axis, marker is another. Other ecosystems approximate this — Go's zap uses structured fields, Python's logging has Filter objects that can inspect arbitrary LogRecord attributes — but SLF4J Markers are one of the cleaner formulations of the idea, and they're underused in codebases that reach for custom log levels when what they actually need is a second axis.

What Surprised Me

We all assume async logging was a performance upgrade with no real downside. It's a trade — lower latency on application threads in exchange for weaker guarantees about what survives a crash. That trade is often worth making. It's not invisible.

I didn't expect caller detection to have such variance across languages. The gap between __FILE__ resolved at compile time and StackWalker walking the call stack at runtime is not a footnote — it's an architectural difference that shows up under load, and most engineers pick a logging framework without knowing which model they've chosen.

Filtering being a pipeline of gates, not a single check, was more nuanced than I expected. I assumed one threshold, one decision. In practice, logger-level filters and appender-level filters can conflict, and events can be dropped at multiple points for different reasons.

The syscall boundary reframed how I think about logging performance. Everything above it is yours — allocations, formatting, buffering. Everything below it is the kernel's. Understanding where that boundary sits, and how often you cross it, makes the buffering trade-offs obvious in a way they weren't before.

Compile-time log elimination felt genuinely strange when I first understood it. The log crate in Rust doesn't just suppress a call when a level is disabled — the code is removed entirely from the binary by the compiler. That's a fundamentally different model from anything Java or Python offer, and it matters in contexts where it matters.

Markers are really interesting. The logs that are easiest to reason about in production are the ones where someone thought carefully about how to filter them — not just what level to assign, but what category they belong to. It's a small design decision that compounds over time.

Resources

These are the rabbit holes that led here.

https://stackoverflow.com/questions/26949503/how-exactly-is-the-logger-a-singleton-and-how-are-different-log-files-created-i — The good old StackOverFlow had a question regarding this.
https://docs.oracle.com/javase/6/docs/technotes/guides/logging/overview.html
https://www.reddit.com/r/java/comments/rdv98z/have_you_ever_wondered_how_javas_logging/ — Down the memory lane.
https://www.loggly.com/ultimate-guide/java-logging-basics/
https://www.marcobehler.com/guides/java-logging
https://signoz.io/guides/java-log/ — table for log level is very good
https://github.com/pinojs/pino — JS Library for logging
https://davidagood.com/logging-in-java/ — Java's logging is crazy
https://github.com/TheTechGranth/thegranths/tree/master/src/main/java/SystemDesign/LoggingFramework — a good basic logger
https://www.youtube.com/watch?v=hOzH7ecc8vg&t=2s — a good explanation for LLD for logger
https://www.youtube.com/live/QV4O9u1N_XU?si=lO4YYFxf-jOk5tTb
https://algomaster.io/learn/system-design/logging — logging best practices

Python Debugging Techniques

Mon, 29 Apr 2024 21:28:41 +0000

While developing a software application in one phase, you need to ensure that it is flawless and error-free. Therefore, to check that the functionality works as expected, we debug the codebase. In SDLC, debugging occurs after successful completion of the software testing phase. In addition, the errors found during the testing phase were fixed by debugging the code. Hence, successful debugging means the complete uncovering of errors and the fixation of bugs, which makes the application robust and efficient. That is why debugging is important in SDLC.

Overview of Common Types of Bugs in Python Programs

There are various types of bugs in the Python program that can be fixed by debugging the code. Common types of bugs you may encounter are syntax, runtime, logical, name, type, index, and filenotfound errors, where each error indicates a different type of problem in the code but can be resolved or fixed by code debugging.

Therefore, in this article, I will cover the details of Python-oriented debugging, with information on debugging tools and the process of debugging your code using different techniques and strategies. Let’s get started.

Understanding Python's Debugging Tools

To debug code effectively, understanding Python debugging tools and setting up the environment is crucial. Here, I will discuss three different tools that can help you debug your Python code efficiently.

1. Introduction to Python's Built-in Debugging Tools

The pdb module is a Python built-in debugger that is part of the Python standard library. It provides a step-by-step method for debugging the code by importing the library. This is a powerful module that allows the Python developer to pause the execution of the code at a breakpoint. Breakpoints are the setup of specific points and then evaluate the values of variables and logic of the conditions or expression. Furthermore, it has a very straightforward way of debugging, which is invoked using the command line interface. It is a perfect tool for simple tasks and for those debuggers who are comfortable working with cmd.

import pdb #import
def checker(a, b):
    pdb.set_trace() #debugging starts here
    if b > a:
        print("B is greater than A")
    elif b == a:
        print("B is equal to A")
    else:
        print("A is smaller than B")
a = 33
b = 200
result = checker(a,b)

2. Exploring Third-Party Debugging Tools and IDE Integrations

The second popular method for debugging Python code is to set up an integrated debugging environment (IDE). These debugging IDEs will ensure an effective way of writing, running, and debugging the code. Furthermore, these IDS provide you with booting features of breakpoints, variable inspection, execution workflow (with step-into, step-out, and step-over), watch expression, stack trace, and advanced code editing features that make debugging much easier. Several debugging IDEs are PyCharm, Spider, Visual Studio Code with Python extension, PyDev (plugin for Eclipse), JupyterLab, Wing IDE, etc., and some third-party tools, including PySnooper and pdb++.

Note: Breakpoints are the markers set by a programmer to temporarily stop the execution for debugging purposes.
Variable inspection is checking the current value of a variable while a program is paused.
A stack trace is a report of the active stack frames at a certain point in time during program execution, typically captured when an exception occurs.

3. Configuring Debugging Environments for Efficient Debugging Workflows

Use Integrated Development Environments (IDEs): IDEs like PyCharm, Visual Studio Code, or Jupyter Notebook offer debugging tools that can streamline the debugging process.
Set Breakpoints: Place breakpoints in your code to pause execution at specific points and inspect variables, helping to identify the issues.
Step Through Code: Utilize features like stepping into, over, or out of functions to understand code execution flow better.
Inspect Variables: Utilize IDE features to examine variable values in real-time, aiding in the detection of unexpected behavior.
Use Debugging Libraries: Libraries like pdb or ipdb provide additional debugging functionalities to enhance your debugging workflow.
Logging: Incorporate logging statements in your code to track program flow and variable values, providing insights into the program's behavior.
Unit Tests: Write unit tests to catch bugs early in the development phase, making debugging more efficient.

Using Print Statements for Debugging

1. Leveraging Print Statements for Basic Debugging

Debuggers are the preferred way of debugging your programs, but if you don't have access or, in some cases, your program is of a smaller level, then leveraging the print() function for debugging is an efficient way. You can tackle many bugs with just the print statement at the right place where you think an exception or abnormal behavior might occur. This is one of the simplest and most basic ways to debug your program before setting up the entire debugging environment on your machine. Therefore, the print() function will help you debug your code and fix bugs at an early stage, which is the priority of developers.

2. Tips for Effective Use of Print() to Trace Program Flow

Print statements are versatile tool for every programming language. It not only helps you print your code output but is also used for debugging. In Python, there are some tips for the effective use of the print statement that help you trace the program flow:

Use the print() function to display the output result, message, and outcome of conditional and logical statements and other logging information of code that helps you understand the program flow.
For effective debugging, strategically place the print() function at the right place in your code where an unexpected behavior might occur to trace the execution of the program. Then, evaluate the output to identify where the code is breaking.
Utilized print() functions in conditional statements to verify whether the logic of your program generates correct outcomes or not.

3. Printing Variable Values and Debugging Output Formatting

Suppose we have a function that calculates the average of a list of integers. However, the output is not what we expected, therefore we need to debug the code using the print statements. This is how we can use the print statement:

def average(nums): #function for calculating the average of a list
    sum = 0
    for list in nums:
        sum += list
    print(f"Sum of List", sum) #to check that the sum is not zero 
    print(f"Length of List", len(nums)) # to check that the list size is not zero
    return sum / len(nums)

number = [5,3,8,9,2,7]
#the format for print the output for effective readability
print(f"The Average of {number} is:", average(number))

In this example, we have a function called average() that takes the list as a parameter and then returns the average. We created a list to test the function. Next, we use the print statement at specific points, displaying the values of the sum alongside the length of the list. This allows us to evaluate the output and quickly verify the point that deviates from the program's normal flow. In addition, f-strings are used in the program for effective readability and to follow the debugging output formatting.

Tip: Always start debugging small sections of your code once at a time to isolate issues and make troubleshooting more manageable.

Debugging with Logging

1. Introduction to Python's Logging Module

For small programs, the print() function is completely fine but cannot be the best practice, when dealing with large Python programs that have complicated logic. Therefore, you should have a persistent log that stores all the data of your program such as information, warnings, exceptions, and errors. Python has introduced a built-in module for logging. You need to import the library into your program, and no further installation is required.

2. Configuring Logging Levels and Formatting for Debugging

In Python, the logging module consists of 5 different levels. Each level defines the category of the log message based on the type of severity you encountered during the program execution. These Five levels are as follows:

Debug: Used in the program development phase to diagnose code issues.
Info: Used to provide a successful program case in which there is no error in the program.
Warning: Used to indicate the warning at runtime that might occur in the future. e.g. removal of module older version.
Error: Used to print all errors and bugs encountered during program execution and cause the program to deviate from its normal flow.
Critical: A situation or an error in the program that violates the requirements and crashes the application suddenly.

Enough for logging levels? Now, understand the debugging format when you use the logging module. First, we import the logging library.

import logging #import logging library

Then, set the logging level to debug, as we are using it for debugging purposes. Here, the setting level refers to the level at which we want to start logging. So, write debug to log all the info and start from there.

logging.basicConfig(level=logging.DEBUG, format = '%(name)s - %(levelname)s - %(message)s') # create and config the logger

Then, you can use all the levels for printing the actual messages like this.

logging.debug("A Debug Logging Message")  
logging.warning("A Warning Logging Message")  
logging.error("An Error Logging Message")

That is how you set up the log messages at the debug level, and I hope you understand the basic formatting of debugging using logging.

3. Using Logging to Trace Program Execution and Capture Errors

Here, we will understand how to trace the program execution and capture the error message using the logging module with the help of a coding example. Suppose we have a program with a division function, but encounter an error of zero division exception. For this, we need to debug the code to understand the complete log of the program execution including the timestamp and the line of code where the error occurs using the logging module instead of the print statement.

import logging #import logging library
#configure logging with debug mode
logging.basicConfig(filename = 'file.log', filemode='w', level=logging.DEBUG, format = '%(name)s - %(asctime)s - %(levelname)s - %(message)s') # create and config the loger 

#function that can contain an error
def division(num1, num2):
    try:
        num3 = num1 / num2
    except ZeroDivisionError as e:
        logging.error("Error Occur %s", e)
    else:
        logging.debug("Result is %s", num3)

#program flow
logging.info("Info Message: Program Started")
division(8,2)
division(8,0) 
logging.info("Execution Ends ")

Note: Understand the error message carefully because it will guide you to the root cause of the error occurrence.

In the output, it clearly shows that the first function call gives the correct output, but when the second time we call the same function with different values the error occurs, and all the log info of the program execution is stored in the file. So, this is how you can trace your program execution and configure all the relevant logging information for debugging and analysis.

Interactive Debugging with Debuggers

1. Using Interactive Debuggers to Inspect Code Execution

Now, let's move towards interactive debuggers, by which you can debug your code and find the result upon each execution. Here, I use VS code with the Python extension for interactive debugging. In addition, I will discuss a step-by-step guide to debug the simple Python code for a better understanding of VS code.

1-1. Open Interactive Debugger

Begin by opening VS code on your computer, then move toward your Python code directory. After that, open your code file. The content inside my code file is:

even_numbers = [2,4,6,8,10] #list of even number
square = (num ** 2 for num in even_numbers) #generator expression in single line
for numbers in square:
    print(numbers)

1-2. Setting Breakpoints

In your Python code, set the breakpoint by clicking on the left margin of the line numbers from where you want debugging to start. Here, I am setting the three breakpoints.

Caution: Be cautious while using breakpoints because sometimes using too many breakpoints or placing them at the wrong location can cause the debugging process confusing.

1-3. Start Debugging Process

Then, click on the "Run and Debug" icon in the sidebar, and choose the appropriate Python interpreter for debugging. This will start the debugging process and will pause the execution at every breakpoint you set.

1-4. Stepping Through Code

You have four different debugging controls, that will appear when the debugging process starts. These are step-over, step-into, continue, and pause. You can use any one of them, for executing the current line and shifting towards the next, you have to click on step over control. To step into control, you can move into the function you called on the current line. Whereas pause and continue are straightforward, so there is no need to explain.

2. Examining Variable Values and Stack Traces During Debugging Sessions

You can also examine the variable values and stack traces in the debug sidebar while debugging your Python code. This will show you all the variables in the current scope with their values. Whereas the stack traces will show the chain of the function calls and lead you to the root that causes the error.

FAQs

Q: Why is debugging necessary?
A: It helps you to find errors or bugs at an early stage and makes the testing phase easier.

Q: How can I start debugging?
A: After writing the code, you can use the print() function or logging or can set the breakpoints for interactive debugging.

Wrapping Up

Finally, we discussed that debugging is an important part of the software development phase. Mostly, developers who work on small-scale projects just ignore debugging and shift toward fixing the bugs without finding which part caused the error. Therefore, adding debugging to your programming part, and also exploring new debugging tools can enhance your debugging process by effectively reducing time consumption. We learned different strategies that include simple print() statements to logging modules, but you can incorporate a debugger as it is the preferred choice of many developers and industries. I hope you like this guide. If you have any questions, then feel free to give feedback. Thank you for following this guide. Happy Coding!