Coder Legion - Recent posts tagged pytorch

AWS PyTorch: Build, Train, and Deploy Deep Learning Models in the Cloud

Wed, 21 Oct 2026 06:46:29 +0000

title: "AWS PyTorch: Build, Train, and Deploy Deep Learning Models in the Cloud"
published: true
description: "Learn how to build a PyTorch image classifier, store model artifacts in Amazon S3, and move toward managed machine learning training and deployment with Amazon SageMaker AI."

tags: aws, pytorch, machinelearning, python

PyTorch makes it easier to experiment with neural networks, computer vision, natural language processing, and deep learning.

But building a model is only one part of a machine learning project.

When it's time to move from experimentation to production, you need reliable compute, data storage, repeatable training, model artifacts, deployment, monitoring, and cost controls.

This is where AWS can help.

By combining PyTorch, Amazon SageMaker AI, Amazon S3, and Amazon CloudWatch, developers can build machine learning workflows that move from local experiments toward scalable cloud deployments.

In this article, we'll build a simple image classifier and explore how to extend the project into an AWS-based machine learning pipeline.

1. What Is PyTorch on AWS?

PyTorch is an open-source deep learning framework that provides tensor operations, automatic differentiation, GPU acceleration, and tools for creating neural networks.

AWS provides several services that can support PyTorch workloads:

Amazon SageMaker AI: Run managed training jobs and deploy machine learning models.
Amazon S3: Store datasets, model artifacts, and training outputs.
Amazon EC2: Run PyTorch on virtual machines, including suitable GPU instances.
AWS Deep Learning Containers: Use supported, preconfigured machine learning environments.
Amazon CloudWatch: Monitor supported metrics, logs, and application behavior.
AWS IAM: Control access to training data, infrastructure, and model artifacts.

You don't need all these services for every project. Start with what your workload requires and add complexity when it provides a clear benefit.

2. Understand the Architecture

A typical workflow looks like this:

        Training Dataset
               |
               v
         Amazon S3
               |
               v
      Amazon SageMaker AI
          Training Job
               |
               v
         PyTorch Model
               |
               v
       Model Artifacts in S3
               |
               v
       Deployment / Inference
               |
               v
       Application Predictions

Amazon S3 provides durable storage for datasets and artifacts. SageMaker AI can run managed training jobs, while a suitable inference service can expose the trained model to applications.

CloudWatch and related AWS monitoring capabilities help you investigate failures, latency, and resource usage.

For small experiments, you can train locally first. For larger workloads, managed training can reduce infrastructure-management overhead.

3. Hands-On Project: Build a Digit Classifier

We'll train a small neural network on the MNIST handwritten-digit dataset.

Our model will learn to classify grayscale images into one of ten categories, representing the digits from 0 to 9.

Prerequisites

You'll need:

Python 3.
Basic Python knowledge.
An internet connection to download the dataset.
An AWS account only when you move to the cloud steps.

The first part runs locally and does not require AWS resources.

Step 1: Create a project environment

Create a project directory and enter it:

mkdir aws-pytorch-demo
cd aws-pytorch-demo
python -m venv .venv

Activate the virtual environment on Windows:

.venv\Scripts\Activate.ps1

On Linux or macOS:

source .venv/bin/activate

Install the dependencies:

pip install torch torchvision

For GPU training, install a PyTorch build compatible with your GPU and CUDA configuration. Follow the official installation instructions: https://pytorch.org/get-started/locally/.

Step 2: Create the training script

Create train.py:

import torch
from torch import nn
from torch.utils.data import DataLoader
from torchvision import datasets, transforms

device = torch.device(
    "cuda" if torch.cuda.is_available() else "cpu"
)

print(f"Training device: {device}")

transform = transforms.ToTensor()

train_data = datasets.MNIST(
    root="data",
    train=True,
    download=True,
    transform=transform,
)

test_data = datasets.MNIST(
    root="data",
    train=False,
    download=True,
    transform=transform,
)

train_loader = DataLoader(
    train_data,
    batch_size=64,
    shuffle=True,
)

test_loader = DataLoader(
    test_data,
    batch_size=256,
    shuffle=False,
)


class DigitClassifier(nn.Module):
    def __init__(self):
        super().__init__()

        self.network = nn.Sequential(
            nn.Flatten(),
            nn.Linear(28 * 28, 128),
            nn.ReLU(),
            nn.Linear(128, 10),
        )

    def forward(self, x):
        return self.network(x)


model = DigitClassifier().to(device)

criterion = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(
    model.parameters(),
    lr=0.001,
)

# Training loop
for epoch in range(3):
    model.train()
    total_loss = 0.0

    for images, labels in train_loader:
        images = images.to(device)
        labels = labels.to(device)

        optimizer.zero_grad()

        outputs = model(images)
        loss = criterion(outputs, labels)

        loss.backward()
        optimizer.step()

        total_loss += loss.item()

    average_loss = total_loss / len(train_loader)

    print(
        f"Epoch {epoch + 1}/3, "
        f"Loss: {average_loss:.4f}"
    )

# Evaluation
model.eval()
correct = 0
total = 0

with torch.no_grad():
    for images, labels in test_loader:
        images = images.to(device)
        labels = labels.to(device)

        outputs = model(images)
        predictions = outputs.argmax(dim=1)

        correct += (predictions == labels).sum().item()
        total += labels.size(0)

accuracy = 100 * correct / total
print(f"Test accuracy: {accuracy:.2f}%")

# Save model parameters
torch.save(
    model.state_dict(),
    "digit_classifier.pth",
)

print("Saved digit_classifier.pth")

Step 3: Train the model

Run:

python train.py

The script downloads MNIST, trains the neural network, evaluates it against the test dataset, and saves the learned parameters.

Your output should include:

Training device: cpu
Epoch 1/3, Loss: ...
Epoch 2/3, Loss: ...
Epoch 3/3, Loss: ...
Test accuracy: ...
Saved digit_classifier.pth

The loss and accuracy values depend on the training run. If a compatible GPU is available, the script can use it automatically.

Step 4: Make a prediction

Create predict.py:

import torch
from torch import nn
from torchvision import datasets, transforms


class DigitClassifier(nn.Module):
    def __init__(self):
        super().__init__()

        self.network = nn.Sequential(
            nn.Flatten(),
            nn.Linear(28 * 28, 128),
            nn.ReLU(),
            nn.Linear(128, 10),
        )

    def forward(self, x):
        return self.network(x)


device = torch.device(
    "cuda" if torch.cuda.is_available() else "cpu"
)

model = DigitClassifier().to(device)

state = torch.load(
    "digit_classifier.pth",
    map_location=device,
    weights_only=True,
)

model.load_state_dict(state)
model.eval()

dataset = datasets.MNIST(
    root="data",
    train=False,
    download=True,
    transform=transforms.ToTensor(),
)

image, actual_label = dataset[0]

with torch.no_grad():
    logits = model(image.unsqueeze(0).to(device))
    predicted_label = logits.argmax(dim=1).item()

print(f"Predicted digit: {predicted_label}")
print(f"Actual digit: {actual_label}")

Run the prediction:

python predict.py

Example output:

Predicted digit: 7
Actual digit: 7

These values are illustrative. The actual result depends on your trained model.

You now have a basic end-to-end deep learning workflow: train, evaluate, save, reload, and predict.

4. Move the Project to AWS

Once the model works locally, you can move the workflow to AWS.

Step 1: Create an S3 bucket

Choose a globally unique bucket name:

aws s3 mb s3://YOUR-UNIQUE-PYTORCH-BUCKET \
  --region us-east-1

Replace the placeholder with your actual bucket name.

Upload the training script:

aws s3 cp train.py \
  s3://YOUR-UNIQUE-PYTORCH-BUCKET/code/train.py

For this example, the script downloads MNIST directly. A production pipeline should manage dataset preparation, storage, and versioning more explicitly.

Step 2: Run training with SageMaker AI

Amazon SageMaker AI supports PyTorch training jobs using supported framework environments and training configurations.

At a high level, you need to:

Configure an execution role with appropriate permissions.
Prepare your training script and dependencies.
Select a supported PyTorch environment.
Configure the training input and output locations.
Choose an appropriate instance type.
Set runtime limits and launch the job.
Review the training logs and saved artifacts.

Use the current official guide for the SDK version you install:

https://docs.aws.amazon.com/sagemaker/latest/dg/pytorch.html

SageMaker SDK interfaces evolve, so use examples matching your installed SDK version rather than mixing older estimator APIs with newer SDK workflows.

Step 3: Store model artifacts in S3

For a locally trained model, you can upload the saved parameters:

aws s3 cp digit_classifier.pth \
  s3://YOUR-UNIQUE-PYTORCH-BUCKET/models/digit_classifier.pth

For a managed training job, configure the output location so SageMaker AI saves the model artifacts automatically.

Keep a record of the dataset version, source-code version, training parameters, framework version, and evaluation metrics. This makes it easier to reproduce and audit experiments.

5. Deploy the Model for Inference

Training and inference are separate stages.

For an application to request predictions, you need a serving interface that loads the model, validates input, executes inference, and returns a response.

With Amazon SageMaker AI, you can deploy supported PyTorch models to a managed endpoint. The required model packaging and inference code depend on the selected deployment approach.

A typical inference flow is:

Load the approved model artifact.
Convert the input into the tensor format expected by the model.
Run the model in evaluation mode.
Convert the output into a response your application can consume.
Log relevant errors and operational metrics.

For the MNIST model, input should represent a grayscale image with dimensions of 28 × 28 pixels. Validate the request before running inference.

For workloads that don't need immediate responses, consider batch inference instead of keeping an online endpoint available continuously.

6. Production Best Practices

A model that works locally isn't automatically production-ready.

Cost optimization

Start with CPU training for small models when practical.
Use GPU instances when benchmarks show a meaningful benefit.
Set job time limits.
Stop endpoints and compute resources when they are no longer required.
Evaluate batch inference for offline workloads.
Include storage, logging, data transfer, and supporting services in cost estimates.

Security

Apply least-privilege IAM policies.
Keep S3 buckets private.
Encrypt sensitive datasets and artifacts.
Avoid embedding credentials in source code.
Restrict access to training data and deployed models.

Reproducibility

Track dataset versions, dependencies, training parameters, random seeds, and model artifacts. Store code in version control and preserve evaluation results for each model version.

Monitoring

Monitor training failures, compute utilization, inference latency, and application errors using supported AWS monitoring capabilities.

Also evaluate model quality over time. A service can remain operational while its predictions become less accurate because real-world data has changed.

Model evaluation

Don't rely on training loss alone.

For classification tasks, consider precision, recall, F1 score, confusion matrices, and performance on representative held-out data. Select metrics based on the real business problem.

7. Where Can You Use PyTorch on AWS?

Common applications include:

Computer vision: Image classification and defect detection.
Natural language processing: Text classification and domain-specific model fine-tuning.
Recommendation systems: Ranking and personalization.
Forecasting: Demand prediction and time-series analysis.
Generative AI: Training or fine-tuning compatible neural network models.

For large-scale distributed training, additional planning is needed for GPU memory, network communication, checkpointing, and data loading.

Final Thoughts

PyTorch provides the flexibility to build deep learning models, while AWS offers several ways to run training workloads, store model artifacts, and deploy predictions.

Start with a small experiment, validate the results, and then introduce cloud infrastructure where it provides measurable value.

The real goal is not just to train a model in the cloud. It's to create a secure, reproducible, observable machine learning workflow that can grow with your application.

Official Resources

AWS #PyTorch #MachineLearning #DeepLearning #AmazonSageMaker

How to Build a Deep learning model to detect Ships

Mon, 19 Oct 2026 16:30:01 +0000

For anyone studying practical computer vision and deep learning deployment:
A frequent challenge in applied vision projects is balancing spatial accuracy against computational overhead during training. Selecting an architecture variant like YOLOv8l provides the parameter capacity necessary to resolve multi-scale objects in clustered environments, but success depends entirely on proper configuration and validation monitoring. This tutorial breaks down the mechanics of training a custom detector from scratch, detailing how loss convergence curves and validation checkpoints guide the selection of deployable weights.

The progression moves methodically from environment initialization to final model inference. It covers dataset re-indexing, parsing raw coordinate tags into standardized label files, executing supervised training runs while monitoring loss metrics, and implementing an OpenCV inference script to handle bounding box coordinates, class labels, and confidence thresholds on unseen imagery.

The Resources:

Reading on Medium: https://medium.com/object-detection-tutorials/yolov8-custom-object-detection-full-code-walkthrough-23fa476f834e

Detailed written explanation and source code: https://eranfeit.net/yolov8-custom-object-detection-full-code-walkthrough/

Join my Newsletter : https://eranfeit.net/advance-your-skills-in-computer-vision-ai/

Deep-dive video walkthrough: https://youtu.be/zAyPboTjcAA
This material is published solely for educational purposes. Feel free to leave constructive feedback or questions about threshold tuning and batch performance below.

GPU-Accelerating MSCRED with CUDA, im2col, GEMM, and a Custom PyTorch Extension

Fri, 11 Sep 2026 18:07:01 +0000

Introduction

During my M.Tech in Data Science and Artificial Intelligence at IIT Bhilai (2021–2023), I conducted thesis research on multivariate time-series anomaly detection and GPU acceleration using industrial process signals.

My thesis, titled “Anomaly Detection of Multivariate Time Series Data and Acceleration Using GPUs,” explored an MSCRED—Multi-Scale Convolutional Recurrent Encoder-Decoder—and ConvLSTM-based pipeline for modelling relationships between signals and detecting abnormal temporal patterns.

Although the model produced useful anomaly-detection results, its convolution-heavy encoder created a significant CPU bottleneck. I therefore implemented GPU-accelerated operations using CUDA, transformed convolution into an im2col plus GEMM workflow, investigated global-memory and shared-memory implementations, and integrated the accelerated operations with PyTorch.

In the measured workload, encoder runtime decreased from 300 seconds on the CPU to 23.664 seconds on the GPU, representing an approximately 12.68× encoder speed-up.

Research attribution: MSCRED and ConvLSTM are established architectures introduced by the researchers cited in the References section. This article describes how I used, adapted, profiled, and accelerated selected operations as part of my M.Tech thesis.

Data confidentiality: The original research involved industrial process data. This article discusses the architecture, optimisation methodology, and aggregate benchmark results without publishing proprietary signals, internal process details, confidential datasets, or restricted source code.

The Anomaly-Detection Problem

Industrial systems often generate many related sensor and process-control signals at the same time. A single signal may appear normal when viewed independently, while the relationship between several signals may indicate an abnormal operating condition.

This makes multivariate time-series anomaly detection more difficult than analysing one signal at a time.

The model must learn:

Relationships between multiple signals
Changes across different time windows
Temporal dependencies between successive observations
Deviations from normal operating patterns
Reconstruction differences that may indicate anomalies

The goal of the research was to detect unusual multivariate patterns while preserving the temporal and spatial relationships present in the data.

MSCRED and ConvLSTM Architecture

MSCRED models relationships between multiple time-series signals by constructing signature matrices over different temporal windows.

At a high level, the pipeline contains:

Signature-matrix generation from multivariate signals
A convolutional encoder for spatial feature extraction
ConvLSTM layers for temporal modelling
A decoder that reconstructs expected signal relationships
Reconstruction-error analysis for anomaly scoring

The encoder processes the signature matrices through convolutional operations, while ConvLSTM captures how the encoded representations evolve over time.

Multivariate signals
        ↓
Signature matrices
        ↓
CNN encoder
        ↓
ConvLSTM
        ↓
Decoder
        ↓
Reconstruction error
        ↓
Anomaly score

Why Signature Matrices Are Useful

A signature matrix represents pairwise relationships between signals over a selected time interval.

By generating these matrices at multiple scales, the model can observe both short-term and longer-term relationships. This helps identify anomalies that may not be visible in an individual signal.

Role of the Encoder

The convolutional encoder extracts spatial patterns from the signature matrices. It reduces the raw matrix representation into compact feature maps that capture important relationships between signals.

Role of ConvLSTM

ConvLSTM models how those spatial representations change over time.

Unlike a standard fully connected LSTM, ConvLSTM preserves the spatial structure of feature maps while learning temporal dependencies.

Role of the Decoder

The decoder reconstructs the expected signature matrices.

When the reconstructed output differs significantly from the observed input, the reconstruction error can indicate an anomalous pattern.

Anomaly-Detection Evaluation

The purpose of the MSCRED and ConvLSTM pipeline was to detect abnormal relationships and temporal behaviour across multiple industrial process signals.

The model learned normal multivariate patterns by reconstructing signature matrices. During evaluation, the difference between the observed signature matrices and the reconstructed outputs was used to calculate a reconstruction error.

Observed signature matrix
            ↓
MSCRED reconstruction
            ↓
Reconstruction error
            ↓
Thresholding
            ↓
Normal or anomalous

A larger reconstruction error indicated that the observed relationships differed from patterns learned during normal operation and could therefore represent an anomaly.

The thesis evaluated anomaly-detection quality using the F1 score. Under the reported experimental setup, the best recorded F1 score was 0.941.

This result represents model-quality evaluation and should be interpreted separately from the CUDA runtime measurements presented later. The GPU memory optimisations were designed to reduce computation time, not to improve predictive accuracy.

Where the Performance Bottleneck Appeared

The encoder repeatedly applied convolutional operations to signature matrices.

On the CPU, these operations involved substantial nested computation across:

Input channels
Output filters
Spatial positions
Kernel dimensions
Batch elements

This made the encoder a strong candidate for GPU acceleration because many convolution calculations could be performed in parallel.

The objective was not to redesign the anomaly-detection model. Instead, it was to preserve its expected computational behaviour while accelerating selected expensive operations.

Transforming Conv2D into `im2col` and GEMM

A convolution repeatedly applies a kernel to overlapping regions of an input feature map.

The im2col, or image-to-column, transformation rearranges those local regions into columns of a matrix.

After this transformation, convolution can be expressed as a general matrix multiplication:

Output matrix = Weight matrix × im2col(Input)

This conversion is useful because matrix multiplication is highly parallel and can be executed efficiently on a GPU.

Conceptual Transformation

Input feature map
        ↓
Extract overlapping kernel windows
        ↓
Arrange windows as matrix columns
        ↓
Multiply by reshaped filter matrix
        ↓
Reshape result into output feature map

Why Use GEMM?

GEMM stands for General Matrix-Matrix Multiplication.

Modern GPU architectures are designed to perform large numbers of multiply-and-accumulate operations in parallel. Expressing convolution as GEMM allows the workload to use that parallelism.

Trade-Off

im2col introduces additional memory usage because overlapping input regions are copied into an intermediate matrix.

The method is most effective when the computational benefit of parallel matrix multiplication outweighs the cost of creating and storing the intermediate representation.

CUDA Kernel Design

The CUDA implementation divided the computation across many GPU threads.

Each thread was responsible for part of the output, while thread blocks grouped related work.

Important kernel-design considerations included:

Mapping output elements to threads
Selecting thread-block dimensions
Coalescing memory access where possible
Reducing unnecessary global-memory operations
Avoiding excessive synchronisation
Preserving correct tensor indexing
Managing intermediate buffers efficiently

The implementation required careful validation because incorrect indexing in GPU code can produce plausible-looking but incorrect outputs.

Global Memory and Shared Memory

The first GPU implementation relied primarily on global memory.

Global memory provides access to the complete input, but its latency is considerably higher than that of on-chip shared memory.

The optimised implementation staged reusable tiles inside shared memory. Threads within the same block could then reuse nearby values without repeatedly loading them from global memory.

The shared-memory optimisation was intended to:

Reduce repeated global-memory reads
Improve data locality
Increase data reuse within each thread block
Reduce memory-related kernel overhead
Improve matrix-multiplication efficiency

Conceptual Data Flow

Global memory
      ↓
Load reusable tile
      ↓
Shared memory
      ↓
Threads perform repeated calculations
      ↓
Write result to global memory

Shared memory is limited in capacity, so tile dimensions and thread-block organisation must be selected carefully.

Poorly selected configurations can:

Reduce occupancy
Increase synchronisation overhead
Create bank conflicts
Use too much shared memory per block
Limit the number of active thread blocks

Integrating the CUDA Implementation with PyTorch

To use the accelerated operations inside the existing model pipeline, I connected the CUDA implementation to PyTorch through a custom extension.

The integration enabled PyTorch tensors to be passed to the CUDA implementation and the computed results to be returned to the model pipeline.

Important integration concerns included:

Tensor dimensions
Device placement
Data types
Contiguous-memory assumptions
Compilation and linking
Error handling
CPU-versus-GPU output comparison

Correctness validation was essential. A faster kernel is not useful if it changes tensor shapes, produces incorrect indexing, introduces unacceptable numerical differences, or behaves inconsistently for the tested configurations.

The custom extension made it possible to retain PyTorch for model development while using lower-level CUDA operations for selected performance-critical components.

Experimental Setup

The benchmark used the following environment:

GPU: NVIDIA RTX A6000, 48 GB
CPU: 16-core Intel Xeon Gold 5218
System memory: 64 GB
Data points: 36,000
Number of features: 30
Batch size: 128
CPU threads: 32

Timing Methodology

The timings reported here are measurements from my thesis experiments using the specified hardware and workload. The CPU and GPU implementations used the same input dimensions and batch configuration.

The results should be treated as comparative experimental measurements rather than universal library benchmarks. Performance may vary with implementation details, CUDA configuration, tensor dimensions, data-transfer costs, compiler settings, and hardware.

Performance Benchmark Results

Component	CPU	GPU: global memory	GPU: shared memory
Encoder	300 s	27.492 s	23.664 s
ConvLSTM	1,200 s	909.208 s	847.260 s
Total runtime	2,700 s	1,594 s	1,481 s

Results at a Glance

Anomaly-detection quality: best recorded F1 score of 0.941
Encoder acceleration: approximately 12.68×
End-to-end acceleration: approximately 1.82×
Total runtime reduction: approximately 45.1%

The shared-memory implementation reduced encoder runtime from 300 seconds to 23.664 seconds, producing an approximately 12.68× encoder speed-up.

For the complete measured pipeline, runtime decreased from 2,700 seconds to 1,481 seconds. The total-runtime measurement represents the complete measured pipeline rather than only the encoder and ConvLSTM components shown separately.

Why the Encoder Improved More Than ConvLSTM

The convolutional encoder benefited strongly from GPU execution because its operations provided substantial data parallelism.

ConvLSTM achieved a smaller improvement because of:

Recurrent dependencies
Sequential temporal processing
Memory movement
Repeated state updates
More limited parallelism across time steps

Consequently, the 12.68× encoder acceleration did not translate into a 12.68× end-to-end speed-up.

After the encoder was accelerated, ConvLSTM became the dominant runtime bottleneck.

This is an important performance-engineering lesson: accelerating one component can move the bottleneck to another part of the system.

Correctness Validation

The CUDA implementation was compared with the reference implementation for the tested tensor shapes and configurations.

The optimisation work focused on reducing execution time while preserving the expected computational behaviour. Model-quality evaluation and runtime evaluation were treated as separate concerns.

Engineering Challenges

Several engineering challenges appeared during the implementation.

1. Correct Tensor Indexing

The CUDA kernel had to reproduce the same logical output as the reference convolution operation.

A small indexing error could generate incorrect feature maps without necessarily causing a runtime failure.

2. Memory Management

The im2col transformation created additional intermediate data.

This required careful allocation, reuse, and release of GPU memory.

3. Synchronisation

Threads sharing data through shared memory had to synchronise at appropriate points.

Missing or unnecessary synchronisation could cause incorrect results or reduced performance.

4. Kernel Configuration

Thread-block dimensions and tile sizes affected:

Occupancy
Shared-memory usage
Memory access
Work distribution
Overall kernel performance

5. PyTorch Integration

The extension had to account for tensor shapes, data types, device placement, memory layout, compilation, linking, and output comparison.

6. Measuring the Correct Bottleneck

Improving one component changed the relative cost of other components.

Profiling therefore had to be repeated after major optimisations.

Engineering Lessons

1. Optimising One Component Does Not Guarantee an Equivalent End-to-End Improvement

Once the encoder was accelerated, ConvLSTM represented a larger proportion of the remaining runtime.

2. `im2col` Makes Convolution GPU-Friendly but Introduces Memory Overhead

The transformation enables GEMM-based execution while creating a larger intermediate representation.

3. Shared Memory Is Valuable When Threads Repeatedly Reuse Nearby Data

Its benefit depends on access patterns, tile sizes, synchronisation, and thread-block configuration.

4. Profiling Must Guide Optimisation

Performance work should focus on measured bottlenecks rather than assumptions about which component is slow.

5. Correctness Is as Important as Speed

GPU output should be compared against a trusted reference implementation before performance improvements are accepted.

6. Custom PyTorch Extensions Offer Flexibility at the Cost of Additional Complexity

Device handling, tensor shapes, data types, compilation, numerical validation, and fallback behaviour all require careful attention.

Frequently Asked Questions

What is MSCRED?

MSCRED stands for Multi-Scale Convolutional Recurrent Encoder-Decoder.

It is an architecture designed to model relationships between multiple time-series signals and detect abnormal behaviour through reconstruction error.

What does `im2col` do?

im2col rearranges overlapping convolution windows into matrix columns, allowing convolution to be expressed as matrix multiplication.

Why did shared memory help?

Shared memory allowed threads in the same block to reuse data without repeatedly loading it from slower global memory. This reduced memory-access overhead for reusable tiles.

Did GPU memory optimisation improve the F1 score?

That was not the purpose of the optimisation, and this article does not claim that it did.

The best recorded F1 score describes anomaly-detection quality in the thesis experiments. The CPU and GPU timing results describe computational performance. They should be interpreted separately.

Why was the encoder speed-up much larger than the total speed-up?

After the encoder became faster, ConvLSTM represented a larger share of total runtime. The overall system was therefore limited by the remaining bottleneck.

Is a Custom CUDA Kernel Always Faster Than PyTorch Operations?

No.

Optimised PyTorch and CUDA libraries are already highly efficient for many operations. A custom kernel is most useful when:

The workload has a specialised access pattern
Existing operations create unnecessary overhead
Profiling identifies a clear bottleneck
Correctness and maintenance costs are justified

Can the Original Industrial Dataset Be Shared?

No.

The original dataset contains industrial process information. A public reproduction would require synthetic or appropriately anonymised multivariate time-series data.

Are These Benchmark Results Universal?

No.

Performance depends on GPU architecture, CPU configuration, tensor dimensions, batch size, kernel implementation, memory layout, CUDA and PyTorch versions, compiler settings, and workload characteristics.

The reported numbers apply to the specific experimental setup described in this article.

Limitations and Data-Privacy Considerations

The reported results were obtained on a specific hardware configuration and workload.

Different GPUs, tensor dimensions, batch sizes, memory layouts, CUDA versions, and kernel configurations may produce different results.

The original dataset cannot be published because it contains industrial process information. A public reproduction would therefore require synthetic or appropriately anonymised multivariate time-series data.

This article presents aggregate benchmark measurements and technical methodology rather than the complete industrial dataset or restricted implementation.

Future Work

Future work could focus on:

Profiling and optimising the ConvLSTM component
Reducing intermediate memory allocation
Reducing host-to-device and device-to-device memory movement
Comparing the custom implementation with optimised CUDA libraries
Evaluating additional tile and thread-block configurations
Testing across different GPU architectures
Reproducing the workflow using synthetic public data
Adding automated correctness and regression tests
Measuring kernel-level execution using GPU profiling tools

Conclusion

This work was completed as part of my M.Tech thesis research at the Indian Institute of Technology Bhilai.

It demonstrated how low-level GPU optimisation can be integrated into a deep-learning anomaly-detection pipeline.

The convolutional encoder benefited substantially from:

CUDA parallelisation
im2col-based lowering
GEMM
Shared-memory data reuse
Custom PyTorch integration

Encoder runtime decreased from 300 seconds to 23.664 seconds in the measured workload.

The broader result also illustrated an important systems principle: once one stage is accelerated, another stage may become the new performance bottleneck.

In this case, ConvLSTM limited the end-to-end improvement even after the encoder achieved a substantial speed-up.

The work reinforced that successful performance engineering requires both low-level optimisation and system-level measurement.

References

Zhang, C., Song, D., Chen, Y., Feng, X., Lumezanu, C., Cheng, W., Ni, J., Zong, B., Chen, H., and Chawla, N. V. (2019). A Deep Neural Network for Unsupervised Anomaly Detection and Diagnosis in Multivariate Time Series Data. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01), 1409–1416. https://doi.org/10.1609/aaai.v33i01.33011409
Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W.-K., and Woo, W.-C. (2015). Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting. Advances in Neural Information Processing Systems 28, 802–810. https://proceedings.neurips.cc/paper_files/paper/2015/hash/07563a3fe3bbe7e3ba84431ad9d055af-Abstract.html
NVIDIA. CUDA C++ Programming Guide. https://docs.nvidia.com/cuda/cuda-c-programming-guide/
PyTorch. Custom C++ and CUDA Operators. https://docs.pytorch.org/tutorials/advanced/cpp_custom_ops.html

Citation and Feedback

If this article informs academic or technical work, please cite the original MSCRED and ConvLSTM papers above.

To refer specifically to my thesis implementation, optimisation process, or reported benchmarks, you may cite this article as:

Vutnoor, R. (2026). “GPU-Accelerating MSCRED with CUDA, im2col, GEMM, and a Custom PyTorch Extension.” DEV Community.

If you found the article useful, a reaction, comment, or constructive technical question on DEV would be appreciated.

I am Ranjith Vutnoor, an AI/ML Software Engineer and IIT Bhilai alumnus working on production RAG systems, LLM evaluation, model fine-tuning, PyTorch, and GPU-accelerated machine learning.

Building a Real-Time Network Anomaly Detector with PyTorch and Go

Tue, 26 May 2026 07:41:30 +0000

description: "4-service microservices architecture for live network intrusion detection: hybrid ML model, sub-second alerts, and an 11-page React operator dashboard."
tags: pytorch, go, machinelearning, microservices
series: engineering-deep-dives
cover_image:

canonical_url: https://satyasaikilaru.vercel.app/blog/network-anomaly-detector

TL;DR — I built a full-stack intrusion detection platform: a hybrid ML model (Focal Loss classifier + autoencoder for zero-days), a 4-service Go/Python backend, and an 11-page React dashboard. R2L recall doubled (14% → 29%), zero-day detection works via reconstruction error, and the whole pipeline runs in <2ms per prediction.

The Problem No One Talks About

Every network security tutorial shows you how to train a classifier on NSL-KDD and call it done. But classifiers have a fatal blind spot: they can only detect attacks they've trained on.

The NSL-KDD test set alone contains 17 attack subtypes that don't appear in training. In production, this ratio is infinitely worse — every new CVE, every new botnet, every novel C2 protocol is invisible to a classifier until you retrain.

I wanted to build something that catches both: known attacks with high precision and novel threats it's never seen before.

Architecture at a Glance

┌─────────────────────────┐
│   React Dashboard :5173 │  11 pages, ECharts, AG Grid, Firebase Auth
│   Apollo Client (GQL+WS)│
└───────────┬─────────────┘
            │
┌───────────▼─────────────┐
│  GraphQL Gateway :4000  │  gqlgen, Firebase Admin SDK, SQLite
│  Auth · Persistence     │
└───────────┬─────────────┘
            │ HTTP proxy
┌───────────▼─────────────┐
│  Go API Backend :8080   │  REST, WebSocket broadcast, Simulation
│  Packet capture control │
└───────────┬─────────────┘
            │ HTTP /api/predict
┌───────────▼─────────────┐
│  FastAPI ML Server :8000│  PyTorch → ONNX inference, <2ms/sample
└─────────────────────────┘

Why 4 services? The ML server needs high memory and bursty CPU. The Go API needs low memory and high concurrency. Scaling them as one binary wastes money. Each service deploys and scales independently.

The Hybrid ML Model

Problem 1: Rare attacks are invisible to standard loss functions

NSL-KDD class distribution:

Class	Training %	What it is
Normal	53%	Legitimate traffic
DoS	36%	Denial of service
Probe	9.4%	Port scanning, reconnaissance
R2L	0.8%	Remote-to-Local exploits
U2R	0.04%	User-to-Root privilege escalation

Standard CrossEntropy loss treats all samples equally — so the model essentially ignores R2L and U2R. Why learn to detect 0.04% of the data when you can boost accuracy by getting the 53% right?

Fix: Focal Loss.

FL(p_t) = -α_t (1 - p_t)^γ log(p_t)

Focal Loss down-weights easy examples (the Normal traffic the model already handles) and forces it to focus on hard, rare samples. The γ parameter controls how aggressively.

Result:

Metric	CrossEntropy (V1)	Focal Loss (V2)	Change
Probe Recall	68%	88%	+20%
R2L Recall	14%	29%	+15% (2x)
Macro F1	0.558	0.589	+5.5%

Problem 2: Zero-days are invisible to any classifier

A classifier is a lookup table — "I've seen this pattern before, it's DoS." For the 17 novel attack subtypes in the test set (and the infinite unknown attacks in the wild), we need something fundamentally different.

Fix: Autoencoder trained only on Normal traffic.

The autoencoder learns to compress and reconstruct "what normal looks like." At inference:

Low reconstruction error → looks normal → pass
High reconstruction error → looks like nothing I've ever seen → anomaly flag

This is zero-day detection by definition: it catches anything that deviates from normal, regardless of whether it matches any known signature.

The 3-Vote Consensus

Two detectors means disagreements. Policy:

Classifier	Autoencoder	Action
Attack	High error	High confidence — auto-block above threshold
Normal	High error	Possible zero-day — escalate to operator
Attack	Low error	Known pattern — treat at face value
Normal	Low error	Green — pass

The second row is the gold: a connection that looks normal per-feature but whose overall shape is anomalous. That's where zero-days hide.

The Dashboard — 11 Pages

Not just a prediction endpoint. Operators need context:

Page	Why it exists
World Map	Animated attack routes with GeoIP — "where is this coming from?"
Traffic Feed	AG Grid, bulk select, CSV export — "show me everything"
Incidents	AI threat summary + timeline by class — "what's happening now?"
Analytics	Time-series, attack distribution — "what's the trend?"
Network Graph	Force-directed IP topology — "who's talking to whom?"
Model Compare	Radar chart, confusion matrix — "how good are we?"
SOAR	Auto-block on confidence thresholds — "respond automatically"
Settings	Profile, theme, webhooks, Slack — "configure everything"
System Status	Service health, latency sparklines — "is the platform healthy?"

Firebase Auth handles Google Sign-In + Email/Password. First user = admin in SQLite; subsequent = viewers. RBAC enforced at the gateway, not just the UI.

Performance

Class	Precision	Recall	F1
Normal	91.0%	88.9%	90.0%
DoS	95.8%	85.6%	90.4%
Probe	60.4%	88.3%	71.7%
R2L	53.9%	29.4%	38.0%
U2R	2.2%	41.8%	4.2%

Overall: 80.0% accuracy · 0.589 Macro F1

R2L/U2R are still the weakest — genuinely hard with <1% of training data. The autoencoder compensates by catching them as distribution anomalies even when the classifier misses.

What I'd Do Differently

Add a χ² frequency-distribution detector — the same math that cracks a Caesar cipher (compare observed letter frequencies to English) applied to network feature distributions. Catches slow horizontal scans that per-packet models miss.
Per-host behavioral baselines — a global Normal model doesn't notice "this IP was quiet for 6 months and just started scanning port 445." A per-host baseline does.
Adversarial training — FGSM/PGD perturbations during training to harden against evasion. If someone discovers your model, they'll craft packets to slip past it.

Stack

Go · Python · PyTorch · ONNX · FastAPI · React 19 · Vite · Tailwind CSS · ECharts · AG Grid · Apollo Client · GraphQL (gqlgen) · Firebase Auth · SQLite · scapy · GeoIP2

Portfolio · GitHub

If you're building something similar or want to discuss the architecture — connect with me on LinkedIn.