https://coderlegion.com/tag/shell-scripting Powered by Question2Answer https://coderlegion.com/24293/from-shell-scripts-to-scalable-automation-why-ansible <p>Most of us start our automation journey with 𝗦𝗵𝗲𝗹𝗹 𝗦𝗰𝗿𝗶𝗽𝘁𝗶𝗻𝗴—and that's exactly how it should be.</p> <p>It's simple, powerful, and a great way to automate repetitive tasks.</p> <p>But as your infrastructure grows, you'll eventually hit its limits.</p> <p>🚨 Imagine this...<br> You need to configure 100 or even 1,000 servers.<br> Yes, you can SSH into each server and execute a shell script.<br> But managing automation at this scale becomes challenging.</p> <p>⚠️ Challenges with Shell Scripting<br> ❌ No Idempotency — Running the same script multiple times can repeat tasks and cause errors unless you add custom checks.</p> <p>❌ Limited Error Handling — You must manually validate exit codes ($?) and handle failures yourself.</p> <p>❌ Complex at Scale — Large scripts quickly become difficult to maintain with retries, logging, validations, and conditions.</p> <p>❌ Limited Scalability — Shell executes sequentially by default. Parallel execution is possible using &amp; and wait, but managing jobs, failures, and thousands of SSH connections adds complexity.</p> <p>❌ No Built-in Inventory — You manage server IPs and SSH logic yourself.</p> <p>❌ Distribution-Specific Logic — Different Linux distributions often require different commands.</p> <p>✅ So, why Ansible?</p> <p>Ansible is an open-source, agentless automation tool that follows a push-based architecture.</p> <p>It connects to remote servers over SSH, pushes tasks from the control node, and executes them without installing any agent on the managed nodes.</p> <p>Key Advantages<br> ✅ Idempotent by default</p> <p>✅ Inventory Management</p> <p>✅ Parallel Execution</p> <p>✅ Better Error Handling</p> <p>✅ Configuration Management</p> <p>✅ Rich Modules</p> <p>✅ Readable &amp; Scalable YAML Playbooks</p> <p>💡 Shell Scripting is still a fundamental skill for every DevOps engineer.</p> <p>But as your infrastructure grows, writing more shell code isn't always the answer.</p> <p>Sometimes, choosing the right tool is.</p> <p>The question isn't whether Shell Scripting is good or bad.</p> <p>The real question is: At what point does your automation stop scaling?</p> <p>Follow my profile for more practical DevOps content and real-world automation insights. 🚀</p> Articles https://coderlegion.com/24293/from-shell-scripts-to-scalable-automation-why-ansible Sat, 08 Aug 2026 17:40:13 +0000 https://coderlegion.com/24294/from-shell-scripts-to-scalable-automation-why-ansible <p>Most of us start our automation journey with 𝗦𝗵𝗲𝗹𝗹 𝗦𝗰𝗿𝗶𝗽𝘁𝗶𝗻𝗴—and that's exactly how it should be.</p> <p>It's simple, powerful, and a great way to automate repetitive tasks.</p> <p>But as your infrastructure grows, you'll eventually hit its limits.</p> <p>🚨 Imagine this...<br> You need to configure 100 or even 1,000 servers.<br> Yes, you can SSH into each server and execute a shell script.<br> But managing automation at this scale becomes challenging.</p> <p>⚠️ Challenges with Shell Scripting<br> ❌ No Idempotency — Running the same script multiple times can repeat tasks and cause errors unless you add custom checks.</p> <p>❌ Limited Error Handling — You must manually validate exit codes ($?) and handle failures yourself.</p> <p>❌ Complex at Scale — Large scripts quickly become difficult to maintain with retries, logging, validations, and conditions.</p> <p>❌ Limited Scalability — Shell executes sequentially by default. Parallel execution is possible using &amp; and wait, but managing jobs, failures, and thousands of SSH connections adds complexity.</p> <p>❌ No Built-in Inventory — You manage server IPs and SSH logic yourself.</p> <p>❌ Distribution-Specific Logic — Different Linux distributions often require different commands.</p> <p>✅ So, why Ansible?</p> <p>Ansible is an open-source, agentless automation tool that follows a push-based architecture.</p> <p>It connects to remote servers over SSH, pushes tasks from the control node, and executes them without installing any agent on the managed nodes.</p> <p>Key Advantages<br> ✅ Idempotent by default</p> <p>✅ Inventory Management</p> <p>✅ Parallel Execution</p> <p>✅ Better Error Handling</p> <p>✅ Configuration Management</p> <p>✅ Rich Modules</p> <p>✅ Readable &amp; Scalable YAML Playbooks</p> <p>💡 Shell Scripting is still a fundamental skill for every DevOps engineer.</p> <p>But as your infrastructure grows, writing more shell code isn't always the answer.</p> <p>Sometimes, choosing the right tool is.</p> <p>The question isn't whether Shell Scripting is good or bad.</p> <p>The real question is: At what point does your automation stop scaling?</p> <p>Follow my profile for more practical DevOps content and real-world automation insights. 🚀</p> Articles https://coderlegion.com/24294/from-shell-scripts-to-scalable-automation-why-ansible Sat, 08 Aug 2026 17:39:49 +0000 https://coderlegion.com/24086/deploying-a-microservices-e-commerce-application-on-aws-using-shell-scripting <p><strong>This article was originally published on Hashnode and is being republished here.</strong> </p> <p>It covers deploying a microservices-based e-commerce application on AWS using Shell Scripting, including server setup, service configuration, and application deployment.</p> <p>Over the past few days, I learned Shell Scripting and decided to apply everything I had learned by building a complete project from scratch to gain real hands-on experience.</p> <p>My goal wasn't just to deploy the RoboShop e-commerce application on AWS using Shell Scripting. I also wanted to understand what happens behind the scenes—how services start, how they communicate with each other, how to automate repetitive deployment tasks, and most importantly, how to troubleshoot issues when things go wrong.</p> <p>Throughout this project, I learned how to debug deployment issues by analyzing HTTP status codes, checking Nginx logs, application logs, and database logs, identifying whether the problem was in the upstream or downstream service, and writing proper deployment logs to make troubleshooting easier.</p> <p>More than the automation itself, the debugging experience gave me strong hands-on knowledge of application deployment, troubleshooting, and real-world infrastructure automation.</p> <hr> <p><strong>What is the RoboShop E-Commerce Project?</strong><br> RoboShop is a microservices-based e-commerce application consisting of around 10 services, including Frontend, Catalogue, User, Cart, Shipping, Payment, MongoDB, MySQL, Redis, RabbitMQ, and Dispatch. Each service has its own responsibility and communicates with other services to complete the order flow.</p> <p>This project is a great way to understand how multiple services are deployed, connected, and managed in a real-world environment.</p> <hr> <p><strong>Application Components</strong><br> The RoboShop application is built using multiple technologies, with each service having a specific responsibility.</p> <p><strong>Service</strong> and <strong>Technology</strong><br> Mongodb (NoSQL Database)<br> Catalogue (Node.js)<br> Redis (Caching Server)<br> User (Node.js)<br> Cart (Node.js)<br> Mysql (SQL Database)<br> Shipping (Java)<br> Rabbitmq (Message Broker)<br> Payment (Python)<br> Dispatch (Go (Golang))<br> Frontend (Nginx (Reverse Proxy))</p> <hr> <p><strong>How Requests Flow</strong><br> The Frontend uses Nginx as a reverse proxy. Instead of clients communicating directly with backend services, every request first reaches Nginx. Based on the requested URL, Nginx forwards the request to the appropriate backend service.</p> <pre><code>location /api/catalogue/ { proxy_pass http://catalogue:8080/; } location /api/user/ { proxy_pass http://user:8080/; } location /api/cart/ { proxy_pass http://cart:8080/; } location /api/shipping/ { proxy_pass http://shipping:8080/; } location /api/payment/ { proxy_pass http://payment:8080/; } </code></pre> <p>Using a reverse proxy provides several benefits:</p> <p>Clients communicate only with Nginx, not directly with backend services.</p> <p>The actual backend server details remain hidden.</p> <p>Nginx routes requests to the correct service based on the URL.</p> <p>It also makes it easier to manage load balancing, SSL termination, and routing from a single entry point.</p> <hr> <p><strong>Deployment Approach</strong><br> Although the services are built using different technologies such as Node.js, Java, and Python, the deployment process is almost the same for every service.</p> <p>The deployment flow includes:</p> <ul> <li><p>Installing the required runtime</p> </li> <li><p>Creating the application user and directory</p> </li> <li><p>Downloading and extracting the application</p> </li> <li><p>Installing application dependencies</p> </li> <li><p>Configuring a systemd service</p> </li> <li><p>Loading the database schema (where required)</p> </li> <li><p>Starting and enabling the service</p> </li> </ul> <p>Since this workflow was common across almost every service, I moved all the reusable functions into a single common.sh file and reused it across all deployment scripts. This reduced duplicate code and made the scripts much easier to maintain.</p> <hr> <p><strong>What I Automated</strong><br> During this project, I automated the following tasks:</p> <p>✅ Validated every command using Linux exit codes (0 = Success, non-zero = Failure) to stop execution immediately if a step failed.</p> <p>✅ Implemented idempotency, allowing the scripts to be executed multiple times safely.</p> <p>✅ Moved all reusable deployment logic into a common common.sh file.</p> <p>✅ Automated application deployment for all services, including runtime installation, dependency installation, and systemd configuration.</p> <p>✅ Automated EC2 instance creation using AWS CLI.</p> <p>✅ Automated Route53 DNS record creation using AWS CLI.</p> <p>✅ Generated detailed log files for every deployment step to simplify troubleshooting.</p> <p>✅ Performed root user validation before executing privileged operations.</p> <p>Instead of repeating the same steps for every server, I automated the entire deployment using Shell Scripting. This made deployments consistent, repeatable, and much faster.</p> <hr> <p><strong>Debugging Challenges</strong><br> This was the most interesting part of the project and where I learned the most. Writing the scripts was only one part, Troubleshooting the issues taught me how to debug deployments by following logs and tracing requests step by step instead of guessing.</p> <p>After deploying the application, I expected everything to work. However, when I opened the application, the catalogue products were not loading.</p> <p>Here's how I approached the issue:</p> <p>Checked the browser's HTTP status code and found a 504 Gateway Timeout.</p> <p>Looked at the Nginx access.log on the frontend and found that<br> =&gt; Nginx forwarded the request to the upstream (backend) service.</p> <p>=&gt;The upstream service did not return a response within the configured timeout. (waiting for something to get from database)</p> <p>Verified that the Catalogue service was running using systemctl status</p> <p>Checked the Catalogue service logs using: journalctl -u catalogue -f</p> <p>The logs showed that the Catalogue service couldn't connect to MongoDB.</p> <p>Logged into the MongoDB server and found that the MongoDB service was stopped.</p> <p>Started the MongoDB service, and the Catalogue service immediately began working.</p> <hr> <p><strong>💡What I learned:</strong><br> This issue taught me not to jump to conclusions or restart services blindly. Instead, I learned to follow the request flow—from the browser, to Nginx, to the application logs, and finally to the database—to identify the actual root cause.</p> <p>This debugging process gave me much more confidence in troubleshooting real deployment issues and understanding how different services interact in a microservices environment.</p> <hr> <p><strong>🚀 What's Next?</strong><br> In the next part of this project, I'll automate the application configuration using Ansible, a configuration management tool. This will help eliminate manual server configuration and make deployments even more consistent and scalable.</p> <hr> <p><strong>🎉 Final Thoughts</strong><br> This project helped me turn my Shell Scripting knowledge into practical experience. Along the way, I learned a lot by intentionally stopping services, analyzing logs, and troubleshooting the resulting issues.I gained valuable hands-on experience in automation, deployment, and debugging. I'll continue building more projects to strengthen my practical skills and gain even more real-world experience.</p> <p>If you're interested in exploring the implementation, feel free to check out the complete project source code from below.<br> <strong><a rel="nofollow" href="https://github.com/pavankumar0816/shell-roboshop-common">https://github.com/pavankumar0816/shell-roboshop-common</a></strong></p> <hr> <p>I am including a few screenshots of the deployed application:<br> <img src="https://coderlegion.com/?qa=blob&amp;qa_blobid=14644443864324811477" alt=""></p> <p><img src="https://coderlegion.com/?qa=blob&amp;qa_blobid=13835948379371085014" alt=""></p> <p><img src="https://coderlegion.com/?qa=blob&amp;qa_blobid=1458508592828395889" alt=""></p> <p><img src="https://coderlegion.com/?qa=blob&amp;qa_blobid=18310175374989591968" alt=""></p> <p><img src="https://coderlegion.com/?qa=blob&amp;qa_blobid=10844257763862837480" alt=""></p> <p><img src="https://coderlegion.com/?qa=blob&amp;qa_blobid=9979187544920098741" alt=""></p> Articles https://coderlegion.com/24086/deploying-a-microservices-e-commerce-application-on-aws-using-shell-scripting Tue, 04 Aug 2026 10:45:47 +0000