Coder Legion - Recent posts tagged workflows

Deferring Publication After a Budget-Safe Draft Revision

Fri, 16 Oct 2026 08:38:28 +0000

A Cloudflare Workflow can spend most of its bounded external request budget before publication begins, risking partial failures when generation and publishing share a single run.

Quick solution:

if (!hasPublicationBudget) {
  await commitDraftOnly(revisionData);
  return { promotionDeferred: true, readyPath: revisionData.path };
}

Reserving candidate commit, promotion preflight, live checks, site writes, and vault finalization before model calls prevents generation from creating a draft that the same run cannot safely publish. When an explicit revision-only run cannot reserve the full tail, retain only the five-request draft commit reserve.

Understanding the Request Budget Conflict

Cloudflare documents strict subrequest ceilings on its free plan, while GitHub documents base tree writes for atomic tree changes via REST API endpoints for Git trees as outlined in the GitHub REST API Git trees documentation. When a workflow handles candidate commits, promotion preflights, and final vault writes, it can exhaust its budget before publishing.

For more details on platform limits, consult the Cloudflare Workflows limits documentation. The five-request commit budget and the later publish-only step are application safeguards rather than limits specified by either provider. For related architectural guidance, you can also review Managing Concurrent Git Commits During Automated Publishing.

Implementing a Deferred Publication Fallback

When a revision-only repair occurs, you have a different recovery option available. Instead of failing the entire run, commit the reviewed revision and promote that exact ready draft in a later run.

Keep the full publication reserve unchanged for normal candidates and same-run promotion.
When an explicit revision-only run cannot reserve the full tail, retain only the five-request draft commit reserve.
Generate and review the revision, commit its Markdown, quality record, and raster metadata, then return the resulting ready-draft path with an explicit deferred publication result.
Skip publication selection in that run. After it completes, use a separate publish-only run scoped to the exact ready-draft path.
Keep manual live runs serial so a second workflow cannot race the vault SHA guard.

Verification and Execution Checklist

Verify the five-request reserve constraint during low-budget execution.
Confirm generation and commit succeed after the full publication reserve fails.
Validate that the exact-path deferred result is correctly returned.
Ensure manual live runs remain serial to prevent race conditions on the vault SHA guard.

Selecting the Latest Targeted Revision

Fri, 16 Oct 2026 08:27:59 +0000

A source often has multiple review revisions, and if a targeted workflow scans only a bounded prefix, an older revision can displace the newest one.

Quick solution:

// Sort matching revision drafts by descending -rN before path tie-break
const selected = drafts
  .filter(d => d.source === targetSource)
  .sort((a, b) => parseRevision(b) - parseRevision(a) || a.path.localeCompare(b.path))[0];

Prerequisites include a list of draft objects containing explicit revision numbers. A caution to keep in mind is that untargeted batches still require stable default ordering to avoid regression.

Diagnosing Revision Queue Scan Failures

When a workflow inspects a limited batch of files, lexicographical sorting on paths can place an older revision at the front of the queue. For instance, revision 16 might appear before revision 24 due to naming conventions or directory structures. Because the inspection budget is bounded, the workflow processes the stale revision and exits before reaching the correct draft.

To understand this behavior in practice, you can review how modern API routing and queue boundaries operate in platform documentation such as Cloudflare Workflows or when managing object trees via the REST API endpoints for Git trees, and also consider Automating Android Ci Workflows With Github.

Implementing Newest-First Revision Sorting

To ensure the correct draft is chosen, modify your sorting logic to prioritize descending revision numbers before falling back to path comparisons. This approach guarantees that even if a scan window is bounded, the highest revision number is always evaluated first.

Option	When to Use It	Trade-off or Failure Mechanism	Recommendation
Lexicographical Path Sort	Ordinary untargeted batch operations.	Fails to prioritize newer revision numbers when path names differ.	Use only for default untargeted runs.
Descending Revision Sort	Targeted source runs with multiple drafts.	Requires reliable parsing of revision tags like `-rN`.	Use for all targeted workflow queues.

Verification Checklist

Filter drafts explicitly by the requested source target.
Parse the revision identifier into a numeric value.
Sort matching drafts in descending order by revision number.
Apply path tie-breaking only after revision sorting.
Confirm via test that the highest revision is selected.

Conclusion

By following the steps outlined above and implementing the newest-first revision sorting logic, you can ensure that your workflow always selects the latest targeted revision, even in cases where the scan window is bounded. This approach helps prevent older revisions from being processed incorrectly and ensures that your workflow operates efficiently and effectively.

n8n Free Trial: 3 Workflows to Test in 14 Days

Thu, 10 Sep 2026 15:50:00 +0000

Direct answer: use the 14-day n8n Cloud trial to prove three narrow jobs, not to build a giant automation system. Keep Cloud only if at least two workflows run reliably, a named person can diagnose a failure, and the expected production executions fit the paid plan. Cancel if native tools already solve the jobs or the workflows still need constant rescue on day 14.

Partner disclosure. This guide includes a partner link to n8n Cloud. If you sign up through it, I may earn a commission at no extra cost to you.

This guide is a trial experiment for small-business owners and operators. It is not a broad introduction to n8n, a Cloud versus self-hosted comparison, or a complete pricing table. The job here is to leave the trial with evidence: three test records, three failure drills, an execution estimate, and a clear keep-or-cancel decision.

Know the trial limits before day one

Current trial fact	What it means for your test
14 days	Prepare sample data and app access before signup so the clock is used for testing.
No credit card required for the Starter and Pro Cloud trial	The trial ends unless you manually subscribe. There is no ongoing free n8n Cloud plan.
Pro features under Starter compute limits	You can evaluate features such as execution search and workflow history, but not Pro-level compute capacity.
800 AI Assistant credits during the trial	Use the preview for one controlled build. Do not make AI credits the reason the core workflows pass.
Manual executions do not use the production quota	Develop and test manually first. Activate schedules and webhooks only when acceptance checks are ready.

Official-source mismatch, checked August 8, 2026: the current n8n pricing FAQ says the trial uses Starter's 2,500-execution limit, while the dedicated free-trial documentation still says 1,000 executions. Treat the quota shown inside your workspace as binding. Until it confirms otherwise, plan as if you have 1,000 production executions.

Trial and plan details can change. The current pricing page also lists 320 MiB RAM and 10 millicores of burstable CPU for trial and Starter compute. That is enough for small tests, but large files, long code steps, and memory-heavy AI work can distort the result. Use small, representative records first.

A useful trial begins with a small test pack. Create five dummy leads, one week of sample sales or booking rows, and six fake invoices with different due dates and statuses. Use test inboxes and test chat channels. Do not start with live customer records, automatic payment actions, or messages that can reach real customers.

Name one owner. This person builds the workflow, records failures, and decides whether it passed.
Write the expected output. For every sample record, state what should happen and what must not happen.
Check the native route. If your form, accounting tool, or CRM already provides the needed alert or reminder, test that first.
Prepare credentials. Know which test accounts need access, but add credentials through n8n's credential cards. Never paste secrets into a prompt.
Create a result sheet. Record workflow, input, expected result, actual result, execution count, failure, recovery time, and decision.

A practical 14-day schedule

Days	Work	Evidence to keep
Before signup	Prepare dummy data, test accounts, acceptance checks, and the result sheet.	Test pack is ready before the clock starts.
1-2	Set the workspace timezone, connect test credentials, and run one manual trigger.	Correct timestamps and one clean credential test.
3-5	Build and break the lead alert.	Valid, invalid, and duplicate lead results.
6-8	Build and break the weekly summary.	Correct totals, empty-data handling, and delivery proof.
9-11	Build and break the invoice reminder.	Due, paid, disputed, and duplicate invoice results.
12	Run credential, timeout, duplicate, and missing-data failure drills.	Recovery steps and time to diagnose each failure.
13	Estimate monthly production executions and name the support owner.	Normal-month and busy-month estimates.
14	Use the scorecard, export workflows, and keep or cancel.	A written decision, not a vague intention to revisit later.

Workflow one: route a qualified lead alert

Start with a form or webhook, normalize the fields, reject records missing an email or phone number, and send a notification to the person who owns follow-up. Use email, Slack, Telegram, Google Chat, or another channel your team already checks. The workflow is valuable when it filters and routes leads, not merely when it copies every form submission into another inbox.

Minimum path: Form or Webhook → Edit Fields → If → destination app.

Send three valid leads with different service interests. Each should reach the correct owner with a timestamp and source.
Send a lead with no usable contact detail. It should enter a review route, not disappear or trigger normal follow-up.
Send the same lead twice. The owner should not receive two identical alerts if duplicates are realistic in your process.
Disable or invalidate the destination credential. Confirm that the failure is visible and that the owner knows how to retry safely.

Pass: five valid test leads reach the right owner within the expected time, invalid records are held for review, and a duplicate does not create duplicate work. Fail: the team still checks the form manually because it does not trust the alert.

Workflow two: deliver a weekly operating summary

Use a Schedule Trigger to read a small sheet, table, or database and calculate three numbers that affect a real decision. A bakery might track orders, average order value, and unfulfilled pickups. A clinic might track new inquiries, booked appointments, and unanswered leads. Compute the numbers deterministically before asking AI to explain them.

Minimum path: Schedule Trigger → source app → filter or aggregate → email or chat.

Run the workflow manually against a known sample and compare every total with a hand calculation.
Test an empty week. The summary should say there is no data, not invent a trend or fail silently.
Test one malformed row and one late record. Decide whether to exclude, flag, or stop.
Activate the schedule once and verify the workspace timezone, delivery time, and production execution record.

Pass: the owner receives the correct numbers at the correct time and can trace them to source rows. Fail: the summary is polished but the totals cannot be reproduced.

Workflow three: create a safe invoice reminder

First check the native reminder feature in your accounting platform. Use n8n when the job needs data from more than one system, different rules by customer or invoice, an internal approval, escalation to an account owner, or a stop condition that the native feature cannot express.

During the trial, create an internal task or a draft reminder instead of sending directly to a customer. Use fake invoices with statuses such as due soon, overdue, paid, disputed, and missing an email address. Only consider automatic external sending after someone has approved the wording, timing, opt-out handling, and stop rules.

Minimum path: Schedule Trigger → invoice source → due-date and status checks → draft or internal approval.

A due invoice creates one draft with the correct customer, amount, invoice number, and due date.
A paid or disputed invoice creates no reminder.
A missing email creates an internal review task instead of an invalid send attempt.
Running the workflow twice does not create two reminders for the same invoice and stage.

Pass: the workflow applies status and duplicate safeguards before any customer-facing action. Fail: a changed invoice status can still produce an incorrect or repeated reminder.

Use the new AI Assistant for one controlled build

n8n's current AI Assistant is a Cloud-only preview that can create, edit, test, and troubleshoot workflows inside the editor. It replaces the older assistant experience as the actively developed path. Preview features can change and can produce incorrect workflows, so use it to accelerate one draft, then inspect every node and test the result yourself.

Example prompt: Create a test workflow that receives a lead by webhook, requires either email or phone, routes valid records to my test Slack channel, sends invalid records to a review branch, and records the result. Use dummy data. Ask before activating the workflow or changing credentials.

Add credentials through the credential card when asked. Do not paste API keys, passwords, or live customer data into the chat. n8n says the assistant asks for confirmation before high-impact actions, but the operator still owns permissions, message content, data exposure, and the final activation. Trial credits are token-based, so long conversations and large workflows consume more than a short, focused request.

Understand what consumes an execution

n8n counts a complete production workflow run as one execution, regardless of how many steps it contains. Manual runs from the editor do not count. Triggered production runs do count, including failed scheduled runs. Polling triggers count when they find new data, and each webhook request normally starts an execution. Sub-workflows and error workflows do not add separate billable executions.

Frequency changes the economics quickly. A daily summary is about 30 executions per month. A five-minute schedule is roughly 8,640 to 8,928 scheduled runs in a 30- or 31-day month, even before other workflows and retries. Use event triggers when appropriate, avoid pointless polling, and estimate both a normal month and a busy month before choosing a plan.

Run failure drills before you trust the workflows

A successful happy-path demo only proves that one sample worked once. On day 12, break each workflow deliberately and record what the operator sees, whether partial actions can repeat, and how long recovery takes.

Remove one required field and send an empty source.
Expire or disconnect one test credential.
Submit the same record twice.
Simulate an app timeout or rate-limit response.
Change an invoice from overdue to paid after it enters the queue.
Run a scheduled workflow across the wrong timezone boundary.

A useful workflow exposes the failure, avoids unsafe partial repeats, and gives the owner a clear retry path. If the same error can send a second customer message, charge twice, or overwrite good data, the workflow is not ready for production.

Use this keep-or-cancel scorecard

Question	Keep signal	Cancel or redesign signal
Did at least two workflows solve a real job?	The owner stopped a repeatable manual check or handoff.	The workflows were demos with no process owner.
Were results reliable?	Acceptance checks and failure drills passed.	The team still verifies every result manually.
Can someone support it?	A named owner can find executions, diagnose failures, and retry safely.	Only the original builder understands the workflow.
Does the usage fit?	Busy-month executions leave reasonable headroom.	Polling, retries, or volume already exceed the likely plan.
Is n8n the simplest route?	Filtering, transformation, routing, approval, or error handling justifies it.	A native feature solves the job with less maintenance.

Keep Cloud when at least two workflows pass, the support owner is clear, and the paid-plan cost is smaller than the time and errors removed. Cancel when native tools win, the workflows remain fragile, or no one will own them. A cancelled trial that prevents a poor automation purchase is still a successful test.

Export before the trial ends

There is no permanent free n8n Cloud plan. If you do not subscribe, the trial expires. n8n's current Cloud documentation says the workspace owner can download workflows from a recent backup for 90 days after the trial, after which the data is permanently deleted. Export your workflow JSON files and result sheet on day 14 instead of relying on that recovery window.

If the trial passes, activate only the workflows whose safeguards are complete. Start with one production workflow, monitor its executions, and add the next one after the owner has handled a real failure. The point is not to automate everything. It is to prove that one small system can save time without creating a larger support problem.

Sources checked

Product limits and feature behavior were verified on August 8, 2026. n8n can change trial terms, pricing, preview features, and plan limits.

n8n partner link

Test one useful workflow before you automate m

Closing the Loop: From Contradiction Detection to Automated Resolution

Thu, 20 Aug 2026 21:51:09 +0000

The Problem

Every knowledge base accumulates contradictions. A new source arrives that disagrees with something already compiled, an updated report revises a market-share figure, two documents describe the same event differently. Under normal conditions these conflicts merge silently and get served to anyone who queries.

The manual fix is conceptually simple: find conflicting pages, review them, decide which source wins, rewrite the content, promote the page back to active. In practice this becomes a recurring burden. You have to track which pages are affected, remember why each was flagged, and repeat the process every time new sources arrive.

Two things make it worse. First, the affected pages keep answering queries while broken — a contradicted page with stale or incorrect information is indistinguishable from a clean one in search results. Second, the effort scales linearly with how many pages are affected.

Synthadoc v1.3.0 addresses both halves: automated detection through the adversarial lint gate, and structured interactive remediation through the contradiction resolver workflow.

Detection: The Adversarial Lint Gate

The adversarial reviewer has been in Synthadoc since v0.5.0. It runs a second, independent LLM pass over each wiki page, specifically tasked with identifying unsupported claims, overstated superlatives, and contestable facts. Warnings land in page frontmatter and surface in lint reports.

Before v1.3.0, that was the end of the story. Warnings were informational. A page with ten adversarial warnings still answered queries at full weight alongside a clean, reviewed page.

The adversarial lint gate changes this. Configure a threshold in config.toml:

[lint]
adversarial_gate_threshold = 2
adversarial_max_per_page   = 3

When any active or stale page reaches that warning count during a lint run, it is automatically demoted to contradicted. The transition is recorded in the lifecycle audit trail — "auto-demoted: 2 adversarial warning(s) ≥ gate threshold 2" — and the page is immediately excluded from the BM25 search index. The next query will not see it. No restart required, no polling delay.

New wikis created with synthadoc init have the gate active at threshold 3 out of the box. Existing wikis upgrading from an earlier release are unaffected — if your config.toml has no adversarial_gate_threshold key, the gate is disabled until you add one. Compliance-sensitive environments typically run at 1.

One Click to Remediation

After lint demotes pages, the web UI signals the problem through two simultaneous paths: a persistent hint chip in the sidebar ("⚔ N contradicted → Run resolver"), and a pre-filled suggestion that appears below any assistant reply mentioning contradicted pages. Click either and the contradiction resolver starts.

From the CLI, with no web UI required:

synthadoc workflow run --name contradiction-resolver
synthadoc workflow run --name contradiction-resolver --slug grace-hopper
synthadoc workflow run --name contradiction-resolver --type adversarial

How the Resolver Works

The resolver is an interactive agentic loop. Everything it does requires your approval before anything is written.

Step 1 — Scope and cost estimate. Before any LLM call, the workflow lists the pages in scope and shows the estimated token count and cost. You approve to proceed or cancel.

Step 2 — Diagnosis and proposal. For each page, the agent reads the current content, the adversarial lint_warnings or source-conflict contradiction_note, and the original source file where available. It selects a strategy, generates a rewrite, and shows you the exact before/after diff.

Step 3 — Approval gate. You approve the change, skip this page, or ask for a different strategy. Nothing is written until you say so.

Step 4 — Scoped re-lint. After an approved write, lint runs on only that page — adversarial and contradiction checks only. If the page passes, it is promoted automatically: contradicted → active, with a lifecycle event naming the strategy used.

Step 5 — Strategy rotation on failure. If re-lint fails, the agent diagnoses why and selects a different approach: content rewrite → web ingest for better sourcing → forced source re-ingest → cross-page resolution. Maximum four attempts per page; escalation with a plain-language diagnosis after the cap.

Step 6 — Ground-truth confirmation. After all pages are processed, the workflow runs synthadoc status and appends the live lifecycle counts. The final output is not "we tried to fix three pages" — it is contradicted: 0.

Your Escape Hatch: Content Snapshots

The diff-before-write approval gate is the primary guard. But if you approve a change and later decide the resolver's rewrite missed the editorial nuance you had in mind, content snapshots give you a fallback.

Every lifecycle event — including the contradicted → active transition the resolver writes — captures a frozen copy of the page body at that moment.

From Obsidian: open the Synthadoc plugin → Manage Page Lifecycle → switch to the Content Snapshots tab. Find the page, click View to see the diff against the current file, then Rollback to this version — no terminal required.

From the CLI:

synthadoc lifecycle history grace-hopper
synthadoc lifecycle rollback grace-hopper --index 2 --reason "reverting resolver rewrite"

Either way, rollback saves the current content before overwriting, so the operation is always undoable. After rollback the page returns to contradicted on the next lint run, at which point you can re-run the resolver with a different strategy, edit manually, or archive the page.

Architecture: How the Pieces Connect

Lifecycle management and search index invalidation are a shared service used by both the lint agent and the contradiction resolver. Whichever path transitions a page state, the search corpus updates immediately. There is no eventual consistency window and no restart required.

The resolver is one of six AgenticWorkflow implementations registered with the action agent. They share the same loop machinery — tool dispatch, SSE progress events, confirmation timeout, tool budget — and differ only in their system prompt and the tool set they operate on.

A unified CLI command surfaces all of them:

synthadoc workflow list     # see everything available
synthadoc workflow run --name <name> [--slug SLUG] [--type TYPE]

What We Learned

Scoped re-lint was the hardest design decision. Running a full wiki lint pass after each strategy attempt is correct but slow enough to make the workflow feel unresponsive. Scoped re-lint — checking a single slug with adversarial and contradiction passes only — is what makes per-attempt feedback feel interactive. Threading that path through existing lint infrastructure without affecting full-wiki behaviour required care.

The approval gate is load-bearing, not cosmetic. A workflow that applies all changes and then asks whether you wanted that is functionally useless. Showing the diff first and applying only on explicit approval is the feature — it is what separates an agent that acts on your behalf from one that acts instead of you.

System prompts are the policy layer. Strategy selection, iteration cap, escalation criteria, the ground-truth status check at the end — all of these live in the system prompt, not in Python. When we needed to raise the attempt cap and allow strategy reuse on later attempts, four lines of prompt text covered it. No code changed.

What's Next

The pluggable workflow architecture was designed for exactly this kind of extension. Adding a new workflow means implementing three methods on AgenticWorkflow, writing a system prompt, and registering the class. The CLI and web UI routing are inherited automatically — synthadoc workflow list picks it up immediately, and synthadoc workflow run --name <name> runs it without any other plumbing.

The next candidates:

Orphan page resolver — pages with no inbound wikilinks are a known structural problem. A workflow that proposes index entries and cross-links for each orphan, shows the diff, and applies approved changes follows the same loop directly.
Broken citation resolver — pages where citation markers point to deleted source lines are detectable at lint time. An agentic loop that re-reads the source, finds the correct line range, and proposes an updated marker is a natural fit.

Neither requires changes to the core. The loop machinery, the confirmation gate, the SSE progress events, and the CLI integration are all shared. Only the system prompt and the tools it can call are new.

Synthadoc v1.3.0 is an open-source LLM knowledge compilation engine that turns raw documents into structured, local-first wikis. A transparent, human-readable alternative to traditional RAG, which can be self-managed and self-improved without the use of any tools.

GitHub
PyPI

I use AI coding agents daily.

Sun, 16 Aug 2026 13:37:07 +0000

My AI coding agent kept claiming "done" without proof. So I built gates for it.

I use AI coding agents daily. They're fast. They're also confident when wrong.

Last month mine recommended a library which doesn't exist. Another run claimed "build complete" with zero tests executed. Sound familiar?

So I built Meta-Architect. Open-source skill pack for Codex. Six fail-closed gates before anything ships:

$arch: architecture first, with a written decision record
$sage: every dependency claim graded VERIFIED, PARTIAL, or MISSING against upstream repos
$flow: logic and state review
$vet: security and dependency audit
$vibe: DX/UX review
$build: stays locked until all five pass

Red stays red. The agent stops instead of pretending.

What it won't do:

Replace your agent, runtime, or model
Run unattended
Claim "production-ready" without receipts

Current state: v0.1.13, Codex-only, MIT license. Claude Code and Cursor support on the roadmap.

github.com/JustineDevs/meta-architect

What's the worst thing your AI agent shipped while claiming success? Best answer becomes my first gate test case.

They Asked for My AI Rules. But I Could Not Just Hand Them Over.

Mon, 13 Jul 2026 18:56:49 +0000

A team lead announces that the team will start using AI-assisted development. Everyone nods. Nobody asks what that actually means on Monday morning.

Some times ago I was in that position. A project I was working on needed to start using AI-assisted development, and the team was new to it. Nobody had rules written down for an agent to follow. Nobody had skills defined for it to load. There was no shared idea of how this should work inside our specific repo. Someone had to go first. That someone was me.

The rules worked because I built them for one repo

I spent time curating a set of rules and skills for that project. Not generic ones. I shaped them tightly around how that repo was actually structured, its conventions, its layout, the things a new engineer usually has to learn by asking around. I wanted an agent working inside that codebase to already know what a human teammate would have picked up in the first two weeks.

I gave a demo. It landed well. Well enough that it got shared further across team, as something other teams could learn from.

I gave the demo again. Same reaction. Then a few developers reached out for the actual rules and skills files.

I said sure, and then I actually looked at what I would be handing them.

The problem showed up the moment other people wanted in

It was not copy-paste-able. The rules referenced folder names, module boundaries, and patterns specific to one repo. Handing them over as-is would have meant handing over advice that was wrong for their project, dressed up as a shortcut.

So I told them to use it as a reference. Look at the structure, understand the reasoning, adapt it to your own repo.

That is correct advice. I watched people nod at it and then quietly missing it.

I was solving the wrong problem the whole time

I had been thinking about this as a documentation problem. Write good rules, explain them well, let people copy the idea. What I actually had was a generation problem. The rules that worked were the ones rendered specifically for a stack, not the ones written generically and left for someone to translate.

The thing that was supposed to spread across other teams did not land as expected. What reached them was a reference nobody had time to turn into anything real. That gap sat there whether I looked at it or not.

What I wanted to build was not another generic harness. I wanted something that looked at a repo's actual stack and produced rules and skills that worked from day one, not rules that needed a translation step before they were useful.

Nobody on a team agrees on one AI tool

Once I sat with that problem, a second one showed up right behind it, and this one turned out to matter just as much.

Every adoption plan I had seen quietly assumed one AI tool per team. In practice, that assumption breaks almost immediately. One engineer wants Cursor. Another has already built muscle memory in Claude Code. Someone else is testing Windsurf, or is stuck on Copilot because that is what the company licensed, or wants to try Antigravity because something new just came out.

Nobody wants to be told which editor or which agent to use, and forcing a single choice on a team is its own kind of friction, separate from the rules problem entirely. An adoption effort that only works for one tool is not really team adoption. It is one person's adoption, dressed up as a rollout.

What the tool had to actually do

That is the shape the problem ended up taking. Not "write better rules." Generate rules and skills that are rendered for a specific stack, and make sure they work no matter which of the tools a given engineer has already chosen.

That is what became FARE and BARE, frontend and backend AI starter recipes, published as npm CLI tools. Point either one at a project and it generates rules, skills, and workflow context shaped around that project's actual stack, with adapters for five tools out of the box: Cursor, Claude Code, Windsurf, Copilot, and Antigravity. The output is meant to be used immediately, not adapted first.

The part worth being honest about is that this did not remove the original judgment call. Something still has to decide what "good rules for this stack" means because it's always well opinionated. The CLI just means that decision gets made once, by the tool, instead of once per engineer, per project, under deadline pressure, with wildly inconsistent results.

I did not build everything I wanted to

The scope I first sketched out was larger than what shipped. There were ideas around deeper project-specific customization, additional integrations, and more automated context tracking that I dropped or pushed to later releases. Some of that was a reasonable v1 boundary. Some of it was me wanting to build the interesting version instead of the useful one, and catching myself doing it.

Keeping it simple was not the easy default. It was a decision, and one I had to make more than once while writing it.

I do not know yet whether this is the right shape for other teams' problems or just the right shape for the one I kept running into. If you try FARE or BARE on your own project, I would genuinely like to know where it holds up and where it does not.

Awaiting your feedback!
Frontend Ai starter REcipes
Backend Ai starter REcipes

Orchestrating Complex Workflows: From Sequential to Parallel to Intelligent

Mon, 13 Jul 2026 16:19:29 +0000

Your workflow has 10 steps. Today they run sequentially (one after another). Step 5 depends on Step 2, but Step 3-4 could run in parallel. You’re wasting time. I built Socratic-workflow to intelligently schedule tasks: parallel where possible, sequential where necessary, with automatic retry, branching based on conditions, and human approval gates. Same work. 40% faster.

The Problem: Dumb Sequential Workflows
Scenario 1: The Artificial Wait
Your data pipeline has these steps:

Extract data from database (2 minutes)
Validate data (1 minute)
Transform data (2 minutes)
Enrich with external data (3 minutes)
Load to data warehouse (2 minutes)
Update metrics dashboard (1 minute)
Send notifications (1 minute)
Archive raw files (1 minute)
Log completion (30 seconds)
Delete temporary files (30 seconds)
Running sequentially: 2+1+2+3+2+1+1+1+0.5+0.5 = 13.5 minutes

But wait:

Step 6 (metrics) doesn’t need Step 5 (load to warehouse)
Step 7 (notifications) doesn’t need anything
Step 8 (archive) doesn’t need anything
Steps 6, 7, 8 could run in parallel with Step 5
If parallelized:

Step 1-5: Sequential (dependencies) = 10 minutes
Step 6-8: Parallel with Step 5 = overlap (3 minutes)
Step 9-10: Sequential at end = 1 minute
Total: 10 + 1 = 11 minutes

Savings: 13.5 – 11 = 2.5 minutes per run

If this runs 20 times per day: 50 minutes saved per day = 5 hours per week = 260 hours per year

Scenario 2: The Fragile Coordination
You have multiple workflows that need to work together:

Workflow A (Recommendation generation):

Get user data
Get product data
Generate recommendations
Rank by profit margin
Store in cache
Workflow B (User notification):

Get user preferences
Get notifications to send
Personalize message
Send email
Log delivery
These should coordinate:

Workflow A should complete before Workflow B runs
If Workflow A fails, Workflow B shouldn’t even start
If Workflow B fails partway, it should retry
If either fails permanently, alert human
Managing this coordination manually means:

Writing scripts that call scripts
Managing dependencies with cron jobs
Handling failures with try-catch blocks
Manual coordination between teams
One small change breaks the whole system.

Scenario 3: The Conditional Complexity
Your workflow needs to branch based on conditions:

Get user data
Check user location

├─ If US → Route to US workflow (Step 3a, 4a, 5a)

├─ If EU → Route to EU workflow (Step 3b, 4b, 5b)

└─ If other → Route to default workflow (Step 3c, 4c, 5c)
Merge results
Send response
Without proper orchestration, this becomes spaghetti code. Branches are hard-coded. Changes require code rewrites. Adding a new region means debugging the entire workflow.

Why Standard Job Schedulers Fail
Traditional tools (cron, Jenkins, Airflow) handle sequential workflows and simple parallel work.

They break when you need:

Complex dependencies: “Step 5 depends on both Step 2 and Step 4” Dynamic branching: “Route to different steps based on Step 3’s output” Human approval: “Wait for human approval before proceeding” Intelligent retry: “Retry Step 5 up to 3 times, with exponential backoff, but only if failure is transient” Cost optimization: “Run steps in parallel to save time, but respect resource budgets” Fairness: “Ensure no workflow is starved by others running parallel steps”

You end up with complex scripts that:

Are hard to understand
Break when something changes
Don’t handle errors well
Waste resources
The Solution: Socratic-Workflow
Socratic-workflow is a declarative workflow orchestration system.

Instead of writing code, you declare what needs to happen. The system figures out the best way.

Core Idea: Declarative Workflows
Old way (code):

def run_pipeline():

    try:

        user_data = extract_user_data()

    except:

        log_error("extraction failed")

        alert_human()

        return

    

    try:

        product_data = extract_product_data()

    except:

        log_error("product extraction failed")

        alert_human()

        return

    

    recommendations = generate_recommendations(user_data, product_data)

    ranked = rank_by_margin(recommendations)

    cache_recommendations(ranked)

    notify_user(ranked)

    log_completion()

New way (declarative):

workflow: recommendation_pipeline

steps:

  extract_user:

    action: extract_user_data

    on_failure: alert_human

    

  extract_product:

    action: extract_product_data

    on_failure: alert_human

    

  generate:

    action: generate_recommendations

    depends_on: [extract_user, extract_product]

    on_failure: alert_human

    

  rank:

    action: rank_by_margin

    depends_on: generate

    

  cache:

    action: cache_recommendations

    depends_on: rank

    

  notify:

    action: notify_user

    depends_on: rank

    

  log:

    action: log_completion

    depends_on: [cache, notify]

The system:

Understands dependencies
Parallelizes where possible
Handles failures correctly
Is easy to understand and modify

The Architecture: How Orchestration Works
Component 1: Workflow Definition
A workflow is a DAG (Directed Acyclic Graph):

Nodes = steps

Edges = dependencies

Example:

┌─────────────┐

│  ExtractUser │

└──────┬──────┘

       │

┌──────▼──────┐     ┌──────────────┐

│ GenerateRecs │◄────│ ExtractProduct │

└──────┬──────┘     └──────────────┘

       │

┌──────▼──────┐

│  RankByMargin │

└──────┬──────┘

       │

  ┌────┴─────┐

  │           │

┌──▼───┐ ┌──▼────┐

│ Cache │ │Notify │

└──┬───┘ └───┬───┘

  │          │

  └────┬─────┘

       │

  ┌────▼────┐

  │ LogDone  │

  └─────────┘

The system:

Identifies that ExtractUser and ExtractProduct are independent → run in parallel
Identifies that Cache and Notify only depend on RankByMargin → both can run in parallel
Respects all dependencies
Parallelizes aggressively within constraints
Component 2: Task Scheduling
Intelligent scheduler assigns tasks to workers:

Available workers: 4

Timeline:

Time 0:

├─ Worker 1: ExtractUser

├─ Worker 2: ExtractProduct

├─ Worker 3: Idle

└─ Worker 4: Idle

Time 2 (assuming both extracts take 2 min):

├─ Worker 1: GenerateRecs

├─ Worker 2: Idle

├─ Worker 3: Idle

└─ Worker 4: Idle

Time 5 (assuming generate takes 3 min):

├─ Worker 1: RankByMargin

├─ Worker 2: Idle

├─ Worker 3: Idle

└─ Worker 4: Idle

Time 7 (assuming rank takes 2 min):

├─ Worker 1: Cache

├─ Worker 2: Notify

├─ Worker 3: Idle

└─ Worker 4: Idle

Total: 9 minutes (vs 15 minutes sequential)
Component 3: Failure Handling
Each step can have strategies for different failures:

step: call_external_api

action: get_enrichment_data

retry:

max_attempts: 3

backoff: exponential

interval: 2s

on_failure:

transient: retry # Network error? Retry

permanent: skip # API doesn't have data? Skip this step

critical: abort # Critical dependency failed? Stop everything

timeout: 30s
When a step fails:

System determines failure type
Applies appropriate strategy
Continues or stops accordingly
Logs everything for debugging
Component 4: Conditional Routing
Workflows can branch based on data:

step: classify_request

action: classify_customer_type

output: customer_type

Next step depends on output

if: customer_type == "premium"

then: [premium_workflow_steps]

if: customer_type == "standard"

then: [standard_workflow_steps]

else: [default_workflow_steps]
Component 5: Human Approval Gates
Workflows can pause and wait for human approval:

step: propose_recommendation

action: generate_top_10_recommendations

output: candidates

Before proceeding, wait for human approval

approval:

required: true

timeout: 24h

reviewers: [*Emails are not allowed*, *Emails are not allowed*]

next_step: execute_recommendation # Only runs after approval
Real Examples
Example 1: Data Pipeline
workflow: daily_data_pipeline

steps:

extract:

action: extract_from_source

parallelism: 3  # Extract from 3 sources in parallel

validate:

depends_on: extract

action: validate_data

retry:

  max_attempts: 3

transform:

depends_on: validate

action: transform_data

enrich:

depends_on: transform

action: enrich_with_external_data

timeout: 30m

load:

depends_on: enrich

action: load_to_warehouse

# These run in parallel with load

notify:

depends_on: load

action: send_notification

cleanup:

depends_on: load

action: delete_temp_files

dashboard:

depends_on: load

action: update_dashboard

final:

depends_on: [notify, cleanup, dashboard]

action: log_completion

Time saved: Sequential takes 10 + 5 + 2 + 3 + 2 + 1 + 1 + 1 = 25 minutes With parallelization: 10 + 5 + 2 + 3 + 2 = 22 minutes (steps 6-8 parallel) Savings: 3 minutes per run = 15% time reduction

Example 2: Recommendation with Approval
workflow: recommendation_deployment

steps:

generate:

action: generate_recommendations

evaluate:

depends_on: generate

action: evaluate_quality

check_quality:

depends_on: evaluate

condition:

  if: evaluation.quality_score > 0.90

    then: proceed

  else: require_review

review:

condition: required

action: human_review

reviewers: [data_science_lead]

timeout: 24h

approve:

depends_on: review

condition:

  if: reviewer.approved

    then: deploy

  else: iterate

iterate:

condition: required

action: improve_recommendations

depends_on: approve

loops_back_to: evaluate

max_iterations: 3

deploy:

depends_on: approve

action: deploy_to_production

monitor:

depends_on: deploy

action: start_monitoring

Example 3: Multi-Region Deployment
workflow: deploy_to_all_regions

steps:

build:

action: build_application

test:

depends_on: build

action: run_tests

retry:

  max_attempts: 2

# Deploy to all regions in parallel

deploy_us:

depends_on: test

action: deploy

region: us

deploy_eu:

depends_on: test

action: deploy

region: eu

deploy_asia:

depends_on: test

action: deploy

region: asia

# Verify all regions

verify_all:

depends_on: [deploy_us, deploy_eu, deploy_asia]

action: run_smoke_tests

parallelism: 3  # Test all 3 regions in parallel

rollback_on_failure:

depends_on: verify_all

condition:

  if: any_test_failed

    then: rollback

rollback:

action: rollback_deployment

targets: [all_regions]

notify: [ops_team]

complete:

depends_on: [verify_all, rollback]

action: send_completion_notification

Time saved: 3 deploys in parallel = 3x faster than sequential

Implementation: Creating Workflows
Step 1: Define Workflow
from socratic_workflow import Workflow, Step

workflow = Workflow(name="recommendation_pipeline")

Define steps

workflow.add_step(

name="extract_user",

action="extract_user_data",

timeout="5m",

on_failure="alert_human"

)

workflow.add_step(

name="extract_product",

action="extract_product_data",

timeout="5m",

on_failure="alert_human"

)

workflow.add_step(

name="generate",

action="generate_recommendations",

depends_on=["extract_user", "extract_product"],

timeout="10m"

)

workflow.add_step(

name="rank",

action="rank_by_margin",

depends_on="generate"

)

workflow.add_step(

name="cache",

action="cache_recommendations",

depends_on="rank"

)

workflow.add_step(

name="notify",

action="notify_user",

depends_on="rank"

)

workflow.add_step(

name="complete",

action="log_completion",

depends_on=["cache", "notify"]

)
Step 2: Execute Workflow

Run the workflow

execution = await workflow.execute()

Monitor progress

print(execution.status) # "running"

print(execution.progress) # 0.3 (30% complete)

print(execution.current_steps) # ["extract_user", "extract_product"]
Step 3: Handle Completion

Wait for completion

result = await execution.wait()

print(result.status) # "completed" or "failed"

print(result.duration) # 11 minutes 30 seconds

print(result.results) # {step_name: result_data}
The Philosophy: Express Intent, Not Implementation
Traditional workflows force you to express how work happens.

“Do step 1, then step 2, then step 3, handling failures like this…”

Socratic-workflow lets you express what needs to happen.

“These 3 things need to happen. These 2 depend on those. If this fails, do that.”

The system figures out the best how:

Which things can run in parallel
How to retry failures
How to handle timeouts
How to allocate resources
You focus on intent. The system handles execution.

Use Socratic-Workflow for Orchestration
The workflow orchestration system described in this post is production-ready and open source:

GitHub: https://github.com/Nireus79/Socrates
PyPI Package: https://pypi.org/project/socratic-workflow/
Documentation: https://github.com/Nireus79/Socrates/tree/main/socratic-workflow

Quick Start

Install the workflow system

pip install socratic-workflow

from socratic_workflow import Workflow

Create a workflow

workflow = Workflow(name="my_pipeline")

Add steps

workflow.add_step(

name="extract",

action="extract_data",

timeout="10m"

)

workflow.add_step(

name="transform",

action="transform_data",

depends_on="extract"

)

workflow.add_step(

name="load",

action="load_data",

depends_on="transform"

)

Run it

execution = await workflow.execute()

result = await execution.wait()

print(f"Completed in {result.duration}")

print(f"Status: {result.status}")
Full examples and documentation: https://github.com/Nireus79/Socrates

The Complete Socratic Ecosystem
Socratic-workflow is part of the complete Socrates AI system (11 modules, 2,300+ tests):

Socratic-nexus: Multi-provider LLM client
Socratic-morality: Constitutional governance with 13 modules
Socratic-agents: Multi-agent orchestration with conflict resolution
Socratic-knowledge: Enterprise RAG with multi-tenancy
Socratic-learning: Self-improving agents
Socratic-analyzer: Code quality analysis
Socratic-performance: Real-time monitoring
Socratic-workflow: Workflow orchestration(this post)
Socratic-conflict: Conflict resolution between agents
Socratic-docs: Auto-documentation
Socratic-maturity: Project maturity tracking
All modules work together seamlessly. Use individual packages or the complete platform.

Available on PyPI under MIT License: https://pypi.org/user/Nireus79/

Anatomy of a Kestra Flow: How It All Fits Together

Thu, 19 Mar 2026 15:48:35 +0000

Join Developer Advocate Amara Graham as she walks through 4 example flows of increasing complexity to show you different workflow components, required properties, and tactics for making your flows easier to read and maintain.

Try Kestra for free: https://go.kestra.io/yt/try-kestra

All flows in this video, except the hello world boilerplate, are available as blueprints:
log_off_flow (available in two options: Notion or Confluence)
With Notion
With Confluence
airbyte-sync-parallel-with-dbt
argocd-single-tenant-wave-rollout

Getting Started with the Vercel AI SDK (Beginner to Working Chat App)

Wed, 21 Jan 2026 12:44:06 +0000

If you’re building AI features like chatbots, content generators, AI agents, or streaming assistants, you’ll quickly realize that the “LLM call” is only 20% of the work.

The other 80% is:
✅ streaming responses
✅ UI updates in real-time
✅ tool calling
✅ handling errors & retries
✅ server/client integration
✅ building a clean developer experience

That’s where the Vercel AI SDK shines.

In this blog, you’ll learn how to get started with the Vercel AI SDK and build a simple streaming chat app using Next.js.

What is the Vercel AI SDK?

The Vercel AI SDK is a set of libraries and utilities to build AI-powered apps with great UX:

Streaming responses (token-by-token)
React hooks for chat and completion
Easy server routes (Next.js App Router)
Tool/function calling support
Provider integrations (OpenAI, etc.)

It’s designed for modern full-stack apps, especially in the Next.js ecosystem.

What You’ll Build

By the end, you’ll have:

✅ A Next.js app
✅ A chat UI
✅ An API route that streams AI responses
✅ Working end-to-end in minutes

Step 1: Create a Next.js App

Run:

npx create-next-app@latest vercel-ai-sdk-demo
cd vercel-ai-sdk-demo
npm run dev

Choose:

TypeScript ✅
App Router ✅
Tailwind (optional but recommended) ✅

Step 2: Install the Vercel AI SDK

Install the SDK:

npm install ai

Then install an AI provider package (OpenAI is common):

npm install @ai-sdk/openai

Step 3: Add Your API Key

Create .env.local:

OPENAI_API_KEY=your_api_key_here

Restart the dev server after adding env vars.

Step 4: Create the Streaming Chat API Route

In Next.js App Router, create:

app/api/chat/route.ts

import { openai } from "@ai-sdk/openai";
import { streamText } from "ai";

export async function POST(req: Request) {
  const { messages } = await req.json();

  const result = await streamText({
    model: openai("gpt-4o-mini"),
    messages,
    system: "You are a helpful assistant. Keep answers short and clear.",
  });

  return result.toDataStreamResponse();
}

What’s happening here?

messages contains the full chat history
streamText() streams the model output
toDataStreamResponse() returns a streaming response to the UI

Step 5: Build the Chat UI with `useChat()`

Now create a UI page:

app/page.tsx

"use client";

import { useChat } from "ai/react";

export default function Home() {
  const { messages, input, handleInputChange, handleSubmit, isLoading } =
    useChat({ api: "/api/chat" });

  return (
    <main style={{ maxWidth: 700, margin: "40px auto", padding: 16 }}>
      <h1 style={{ fontSize: 24, fontWeight: 700 }}>Vercel AI SDK Chat</h1>

      <div style={{ marginTop: 20, border: "1px solid #ddd", padding: 16 }}>
        {messages.map((m) => (
          <div key={m.id} style={{ marginBottom: 12 }}>
            <strong>{m.role}:</strong> {m.content}
          </div>
        ))}
      </div>

      <form onSubmit={handleSubmit} style={{ marginTop: 16, display: "flex" }}>
        <input
          value={input}
          onChange={handleInputChange}
          placeholder="Ask something..."
          style={{ flex: 1, padding: 10, border: "1px solid #ccc" }}
        />
        <button
          type="submit"
          disabled={isLoading}
          style={{ marginLeft: 8, padding: "10px 14px" }}
        >
          Send
        </button>
      </form>
    </main>
  );
}

Step 6: Run It

Start your app:

npm run dev

Open:
http://localhost:3000

Now type a message and watch the AI respond in real-time.

Understanding `messages`

messages is an array like:

[
  { role: "user", content: "Hello" },
  { role: "assistant", content: "Hi! How can I help?" }
]

The SDK manages:

appending new messages
streaming assistant messages
updating UI automatically

Add a Better UI (Optional)

You can style messages by role:

{messages.map((m) => (
  <div
    key={m.id}
    style={{
      padding: 12,
      borderRadius: 12,
      marginBottom: 10,
      background: m.role === "user" ? "#f2f2f2" : "#e7f5ff",
    }}
  >
    <strong>{m.role}:</strong> {m.content}
  </div>
))}

Bonus: Add Tool Calling (Basic Example)

Tool calling allows your AI to call functions like:

search database
fetch weather
calculate tax
call APIs

Example concept:

tools: {
  getTime: {
    description: "Get current server time",
    execute: async () => new Date().toISOString(),
  },
}

This is where Vercel AI SDK becomes powerful for agentic workflows.

Common Issues & Fixes

❌ “OPENAI_API_KEY missing”

Fix:

Ensure .env.local exists
Restart dev server
Check variable name

❌ No streaming happens

Fix:

Use useChat() properly
Ensure API route returns toDataStreamResponse()

❌ Build fails on deployment

Fix:

Add env var in Vercel dashboard
Ensure route uses server runtime properly

Production Tips (Best Practices)

✅ Keep prompts stable

Use a clear system prompt and avoid changing it too often.

✅ Limit message history

For performance, send only last N messages.

✅ Add safety

Filter harmful content or enforce policies.

✅ Add logging

Log prompts, latency, and tool usage.

What to Build Next

Once you have a basic chat working, you can extend it into:

AI Customer Support Assistant
RAG-powered Knowledge Bot
AI Agent for Jira / GitHub tasks
Meeting Summary Generator
AI Form Auto-Fill Assistant

Conclusion

The Vercel AI SDK is one of the fastest ways to build AI apps with excellent streaming UX and clean developer experience.

If you’re building in Next.js, this is a must-try toolkit.

Note: Create agents using Vercel
https://emilyxiong.medium.com/create-an-ai-agent-with-vercel-ai-sdk-e690b807eb2a

Sprint Workflow: A Beginner's Guide to Agile Development

Tue, 28 Oct 2025 09:33:28 +0000

In this article, we explore the sprint process in software development. You'll learn to manage tasks using a sprint board, track progress, and enhance team efficiency for successful project deliveries!

What Is a Sprint?

A sprint is a short, focused period of work during which a development team tackles specific tasks or goals (for my team, it’s two-week periods). In modern software projects, sprints help teams stay organized, track progress, and deliver updates efficiently.

What is Agile Development

Agile development is a way of building software that focuses on flexibility, teamwork, and continuous improvement. Instead of planning everything up front, teams work in short sprints to deliver small, usable project pieces. After each sprint, the team reviews what went well and what can be improved, making it easier to adapt to changes and deliver better results. Agile helps teams stay organized, respond quickly to feedback, and progress projects.

What is a Sprint Board & Ticket?

I like explaining a ticket by analogy—think of a ticket as an index card pinned to a corkboard (Sprint Board). In the following example, the corkboard has five column headers labeled as follows:

To Do
In Progress
In Review
Testing
Done

The index card lists tasks and moves through each column of the corkboard as the tasks are completed.

To Do	In Progress	In Review	Testing	Done
(TICKET)

To Do

Let’s write two tasks on the imagined index card:

Update the README file to include the new login feature
Add a URL link to the deployed project.

The provided index card (sprint ticket) will have an auto-generated number; let’s say it’s 12.

Now, I will assign YOU to complete ticket number 12. After reviewing the assigned tasks, you will let the team know how long it will take you to complete. This is done using a point system.

A point represents a time variable that can differ among teams. Our team’s point system is as follows:

point-0: within 2 hours
point-1: within 1/2 day
point-3: within 1 day
point-5: within 3 days
point-8: within 1 week

Since this is your first ticket, a full day will be acceptable for the task as you will need time to write documentation for the new login procedure and add a URL to the project. Therefore, you should mark your ticket with 3 points (which represents a day).

In Progress

Once you start working on your assigned ticket, you move it to the “In Progress“ column. When you complete it, you move it to the “In Review“ column and notify your team.

In Review

The review phase of your ticket involves an assigned team member reviewing it to ensure that all of the assigned tasks were completed. In this demonstration, you needed to write documentation in the README file detailing the new login procedure and adding a URL to the project.

⚠ Important: You are NOT to go beyond the tasks on your ticket! For example, you would not add additional information to the README file regarding other aspects of the project. Although your intentions are good, since it goes beyond the scope of the ticket, it will be refused because it will cause confusion and disorganization. Further README file updates should be added to new future tickets.

Testing

Once the ticket passes the review stage, it is moved to testing to ensure the code functions properly; in this simple example, the project link must be clickable and direct to the correct URL. If your ticket passes the testing phase, it is closed and moved to the “Done“ column.

Done

The done column is filled with completed tickets, showcasing all of the project's updates. Great work on completing your ticket and properly maintaining a professional team workflow using a sprint ticket and a sprint board!

Additional Sprint Tasks

Now that we have covered a simple version of a sprint ticket and a sprint board, I can offer you more details of my team’s process.

Additional Sprint Board Columns:

No Status: Tasks or tickets that haven’t been assigned a workflow stage yet.
Back Log: A list of tasks or ideas waiting to be worked on in future sprints.
Epics: Large, high-level tasks or projects broken down into smaller tickets.
Blocked: Tasks that cannot move forward due to an issue or dependency.

Leftover Tickets

What do we do with leftover tickets that weren’t completed during the sprint? They are usually moved back to the backlog or carried over to the next sprint for further work. Reviewing leftover tickets helps the team understand what caused delays and improves future planning, ensuring that important tasks aren’t forgotten.

Team Meetings

Stand-ups: Short daily meetings where team members share progress, plans, and blockers.
Weekly Meetings: Longer meetings held once a week to review progress, plan upcoming work, and discuss any issues.
One-on-one: Private meetings between two team members (often a manager and developer) to discuss individual goals, feedback, or concerns.

Future Sprint Planning

Sprint planning is the meeting where the team decides which tasks or tickets will be tackled in the upcoming sprint. During this session, team members review the backlog, estimate the effort required for each item, and agree on the sprint goals. The result is a clear plan that outlines what will be accomplished and who is responsible for each task, helping everyone stay focused and aligned throughout the sprint.

About Us

What is Gridiron Survivor?

Gridiron Survivor is an apprenticeship program created by Shashi Lo, a Senior UX Engineer at Microsoft. It aims to provide developers entering the tech industry with vital work experience. The program focuses on practical training in project management, coding practices, and team collaboration, offering mentorship and skills essential for success in their initial tech roles.

What is the Elfgorithm App?

Elfgorithm is an AI-driven Secret Santa app set to launch in winter 2025. It streamlines gift exchanges by removing the guesswork from Secret Santa activities. The app manages gift-giving details and provides personalized gift suggestions, ensuring you find the perfect presents for everyone.

Gridiron Survivor Sponsors

A very special thanks to our sponsors!

GitKraken: A popular Git client that provides a graphical interface to manage Git repositories. It is known for its user-friendly design and features that simplify version control, making it easier for developers to collaborate and manage their code.
Frontend Mentor: An online platform that offers front-end coding challenges. It helps developers improve their skills by providing real-world projects to work on, along with a supportive community for feedback and learning.
Vercel: A cloud platform for static sites and serverless functions. It is designed to optimize the workflow of developers by providing tools for building, deploying, and scaling modern web applications with ease. Vercel is known for its seamless integration with frameworks like Next.js.

Be sure to listen to the HTML All The Things Podcast!

???? I also write articles for the HTML All The Things Podcast, which you can read on their website: https://www.htmlallthethings.com/.

Be sure to check out HTML All The Things on socials!

Affiliate & Discount Links!

With CodeMonkey, learning can be all fun and games! CodeMonkey transforms education into an engaging experience, enabling children to evolve from tech consumers to creators. Use CodeMonkey's FREE trial to unlock the incredible potential of young tech creators!

With a structured learning path tailored for various age groups, kids progress from block coding to more advanced topics like data science and artificial intelligence, using languages such as CoffeeScript and Python. The platform includes features for parents and teachers to track progress, making integrating coding into home and classroom settings easy.

Through fun games, hands-on projects, and community interaction, CodeMonkey helps young learners build teamwork skills and receive recognition for their achievements. It fosters a love for coding and prepares children for future career opportunities in an ever-evolving tech landscape.

To learn more about CodeMonkey, you can read my detailed review article!

Affiliate Links:

Advance your career with a 20% discount on Scrimba Pro using this affiliate link!

Become a hireable developer with Scrimba Pro! Discover a world of coding knowledge with full access to all courses, hands-on projects, and a vibrant community. You can read my article to learn more about my exceptional experiences with Scrimba and how it helps many become confident, well-prepared web developers!

Important: This discount is for new accounts only. If a higher discount is currently available, it will be applied automatically.

How to Claim Your Discount:

Click the link to explore the new Scrimba 2.0.
Create a new account.
Upgrade to Pro; the 20% discount will automatically apply.

Disclosure: This article contains affiliate links. I will earn a commission from any purchases made through these links at no extra cost to you. Your support helps me continue creating valuable content. Thank you!

Conclusion

A sprint is a short, focused period of work during which a development team completes specific tasks or goals. Sprints are an essential part of Agile development, a method of software creation that stresses flexibility, teamwork, and continuous improvement.

Sprint tasks are tracked using tickets that include detailed instructions. As work progresses, these tickets are moved across titled columns on a Sprint Board—To Do, In Progress, In Review, Testing, and Done, among others. Each ticket is assigned an estimated completion time by the assignee using a point system. Points represent time intervals that can differ between teams, such as point-1 for half a day, point-3 for one day, and point-5 for three days. Leftover tickets that weren’t completed during a sprint are typically moved back to the backlog or carried over to the next sprint for further work.

Regular team meetings keep everyone aligned and productive. Daily stand-ups help share progress and blockers, weekly meetings review progress and plan ahead, and one-on-ones focus on individual goals and feedback. Future sprint planning prepares the team for the next cycle by reviewing the backlog, estimating tasks, and setting clear goals. This ongoing process helps teams stay organized, adaptable, and successful.

Ready to implement Agile? Begin planning your next sprint and discover how this workflow can enhance your team’s productivity and project success!

Let’s connect! I’m active on LinkedIn and Twitter.

Unlocking AI-Native Development: Key Highlights from GitHub Galaxy 2025

Sun, 10 Aug 2025 19:31:41 +0000

Introduction
Recently, GitHub hosted Galaxy 2025, an all-virtual event spotlighting the future of AI-native software development. This gathering offered a firsthand look at how GitHub is redefining developer workflows — from enterprise-grade security to seamless AI-powered collaboration.

Highlights from Galaxy 2025

Keynote: Where AI Meets Enterprise Excellence
GitHub emphasized AI-native and agentic experiences, underscoring the importance of developer choice and robust security in modern software development ([GitHub Galaxy][2]).
Hands-On Secure Workflows
The event spotlighted secure DevOps workflows in action — showcasing tools that streamline collaboration while addressing vulnerabilities at scale.
Product Roadmap & GitHub Vision
Attendees gained insights into GitHub’s strategic vision and upcoming features designed to empower teams with AI-powered toolchains and automation.
Agentic AI & Developer Flow
Topics such as agentic AI, maintaining developer focus, and enhancing productivity at scale were central to discussions — critical as organizations strive to innovate faster than ever ([GitHub Universe][3], [X (formerly Twitter)][4]).

Why It Matters for Your Organization

GitHub Galaxy 2025 wasn’t just an event—it was a glimpse into the future of software development. Whether you're leading development teams or shaping digital strategies, the insights shared are a blueprint for staying competitive and secure in an AI-forward world.

Final Thoughts

The Galaxy logo itself—dynamic and charged with energy—reflects GitHub’s bold direction toward AI-native development. Let this be your inspiration to explore how AI-first workflows and enterprise-grade security can elevate your development lifecycle.

For more info, check out this link:
https://galaxy.github.com/

Let’s stay connected:

Instagram: https://www.instagram.com/angular_development/

Facebook: https://m.facebook.com/learnangular2plus/

Threads: https://www.threads.net/@angular_development

Medium: https://medium.com/@eraoftech

coderlegion: https://coderlegion.com/user/Sunny

Quora: https://neweraofcoding.quora.com/

YouTube: https://www.youtube.com/@neweraofcoding

LinkedIn: https://www.linkedin.com/company/infowebtech/

Hashnode: https://neweraofcoding.hashnode.dev/

GitHub: https://github.com/angulardevelopment/ | sunny7899

BlueSky: https://bsky.app/profile/neweraofcoding.bsky.social

Substack Newsletter: https://codeforweb.substack.com/

Pinterest: https://in.pinterest.com/tech_nerd_life/

dev.to: https://dev.to/sunny7899

Looking for web dev trainings: https://beginner-to-pro-training.vercel.app/

Software development services: https://infowebtechnologies.vercel.app/

Contribution to the web development community: https://code-for-next-generation.vercel.app/

Book a session: https://topmate.io/softwaredev

Telegram Channel: https://t.me/neweraofcoding

Slack Community: Invite

Discord Community: http://discord.gg/Nuc9YRngHz

Buy me a coffee on Ko-fi: https://ko-fi.com/softwaredev

Ebooks: https://apexsunshine.gumroad.com

Thank you for being a part of the community. Happy coding!

Two Ways to Reduce Bottlenecks with the Theory of Constraints

Mon, 30 Jun 2025 12:26:04 +0000

Working out how to apply Theory of Constraints in real-world situations can be hard, but there are two simple ways to kickstart it.

A chain is only as strong as its weakest link. That’s the idea behind the Theory of Constraints. Just a few weak links will limit any system, but you can improve the whole system’s performance by focusing on improving those links.

Whether your system is a software delivery process, a large production line or your household chores, the only way to improve the whole system is to work at the bottlenecks.

Theory Is Great, in Theory

There are two ways to learn about the Theory of Constraints. You can read Eliyahu Goldratt’s business novel, The Goal, or read the accompanying handbook, Theory of Constraints.

Either option explains the set of focusing steps you use to apply the theory.

Identify the system’s constraint.
Decide how to exploit the constraint.
Subordinate everything else to this decision.
Elevate the constraint.
If the constraint resolves, go back to step one.

The theory is wide-reaching. It can be applied to any system. For example, a production line in a factory, managing software delivery or a group of Scouts trying to hike to their destination with their equipment. Because of this, the language is likely to be a little abstract for your real-world situation.

Specifically, the terms exploit, subordinate and elevate might be unclear.

Let’s simplify the language a little, then look at two practical ways to apply the Theory of Constraints to your work.

Exploit, Subordinate and Elevate

A “constraint” in the Theory of Constraints is essentially a bottleneck. You are trying to flow work to completion to get some value, and many steps are involved, possibly many people. If you look closely at this flow of work, you’ll find there is one stage where things get stuck.

You might be making computers and have a pile of almost finished products stacked, waiting for someone to install a graphics card. Well, the graphics card stage is your bottleneck or constraint.

You can spot this problem thanks to the pile of computers sitting in front of the graphics card workstation, but you’ll also notice workstations after the bottleneck are starved of work while those ahead of it are producing more laptops that only add to the size of the queue.

If your work is less physically visible, having a visual system like a Kanban board can show you the problem. Seeing work items stuck in a queue in front of a specific task shows you ther’s a bottleneck.

When you can see this bottleneck, you’re ready to apply the three mystical terms.

You can exploit the constraint by improving it with existing people and resources. If you can transfer someone to work at the constraint, your overall production (finished laptops, software features or clean clothes) will increase.

You then subordinate everything else to the pace of your constraint. That means setting the speed of the upstream process to match the speed of the bottleneck. If you can fit 10 graphics cards into laptops in a day, you can assemble only 10 laptops a day. Everything from start to finish runs at the pace of the stage with the bottleneck.

Once the system is running at a matched speed, you look at ways to elevate the constraint. That means finding ways to increase the throughput from 10 graphics cards daily to something more than 10. Assuming you’re not using existing people and resources to elevate the constraint, you’ll need to look at the costs and benefits of increasing throughput. If you doubled the size of the team or introduced tools to support the process to increase throughput to 20 graphics cards a day, how much would you invest to get those extra sales?

You continue to apply this three-stage process until the constraint moves. For example, you are producing 20 graphics a day while selling only 18 laptops a day. Now you apply the focusing steps to the sales stage instead.

This theory can be applied to physical or knowledge work. Work is like water flowing down a canal with many locks. You can’t allow the water from higher stretches to flood the lower ones or allow a section of the canal to run dry.

Let’s look at two practical ways to apply these focusing steps in real life.

Start on the Right

In software development, work is usually visualized on a task board that follows work as it flows from left to right. The columns on the board represent queues and work stages, such as “waiting for testing” and “in test.”

To apply the Theory of Constraints, first ensure the task board accurately represents every hand-off and queue in your process. Once you have represented the real flow of work, the board will quickly show you where bottlenecks are.

You can apply the focusing steps to the bottleneck, but there’s a simpler option for cross-functional teams. Always work on the right-most work item before anything else. If a feature is ready to deploy to production, deploy that before testing the feature to the left.

By always progressing the work closest to completion, you get the exploit and subordinate focusing steps with no effort. Your cross-functional team is self-leveling, even for dynamic bottlenecks. The speed of the system is automatic, and the team will apply their time flexibly to the tasks that require their attention.

When using this approach, you just need to keep inertia out of your improvement process. You won’t have a stack of work telling you what needs to be improved, so you’ll need to pay attention to which tasks must be improved — or elevated, in Theory of Constraints terminology.

Sometimes, the simplicity of this approach is what makes it so powerful.

Schedule to the Bottleneck

If you don’t have a cross-functional team, you can schedule to the bottleneck. For example, if you have a separate testing team, you should produce features only at the speed they can be tested without increasing the queue size.

This is where adding work in process (WIP) limits to your task board can help. The most important WIP limit is the one for the queue in front of the bottleneck, but you may need to add limits to other columns to the left of the constraint. You don’t need limits after the bottleneck; if you find yourself adding WIP limits on the right, it means your constraint has moved.

When you schedule to the bottleneck, ideally you want to provide work to the constraint just as they need the next task. However, most teams use a buffer to make sure the bottleneck is never idle.

Scheduling to the bottleneck takes more effort than starting on the right. However, its one advantage is highlighting the constraint very clearly. Nobody on the team has any doubts where improvement efforts should be focused.

The Powerful “Aha” Moment

The reason the Theory of Constraints has such an impact is that it highlights how all the work you do above what your bottleneck can handle is wasted. In the laptop example, no amount of assembled laptops you produce makes a difference to how many you sell if you can’t complete them with a graphics card.

The answer can be as simple as moving someone from assembly to graphics installation, or you may need better tools to support this stage in the process. The Theory of Constraints sets you on the right path and provides concrete motivation for improving the right thing.

Mastering Web Scraping with Python

Thu, 27 Jun 2024 21:44:52 +0000

Data gathering is made possible via web scraping, which automates the process of extracting information from websites. Powerful libraries like Scrapy, BeautifulSoup, and Selenium, together with its straightforward syntax, make Python the best choice. For analysis purposes, such as pricing tracking, review gathering, or news tracking, proficiency with web scraping is essential. In this article, we'll cover web scraping basics, usage of tools like BeautifulSoup, Selenium, and Scrapy and Requests for API scraping.

Understanding Web Scraping

Web scraping makes use of a variety of tools and techniques to extract information.

Tools

BeautifulSoup: This Python module parses HTML and XML documents and can easily extract information by building a parse tree from the source code of webpages. Learn more here.
Scrapy: It is an effective Python library that uses self-contained crawlers called spiders, that extract data from websites. Learn more here.
Selenium: It's a browser automation tool that runs a web browser in a regular manner by scraping dynamic material generated using JavaScript. Learn more here.
Requests: It is an HTTP library for Python that allows you to send and handle HTTP requests and responses. Learn more here.

Legal Considerations

Many websites' Terms of Service (TOS) specifically ban scraping.
Scraping data may breach copyright laws by extracting and using material without authorization.
Always examine and respect a website's robots.txt file, which specifies which portions of the site can and cannot be scraped.

Installing and Configuring Web Scraping Libraries

Note: The following codes are run on the terminal or in a code editor or IDE like Visual Studio Code.

The libraries can be installed as follows:

BeautifulSoup:
```
  pip install beautifulsoup4
```
Scrapy:
```
  pip install scrapy
```
Selenium:
```
  pip install selenium
```
Requests:
```
  pip install requests
```

Caution: Make sure `pip` is installed on your system.

Working in a virtual environment is recommended since it reduces conflicts with other projects and helps manage dependencies.

Creating a virtual environment:

python -m venv sampenv

Activation:

On Windows:
```
  sampenv\Scripts\activate
```
On Linux and macOS:
```
  source sampenv/bin/activate
```

Tip: Working in a virtual environment is recommended since it reduces conflicts with other projects and helps manage dependencies.

Basic Web Scraping with BeautifulSoup

BeautifulSoup is a Python library that works on HTML and XML files to extract data by creating a parse tree from a page source to traverse and find elements, attributes or text using methods like find() and find_all().

Extracting Tags

You can extract HTML tags like <p>, <div>, <a>, etc. using find() and find_all() methods.
Syntax:

find(): soup.find(tag_name, attrs={})
find_all(): soup.find_all(tag_name, attrs={})

Extracting Attributes

HTML tag attributes can be extracted using the get() method or a dictionary-style access from the find() method.
Syntax:

tag = soup.find('tag_name')
get(): tag.get('attr_name')
Dictionary-style access: tag['attr_name']

Extracting Text

Text contents of HTML elements can be accessed using the get_text() method or the .text attribute.
Syntax:

tag = soup.find('tag_name')
get_text(): tag.get_text()
.text: tag.text

Let's take a look at an example to understand the methods better:

from bs4 import BeautifulSoup

html_content = """
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Grocery List</title>
</head>
<body>
    <div id="content">
        <h1>Welcome to my List</h1>
        <p>This is a list of my <strong>grocery items</strong>.</p>
        <ul>
            <li>Milk</li>
            <li>Bread</li>
            <li>Carrots</li>
            <li>Butter</li>
        </ul>
        <a href="https://samp.com">Sample</a>
    </div>
</body>
</html>
"""

soup = BeautifulSoup(html_content, 'html.parser')
title_tag = soup.find('title')     # extracts Title tag
h1_tag = soup.find('h1')     # extracts H1 tag
p_tag = soup.find('p')
ul_tag = soup.find('ul')
a_tag = soup.find('a')

print("Title:", title_tag.text)    # extracts text from Title tag
print("H1:", h1_tag.get_text())    # extracts text from H1 tag
print("Paragraph:", p_tag.text)
print("List Items:")
for li in ul_tag.find_all('li'):    # extracts list items
    print("-", li.text)
print("Link Href:", a_tag['href'])    # displays 'href' attribute's value

The code contains a sample HTML script, from which tags, attributes and texts are extracted.

Output:

Scraping Dynamic Websites with Selenium

Selenium is a great tool for scraping dynamic webpages using JavaScript. It enables button clicks, form submissions, and page navigation by allowing programmatic control of a web browser. In order to use Selenium, you need to set up Selenium WebDriver for your browser.

Google Chrome

Download ChromeDriver from chromedriver.chromium.org

Set up ChromeDriver:

  from selenium import webdriver
  from selenium.webdriver.chrome.service import Service
  from webdriver_manager.chrome import ChromeDriverManager
  service = Service(ChromeDriverManager().install())
  driver = webdriver.Chrome(service=service)

Mozilla Firefox

Download GeckoDriver from github.com/mozilla/geckodriver/releases.

Set up GeckoDriver:

  from selenium import webdriver
  from selenium.webdriver.firefox.service import Service
  from webdriver_manager.firefox import GeckoDriverManager
  service = Service(GeckoDriverManager().install())
  driver = webdriver.Firefox(service=service)

Microsoft Edge

Download EdgeDriver from developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/.

Set up EdgeDriver:

  from selenium import webdriver
  from selenium.webdriver.edge.service import Service
  from webdriver_manager.microsoft import EdgeChromiumDriverManager
  service = Service(EdgeChromiumDriverManager().install())
  driver = webdriver.Edge(service=service)

Extracting Elements

find_element() Method

To discover a single web element that meets the given criteria, use the find_element() function.

Syntax:

element = driver.find_element(By.ID, 'ID_value')

find_elements() Method

Use the find_elements() function to find numerous web items that fit certain criteria. It will return a list of WebElement objects that represent all matched things.

Syntax:

elements = driver.find_elements(By.CLASS_NAME, 'Class_name_val')

get_attribute() Method

It retrieves the value of an attribute like href, src, etc. from a web element.

Syntax:

link = driver.find_element(By.TAG_NAME, 'Tag_value')
href_value = link.get_attribute('attr_value')

text Property

It extracts text content from elements like paragraphs, headings, or list items.

Syntax:

element = driver.find_element(By.CLASS_NAME, 'Class_val')
txt = element.text

send_keys() Method

This method is helpful for input fields since it mimics typing into text fields or search boxes.

Syntax:

inp_box = driver.find_element(By.NAME, 'Name_value')
inp_box.send_keys('Hello!')

click() Method

This method is used to click buttons, links, checkboxes, or any clickable element.

Syntax:

button = driver.find_element(By.ID, 'ID_value')
button.click()

execute_script() Method

JavaScript code for advanced interactions, such as scrolling and event triggering, is executed via this method.

Syntax:

driver.execute_script("your_JS_code")
Example for scrolling:
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")

Let's take a look at an example to understand it better:

samplepage.html:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Sample Page for Selenium</title>
    <script>
        document.addEventListener('DOMContentLoaded', function() {
            document.getElementById('dynamicButton').addEventListener('click', function() {
                document.getElementById('dynamicContent').innerHTML = '<p class="dynamicText">I hope you are doing well.</p>';
                document.getElementById('dynamicContent').innerHTML += '<a href="https://samp.com" class="dynamicLink">Example Link</a>';
            });
        });
    </script>
</head>
<body>
    <h1 id="pageTitle">Welcome!</h1>
    <input type="text" id="inputField" name="inputName" placeholder="Enter something...">
    <button id="dynamicButton">Load Content</button>
    <div id="dynamicContent"></div>
</body>
</html>

Python script for data extraction:

from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from webdriver_manager.chrome import ChromeDriverManager

service = Service(ChromeDriverManager().install())
driver = webdriver.Chrome(service=service)
driver.get("samplepage.html")    # opens the webpage

wait = WebDriverWait(driver, 20)
page_title = wait.until(EC.presence_of_element_located((By.ID, 'pageTitle')))     # waits for 20 sec for page to load
page_title = driver.find_element(By.ID, 'pageTitle')
print("Page Title:", page_title.text)
input_field = driver.find_element(By.NAME, 'inputName')
input_field.send_keys('Hello, Alayne!')    # enters given data
print("Input Field Value:", input_field.get_attribute('value'))
dynamic_button = driver.find_element(By.ID, 'dynamicButton')
dynamic_button.click()    # clicks the button
wait = WebDriverWait(driver, 10)
dynamic_text = wait.until(EC.presence_of_element_located((By.CLASS_NAME, 'dynamicText')))
print("Dynamic Text:", dynamic_text.text)
dynamic_link = driver.find_element(By.CLASS_NAME, 'dynamicLink')
print("Dynamic Link Href:", dynamic_link.get_attribute('href'))
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")    # scrolls
driver.quit()    # close the browser

The code extracts data from the given JavaScript file and also enters data into the text field, clicks a button and scrolls through the webpage.

Output:

Scraping with Scrapy

A Python-based framework for online crawling and scraping, Scrapy provides a number of tools for effectively extracting data from webpages. It effortlessly pulls data, follows links, and manages requests.
An item is a container that item pipelines use to process and store scraped data. An item pipeline component is any Python class that implements a method on items.

Let's understand how to set up and define pipelines in for a Scrapy project using an example. We will be scraping this webpage

Setting up Scrapy Projects and Defining Item Pipelines

Open your terminal and go to the directory where you want to create a Python project.
Run the following command to create a Scrapy project consisting of files necessary for configuring the project.
```
 scrapy startproject scrapy_test
```

This creates a Scrapy project named scrapy_test.

Go to items.py in the scrapy_test folder. Here, you'll define a Scrapy item class, acting as a blueprint for your extracted data. Use scrapy.Field() to create field instances capable of storing any data type.
```
 import scrapy
 class  ScrapyTestItem(scrapy.Item):
 text = scrapy.Field()
 author = scrapy.Field()
 tags = scrapy.Field()
```

This code creates text, author and tags fields to store data from the webpage.

To set up the project settings, navigate to settings.py. This is where you design and configure the order of execution for the scraped item processing pipelines. Lower values indicate higher priority.
```
 ITEM_PIPELINES = {
 'scrapy_test.pipelines.JsonWriterPipeline': 300,
 'scrapy_test.pipelines.HtmlWriterPipeline': 800,
 }
```

This ensures that JsonWriterPipeline executes before HtmlWriterPipeline. These pipelines are defined in the next step.

Navigate to pipelines.py. This file handles data processing, including cleansing, verifying, storing, and exporting. Multiple pipelines can be defined here.

 import json
 class JsonWriterPipeline:
     def open_spider(self, spider):
         self.file = open('quotes.json', 'w')

     def close_spider(self, spider):
         self.file.close()

     def process_item(self, item, spider):
         line = json.dumps(dict(item), ensure_ascii=False) + "\n"     # extracted data is converted to JSON formatted string, and non-ASCII characters are handled properly.
         self.file.write(line)
         return item

 class HtmlWriterPipeline:
     def open_spider(self, spider):
         self.file = open('quotes.html', 'w')
         self.file.write('<html><body><ul>')

     def close_spider(self, spider):
         self.file.write('</ul></body></html>')
         self.file.close()

     def process_item(self, item, spider):
         self.file.write(f"<li><p>{item['text']}</p><p>{item['author']}</p><p>{', '.join(item['tags'])}</p></li>")     
         return item

This code formats the fields and stores the extracted data into two files, quotes.json and quotes.html.

Open scrapy_test/spiders, then create scrapy_code.py. This file will include spider classes to define URLs to crawl, process answers, and retrieve data using XPath or CSS selectors. The item classes in items.py are used by the spider.

 import scrapy
 from scrapy_test.items import ScrapyTestItem

 class QuotesSpider(scrapy.Spider):
     name = 'quotes'
     start_urls = [http://quotes.toscrape.com]

     def parse(self, response):
         for quote in response.css('div.quote'):
             item = ScrapyTestItem()
             item['text'] = quote.css('span.text::text').get()
             item['author'] = quote.css('span small::text').get()
             item['tags'] = quote.css('div.tags a.tag::text').getall()
             yield item

This code defines a spider named quotes. Using a CSS selector, it selects all elements with the class quote and creates a ScrapyTestItem instance for each. The get() method retrieves the text content and stores it in the respective fields.

In your terminal, navigate to the root directory of the project (scrapy_test) and run the command:
```
 scrapy crawl quotes
```

Where quotes is the spider we created.

The extracted data is stored in quotes.json and quotes.html in the project's root directory.

Output:

Scraping APIs and JSON Data

Scraping APIs provide an organized way to obtain structured data. These APIs make endpoints available for accessing structured data in formats like XML and JSON. Using Requests, you may retrieve the required data by submitting queries to these endpoints and interpreting the returned responses.
Accessing APIs might involve the usage of API keys or tokens. Keys or tokens are specified in the request header.

Making HTTP Requests and Accessing APIs

Acquire the API key or token by generating the key/token in the platform of your choice.

Define the URL of the webpage you want to scrape and include the key/token in the request header. Also mention the content type.

 url = https://website_name.com/data
 api_key = 'abcxyz'
 header = { 'Authorization': f'Bearer {api_key}', 'Content-Type': 'application/json' }

Send an HTTP GET request to the specified URL with the provided headers using the get() method. It returns a Response object containing the server's response to the HTTP request, which can be used to extract the status code, content, etc.
```
 response = requests.get(url, headers=header)
```

If the status code of the response is 200, the request was successful and the contents can be extracted in text or JSON format.

 if response.status_code == 200:
     print(response.headers)
     print(response.text)    # or response.json()
 else:
     print(f"Error: {response.status_code}")

Let's take a simple example using the URL 'https://jsonplaceholder.typicode.com/todos/12'

    import requests
    url = 'https://jsonplaceholder.typicode.com/todos/12'
    header = {'Content-type': 'application/json'}
    response = requests.get(url, headers=header)
    if response.status_code == 200:
        print(response.headers)
        print(response.json())
    else:
        print(f"Error: {response.status_code}")

This code does not involve the use of keys/tokens.

Output:

You can also use Requests to download ZIP file. For example, to download a Kaggle dataset:

import os
import requests

key = '82jfvyg3j4jngjr'    # Kaggle API key
dataset_url = 'https://www.kaggle.com/datasets/vivek468/superstore-dataset-final'
save_path = 'C:\\Downloads'    # replace with the path you want to store the file in

response = requests.get(dataset_url, headers={'Authorization': f'Bearer {key}'}, stream=True)
if response.status_code == 200:
    with open(os.path.join(save_path, 'dataset.zip'), 'wb') as f:    # download the file
        for chunk in response.iter_content(chunk_size=1024):
            f.write(chunk)
    print('Download successful.')
else:
    print('Failed to download dataset. Status code:', response.status_code)

Output:

Conclusion

Thus, a potent method for obtaining data from websites is Web Scraping, which makes it possible to gather and analyze data effectively. The requests library, BeautifulSoup, Selenium, and Scrapy provide special benefits for various scraping requirements and enable developers to explore the web and retrieve important data easily. I encourage you to explore these tools further to unlock the full potential of web scraping to interact with and analyze web data.

Streamlining Workflows Task Automation with Python

Sun, 02 Jun 2024 21:25:31 +0000

In this comprehensive guide, we embark on a journey into the realm of task automation using Python. We'll explore how Python can streamline workflows by automating repetitive tasks. From setup to practical examples, you'll learn essential libraries, scheduling, testing, and deployment.

Task automation

Task automation helps in enhancing the productivity by automating repeating tasks, reducing manual effort, and minimizing errors. This allows the organizations to streamline operations and redirect resources. Automation tools like Python comes in very handy for these tasks.

Why Python for Task Automation?

Simplicity: Python's clear and easy-to-read syntax makes it ideal for automating tasks with minimal code complexity.
Extensive Libraries: A wealth of libraries, like os, shutil, requests, and pandas, provide tools for various automation tasks, from file management to data processing.
Cross-Platform: Python runs on multiple operating systems, allowing scripts to be used in diverse environments.
Strong Community: Python's large community offers extensive resources, tutorials, and support for automation-related queries.
Rapid Prototyping: Python's simplicity enables quick development and testing of automation scripts, allowing for rapid iteration and refinement.

Getting Started with Python for Task Automation

Setting Up the Python Environment

The steps are given below:

Install Python: Download and install the latest version of Python from the official website (https://www.python.org/downloads/).
Choose an IDE: Select an Integrated Development Environment (IDE) for Python development.
Install Required Packages: Use pip, the Python package manager, to install any additional packages or libraries. You need to install packages like requests, beautifulsoup4, or pandas for this guide using pip install.
Check Installation : Make sure that python is installed correctly by using python --version,
You have installed python successfully now write your first code and explore different concepts.

Writing Your First Automation Script

Let's create a simple Python script to automate a basic task, such as file management.

#Simple Python Script for File Management
import os

def organize_files(directory):
    for filename in os.listdir(directory):
        if filename.endswith('.txt'):
            # Move text files to a 'TextFiles' directory
            os.rename(os.path.join(directory, filename), os.path.join(directory, 'TextFiles', filename))
directory_path = 'C:\\Users\\dell\\Desktop\\temp'
organize_files(directory_path)

Essential Python Libraries for Automation

OS Module

The os module in Python provides a platform-independent way of interacting with the operating system, offering functions for various operating system-related tasks such as file and directory operations, process management.

Example :

import os
cwd = os.getcwd()
print("Current directory:", cwd)

new_dir = os.path.join(cwd, 'new_directory')
if not os.path.exists(new_dir):
    os.makedirs(new_dir)
    print("New directory created:", new_dir)
else:
    print("Directory already exists:", new_dir)

files = os.listdir(cwd)
print("Files in current directory:", files)

old_file = os.path.join(cwd, 'old_file.txt')
new_file = os.path.join(cwd, 'new_file.txt')
os.rename(old_file, new_file)
print("File renamed from 'old_file.txt' to 'new_file.txt'")

Note: While renaming the files make sure to create the file whose name you want to change.

Shutil Module

The shutil module offers a high-level interface for file operations, including file copying, moving, and deletion.

Example :

import shutil
source_file = 'C:\\Users\\dell\\Desktop\\temp\\source.txt'
destination_file = 'C:\\Users\\dell\\Desktop\\temp\\destination.txt'

try:
    shutil.copy(source_file, destination_file)
    print("File copied successfully!")
except FileNotFoundError:
    print("Source file not found.")
except PermissionError:
    print("Permission denied.")
except Exception as e:
    print("An error occurred:", e)

Note: Make sure to pass the correct path of the source and destination file in the same way as mentioned above.

Subprocess Module

The subprocess module in Python provides a powerful way to spawn new processes. It allows you to execute system commands, run external programs, and interact with them programmatically from within your Python script.

Example :

import subprocess
try:
    #Run the command and capture output
    result = subprocess.run(['-l'], capture_output=True, text=True)
    # Print the command output
    print("Command output:", result.stdout)
except FileNotFoundError:
    print("Command not found.")
except Exception as e:
    print("An error occurred:", e)

Scheduled Module

The schedule module in Python provides us withe the most simple and intuitive way to schedule repeating tasks and automate job execution. With schedule, you can automate tasks like data backups, report generation, and system maintenance.

Example :

import schedule
import time

def print_message():
    print("This message is printed every second.")
schedule.every(1).seconds.do(print_message)
while True:
    schedule.run_pending()
    time.sleep(1)  # Wait for 1 second

Automating File and Data Management

File Organization and Cleanup

Automating file organization, renaming, and cleanup tasks can significantly improve efficiency and maintain a tidy file system. Python provides powerful tools for automating these processes such as os and shutil.

import os
import shutil

def organize_files(directory):
    # Create directories if they don't exist
    if not os.path.exists(os.path.join(directory, 'Images')):
        os.makedirs(os.path.join(directory, 'Images'))
    if not os.path.exists(os.path.join(directory, 'Documents')):
        os.makedirs(os.path.join(directory, 'Documents'))

    for filename in os.listdir(directory):
        if filename.endswith(('.jpg', '.png', '.gif')):
            # Move image files to 'Images' directory
            shutil.move(os.path.join(directory, filename), os.path.join(directory, 'Images', filename))
        elif filename.endswith(('.pdf', '.docx', '.txt')):
            shutil.move(os.path.join(directory, filename), os.path.join(directory, 'Documents', filename))
    directory_path =  'C:\\Users\\dell\\Desktop\\temp'
    organize_files(directory_path)

Data Processing and Analysis

Automate data processing tasks using libraries like pandas for parsing, cleaning, and transforming datasets.

import pandas as pd
    data = {
        'Name': ['Alice', 'Bob', 'Charlie', 'David'],
        'Age': [25, 30, 35, 40],
        'City': ['New York', 'Los Angeles', 'Chicago', 'Houston']
    }
df = pd.DataFrame(data)
print("DataFrame:")
print(df)

Web Scraping and Data Retrieval Automation

Introduction to Web Scraping

Web scraping is the process of extracting data from websites. It involves retrieving HTML content from web pages and then parsing and extracting the desired information.
It is commonly used foe the following :-

Data Collection: Gathering information for analysis or research.
Lead Generation: Extracting contact information for sales.
Content Aggregation: Collecting and curating content from multiple sources.
Search Engine Indexing: Indexing web pages for search engines.

Using BeautifulSoup and Requests

Beautiful Soup and requests form a powerful combination for web scraping tasks. Requests is used to fetch the HTML content of web pages, while Beautiful Soup is employed to parse and extract data from the HTML, enabling to extract valuable information from websites through automation.

import requests
from bs4 import BeautifulSoup
#URL of the website to scrape
url = 'https://www.google.com'
response = requests.get(url)

#Check if the request was successful (status code 200)
if response.status_code == 200:
    # Parse the HTML content of the page
    soup = BeautifulSoup(response.text, 'html.parser')
    links = soup.find_all('a')
    for link in links:
        print(link.text)
        print("Retrival of data successfull")
else:
    print("Failed to retrieve data from the website.")

Automating Web Tasks

Automate tasks such as data scraping, form submission, and content extraction from websites helps people to extract and discover more about the websites as well as reduce the time for filling of the form manually.

import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart

#Set up SMTP settings
smtp_host = 'smtp.hostinger.com'
smtp_port = 465
sender_email = '*Emails are not allowed*'
receiver_email = '*Emails are not allowed*'
password = 'password'

#Create a multipart message
message = MIMEMultipart()
message['From'] = sender_email
message['To'] = receiver_email
message['Subject'] = 'Demo mail'

#Add message body
message.attach(MIMEText('Welcome!! This is Test Mail.', 'plain'))

try:
    #Create SMTP session
    server = smtplib.SMTP_SSL(smtp_host, smtp_port)
    server.login(sender_email, password)

    #Send email
    server.sendmail(sender_email, receiver_email, message.as_string())
    print("Email sent successfully!")

except Exception as e:
    print(f"Failed to send email. Error: {e}")

finally:
    #Close SMTP session
    server.quit()

System Administration and Maintenance Automation

Server Monitoring and Reporting

Automating server monitoring and generating status reports is crucial for maintaining the health and performance of systems. By scripting these tasks, administrators can proactively detect issues and analyze system behavior to give more informations.

import psutil
import datetime

def monitor_server():
    cpu_usage = psutil.cpu_percent(interval=1)
    memory_usage = psutil.virtual_memory().percent
    disk_usage = psutil.disk_usage('/').percent
    network_io = psutil.net_io_counters()
        timestamp = datetime.datetime.now().strftime("%Y-%m-%d %H:%M:%S")
    #Print server metrics
    print("Timestamp:", timestamp)
    print("CPU Usage:", cpu_usage, "%")
    print("Memory Usage:", memory_usage, "%")
    print("Disk Usage:", disk_usage, "%")
    print("Network I/O (Bytes Sent/Received):", network_io.bytes_sent, "/", network_io.bytes_recv)
monitor_server()

Backup and Recovery Automation

Automating data backup helps in disaster recovery and system restoration tasks. It is essential to ensure that the data integrity and continuity is maintained.

import shutil
import os
import datetime

def backup_data(source_dir, dest_dir):
    #Generate timestamp for backup folder
    timestamp = datetime.datetime.now().strftime("%Y%m%d%H%M%S")
    
    #Create a backup folder with timestamp
    backup_folder = os.path.join(dest_dir, f"backup_{timestamp}")
    os.makedirs(backup_folder)
    
    #Copy files from source directory to backup folder
    for root, dirs, files in os.walk(source_dir):
        for file in files:
            source_path = os.path.join(root, file)
            dest_path = os.path.join(backup_folder, os.path.relpath(source_path, source_dir))
            shutil.copy2(source_path, dest_path)
    
    print("Backup completed successfully.")

source_directory = 'C:\\Users\\dell\\Desktop\\src'
destination_directory = 'C:\\Users\\dell\\Desktop'
backup_data(source_directory, destination_directory)

Task Scheduling and Job Automation

Introduction to Task Scheduling

Task scheduling involves automating the execution of tasks or jobs at predefined times or intervals.

import schedule
import time
def print_message():
    print("Scheduled task: Hello, world!")

schedule.every(5).seconds.do(print_message)

while True:
    schedule.run_pending()
    time.sleep(1)  # Sleep for 1 second to avoid high CPU usage

Using Cron Jobs (Unix) and Task Scheduler (Windows)

import os
def create_task():
    #Define the command to run (in this case, open Notepad)
    command = 'notepad.exe'
    
    #Use os.system to run the command to create a scheduled task
    os.system(f'schtasks /create /sc once /tn "open_notepad" /tr "{command}" /st 00:00')

    if __name__ == "__main__":
        create_task()

Testing and Deployment Automation

Introduction to Testing Automation

Automated testing involves the use of software tools and scripts to execute test cases, validate functionality, and identify defects in a software application.

Using pytest and unittest

pytest is a mature and feature-rich testing framework known for its simplicity and extensibility, which provides powerful features such as fixtures, parameterized testing, and plugins.

class Calculator:
    def add(self, a, b):
        return a + b
   def subtract(self, a, b):
         return a - b
calculator = Calculator()
def test_addition():
    assert calculator.add(2, 3)==5
    assert calculator.add(5, -1)==4
def test_subtraction():
    assert calculator.subtract(5, 3)==2
    assert calculator.subtract(10, 7)==3

unittest is a built-in Python framework for automated testing that allows you to create, organize, and run test cases to test the validation of the data. It supports test case classes, assertions, setup and teardown methods. For more details prefer to unnitest Documentation

import unittest

def add(x, y):
    return x + y

class TestMathOperations(unittest.TestCase):

    def test_addition(self):
        self.assertEqual(add(2, 3), 5)
        self.assertEqual(add(-1, 1), 0)
        self.assertEqual(add(0, 0), 0)

if __name__ == '__main__':
    unittest.main()

Best Practices for Task Automation with Python

When creating task automation scripts in Python, you need to follow the best practices to ensure that your code is reliable, maintainable, and easy to understand. Here are some of the best practices that are listed below :-

Automated Testing: Utilize unittest or pytest to create automated tests, ensuring your automation scripts works well.
Use Functions and Classes: Organizing your code with functions and classes helps to build a better structure.
Robust Error Handling: Using the try/except blocks helps you to manage the possible errors and helps in avoiding them.
Follow PEP 8: Adhere to the Python style guide for consistent code formatting, using tools like black or autopep8.
Modular Design: Break your code into small, independent functions or modules to increase reusability and ease testing.
Clear Comments and Docstrings: Add concise comments to explain complex code, and use docstrings for function and class descriptions.

Conclusion

To sum it up, Python task automation can significantly boost efficiency and productivity by streamlining repetitive tasks and allowing you to focus on more critical work. Through Python's user-friendly syntax, extensive libraries, and strong community support, you can automate a variety of tasks, from file management and data processing to system administration and web scraping. As you dive into automation, follow best practices for maintainable and reliable scripts. Remember, to explore more about the Python Modules by visiting their official website.

Automating With Python: A Developer’s Guide

Tue, 14 May 2024 22:30:57 +0000

In this guide, we'll explore Automation with Python. Automation is key in today's fast-paced world, saving time and effort across various tasks. Python, which is a top choice for Automation. We'll cover its essential libraries along with examples for practical uses. By the end, you'll be ready to automate tasks efficiently using Python.

Understanding Automation

What is automation

Automation is the use of technology to perform different tasks with minimal human intervention. It involves the implementation of systems or processes that can operate automatically, reducing the need for manual labor and increasing efficiency.

Why Automation is Important and Required:

Efficiency: Streamlines processes, saving time and effort.
Accuracy: Minimizes human error, ensuring precision.
Cost Reduction: Lowers labor costs, increases productivity.
Scalability: Adapts easily to changing demands.
Productivity: Frees up time for strategic work.
Data Analysis: Provides valuable insights for decision-making.

Introduction of Python in Automation

Python is a powerful language for automation tasks due to its simplicity, versatility, and extensive libraries. You can also visit Python official site for more information.

Python Modules for Automation

Given below are the essential python Modules to automate various tasks along with their examples for better understanding.

1. Selenium : A powerful tool for automating web browsers, which allows you to interact with web elements programmatically. It's widely used for web scraping, testing and fill forms in web applications.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
import time

driver = webdriver.Chrome()
driver.get("https://www.google.com")
search_input = driver.find_element(By.NAME, "q")

search_input.send_keys("Python automation")
search_input.send_keys(Keys.RETURN)

time.sleep(60)
driver.quit()

2. PyAutoGUI : This helps us to automate GUI interactions. It is useful for automating tasks involving graphical user interfaces, such as opening applications

import pyautogui
import time
try:
    pyautogui.press('win')
    pyautogui.write('notepad')
    pyautogui.press('enter')
    time.sleep(4)  # Wait for Notepad to open

    pyautogui.write('Hello')
    time.sleep(2)  # Let's type slowly for demonstration
    pyautogui.hotkey('ctrl', 's')
    time.sleep(4)  # Wait for Save As dialog to open
    pyautogui.write('Hello.txt')
    time.sleep(2)  
    pyautogui.press('enter')
    time.sleep(2)  

    print("Task completed successfully!")
except Exception as e:
    print("An error occurred:", e)

3. Requests : This is a simple yet elegant HTTP library for Python, which allows you to send HTTP requests easily. It's commonly used for web scraping, accessing APIs.

import requests
try:
  response = requests.get("https://www.google.com")
  if response.status_code == 200:
      print("Request to Google successful!")
      print("HTML Content:")
      print(response.text)
  else:
      print("Failed to retrieve data from Google. Status code:", response.status_code)
except Exception as e:
  print("An error occurred:", e)

4. Beautiful Soup : It provides us with different functions for parsing HTML and XML documents. It allows you to extract data from web pages effortlessly, facilitating automation of data extraction processes.

from bs4 import BeautifulSoup
import requests

response = requests.get("https://www.example.com")
#Parse the HTML content
soup = BeautifulSoup(response.content, "html.parser")
for link in soup.find_all("a"):
    print(link.get_text())

5. Pandas : pandas is a powerful data manipulation library that help us to form data frames. It's commonly used for automating data processing tasks, such as cleaning, transforming, and analyzing datasets.

 import pandas as pd
 data = {
    'Name': ['Alice', 'Bob', 'Charlie', 'David'],
    'Age': [25, 30, 35, 40],
    'City': ['New York', 'Los Angeles', 'Chicago', 'Houston']
  }
df = pd.DataFrame(data)
print("DataFrame:")
print(df)

Getting Started with Automation

How to Automate a Task: A Step-by-Step Guide

Step 1: Identify Task

Before moving to any planning the first step is to identify the repetitive and time-consuming tasks.

Step 2: Divide the Task into Smaller Steps

Break down the task into smaller, manageable modules. Then start working on each module independently.

Step 3: Research Python Libraries

Research for those Python libraries that can help us in automating each step of the task.

Step 4: Write the Code

Write Python code to automate each step of the task. Given below is a short example to get you started with scripting:

import os
import shutil

file_types = set()
for filename in os.listdir('directory_path'):
    if os.path.isfile(filename):
        file_types.add(filename.split('.')[-1])
        
for file_type in file_types:
    os.makedirs(file_type, exist_ok=True)
    
for filename in os.listdir('directory_path'):
    if os.path.isfile(filename):
        file_type = filename.split('.')[-1]
        shutil.move(filename, os.path.join(file_type, filename))

Step 5: Test the Code

Test the code with sample data to ensure that it performs as expected. Verify that files are organized correctly into their respective folders.

Step 6: Update the Code

Refine and update the code as needed based on test results and feedback. Ensure it handles edge cases and exceptions gracefully.

Introduction to Automation Scripting Languages

Scripting and automation have become indispensable tools for individuals and organizations alike.

Understanding Automation Scripting Languages:
Automation scripting languages are programming languages that are specifically designed to automate repetitive tasks. These languages are having the predefined syntax and functionalities that are useful for automating tasks.

Characteristics of Automation Scripting Languages:

Readable Syntax: Clear and concise code for easier comprehension.
Abundance of Libraries: Extensive pre-built functions streamline development.
Interpretation or Compilation: Code execution either directly (interpreted) or translated into machine language (compiled).
Cross-Platform Compatibility: Capable of running on various operating systems.

Popular Automation Scripting Languages:

Python: Versatile, with rich libraries; used for automation, web dev, and data analysis.
Bash (Shell): Default on Unix; excels in system tasks and command-line ops.
PowerShell: Microsoft's automation tool for Windows; robust system management.

Essential Python Libraries for Automation

OS Module: The os module in Python provides a platform-independent way of interacting with the operating system.

Example :

import os
cwd = os.getcwd()
print("Current directory:", cwd)

new_dir = os.path.join(cwd, 'new_directory')
if not os.path.exists(new_dir):
    os.makedirs(new_dir)
    print("New directory created:", new_dir)
else:
    print("Directory already exists:", new_dir)

files = os.listdir(cwd)
print("Files in current directory:", files)

old_file = os.path.join(cwd, 'old_file.txt')
new_file = os.path.join(cwd, 'new_file.txt')
os.rename(old_file, new_file)
print("File renamed from 'old_file.txt' to 'new_file.txt'")

Note: While renaming the files make sure to create the file whose name you want to change.

Shutil Module: The shutil module offers a high-level interface for file operations, including file copying, moving, and deletion. You can gain more knowledge from shutil module

Example :

import shutil
source_file = 'C:\\Users\\dell\\Desktop\\temp\\source.txt'
destination_file = 'C:\\Users\\dell\\Desktop\\temp\\destination.txt'

try:
    shutil.copy(source_file, destination_file)
    print("File copied successfully!")
except FileNotFoundError:
    print("Source file not found.")
except PermissionError:
    print("Permission denied.")
except Exception as e:
    print("An error occurred:", e)

Note: Make sure to pass the correct path of the source and destination file in the same way as mentioned above.

Subprocess Module: The subprocess module provides a powerful way to spawn new processes. It allows you to execute system commands, run external programs.For more details visit subpress module

Example :

import subprocess
try:
    #Run the command and capture output
    result = subprocess.run(['-l'], capture_output=True, text=True)
    # Print the command output
    print("Command output:", result.stdout)
except FileNotFoundError:
    print("Command not found.")
except Exception as e:
    print("An error occurred:", e)

Time and Datetime Modules: The time and datetime modules in Python provide functionalities for handling time-related tasks and scheduling automation jobs.For further information check Time module in python

Example :

import time
delay = 5
start_time = time.time()
while time.time() < start_time + delay:
    pass
print("Automation job executed after {} seconds.".format(delay))

Note: When this code is executed than after the delay of 5 seconds the message is displayed

Web Scraping and Automation

Web scraping and Automation, empower users to automate repetitive tasks and extract valuable data from websites automatically. The users are able to interact with the web pages and extract data automatically.

Automating Web Tasks
Automation scripts can interact with web pages, submit forms, extract data, and perform other tasks, enabling the automation of web-based workflows.

Example

import requests
from bs4 import BeautifulSoup
#Define the URL of the webpage
url = "https://www.example.com"
try:
  response = requests.get(url)
  if response.status_code == 200:
      print("Webpage accessed successfully!")
      soup = BeautifulSoup(response.text, 'html.parser')
      links = soup.find_all('a')
      for link in links:
          print(link.text)
      # For example, submitting a login form
      payload = {'username': 'your_username', 'password': 'your_password'}
      login_response = requests.post('https://www.example.com/login', data=payload)
      if login_response.status_code == 200:
          print("Login successful!")
      else:
          print("Login failed.")
  else:
      print("Failed to access the webpage.")
except Exception as e:
  print("An error occurred:", e)

Automating System Tasks

Automating system tasks involves streamlining repetitive processes on both Windows and Linux operating systems, enhancing productivity and efficiency.

Task Automation on Windows and Linux

On Windows, PowerShell is a powerful automation tool that provides a wide range of cmdlets and scripts for managing system tasks. With PowerShell, you can automate tasks such as file manipulation, registry editing, user management, and system configuration.
On Linux, shell scripting with Bash is commonly used for automating system tasks. Bash scripts can perform tasks such as directory traversal, process management, and system monitoring.

Example :

$sourceDir = "C:\Users\dell\Desktop\File1.txt"
$destDir = "C:\Users\dell\Desktop\File2.txt"

$timestamp = Get-Date -Format "yyyyMMdd_HHmmss"
$backupFile = "backup_$timestamp.tar.gz"

Compress-Archive -Path $sourceDir -DestinationPath "$destDir\$backupFile"
Write-Host "Backup created: $destDir\$backupFile"

Data Processing and Automation

Introduction to Data Processing

Data processing is the method of transforming raw data into valuable information or insights. It involves several stages such as data collection, cleaning, transformation, analysis, and visualization.

Using Pandas for Data Automation

When the Python library Pandas is used for data automation then it allows efficient processing, manipulation, and analysis of data. Pandas is a powerful library that provides data structures such as DataFrame.

Example :

import pandas as pd
data = pd.read_csv('data.csv')
data.head(7)
df = pd.DataFrame(data)
df.head()
missing_values = df.isna().sum()
print(missing_values)
df_filled = df.fillna(0)
df_filled.head(7)

Testing and Automation

Testing and automation are crucial aspects of software development, ensuring software reliability and quality. It involves systematically evaluating software to find defects and built test cases for testing purpose.

Overview of Testing Automation

Testing automation involves the use of software tools and frameworks to automate the process of testing software applications. One popular testing automation framework is pytest.

Using pytest for Testing Automation

Pytest is a popular testing framework in Python that is used for testing automation. It offers powerful features for writing and executing test cases efficiently. With pytest, developers can easily create test functions, organize test suites, and assert expected outcomes. Its suitable for both small and large-scale projects.

Example :

class Calculator:
    def add(self, a, b):
        return a + b
   def subtract(self, a, b):
         return a - b
calculator = Calculator()
def test_addition():
    assert calculator.add(2, 3)==5
    assert calculator.add(5, -1)==4
def test_subtraction():
    assert calculator.subtract(5, 3)==2
    assert calculator.subtract(10, 7)==3

Note: Always save your `pytest` file with the following naming convention that is `test_filename`.

Real-World Automation Examples

Backup Automation: Schedule regular backups of critical data to prevent loss in case of hardware failure or data corruption.
Server Maintenance: Automate routine server maintenance tasks such as software updates, disk cleanup, and system reboots.
Email Automation: Implement automated email responses, email filtering, and email forwarding to streamline communication processes.
Data Migration: Automate the migration of data between databases or cloud storage solutions to ensure accuracy and efficiency.
Report Generation: Automatically generate and distribute reports on a scheduled basis, reducing manual effort and ensuring timely delivery.
Customer Support: Implement chatbots or automated ticketing systems to handle common customer inquiries and support requests.

Conclusion

In conclusion it is better to say that, Automation with Python empowers individuals and organizations to streamline workflows, increase productivity, and drive innovation. Python's simplicity, versatility, and rich ecosystem of libraries, automation can be used to automate tasks across diverse domains, from file operations and web scraping to system administration and data processing. Do gather some more information by doing some experiment with different libraries and tools, and apply best practices that you learned in this guide to give efficient and reliable automation solutions.

Leveraging Python for System Administration

Wed, 01 May 2024 21:40:14 +0000

In the world of computer systems, keeping things running smoothly is a top priority, and that's where system administrators come in. Python, with its user-friendly interface and powerful capabilities, is like a trusty option. This article dives into how Python empowers system administrators to automate tasks and optimize resources, offering practical insights and real-world examples to support professionals at all levels in mastering this essential tool.

Understanding System Administration

In system administration, you're responsible for managing, maintaining, and operating computer systems, networks, and associated infrastructure within your organization. System administrators, commonly known as sysadmins, are integral in guaranteeing the efficient operation and security of IT systems to uphold business functions.

User Management: This involves managing user accounts, permissions, access rights, password resets, account lockouts, and user group management for effective access control.
File System Management: You'll manage file systems, including tasks like creating, modifying, and deleting files and directories, setting permissions, managing disk space, and monitoring file system health to prevent data loss or corruption.
Software Installation: Installing and configuring software applications involves selecting, installing, and configuring software packages on servers and workstations to ensure optimal performance and security.
Configuration Management: You'll manage the configuration of system components like servers, network devices, and software applications, ensuring consistency, currency, and compliance with organizational standards and best practices.

Challenges in Traditional System Administration

Traditional system administration tasks are known for being manual and repetitive, often requiring sysadmins to perform the same actions across multiple systems. While essential for IT infrastructure functionality and security, these tasks can be time-consuming, error-prone, and resource-intensive.

Volume of Work: Sysadmins handle daily routine tasks such as managing user accounts, software updates, and configuration changes, which can become overwhelming in large-scale environments.
Human Error: Manual tasks can lead to mistakes like misconfigurations, typos, and oversights, resulting in system downtime, data loss, security vulnerabilities, and compliance violations.
Lack of Consistency: Without predefined processes and automation tools, inconsistencies may arise due to ad-hoc methods and shortcuts, leading to compliance issues.
Scalability and Agility: Manual processes hinder scalability and adaptability to changing business needs, becoming increasingly burdensome and inefficient as organizations grow and IT environments become more complex.

In the world of IT, managing large-scale systems is like navigating a maze filled with challenges and complexities. As organizations grow, so does the size and complexity of their systems, presenting sysadmins with a host of hurdles to overcome.

Automation Complexity: Automating large-scale environments is complex and time-consuming, involving the development, testing, and maintenance of automated workflows across diverse systems. This requires careful planning and coordination.
Visibility and Monitoring: Maintaining visibility into large-scale system health, performance, and security is challenging. Insufficient monitoring tools may create blind spots, hindering prompt issue identification and resolution.
Security Risks: Large-scale systems are prime targets for cyber threats. Ensuring robust security measures like access controls, encryption, and threat detection is crucial but challenging in dynamic environments.
Change Management: Implementing changes, updates, and upgrades in large-scale systems requires meticulous planning and coordination.
Compliance and Governance: Comprehensive governance strategies are crucial for documentation, policy enforcement, and compliance across diverse environments.

Essential Python Libraries for System Administration

Python is a valuable asset for system administrators, offering a range of libraries to streamline tasks and boost productivity. These libraries empower sysadmins to interact with operating systems, automate tasks, and manage infrastructure effectively.

os Module: The os module in Python offers a plethora of functions for interacting with the operating system. It enables sysadmins to navigate file systems, manipulate files and directories, and manage processes efficiently.
subprocess Module: The subprocess module in Python allows scripts to spawn new processes, interact with system programs, and execute shell commands. It provides a higher-level interface than the os.system() function.
sys Module: The sys module in Python provides access to system-specific parameters and functions, allowing interaction with the Python interpreter and retrieval of information about the runtime environment, command-line arguments, and standard input/output streams.
shutil Module: The shutil module in Python provides a high-level interface for file and directory operations, including copying, moving, renaming, deleting, archiving, and compressing files.
paramiko Module: The paramiko module in Python is a powerful library for SSH communication and remote server management. It allows sysadmins to securely connect to remote servers, execute commands, transfer files, and manage SSH keys programmatically.
psutil Module: The psutil module in Python offers a cross-platform interface for accessing information on running processes, system utilization, and resources. It enables sysadmins to monitor system performance, manage processes, and collect detailed data on CPU, memory, disk, and network usage.

Note: Although the aforementioned modules usually come pre-installed with Python, you may install a module using the command: `pip install module_name` if it's not present in your Python environment. Import a module using the command: `import module_name`.

Examples of Python Scripts for System Administration

User Management We can create, modify, and delete user accounts using Python by calling system commands (useradd, passwd, userdel) via the subprocess module.

Creating User Account: The process of adding a new user to the system with specified attributes such as username, password, and home directory.
```
  import subprocess
  
  def create_user(username, password):
      try:
          subprocess.run(['powershell', '-Command', f'New-LocalUser -Name {username} -Password (ConvertTo-SecureString "{password}" -AsPlainText -Force)'])
          print(f"User '{username}' created successfully.")
      except subprocess.CalledProcessError as e:
          print(f"Error creating user '{username}': {e}")

  create_user("Sia", "si@123")
```
The equivalent Linux command can be obtained by replacing the main_com line by:
```
  subprocess.run(['useradd', '-m', username])
```
The script uses the subprocess module to execute a PowerShell command, creating a new local user account on a Windows system. The create_user function constructs a PowerShell command with the provided username and password, utilizing the New-LocalUser cmdlet. Upon successful creation, a confirmation message is displayed; otherwise, an error message is shown.

Output:
Modifying User Account: Making changes to an existing user account, such as updating user information or modifying permissions.
```
  import subprocess

  def modify_user(username, new_password):
      try:
          subprocess.run(['powershell', '-Command', f'Set-LocalUser -Name {username} -Password (ConvertTo-SecureString "{new_password}" -AsPlainText -Force)'])
          print(f"Password for user '{username}' modified successfully.")
      except subprocess.CalledProcessError as e:
          print(f"Error modifying password for user '{username}': {e}")

  modify_user("Sia", "si@456")
```
The equivalent Linux command can be obtained by replacing the main_com line by:
```
  subprocess.run(['passwd', username], input=new_password.encode())
```
The script employs the subprocess module to execute a PowerShell command, altering the password of an existing local user account. The modify_user function accepts parameters for the username and the new password, constructing a PowerShell command using string formatting or passwd command. If the operation succeeds, a confirmation message is shown.

Output:
Deleting User Account: Removing an existing user account from the system, including all associated files and directories.
```
  import subprocess

  def delete_user(username):
      try:
          subprocess.run(['powershell', '-Command', f'Remove-LocalUser -Name {username} -Force'])
          print(f"User '{username}' deleted successfully.")
      except subprocess.CalledProcessError as e:
          print(f"Error deleting user '{username}': {e}")

  delete_user("Sia")
```
The equivalent Linux command can be obtained by replacing the main_com line by:
```
  subprocess.run(['userdel', '-r', username])
```
The script uses the subprocess module to execute a PowerShell command, deleting an existing local user account. The delete_user function accepts a username parameter indicating the user account to be deleted. It constructs a PowerShell command using string formatting, utilizing the Remove-LocalUser or userdel cmdlet. After deletion, a success message is displayed.

Output:

Caution: Ensure that the script is executed with appropriate privileges to perform these actions.

Setting User Permissions and Managing Access Control To set user permissions and manage access control with a script, you can utilize the subprocess module in Python to execute system commands for manipulating file permissions and access control lists (ACLs).

import subprocess

def set_permissions(file_path, user, permission):
    try:
        subprocess.run(['icacls', file_path, '/grant', f'{user}:{permission}'])
        print(f"Permissions set successfully for user '{user}' on '{file_path}'")
    except subprocess.CalledProcessError as e:
        print(f"Error setting permissions: {e}")

file_path = "myfile.txt"
user = "Sia"
permission = "F"  # Full control
set_permissions(file_path, user, permission)

The equivalent Linux command can be obtained by replacing the main_com line by:

subprocess.run(['chmod', permission, file_path])

The script uses the icacls command to modify file permissions in Windows. The /grant option specifies user and permissions, while the permission parameter allows customization using flags like F, M, RX, R, W, D.

File System Management To automate file and directory operations such as creation, deletion, and modification in Python, you can use the os module for basic file and directory operations, and the shutil module for more advanced operations like copying and moving files.

File Creation, Modification and Deletion

  import os

  def create_file(file_path):
      try:
          with open(file_path, 'w') as f:
              pass  # Creating an empty file
          print(f"File '{file_path}' created successfully.")
      except Exception as e:
         print(f"Error creating file '{file_path}': {e}")

  def delete_file(file_path):
      try:
          os.remove(file_path)
          print(f"File '{file_path}' deleted successfully.")
      except Exception as e:
          print(f"Error deleting file '{file_path}': {e}")

  def modify_file(file_path, content):
      try:
          with open(file_path, 'w') as f:
              f.write(content)
          print(f"File '{file_path}' modified successfully.")
      except Exception as e:
          print(f"Error modifying file '{file_path}': {e}")

  file_path = "sample.txt"
  create_file(file_path)
  modify_file(file_path, "I hope you are enjoying so far!")
  delete_file(file_path)

create_file creates an empty file at the specified path, modify_file modifies the content of an existing file with the provided content and delete_file deletes the file at the specified path.

Output:

Directory Creation, Modification and Deletion

  import os

  def create_directory(directory_path):
      try:
          os.makedirs(directory_path, exist_ok=True)
          print(f"Directory '{directory_path}' created successfully.")
      except Exception as e:
          print(f"Error creating directory '{directory_path}': {e}")

  def delete_directory(directory_path):
      try:
          os.rmdir(directory_path)
          print(f"Directory '{directory_path}' deleted successfully.")
      except Exception as e:
          print(f"Error deleting directory '{directory_path}': {e}")

  def rename_directory(directory_path, new_name):
      try:
          os.rename(directory_path, new_name)
          print(f"Directory '{directory_path}' renamed to '{new_name}' successfully.")
      except Exception as e:
          print(f"Error renaming directory '{directory_path}': {e}")

  directory_path = "sampledir"
  new_name = "samp_dir"
  create_directory(directory_path)
  rename_directory(directory_path, new_name)
  delete_directory(new_name)

create_directory creates a new directory at the specified path, rename_directory renames an existing directory to the specified new name and delete_directory deletes the directory at the specified path.

Output:

After renaming:

Software Installation and Configuration

Automating Software Installation and Updates Automating software installation and updates using package managers like pip or system-specific tools can be achieved through Python scripts.

For example, to install and update the numpy package:

  import subprocess
   
  def install_package(package_name):
      try:
          subprocess.run(['pip', 'install', package_name])
          print(f"Package '{package_name}' installed successfully.")
      except subprocess.CalledProcessError as e:
          print(f"Error installing package '{package_name}': {e}")
   
  def update_package(package_name):
      try:
          subprocess.run(['pip', 'install', '--upgrade', package_name])
          print(f"Package '{package_name}' updated successfully.")
      except subprocess.CalledProcessError as e:
          print(f"Error updating package '{package_name}': {e}")

  package_name = "numpy"
  install_package(package_name)
  update_package(package_name)

The install_package function installs a Python package using the pip install command. The update_package function updates a Python package to the latest version using the pip install --upgrade command.

Configuring Software Settings and Parameters You can use the sys module in Python to configure software settings programmatically on Windows, particularly for command-line arguments handling.

  import sys
  import subprocess

  def configure_with_tool(tool_path, arguments):
      try:
          subprocess.run([tool_path] + arguments, check=True)
          print("Configuration successful.")
      except subprocess.CalledProcessError as e:
          print(f"Error configuring with tool '{tool_path}': {e}")

  if __name__ == "__main__":
      if len(sys.argv) < 3:
          print("Usage: python script.py <tool_path> <arguments>")
          sys.exit(1)

      tool_path = sys.argv[1]
      arguments = sys.argv[2:]
      configure_with_tool(tool_path, arguments)

When you run the script from the command line, you'll provide the tool path as the first argument as follows:

python Articles/software_settings.py "Articles//SimpleCalculator.exe"

System Monitoring and Reporting You can use Python to gather system metrics such as CPU usage, memory usage, and disk space using various libraries and system commands.

import psutil

def get_cpu_usage():
    cpu_usage = psutil.cpu_percent(interval=1)
    return cpu_usage

def get_memory_usage():
    memory = psutil.virtual_memory()
    total_memory = memory.total
    available_memory = memory.available
    used_memory = memory.used
    memory_usage_percent = memory.percent
    return {
        "total_memory": total_memory,
        "available_memory": available_memory,
        "used_memory": used_memory,
        "memory_usage_percent": memory_usage_percent
    }

def get_disk_usage():
    partitions = psutil.disk_partitions()
    disk_usage = {}
    for partition in partitions:
        partition_usage = psutil.disk_usage(partition.mountpoint)
        disk_usage[partition.mountpoint] = {
            "total": partition_usage.total,
            "used": partition_usage.used,
            "free": partition_usage.free,
            "percent": partition_usage.percent
        }
    return disk_usage

cpu_usage = get_cpu_usage()
memory_usage = get_memory_usage()
disk_usage = get_disk_usage()
print("CPU Usage:", cpu_usage, "%")
print("Memory Usage:", memory_usage)
print("Disk Usage:", disk_usage)

The get_cpu_usage function retrieves CPU usage as a percentage, while get_memory_usage gathers memory statistics, and get_disk_usage collects disk stats. These functions utilize psutil methods for data retrieval.

Output:

Conclusion

System Administration faces challenges with manual tasks and the need for efficiency. Python offers a solution with its libraries like os, subprocess, and shutil, making it ideal for automation. Python scripts can handle user management, permissions, access control, file system operations, software installation, and system monitoring. This empowers administrators to enhance productivity and proactively manage computer systems effectively.

Coder Legion - Recent posts tagged workflows

Deferring Publication After a Budget-Safe Draft Revision

Understanding the Request Budget Conflict

Implementing a Deferred Publication Fallback

Verification and Execution Checklist

Selecting the Latest Targeted Revision

Diagnosing Revision Queue Scan Failures

Implementing Newest-First Revision Sorting

Verification Checklist

Conclusion

n8n Free Trial: 3 Workflows to Test in 14 Days

Know the trial limits before day one

Prepare the evidence before signup

A practical 14-day schedule

Workflow one: route a qualified lead alert

Workflow two: deliver a weekly operating summary

Workflow three: create a safe invoice reminder

Use the new AI Assistant for one controlled build

Understand what consumes an execution

Run failure drills before you trust the workflows

Use this keep-or-cancel scorecard

Export before the trial ends

Sources checked

Closing the Loop: From Contradiction Detection to Automated Resolution

The Problem

Detection: The Adversarial Lint Gate

One Click to Remediation

How the Resolver Works

Your Escape Hatch: Content Snapshots

Architecture: How the Pieces Connect

What We Learned

What's Next

I use AI coding agents daily.

They Asked for My AI Rules. But I Could Not Just Hand Them Over.

The rules worked because I built them for one repo

The problem showed up the moment other people wanted in

I was solving the wrong problem the whole time

Nobody on a team agrees on one AI tool

What the tool had to actually do

I did not build everything I wanted to

Orchestrating Complex Workflows: From Sequential to Parallel to Intelligent

Next step depends on output

Before proceeding, wait for human approval

Define steps

Run the workflow

Monitor progress

Wait for completion

Install the workflow system

Create a workflow

Add steps

Run it

Anatomy of a Kestra Flow: How It All Fits Together

Getting Started with the Vercel AI SDK (Beginner to Working Chat App)

What is the Vercel AI SDK?

What You’ll Build

Step 1: Create a Next.js App

Step 2: Install the Vercel AI SDK

Step 3: Add Your API Key

Step 4: Create the Streaming Chat API Route

What’s happening here?

Step 5: Build the Chat UI with useChat()

Step 6: Run It

Understanding messages

Add a Better UI (Optional)

Bonus: Add Tool Calling (Basic Example)

Common Issues & Fixes

❌ “OPENAI_API_KEY missing”

❌ No streaming happens

❌ Build fails on deployment

Production Tips (Best Practices)

✅ Keep prompts stable

✅ Limit message history

✅ Add safety

✅ Add logging

What to Build Next

Conclusion

Sprint Workflow: A Beginner's Guide to Agile Development

In this article, we explore the sprint process in software development. You'll learn to manage tasks using a sprint board, track progress, and enhance team efficiency for successful project deliveries!

What Is a Sprint?

What is Agile Development

Step 5: Build the Chat UI with `useChat()`

Understanding `messages`