Asynchronous AI Processing Architecture: Building Non-Blocking User Experiences for Long-Running AI

Fri, 05 Sep 2025 09:32:32 +0000

How to handle AI image generation requests that take 5-15+ seconds without blocking your users or consuming server resources?

The Challenge

AI-powered applications face a fundamental user experience challenge: modern AI tasks like image generation, video processing, or complex language model operations often require 5-30 seconds to complete. Traditional synchronous request-response patterns create several problems:

Connection Timeouts: HTTP requests timing out during long AI processing, leaving users with error messages instead of results.

Resource Blocking: Server connections remain open during processing, consuming valuable server resources and limiting concurrent user capacity.

Poor User Experience: Users stare at loading screens without progress indicators or the ability to navigate elsewhere while waiting.

Scalability Issues: Each processing request ties up a server connection, severely limiting the number of concurrent users your application can handle.

The Async Architecture Solution

The solution is an asynchronous, prediction-based architecture that decouples AI task submission from result delivery. This approach immediately returns control to the user while processing happens in the background.

The core workflow consists of three distinct phases:

Immediate Submission: User submits request → Server validates and creates AI task → Returns task ID instantly
Background Processing: AI service processes task → Sends webhook notification when complete
Result Delivery: Frontend polls for status updates → Displays result when ready

Architecture Components

1. Non-Blocking API Endpoint

The primary endpoint handles task submission without waiting for AI completion:

export async function POST(request: NextRequest): Promise<NextResponse<APIResponse>> {
  try {
    // 1. Validate input parameters
    const validationResult = schema.safeParse(formData);
    if (!validationResult.success) {
      return await createErrorResponse({ errorCode: EValidationErrorCode.INVALID_PARAMETER });
    }

    // 2. Authentication and rate limiting checks
    const session = await auth();
    const throttleResult = await checkAnonymousUserRequestLimit({ userIdentifier });
    if (!throttleResult.success) {
      return await createErrorResponse({ errorCode: throttleResult.errorCode });
    }

    // 3. Create async AI prediction task
    const webhookUrl = `${process.env.API_BASE_URL}/api/webhooks/replicate`;
    const result = await generateJerseyEffectAsync({
      payload: validatedData,
      requestId: generateRequestId(),
      webhookUrl,
    });

    // 4. Store prediction status in KV store with TTL
    await PredictionStatusManager.createPrediction(result.data.predictionId, {
      status: 'starting',
      requestId: result.data.requestId,
      userId: session?.user?.uuid,
      payload,
      createdAt: Date.now(),
    });

    // 5. Launch concurrent background tasks (non-blocking)
    const backgroundTasks = [
      reserveUserCredits({ userUuid, predictionId, creditsAmount }),
      createUserWorkRecord({ userUuid, payload, predictionId }),
    ];
    
    // Execute background tasks without waiting
    Promise.allSettled(backgroundTasks);

    // 6. Return immediately with prediction ID
    return NextResponse.json({
      success: true,
      message: 'Prediction created successfully',
      data: {
        predictionId: result.data.predictionId,
        requestId: result.data.requestId,
        status: 'starting',
        estimatedTime: result.data.estimatedTime,
      },
    });

  } catch (error) {
    return await createErrorResponse({ errorCode: ECommonErrorCode.INTERNAL_SERVER_ERROR });
  }
}

2. Intelligent Status Management

A centralized status manager handles prediction state across the entire lifecycle:

export class PredictionStatusManager {
  // Store prediction with initial status
  static async createPrediction(predictionId: string, data: PredictionData) {
    await putKV(`prediction:${predictionId}`, {
      ...data,
      status: 'starting',
      createdAt: Date.now(),
    }, { expirationTtl: 3600 }); // 1 hour TTL
  }

  // Update status during processing
  static async markAsProcessing(predictionId: string) {
    const current = await this.getPredictionStatus(predictionId);
    if (current.success && current.data) {
      await putKV(`prediction:${predictionId}`, {
        ...current.data,
        status: 'processing',
        updatedAt: Date.now(),
      });
    }
  }

  // Mark completion with results
  static async markAsSucceeded(predictionId: string, imageUrl: string, filePath: string, duration?: number) {
    const current = await this.getPredictionStatus(predictionId);
    if (current.success && current.data) {
      await putKV(`prediction:${predictionId}`, {
        ...current.data,
        status: 'succeeded',
        result: {
          imageUrl,
          filePath,
          generationTime: duration,
        },
        completedAt: Date.now(),
      });
    }
  }
}

3. Webhook Processing Pipeline

The webhook endpoint handles AI service completion notifications with comprehensive error handling:

export async function POST(request: NextRequest): Promise<NextResponse<APIResponse>> {
  performanceMonitor.start('webhook-total');
  
  try {
    // 1. Security verification (prevent replay attacks)
    const body = await request.text();
    const verification = await verifyWebhookRequest(request, body, {
      secret: process.env.REPLICATE_WEBHOOK_SECRET,
      checkTimestamp: true,
    });

    if (!verification.valid) {
      return await createErrorResponse({ errorCode: ECommonErrorCode.SERVICE_UNAVAILABLE });
    }

    // 2. Parse webhook event
    const event: ReplicateWebhookEvent = JSON.parse(body);
    console.log(' Webhook received:', { id: event.id, status: event.status });

    // 3. Route to appropriate handler
    switch (event.status) {
      case 'starting':
      case 'processing':
        await PredictionStatusManager.markAsProcessing(event.id);
        break;

      case 'succeeded':
        await handleSuccessfulPrediction(event.id, event);
        break;

      case 'failed':
      case 'canceled':
        await handleFailedPrediction(event.id, event.error || 'Task canceled');
        break;
    }

    return NextResponse.json({ success: true, processed: true });

  } catch (error) {
    console.error('❌ Webhook processing error:', error);
    return await createErrorResponse({ errorCode: ECommonErrorCode.INTERNAL_SERVER_ERROR });
  }
}

async function handleSuccessfulPrediction(predictionId: string, event: ReplicateWebhookEvent) {
  // 1. Extract result URL from AI service response
  const imageUrl = Array.isArray(event.output) ? event.output[0] : event.output;
  
  // 2. Download and store result to permanent storage
  const imageResponse = await fetch(imageUrl);
  const imageBuffer = await imageResponse.arrayBuffer();
  
  const uploadResult = await uploadFile({
    path: 'jerseys',
    fileName: `${Date.now()}-${predictionId}.png`,
    file: imageBuffer,
    contentType: 'image/png',
  });

  const finalImageUrl = generatePublicUrl(uploadResult.filePath);

  // 3. Update status with final result
  await PredictionStatusManager.markAsSucceeded(
    predictionId,
    finalImageUrl,
    uploadResult.filePath,
    event.metrics?.predict_time * 1000
  );

  // 4. Update user records and confirm billing
  await updateUserWorkStatus(predictionId, {
    workResult: finalImageUrl,
    generationStatus: 'completed',
    generationDuration: event.metrics?.predict_time * 1000,
  });

  await confirmCreditDeduction({ userUuid, predictionId, actualCreditsUsed });
}

4. Concurrent Background Processing

The architecture maximizes efficiency by running multiple operations concurrently without blocking the main response:

// After successful prediction creation, launch background tasks
const concurrentTasks: Promise<void>[] = [];

// Credit reservation task (for paid users)
if (hasPaid && userUuid && requiredCredits > 0) {
  const creditReserveTask = (async () => {
    const reserveResult = await reserveUserCredits({
      userUuid,
      predictionId,
      creditsAmount: requiredCredits,
      taskType: 'text_to_image',
      taskDescription: `Jersey generation: ${JSON.stringify(payload).substring(0, 100)}`,
    });
    
    if (!reserveResult.success) {
      console.error('❌ Credit reservation failed:', reserveResult.message);
    }
  })();
  
  concurrentTasks.push(creditReserveTask);
}

// User work record creation task
if (userUuid) {
  const createWorkTask = (async () => {
    await createUserWorkRecord({
      userUuid,
      payload,
      requestId,
      predictionId,
      isAnonymous: !isAuthenticated,
    });
  })();
  
  concurrentTasks.push(createWorkTask);
}

// Execute all tasks concurrently without waiting for results
if (concurrentTasks.length > 0) {
  Promise.allSettled(concurrentTasks).then((results) => {
    console.log(' Background tasks completed:', {
      total: results.length,
      fulfilled: results.filter(r => r.status === 'fulfilled').length,
      rejected: results.filter(r => r.status === 'rejected').length,
    });
  });
}

// Return immediately to user
return NextResponse.json({ success: true, data: { predictionId, status: 'starting' } });

Frontend Integration Pattern

The client-side implementation uses intelligent polling with exponential backoff:

async function pollPredictionStatus(predictionId: string) {
  const delays = [1000, 2000, 5000, 10000]; // 1s → 2s → 5s → 10s
  let attemptIndex = 0;
  
  while (attemptIndex < 60) { // Maximum 10 minutes polling
    try {
      const response = await fetch(`/api/predictions/${predictionId}/status`);
      const result = await response.json();
      
      if (result.data.status === 'succeeded') {
        displayResult(result.data.result.imageUrl);
        return;
      } else if (result.data.status === 'failed') {
        displayError(result.data.error);
        return;
      }
      
      // Continue polling with progressive delays
      const delay = delays[Math.min(attemptIndex, delays.length - 1)];
      await new Promise(resolve => setTimeout(resolve, delay));
      attemptIndex++;
      
    } catch (error) {
      console.error('Polling error:', error);
      await new Promise(resolve => setTimeout(resolve, 2000));
    }
  }
}

Monitoring and Debugging

Async architectures require comprehensive monitoring to maintain reliability:

// Performance monitoring integration
export class PerformanceMonitor {
  static start(operation: string, metadata?: Record<string, any>) {
    console.log(` Starting ${operation}`, metadata);
    return Date.now();
  }
  
  static end(operation: string, startTime: number) {
    const duration = Date.now() - startTime;
    console.log(`✅ Completed ${operation} in ${duration}ms`);
    
    // Send metrics to monitoring service in production
    if (process.env.NODE_ENV === 'production') {
      sendMetric(`operation.${operation}.duration`, duration);
      sendMetric(`operation.${operation}.count`, 1);
    }
    
    return duration;
  }
}

// Status debugging endpoint
export async function GET(request: NextRequest) {
  const predictionId = request.nextUrl.searchParams.get('predictionId');
  
  if (!predictionId) {
    return NextResponse.json({ error: 'Missing predictionId' }, { status: 400 });
  }
  
  const status = await PredictionStatusManager.getPredictionStatus(predictionId);
  
  return NextResponse.json({
    prediction: status.data,
    debug: {
      kvTtl: await getKV(`prediction:${predictionId}:ttl`),
      queueStatus: await checkReplicateQueue(),
      systemHealth: await checkSystemHealth(),
    }
  });
}

Architecture Benefits

Immediate User Feedback: Users receive instant confirmation that their request is being processed, eliminating the anxiety of waiting without feedback.

Resource Efficiency: Server connections are freed immediately after task submission, allowing the same server to handle hundreds of concurrent users.

Fault Tolerance: Webhook retries and status persistence ensure results are delivered even if temporary network issues occur.

Scalable Processing: AI processing happens on specialized services while your API servers remain available for new requests.

Progressive Enhancement: Frontend can provide real-time status updates, estimated completion times, and allow users to continue using the application while waiting.

Robust Error Recovery: Failed predictions automatically trigger credit refunds and user notifications, while webhook retry mechanisms ensure eventual consistency even during temporary outages.

Production Performance

This architecture has been successfully deployed at Fastjrsy, an AI jersey design generator that processes thousands of image generation requests daily. Key performance metrics:

Response Time: API endpoints respond in under 200ms
Throughput: Handles 50+ concurrent prediction requests per second
Completion Rate: 99.2% of predictions complete successfully
Average Processing Time: Under 5 seconds for speed mode, under 8 seconds for quality mode, 15 seconds for scene generation
User Retention: 40% reduction in page abandonment during generation

The architecture scales efficiently across Cloudflare's global edge network, providing consistent performance regardless of user location or current load.

This async processing pattern transforms long-running AI operations from user experience blockers into smooth, progressive interactions that maintain user engagement and system efficiency at scale.

Building you first AI Agent for a Task Organizer with Still.js and Groq

Sat, 19 Jul 2025 16:07:28 +0000

AI is now widely used for solving various problems, especially in the tech industry and among software developers. This article explores a specific use case of AI, focusing on certain key aspects while keeping the end-user in mind.

In summary, the task organizer is a software tool designed to help users manage tasks over various timeframes. While powerful Project/Task Management solutions like Motion and ClickUp exist, here we’ll demonstrate the capabilities of Still.js by discussing and building a small PoC covering a specific and tiny aspect.

AI Agent vs Agentic AI

According to google AI Overview "AI agents are specialized tools designed for specific, well-defined tasks, while Agentic AI represents a broader concept of autonomous, goal-driven systems that adapt to changing situations, and coordinate actions with minimal human oversight"

From Generative AI to Generative UI

This concept involves generating UI dynamically based on user prompt. Different prompts produce different UI components. In our case, we’ll handle it using a client-side approach.

How our Agent will it work essentially?

The user will write a text with the tasks he'll do, specifying what, how and when
Content is submitted to the Agent/LLM, which generates task(s)
UI parses LLM response and decides, how/what predefined component to render/present
User can then ask the agent to mark tasks as completed.

Different points of the design systems are addressed in here, for the implementation there is a “hands-on youtube video” where a tiny implementation is built from scratch. Bellow is the design system depicting an overview of the solution.

We’ll consider essentially 3 main parts, the AI provider, a custom Backend API, and the UI which we describe as follow:

AI Provider supplies intelligent capabilities

Backend provides a robust and secure integration with the AI, also serves the Frontend

Frontend handles user input and displays the AI's results.

In this use case, Still.js features like runtime form generation and centralized form validation are leveraged to manage task completion. The structure includes three components:

Home (main component),
TaskDay (group of tasks),
Task (individual tasks). Tasks report back to the Home component as form, enabling them to be marked as completed.

Groq infrastructure is what we'll use for LLM, however other AI providers like Google Gemini, ChatGPT, Copilot, LLaMa, or even offline/on-prem options like Ollama could also be used.

Prompts are sent from the Still.js enabled UI to the AI engine via chat, mainly as text, but it could support voice or audio.

In production, long-term memory for LLMs or agents often requires connecting to external sources like vector databases, highlighting the need for a strong backend. Our agent will have a short-term memory which will be handle as the bellow design:

Tool use and workflow management are key in Agentic AI, enabling both agency (thinking) and predictability (acting). For some tasks, a robust backend better supports these capabilities. The agent we'll build has moderate predictability.

In the demo, we’re connecting the UI straight to the AI engine for the sake of the size of the hands-on video tutorial, however this is also a valid scenario when using ephemeral API token.

Are you still here? Less talking, more doing, click here and follow the tutorial to build your first AI Agent.

See you there

Coder Legion - Recent posts tagged generative-ui