agent scheduler: design doc update - #4801
Conversation
Summary of ChangesHello @qi-min, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request introduces a comprehensive design document for a new 'Agent Scheduler' within the Volcano project. The primary goal is to provide a specialized, high-performance scheduling solution for latency-sensitive AI Agent workloads, which the current Volcano Scheduler is not optimally suited for. The design emphasizes rapid task scheduling, high throughput, and seamless collaboration with the existing Volcano Scheduler through an innovative node sharding mechanism. It outlines a detailed architecture, including a multi-worker framework, an optimized scheduling queue, and a conflict-aware binder, all aimed at improving overall cluster efficiency and workload management for diverse computational demands. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
volcano-sh-bot
added
the
size/L
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a design document for a new Agent Scheduler. The document is well-structured and provides a good overview of the proposed architecture, including problem statement, design goals, components, and scheduling workflow. My review focuses on improving clarity, fixing typos and grammatical errors, and pointing out incomplete sections to ensure the design is robust and easy to understand. I've identified some issues in the YAML examples that need to be corrected to be valid.
Signed-off-by: qi-min <qim_34@163.com>
f7fdaac to
6ce688b
Compare
| 2. The scheduling strategy for an Agent might be different from other workloads. Agent workloads may not require topology spread or pod affinity. Instead, they can be scheduled onto nodes with smaller or fragmented resources to better utilize resource fragments and improve overall cluster efficiency. This requires different scheduling strategies configured for different workloads. | ||
|
|
||
|
|
||
| ### Design Goal |
There was a problem hiding this comment.
Define clear KPIs (e.g., "Support 5000 nodes, 1000 Pods/s throughput with P99 scheduling latency < 200ms").
| metadata: | ||
| name: volcano | ||
| spec: | ||
| nodesDesired: #Nodes should be used within this shard. |
There was a problem hiding this comment.
The nodes in nodesDesired is configured by user manually?
There was a problem hiding this comment.
Also how to allocate nodes among multiple schedulers?
There was a problem hiding this comment.
nodesDesired is assigned by sharding-controller, sharding-controller is response for collect metrics and adjust nodes in NodeShard
There was a problem hiding this comment.
After nodeDesired updated, scheduler will follow this flow this sync nodes https://github.com/volcano-sh/volcano/blob/6ce688bcbcbc26a8c8f24697ac6242e1f8783441/docs/design/agent-scheduler.md#sharding-synchronization
|
|
||
| 4. If none of the nodes in the result are available, the Pod is push back to the scheduling queue in high priority for re-scheduling. E.g. both nodes allocated for Pod C are based on v1 that is used in previous bind, so both node are rejected by binder and binder push Pod C back to queue. | ||
|
|
||
| 5. In new allocation based on node with updated binding version, binder will treat the allocation is based on new resource view of node and then allow binding. E.g. node1 (v2) is allocated for Pod D based previous bound Pod information in node, so it is not treated as a conflict. |
There was a problem hiding this comment.
The Pod D is binded to node 1, set node 1 v3 in the binder-flow.png, maybe not correct?
There was a problem hiding this comment.
After Pod D bind 同node1, then update the bindversion of node1 from v2 to v3. Next pod (let's say Pod E) scheduled to Node1 after bind version change is based on node 1 v3, version v3 is higher than v2 which is used in last binding on node 1, so binder will allow binding Pod E to node 1
|
|
||
|  | ||
|
|
||
| 1. Each scheduling result records more than one allocatable nodes (number is configurable), with binding version recorded in each node at the time of allocation. |
There was a problem hiding this comment.
Who generate binding version and how is it updated?
There was a problem hiding this comment.
At step 2. binder will change bind version after a node checking passed.
| ### Architecture Overview | ||
| An independent Agent scheduler is introduced to identify and make fast scheduling for Agent workloads. The scheduler improves the scheduling rate of individual Pods through optimized scheduling strategies and in-time Pod scheduling. It further increases overall scheduling throughput by leveraging parallel scheduling with multiple workers. | ||
|
|
||
| When Agent workloads coexist with other workloads, the sharding controller dynamically divides nodes into shards based on defined policies like resource threshold, node type, etc. Each scheduler obtains schedulable nodes through shard synchronization and selects or prioritizes the corresponding nodes for scheduling. This enables multiple schedulers to perform parallel scheduling of different workloads based on different shards. Refer to the [sharding controller design and shard strategy](shard-controller.md) for details. |
There was a problem hiding this comment.
shard-controller.md missing now
There was a problem hiding this comment.
@hzxuzhonghu shard-controller.md is in pr
| - **Unschedulable Pods Pool**: Stores pods that have failed scheduling and are determined to be unschedulable under current cluster conditions. | ||
|
|
||
| A key enhancement over the standard queue logic is the **Urgent Retry Mechanism** for binding conflicts. | ||
| When the Conflict-Aware Binder detects a conflict (i.e., multiple workers trying to bind to the same node), |
There was a problem hiding this comment.
should this happen? I kind of remember the coordinator in agent-scheduler will allocate nodes to different workers
There was a problem hiding this comment.
based on latest design, all workers in agent-scheduler share nodes in same shard to avoid cutting nodes in shard into small pieces, so a binder is introduced to check conflict
There was a problem hiding this comment.
I am very concerned with the conflicts probability. not sure did you check the conflict metrics during test
There was a problem hiding this comment.
In current test with in small scale, conflict rate is low when candidate nodes number exceed worker count, e.g. 3 candidates in each allocate and 2 workers.
I agree we need add metrics for conflicting and test in large concurrence to find out a proper ratio between candidates number and worker number which may have lowest conflict rate
|
|
||
| 1. When new unscheduled pending pods are watched, they are added to the **activeQ**, the pods will be popped from the **activeQ** and tried to be scheduled. | ||
| 2. If scheduling fails for the pod, it will be added to the **unschedulable pods pool**. | ||
| 3. When cluster events occur (such as node updates, pod deletions, etc.), the scheduler checks pods in the **unschedulable pods pool**, if the event makes a pod potentially schedulable, the pod is moved to either **backoffQ** or **activeQ**, depending on whether it is still within its backoff period. |
There was a problem hiding this comment.
It does not document the backoffQ clearly, when a pod is enqueued
There was a problem hiding this comment.
A pod scheduling state machine or other design diagrams might make this concept clearer.
There was a problem hiding this comment.
Thanks for your suggestions, I have added a scheduling queue diagram, please check @MahaoAlex
And about the backoffQ I have already explain in the front, but I refined it a bit more, please check @hzxuzhonghu :
BackoffQ: Stores pods that are potentially schedulable (e.g., triggered by cluster events) but are waiting for a backoff period to expire. This prevents the scheduler from being overwhelmed by frequent retries, ensuring high scheduling throughput.
BackoffQ primarily avoids invalid events and prevents the repeated checking of unschedulable pods, thereby improving throughput.
Signed-off-by: JesseStutler <chenzicong4@huawei.com>
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: hzxuzhonghu The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
volcano-sh-bot
added
the
approved
7 participants
What type of PR is this?
/kind docs
What this PR does / why we need it:
Design doc for a new scheduler to support fast scheduling of Agent Workload
Proposal details: proposal
Which issue(s) this PR fixes:
Fixes #
#4722
Special notes for your reviewer:
Does this PR introduce a user-facing change?