Dynamic Distributed Scheduling for Data Stream Computing: Balancing Task Delay and Load Efficiency

Abstract

In data stream computing, distributed scheduling is a key technology to improve system performance and resource utilization. Traditional static and rule-based scheduling methods are difficult to adapt to task fluctuations and resource changes in dynamic environments, often resulting in high task delays and resource waste. To solve this problem, this paper proposes a distributed scheduling algorithm based on global optimization, which aims to achieve efficient task allocation and resource utilization by balancing task delay and load balancing. By introducing real data sets and multiple experimental scenarios, this paper conducts a comprehensive evaluation of the algorithm. The experimental results show that the proposed algorithm shows significant advantages under low, medium, and high loads and node failures, especially under high loads and high failure rates. The improvement in task delay and load balancing is particularly obvious. In addition, this paper designs robustness experiments for task bursts and node failures to verify the algorithm's adaptability and resource reallocation efficiency in dynamic scenarios. Although the algorithm performs well under various conditions, there is still room for further improvement. Future work will explore its adaptive ability in ultra-high loads and heterogeneous computing environments. This study provides new optimization ideas for distributed scheduling in data stream computing, which will help promote the efficient and intelligent development of distributed systems.