As more smart devices rely on artificial intelligence to process data locally, improving how these devices run complex neural networks is crucial. A newly published research paper introduces Para-Pipe, a novel method designed to optimize deep learning performance on System-on-Chips (SoCs) found in edge devices like smartphones, drones, and IoT gadgets. This approach promises to balance two important factors: how fast the AI can produce results (latency) and how much work it can handle over time (throughput), all while using energy more efficiently.
Key Takeaways
- Para-Pipe combines pipelining and operator parallelism to speed up neural network inference on heterogeneous SoCs.
- It fine-tunes parallelism both within individual pipeline stages and across different stages to balance latency and throughput.
- Testing on real SoCs, including Amlogic and Black Sesame Technology chips, shows up to 11% better energy efficiency compared to traditional pipelining methods.
- Para-Pipe generates multiple optimal configurations, allowing developers to choose the best trade-off between speed and efficiency for their specific needs.
Modern edge devices often use heterogeneous SoCs, which combine different types of processors such as CPUs, GPUs, and specialized accelerators on a single chip. Running deep learning models efficiently on these chips is challenging because neural networks consist of many interconnected operators (basic computation units) that can often be executed in parallel. Traditional methods distribute these operators across processors in a pipeline, where each stage processes data sequentially. While pipelining improves throughput, it can introduce latency, meaning longer wait times for each individual inference result.
Para-Pipe addresses this challenge by exploiting hierarchical operator parallelism. In simple terms, it looks for opportunities to run multiple operators simultaneously within the same pipeline stage (intra-stage parallelism) and also manages how different stages run in parallel (inter-stage parallelism). The framework intelligently maps parts of the neural network onto the processing units in a way that balances the trade-off between latency and throughput. This hierarchical approach reduces the communication overhead between processors—a common source of delays and energy waste.
The researchers evaluated Para-Pipe on two SoCs: an Amlogic chip featuring ARM big.LITTLE CPUs and a GPU, and a Black Sesame Technology SoC with a deep learning accelerator and two digital signal processors (DSPs). Their experiments showed that configurations optimized for throughput using Para-Pipe achieved significant energy savings—11% better than purely pipelined approaches and 23.3% better than non-pipelined parallel execution. Additionally, Para-Pipe’s flexible design allows it to produce multiple Pareto-optimal setups, giving system designers options to prioritize speed or efficiency depending on the application.
This work highlights the importance of smart scheduling and resource management in edge AI devices, where power and speed constraints are critical. By enabling more efficient use of heterogeneous processing units through hierarchical parallelism, Para-Pipe could help unlock faster and more energy-conscious AI applications on everyday devices. Future research may explore extending this approach to even more diverse hardware platforms and more complex neural network architectures, potentially broadening its impact across the growing edge computing landscape.
Based on research published on arXiv by Yujie Zhang, Huiying Lan, Ehsan Aghapour et al..