CMPUT 605-Dynamic-Neural-Network Software Stack  Software Design and Code Generation for Dynamic Neural Networks Department of Computing Science  University of Alberta

Winter 2024

TIME: TBD - two 1:20 minutes sessions per week.
INSTRUCTOR: José Nelson Amaral
 

Calendar Description:   Study the design and implementation of software stack for dynamic neural networks that aim to deliver more complex and more capable neural networks without significant increases in computational requirements.

Course Description and Goals:

The goal of this course is to understand the design, implementation, and the use of dynamic neural networks with the goal of analysing efficiency and efficacy of the implementation of such networks with hardware acceleration. The first part of the course will consist on a literature review to understand the typical applications that benefit from dynamical neural networks and the design choices for the creation of such networks.

Next, an in-dept study of the various types of neural networks, their application and use should help focus on one or a few network designs to focus on to investigate efficient implementations.

The course will then study existing hardware-acceleration solutions and try to establish how some of these can best be used for the implementation of dynamic neural networks.

Finally the course will study the current and future software-stack solutions for the implementation of dynamic neural networks.

The implementation component of the course will consist in either implementing or modifying existing code generation strategies for dynamic neural networks. For new ideas that are generated during the course, we may first work on feasibility and limit studies to determine the potential benefit of changes to automated code generation tools.

Grading:

Weekly discussion of readings        30%
Project Development                    40%
Final Paper                      30%

Course Plan:

• January 04 - February 12: Literature Review:
  bi-weekly meetings for discussion of papers;
• February 13 - March 17: Review of software infrastructure and planning for implementation and evaluation.
  student designs and plan for implementation and evaluation, collect potential benchmarks, create initial code infrastructure
• March 17 - April 15: Implementation and review:
  bi-weekly meetings to review implementation, and refine design;

Initial Selection of Papers to Review:

• [HanIEEE-TPAMI22] Y. Han, G. Huang, S. Song, L. Yang, H.Wang and Y.Wang, "Dynamic Neural Networks: A Survey," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 11, pp. 7436- 7456, 1 Nov. 2022, doi: 10.1109/TPAMI.2021.3117837.
• [SkardingIEEEAccess21] J. Skarding, B. Gabrys and K. Musial, "Foundations and Modeling of Dynamic Networks Using Dynamic Graph Neural Networks: A Survey," in IEEE Access, vol. 9, pp. 79143-79168, 2021, doi: 10.1109/ACCESS.2021.3082932.
• [LiIEEETPDS21] Mingzhen Li , Yi Liu , Xiaoyan Liu, Qingxiao Sun, Xin You, Hailong Yang , Zhongzhi Luan , Lin Gan, Guangwen Yang, and Depei Qian, "The Deep Learning Compiler: A Comprehensive Survey," IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 32, NO. 3, MARCH 2021.
• [HuaIEEEAccess23] C. Hua, X. Cao, Q. Xu, B. Liao and S. Li, "Dynamic Neural Network Models for Time- Varying Problem Solving: A Survey on Model Structures," in IEEE Access, vol. 11, pp. 65991-66008, 2023, doi: 10.1109/ACCESS.2023.3290046.
• [Li-CPVR20] Yanwei Li, Lin Song, Yukang Chen, Zeming Li, Xiangyu Zhang, Xingang Wang, Jian Sun, "Learning dynamic routing for semantic segmentation," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 8553-8562
• [HowardarXiv17] AG Howard, M Zhu, B Chen, D Kalenichenko, W Wang, T Weyand, M Andreetto, H Adam, Mobilenets: Efficient convolutional neural networks for mobile vision applications, arXiv preprint arXiv:1704.04861, 2017ˇ earxiv.org
• [ChenCPVR20] Yinpeng Chen Xiyang Dai Mengchen Liu Dongdong Chen Lu Yuan Zicheng Liu. "Dynamic Convolution: Attention over Convolution Kernels," IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 11030-11039.
• [MayerACMSurveys20] RUBEN MAYER and HANS-ARNO JACOBSEN, "Scalable Deep Learning on Distributed Infrastructures: Challenges, Techniques, and Tools," ACM Computing Surveys, Vol. 53, No. 1, Article 3, February 2020.
• [YuEuroSys18] Yuan Yu et al, "Dynamic Control Flow in Large-Scale Machine Learning," Eurosys, Porto, Portugal, April, 2018.
• [GuoCVPR19] Qiushan Guo, Zhipen Yu, Yichao Wu, Ding Liang, Haoyi Qin, Junjie Yan, "Dynamic Recursive Neural Network," CVPR, 2019.
• [HarleyICCV17] Adam W. Harley, Konstantinos G. Derpanis, Iasonas Kokkinos, "Segmentation-Aware Convolutional Networks Using Local Attention Masks", ICVV 2017.
• [LiCVPR20] Yanwei Li, Lin Song, Yukang Chen, Zeming Li, Xiangyu Zhang, Xingang Wang1, Jian Sun, "Learning Dynamic Routing for Semantic Segmentation," CPVR20.
• [LiuCVPR99] Chenxi Liu, Liang-Chieh Chen, Florian Schroff, "Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation," CVPR19.

University of Alberta

Computing Science