JISE


  [ 1 ] [ 2 ] [ 3 ] [ 4 ] [ 5 ] [ 6 ] [ 7 ] [ 8 ] [ 9 ] [ 10 ] [ 11 ] [ 12 ] [ 13 ] [ 14 ] [ 15 ]


Journal of Information Science and Engineering, Vol. 36 No. 6, pp. 1339-1351


Student-Centric Network Learning for Improved Knowledge Transfer


HONG-JI WANG1, XIANG XU2, BAO-MIN XU1, SHUANG-YUAN YU1
AND QUAN-XIN WANG3
1School of Computer and Information Technology
Beijing Jiaotong University
Beijing, 100044 P.R. China

2School of Computing Science
Simon Fraser University
Vancouver, V5B3J1 Canada

3Department of Computer Science
Beijing Jiaotong University Haibin College
Huanghua, 061199 P.R. China
E-mail: {18120466; bmxu;shyyu; qxwang}@bjtu.edu.cn; xuxiangx@sfu.ca


In the context of model compression using the student-teacher paradigm, we propose the idea of student-centric learning, where the student is less constrained by the teacher and able to learn on its own. We believe the student should have more flexibility during training. Towards student-centric learning, we propose two approaches: correlation-based learning and self-guided learning. In correlation-based learning, we propose to guide the student with two types of correlations between activations: the correlation between different channels and the correlation between different spatial locations. In self-guided learning, we propose to give the student network the opportunity to learn by itself in the form of additional self-taught neurons. We empirically validate our approaches on benchmark datasets, producing state-of-the-art results. Notably, our approaches can train a smaller and shallower student network with only 5 layers that outperforms a larger and deeper teacher network with 11 layers by nearly 1% on CIFAR-100. 


Keywords: knowledge transfer, student-teacher paradigm, correlation-based learning, selfguided learning, dense convolution

  Retrieve PDF document (JISE_202006_13.pdf)