Conventional Functional connectivity (FC) analysis focuses on characterizing the correlation between two brain regions, whereas the high-order FC can model the correlation between two brain region pairs. To reduce the number of brain region pairs, clustering is applied to group all the brain region pairs into a small number of clusters. Then, a high-order FC network can be constructed based on the clustering result. By varying the number of clusters, multiple high-order FC networks can be generated and the one with the best overall performance can be finally selected. However, the important information contained in other networks may be simply discarded. To address this issue, in this paper, we propose to make full use of the information contained in all high-order FC networks. First, an agglomerative hierarchical clustering technique is applied such that the clustering result in one layer always depends on the previous layer, thus making the high-order FC networks in the two consecutive layers highly correlated. As a result, the features extracted from high-order FC network in each layer can be decomposed into two parts (blocks), i.e., one is redundant while the other might be informative or complementary, with respect to its previous layer. Then, a selective feature fusion method, which combines sequential forward selection and sparse regression, is developed to select a feature set from those informative feature blocks for classification. Experimental results confirm that our novel method outperforms the best single high-order FC network in diagnosis of mild cognitive impairment (MCI) subjects.