In recent years, unscrupulous hacker attacks have led to the information leakage of enterprise and individual network users, which makes the network security issue unprece-dented concerned. Botnet and dark network, which use C & C channel of unknown proto-col format to communicate, are the important parts. With the development of wireless mo-bile networks technology, this problem becomes more prominent. Classifying and identi-fying the unknown protocol features can help us to judge and predict the unknown attack behavior in the Internet of things environment, so as to protect the network security. Firstly, this paper compares the protocol features to be detected with the existing protocol features in the feature base through the vectorization operation of protocol features, selects the feature set with high recognition rate, and judges the similarity between protocols. The ex-tracted composite features are digitized to generate 0-1 matrix, then Principal Component Analysis (PCA) dimension reduction is processed, and finally clustering analysis is carried out. A Clique to Protocol Feature Vectorization (CPFV) algorithm is designed to improve the efficiency of protocol clustering and finally generate a new protocol format. The ex-perimental results show that compared with the traditional Clique and BIRCH algorithms, the proposed optimization algorithm improves the accuracy by 20% and the stability by 15%. It can cluster and identify unknown protocols accurately and quickly