Federated Learning, Knowledge Transfer, and Knowledge Distillation: At the Junction of Green Machine Learning and Granular Computing

Witold Pedrycz (University of Alberta, Canada),
Abstract: The visible trends of Machine Learning (ML) are inherently associated with the diversity of data and innovative ways they are used in order to carry out learning pursuits. The ongoing objectives of the research agenda are also investigated in the context of green ML (usually referred to as green AI). One can identify three ongoing challenges with far-reaching methodological implications, namely (i)designing in the presence of strict constraints of privacy and security, (ii) efficient model building completed with limited data of varying quality, and (iii) a reduction of computing effort knowledge transfer and distillation.
We advocate that to conveniently address these quests, it becomes beneficial to engage the fundamental framework of Granular Computing to enhance the existing approaches (such as e.g., federated learning in case of (i) and transfer knowledge in (iii)) or establish new directions to the problem formulation. Likewise, it is also essential to establish sound mechanisms of evaluation of the performance of the ML architectures. It will be demonstrated that various ways of conceptualization of information granules in terms of fuzzy sets, sets, rough sets, and others may lead to efficient solutions.
To establish a suitable conceptual ML framework, we include a brief discussion of concepts of information granules and Granular Computing. To proceed with a detailed discussion, a concise information granules-oriented design of rule-based architectures is outlined. A way of forming the rules through unsupervised federated learning is investigated along with algorithmic developments. A granular characterization of the model formed by the server vis-a-vis data located at individual clients is presented. It is demonstrated that the quality of the rules at the client�s end is described in terms of granular parameters and subsequently the global model becomes represented as a granular construct. The roles of granular augmentations of models in the setting of granular knowledge distillation are outlined. It is shown how the agenda of green ML is effectively realized by exploring information granules and stressing an importance of the holistic perspective at critical trade-offs among interpretability, enormous computational overhead, and transparency of predictors and classifiers.
The exposition of the material is self-contained; all main prerequisites are covered in a succinct manner.