Panel on Questioning Evaluation and Resource consumption for Deep LMs in Real World Scenarios

Performance evaluation and resource usage are important aspects to be
accounted for, in the design of Deep Learning Models, such as Deep
Language Models for NLP applications. Evaluation is usually based on
standard benchmarks, but these are limited in size, availability, and
applicability to real industry use cases. On the other hand, Deep
Learning Models are reaching today a trillion parameters, with a trend
like a novel new Moore's law, which of course can't be sustainable in
the long term, since the training of these models requires huge amounts
of resources, so that usage is mostly of fine-tuned pre-trained models
or smaller ones. Big AI players are competing to the largest models,
having a carbon footprint like an industrial plant, requiring high
costs, time and resources for each training activity.