Publications
SUE: Sparsity-based Uncertainty Estimation via Sparse Dictionary Learning
The growing deployment of deep learning models in real-world applications necessitates not only high predictive accuracy, but also mechanism to identify unreliable predictions, especially in high-stakes scenarios where decision risk must be minimized. Existing methods estimate uncertainty by leveraging predictive confidence (e.g., Softmax Response), structural characteristics of representation space (e.g., Mahalanobis distance), or stochastic variation in model outputs (e.g., Bayesian inference techniques such as Monte Carlo Dropout). In this work, we propose a novel uncertainty estimation (UE) framework based on sparse dictionary learning by identifying dictionary atoms associated with misclassified samples. We leverage pointwise mutual information (PMI) to quantify the association between sparse features and predictive failure. Our method – Sparsity-based Uncertainty Estimation (SUE) – is computationally efficient, offers interpretability via atom-level analysis of the dictionary, has no assumption about the class distribution (unlike Mahalanobis distance). We evaluated SUE on several NLU benchmarks (GLUE and ANLI tasks) and sentiment analysis benchmarks (Twitter, ParaDetox, and Jigsaw). In general, SUE outperforms or matches the performance of other methods. SUE performs particularly well when there is considerable uncertainty in the model, i.e., when the model lacks high precision.
Adaptive Greedy Layer Pruning: Iterative Layer Pruning with Subsequent Model Repurposing
Reducing the memory requirements during inference time in pre-trained language models (PLMs) constitutes a key challenge. In this paper, we rigorously investigate the possibility of progressively removing layers from PLMs during their fine-tuning process, in such a way that their final task performance degrade minimally. Our proposed approach not only provides a considerable reduction in the inference cost of using PLMs, but it also highlights the importance of distinct layers, via the identification of layers with marginal contribution to downstream task performance. Our experiments, encompassing seven diverse tasks, corroborate that the exclusion of less pertinent transformer layers facilitates more efficient inference without causing serious degradation of task performance. Indeed, we were able to omit up to 2.2x more layers from the investigated PLMs (depending on the backbone model) compared to a strong layer pruning baseline when preserving no less than 95% of the performance of the full backbone model.
Changing the Basis of Contextual Representations with Explicit Semantics
The application of transformer-based contextual representations has became a de facto solution for solving complex NLP tasks. Despite their successes, such representations are arguably opaque as their latent dimensions are not directly interpretable. To alleviate this limitation of contextual representations, we devise such an algorithm where the output representation expresses human-interpretable information of each dimension. We achieve this by constructing a transformation matrix based on the semantic content of the embedding space and predefined semantic categories using Hellinger distance. We evaluate our inferred representations on supersense prediction task. Our experiments reveal that the interpretable nature of transformed contextual representations makes it possible to accurately predict the supersense category of a word by simply looking for its transformed coordinate with the largest coefficient. We quantify the effects of our proposed transformation when applied over traditional dense contextual embeddings. We additionally investigate and report consistent improvements for the integration of sparse contextual word representations into our proposed algorithm.