TransWikia.com

deep learning and uncertainty estimation

Data Science Asked on January 22, 2021

Recently I got very interested in NLP applications of deep learning. Diving into literature (on arXiv for instance) I noticed that is very unpopular to quote and estimate uncertainties on scores of ML tasks. In the era of pretrained language model (i.e. bert, gpt etc.) all further improvements quoted in papers seems to be compatible among each other within 1 or less standard deviations, making all the results statistically compatible with a fluctuation due to stochastic optimization in neural network training procedure (at fixed data-set). I am a physicist, and this looks really confusing to me when compared to the statistical treatment of experimental data performed by routine in laboratories.
I am sure this question has already been discussed in the past in ML/Data Science community, could you point me some review or paper addressing this issue?
Also, could you please share with me your thoughts about?
Thank you very much

One Answer

In Physics, the uncertainty of a value can be calculated or estimated from some well established rules. For various reasons, these rules are not as well established in Machine Learning/AI. The actual uncertainty value in a machine learning model can suffer from several issues, which are difficult to detect or track down. Here are just a few:

  1. New values (values outside the distribution) in the Test dataset that were not seen in Training: How can you know what you haven't seen before
  2. Data collection errors
  3. Not matching the model to the data.
  • Non-random datasets, skew, etc.
  • Feature engineering
  • Model selection

Usually having a highly accurate uncertainty is not necessary, as any change or inaccuracy will usually have little effect in the actual predictions.

Reference:

https://ai.googleblog.com/2020/01/can-you-trust-your-models-uncertainty.html

Answered by Donald S on January 22, 2021

Add your own answers!

Ask a Question

Get help from others!

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP