Sky Cone

Week 2: Modeling Challenges and Strategies

Section 1: Select and train a model

1. Modeling overview

Model-centric AI development

Data-centric AI development

2. Key challenges

3. Why low average error isn't good enough

ethnicity 种族

4. Establish a baseline

Establishing a baseline level of performance

Speech recognition example:

Type	Accuracy	Human level performance
Clear Speech	94%	95%
Car Noise	89%	93%
People Noise	87%	89%
Low Bandwidth	70%	70%

garbled 混乱不清的
Human level performance (HLP)

Ways to establish a baseline

• Human level performance (HLP)
• Literature search for state-of-the-art/open source
• Quick-and-dirty implementation
• Performance of older system

Baseline helps to indicate what might be possible. In some cases (such as HLP) is also gives a sense of what is irreducible error/Bayes error.

5. Tips for getting started

6. Selecting and Training a Model

Question 1
Which of these is a more accurate description of a data-centric approach to ML development?

1. Holding the neural network architecture fixed, work to improve the data to do well on the problem.
2. Holding the training data fixed, work to improve your neural network's architecture to do well on the problem.

Question 2
Say you have an algorithm that diagnoses illnesses from medical X-rays, and achieves high average test set accuracy. What can you now say with high confidence about this algorithm?

1. It does well even on rare classes of diseases.
2. Its diagnoses are roughly equally accurate on all genders and ethnicities, so we are confident it is not biased against any gender or ethnicity.
3. The system can be safely deployed in a healthcare setting.
4. None of the above.  

Question 3
Which of these statements about establishing a baseline are accurate? Check all that apply.

1. Open-source software should not be used to establish a baseline, since the performance of a good open source implementation might be too good and thus too hard to beat.
2. Human level performance (HLP) is generally more effective for establishing a baseline on unstructured data problems (such as images and audio) than structured data problems
3. For unstructured data problems, using human-level performance as the baseline can give an estimate of the irreducible error/Bayes error and what performance is reasonable to achieve.
4. It can be established based on an older ML system

Question 4
On a speech recognition problem, say you run the sanity-check test of trying to overfit a single training example. You pick a clearly articulated clip of someone saying "Today's weather", and the algorithm fails to fit even this single audio clip, and outputs "______". What should you do?

1. Debug the code/algorithm/hyperparameters to make it pass this sanity-check test first, before moving to larger datasets.
2. Use data augmentation on this one audio clip to make sure the algorithm hears a variety of examples of "today's weather" to fit this phrase better.
3. Create a training set of this example repeated 100 times to force the algorithm to learn to fit this example well.
4. Train the algorithm on a larger dataset to help it to fit the data better.