Data And Machine Learning Basics

Learning From Examples, In Detail

In week 1 we saw that machine learning finds rules from examples instead of having a person write them. This week we look at the pieces. A dataset is a collection of examples. Each example is one case, such as one email, one house, or one photo.

Machine learning comes in several kinds. In supervised learning each example comes with the correct answer, and the model learns to predict it. In unsupervised learning there are no answers, and the model looks for structure such as groups of similar items. In reinforcement learning an agent tries actions and receives rewards. Most practical systems you meet in this course are supervised, so we focus there.

Features And Labels

A feature is one measurable property of an example. For a house, features might be the floor area, the number of rooms, and the age of the building. A label is the answer we want the model to predict, such as the sale price. Supervised learning uses a table in which each row is an example, each column is a feature, and one extra column holds the label.

Choosing and preparing features is a large part of the work. A model can only use what it is given. If the price depends on the neighborhood and you leave out the neighborhood, the model cannot recover it. Features must be numbers in the end, so text and categories are converted into numbers first. Week 4 shows a powerful way to do this for words.

There are two common prediction tasks. Regression predicts a number, such as a price. Classification predicts a category, such as spam or not spam. The same ideas of training and evaluation apply to both.

What Training Means

A model has internal settings called parameters. A very simple model for house prices might say: price equals a number times area plus another number. Those two numbers are the parameters. Training means finding parameter values that make the model's predictions close to the true labels.

To measure how close, we use a loss function. The loss is a single number that is large when predictions are poor and small when they are good. For prices, a common choice is the average of the squared differences between predicted and true values.

Training then repeats a simple loop. Make predictions, compute the loss, and nudge each parameter slightly in the direction that lowers the loss. The nudging method is called gradient descent. Picture standing on a hillside in fog and taking small steps downhill until the ground flattens. The size of each step is the learning rate. Steps that are too large overshoot, and steps that are too small take very long.

One full pass through the training data is called an epoch. Models are often trained for many epochs. Neural networks, which we meet in week 3, use this same loop with millions or billions of parameters.

Splitting The Data Honestly

A model must work on new examples, not only the ones it studied. To check this, we hold some data back. We usually make three parts.

The test set must stay untouched until the very end. If you tune your model while looking at the test set, its score no longer tells you how the model will do on new data.

• Training set: the examples the model learns from, often about 70 to 80 percent.
• Validation set: examples used to compare settings and make choices while you build.
• Test set: examples used once, at the end, to report final performance.

Overfitting And Underfitting

Overfitting happens when a model learns the training examples too closely, including their random quirks. It scores very well on training data and poorly on new data. A student who memorizes last year's exam answers without understanding the subject behaves the same way.

Underfitting is the opposite. The model is too simple to capture the pattern, so it does poorly on both training and new data. A straight line cannot describe a curve.

The signal to watch is the gap between training performance and validation performance. A large gap suggests overfitting. Poor scores on both suggest underfitting. Common remedies for overfitting are collecting more data, using a simpler model, stopping training earlier, and adding regularization, which penalizes overly large parameter values. For underfitting, use a more flexible model or better features.

Evaluating A Model

Evaluation means measuring how well a model performs with numbers you can trust. For regression, a common measure is the average error between predicted and true values. For classification, the first measure is accuracy, the share of predictions that are correct.

Accuracy can mislead. Suppose only 1 email in 100 is spam. A model that always answers not spam is 99 percent accurate and useless. For cases like this we use two other measures. Precision asks: of the items the model flagged, how many were truly positive? Recall asks: of all truly positive items, how many did the model find? Which matters more depends on the cost of each mistake. A cancer screening test usually favors recall, because missing a case is costly. A spam filter usually favors precision, because deleting a real email is costly.

A confusion matrix lays out the four outcomes: true positives, false positives, true negatives, and false negatives. Reading it shows exactly which kinds of mistakes a model makes.

A short worked example ties these ideas together. Imagine a clinic that wants to predict missed appointments. Each row is one booking, the features are the day of the week, the lead time, and the patient's earlier attendance, and the label is whether the person came. The team splits the rows by date, trains on the older bookings, tunes on a middle period, and tests on the newest bookings, because that mirrors real use, where the model will always predict the future from the past. Splitting by time instead of at random avoids a quiet form of leakage.

Data Quality And Bias

A model reflects its data. If the data is small, noisy, or mislabeled, the model will be poor. If the data leaves out certain groups, the model may work badly for them. A face recognition system trained mostly on one kind of face is a well-known case of this problem.

Before training, inspect the data. Look at a sample of rows, count missing values, check how balanced the labels are, and ask how the data was collected and by whom. Another trap is data leakage, where information from the answer sneaks into the features. For example, a feature recording whether a refund was issued would reveal whether a customer complained. The model would look excellent in testing and fail in real use.

Good practice is simple. Write down where the data came from, keep the test set separate from the start, and report results with the right measures. Week 8 returns to evaluation of language systems, where the same discipline applies.

Key Terms

Feature

One measurable property of an example that the model uses as input.

Label

The correct answer for an example that the model learns to predict.

Parameter

An internal number in a model that training adjusts.

Loss function

A number that measures how far the model's predictions are from the true labels.

Gradient descent

A method that repeatedly adjusts parameters in the direction that lowers the loss.

Overfitting

When a model fits training data too closely and performs poorly on new data.

Test set

Data held back and used once at the end to report final performance.

Precision

Of the items the model flagged as positive, the share that truly are positive.

Recall

Of all truly positive items, the share that the model found.

Exercise: Train And Test A Classifier In Python

1. Open a free Google Colab notebook, or any Python 3 environment with scikit-learn installed.
2. Paste the code from the slide titled A complete small example and run it. Note the accuracy on the test set.
3. Change test_size from 0.25 to 0.5 and run again. Write down how the score changes and why more held-out data makes the estimate steadier.
4. Replace max_depth=3 with max_depth=None, which lets the tree grow without limit. Print the accuracy on both training and test data.
5. Compare the training and test scores. Write three sentences about whether the larger tree overfits.
6. Print the confusion matrix with sklearn.metrics.confusion_matrix and name which mistake is most common.

Further Reading

• Machine Learning, free online course, Andrew Ng, Stanford and Coursera
• Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, Aurelien Geron, O'Reilly (early chapters)
• Machine Learning Crash Course, Google Developers, free online
• An Introduction to Statistical Learning, James, Witten, Hastie, and Tibshirani, Springer (free PDF)