Clean the Data Before You Train Anything

A model learns exactly what the data teaches it, mistakes included. The duplicated rows, the label someone typed with a trailing space, the blank that meant “still active” to whoever left it: none of it raises an error. It shows up later, as a model that scores well in testing and disappoints in use.

My years in audit left me with a habit that carries straight over: before you trust a total, test the rows under it. These are the five problems I would check for on any dataset before training anything, how to spot each one, and a line or two of pandas where it helps. The examples assume a DataFrame called df, plus train and test where a split matters.

Duplicates

They come from exports run twice, joins that multiplied rows, and the same record typed two slightly different ways. Duplicates give some examples extra weight in training, and when copies land on both sides of a train/test split, the test score measures memory instead of learning.

Exact copies take one call, df.duplicated().sum(). Near-copies need normalising first:

key = df["name"].str.strip().str.lower().str.replace(r"\s+", " ", regex=True)
df.assign(key=key).duplicated(subset=["key", "invoice_no"]).sum()

Read a sample of what you found before dropping anything. Two identical rows are sometimes two real, identical events, and deleting them is its own bug.

Leakage between train and test

Leakage is any route by which the answer reaches the model during training in a way it never will in use. Three forms cover most of it.

The same entity on both sides. If one customer has fifty rows scattered randomly between train and test, the model can score well by recognising the customer rather than learning the pattern. Split by group instead (scikit-learn’s GroupShuffleSplit does this), then check:

shared = set(train["customer_id"]) & set(test["customer_id"])
len(shared)  # should be 0 if the model will meet new customers in use

Time running backwards. If the model will predict next month from this month, test it on a later period than it trained on. A random split lets it learn from the future.

Columns filled in after the outcome. A “closed reason” field in a model that predicts which cases will close is the answer in disguise. The giveaway is a score that looks nearly perfect. When you see one, take the most important features and ask of each: would we actually know this at the moment of prediction?

A quieter version is fitting a scaler or encoder on the whole dataset before splitting. Keep preprocessing inside a scikit-learn Pipeline so it only ever learns from the training rows.

Inconsistent labels

Run value_counts() on the label column and you will often find one meaning spelt four ways: Paid, paid, PAID with a trailing space, Settled. That part is easy. The harder case is the same input carrying different labels:

df["label"] = df["label"].str.strip().str.lower()
df.groupby("text")["label"].nunique().gt(1).sum()  # inputs with conflicting labels

Conflicts usually mean the labelling rule was ambiguous, changed partway through, or was read differently by two people. Do not settle them by majority vote. Write the rule down, decide the edge cases, and relabel. A model cannot be more consistent than the labels it learned from.

Missing values that mean something

A blank is not always “unknown”. An empty end date can mean the contract is still running. An empty discount can mean none was given. And some blanks are not blank at all: a zero, a -1, a 9999 or a date of 1 January 1900 standing in for “nothing entered”. Filling all of these with the column average destroys the information they carried.

df.isna().mean().sort_values(ascending=False).head(10)  # share missing per column
df.groupby(df["end_date"].isna())["target"].mean()       # does missing predict the outcome?

If missingness predicts the outcome, keep it as a feature, df["end_date_missing"] = df["end_date"].isna(), before you fill anything. Then ask one more question: will the field be missing in the same way when the model runs for real? A field that is blank at prediction time and only filled in later is the leakage problem again.

Units and formats

Dates are the classic. 03/04/2024 is 3 April in India and 4 March in the United States, and a parser left to guess will pick one without telling you. State the format, and read whatever does not fit it:

parsed = pd.to_datetime(df["date"], format="%d/%m/%Y", errors="coerce")
df.loc[parsed.isna() & df["date"].notna(), "date"]  # rows that did not match

Amounts have the same problem with more variety: 1,25,000.00 in Indian digit grouping, a Rs. prefix, the occasional “4.5 lakh” typed into a number column. Strip the formatting explicitly and let anything still unreadable become a blank you can find:

cleaned = df["amount"].astype(str).str.replace(r"Rs\.?|[,\s]", "", regex=True)
df["amount"] = pd.to_numeric(cleaned, errors="coerce")

Units are subtler, because every row looks valid. Suppose two branches of a CA firm export the same kind of ledger, one in rupees and one in thousands of rupees. No single value is wrong. Only df.groupby("source")["amount"].median(), with the sources side by side, gives it away.

Make the checks run themselves

Each check here is a few lines. Put them in one script that runs before every training run and stops with a plain message when something is off: duplicates above zero, a label outside the agreed set, a date column with unparsed rows, a source whose median has jumped. Data that was clean when you started does not stay clean, because the systems producing it keep changing. When a model’s score moves and nobody touched the model, the reason is nearly always in the data, and that script is where you will find it first.