How to deduplicate rows in a CSV file with Python

If every row in your CSV is a genuine full duplicate, Python's standard library is enough — no pandas install needed. Here's a script that works on any file, keeping the first occurrence of each row and dropping the rest:

import csv, sys

def dedupe(path_in, path_out):
    seen = set()
    with open(path_in, newline="", encoding="utf-8") as f_in, \
         open(path_out, "w", newline="", encoding="utf-8") as f_out:
        reader = csv.reader(f_in)
        writer = csv.writer(f_out)
        for row in reader:
            key = tuple(row)
            if key not in seen:
                seen.add(key)
                writer.writerow(row)

if __name__ == "__main__":
    dedupe(sys.argv[1], sys.argv[2])

Run it with python3 dedupe.py input.csv output.csv.

When "the whole row" isn't the right definition of duplicate

Most real exports aren't that clean. Two rows are "the same customer" even though the phone number is formatted differently, or the email has different casing, or one row has a typo in the company name. To dedupe by a single column instead of the whole row, swap the key:

key = row[2].strip().lower()  # e.g. column index 2 is the email

That covers a lot of cases. It stops covering them the moment you need more than one rule at once — normalize a date format and dedupe by email and map five inconsistent header names to the ones your next tool expects. At that point you're not writing a five-line script anymore, you're writing and testing a small program, and that's the part people usually pay someone else to do.

The finished version, free

The five lines above dedupe on the whole row. csvclean.py is the finished version of the same idea: dedupe on any column, ignoring case and padding, plus whitespace, ISO dates, header cleanup, and a --report mode that tells you what is wrong before it changes anything. It reads .xlsx directly too — no save-as-CSV step, and no dependency: a spreadsheet is a zip of XML, and Excel dates come back as dates rather than as the day numbers it stores. It also reads 1.234,56 and 1,234.56 correctly, which is harder than it looks.

csvclean.py on GitHub — one file, no dependencies, public domain, tested on Python 3.9 to 3.13. Keep it, change it, ship it in something you sell.

Or skip writing it yourself

Send a sample of the file, and get back a script that dedupes by whichever field actually identifies a duplicate in your data, normalizes dates and number formats, and maps columns to the headers you need — yours to re-run on every future export.

49€ — one-time, fixed, 48h

Buy — Stripe checkout details

← See the other two packages (website scraping, API/webhook bridges)