A year of compressed captures from my busiest scraper takes about four gigabytes. That is less per month than one SIM card costs me. It is also the only reason a parsing bug that ran undetected for 19 days cost me an afternoon instead of three weeks of history.
The bug itself was mundane. A site renamed one CSS class, my selector stopped matching the price element and started matching the shipping estimate, and nothing threw an error. Every row still contained a plausible number. I noticed because a competitor appeared to be selling at four dollars and holding.
If you scrape anything to feed your own decisions, this is the failure mode you should be designing against. Not the scrape that breaks loudly. The one that keeps producing well formed rows that are quietly wrong.
The raw capture is not yours to edit
Write down exactly what came off the wire, then never touch it again.
That means the HTML, or the JSON body, or the exact bytes of the CSV somebody sent you. Not the parsed fields. Not the version with the whitespace stripped. The response, as received, in a folder named for the capture date, compressed, and read only for the rest of its life.
The argument against this is always storage, and the numbers do not support it. Compressed HTML is tiny. A scraper doing a few thousand pages a day will not trouble a cheap disk inside a year.
The argument for it shows up the first time you find a parser bug. With the raw pages, you fix the selector, rerun the parse and the history repairs itself. Without them, the bad rows are the only rows you will ever have for those dates, and your only remaining decision is whether to explain the gap or quietly leave it.
Store what you sent as well. The URL, the request headers, the response code. Six weeks later that is the only evidence you will have of why a page came back strange.
Layers, not one clever script
The convenient way to build a scraper is one file that fetches, parses, normalises and writes a finished table. It is fewer moving parts and it is the thing I did for two years.
The problem is that it collapses four decisions into one irreversible pass. Break the chain into stages that each read one file and write a different one: raw, parsed, typed, deduplicated, joined. Four or five small scripts, each runnable on its own.
Two properties matter more than the stage names. Each stage should be idempotent: run it twice on the same input, get the same output, no duplicates, no appended junk. And each should log its row count, so a jump between two stages is visible the same week it happens.
If rerunning a step feels risky, you will avoid rerunning it, and then you will never fix anything.
Two kinds of duplicate, two different answers
A single dedupe function usually hides two separate questions.
The first: is this the same record I already captured, seen twice? That one is mechanical. Deduplicate on identity. Use a stable ID from the source when there is one, and when there is not, hash a key built from the fields that define the thing. For a business directory scrape, that might be the canonical URL plus the phone number.
Leave the volatile fields out of the key. Put a price or a review count in there and the same record becomes a brand new record every time it changes.
The second question: are these two records that merely resemble each other? That one is a judgement call, and answering it mechanically is how you delete real data. Two branches of the same chain, three streets apart, will look extremely similar to a fuzzy matcher. Collapse them and your count of locations is now wrong in a direction that flatters whatever you are measuring.
My rule is that near duplicates get flagged, never removed. Write a similarity score into its own column and skim the top of that list by hand once a week. Ten minutes, and it has never cost me a real record.
Blank, zero and missing are three different facts
This is the quietest way a clean looking dataset lies to you.
Missing means no value was captured. The request timed out, or the element was absent from the page.
Blank means the field existed and was empty. The seller left the description box alone.
Zero means the value is genuinely zero. No reviews yet. Free shipping. A free tier that really does cost nothing.
Take a review count as the worked example. Fill the missing ones with zero and every listing you failed to scrape now looks like an unpopular listing. Drop the blanks and your listing count shrinks for a reason that has nothing to do with the market. Treat zero as missing and every brand new listing vanishes, usually the exact segment you wanted.
Keep all three apart end to end. A null for missing, an empty string for blank, a zero for zero. Where a tool refuses to hold that distinction, add a companion column that records which of the three states the value was in.
The test: from the cleaned file alone, can you say how many rows lack a review count because the scrape failed, versus how many lack one because the listing has no reviews? If you cannot, the cleaning step threw information away and did not mention it.
Damage you cannot undo later
Three kinds of damage that are hard to reverse once written.
Encoding is the worst because it is permanent. Decode a UTF-8 page as Windows-1252 and a name with an accent in it comes out mangled. Save that and the original bytes are gone. You will find out when a customer tells you their company name is spelled wrong. Keeping bytes in the raw layer and decoding downstream means a wrong guess costs you a rerun rather than the data.
Money is a string until proven otherwise. The same figure can arrive with a comma for the thousands separator, a full stop for the thousands separator, the currency symbol attached, or no symbol at all because the page put it in a sibling element. I store the raw price string in a column next to the parsed number every single time. It has caught two format changes that would otherwise have had my parser reading a tenth of the real value.
Clocks are the sneaky one. Scraped pages love relative timestamps. “Updated 2 hours ago” is relative to the site’s clock, resolved against your server’s clock, in whatever timezone that server sits in. Store UTC, keep the original string, and write down which zone you assumed when the source gave you none. An eight hour offset in a daily series does not look like a bug. It looks like a daily pattern, and you will waste an afternoon explaining it.
Count what you throw away
Any step that removes rows has to report what it removed, somewhere a human will actually see it.
Rows in, rows out, rows dropped, and a one line reason per category. Mine writes a log line per run that reads something like: 11,400 in, 11,106 out, 294 dropped for an unparseable price.
The absolute number is not the interesting part. The change is. When the drop count goes from 200 to 4,000 overnight, something upstream moved and you want that today. Silent drops are how a dataset loses a third of itself with nobody noticing.
Validate at the door
Check the data where it enters the pipeline, not deep inside the analysis where a bad value surfaces as a confusing result instead of an error.
A small schema does the job. Field names, types, and a handful of range checks: price is numeric and above zero and below a ceiling you would find absurd, the timestamp falls inside the last 24 hours, the required fields are present.
When a row fails, quarantine it in a separate file and carry on with the rest. A quarantine file that starts filling up is a message you can act on. A cleaning function that silently coerces bad values into plausible ones is not.
The schema doubles as the only written record of what you believed the data was, which ends up more useful than the checks themselves.
The step I got wrong
Mine was not subtle. I had a normalise step that stripped whitespace and lowercased every text field, because I was joining on seller name and the source was inconsistent about capitalisation.
Sensible, except I applied it to every string column in the frame, including the product identifier. Some of those identifiers were case sensitive. Two genuinely different products whose codes differed only by case became one product.
Five weeks passed before a count refused to reconcile against a manual check and I went looking. Roughly 400 rows had merged into each other.
The repair took an afternoon, because the raw captures were there and I could reparse the lot. The cost was elsewhere: I had made a stocking decision on a merged number and had to unwind it, and every summary I had produced in those five weeks was wrong in a way I could not describe precisely until the reparse finished.
The lesson is narrower than you would think. Lowercasing was never the problem. Applying a transformation to a class of columns was. “Clean all text fields” is how you break the one text field that was not really text. Name the columns explicitly. It takes an extra minute and makes the assumption visible to whoever reads the script next, usually you.
What this actually costs to run
A day of setup, roughly. A capture script that writes raw and stops. Three or four small scripts that each do one job. A log line with counts. One schema file.
None of it is clever. It is mostly declining to do the convenient thing. And it does not make your data correct, which no process can promise. It makes your data recoverable, and that is the difference between finding a parsing bug and being able to do something about it.
More plain English walkthroughs on data quality and the numbers a one person business runs on are at Data Research Analysis Collection.
Get new guides and videos first — join the Telegram channel.