Collecting 40k offers an hour across 100+ marketplaces, with proof attached

The product monitors price offers — the brand's and its competitors' — across many marketplaces, 24/7. The hard rule came from the business side: the whole universe of offers has to be collected inside a one-hour window that starts at :00 and closes at :59.

I designed and run the collection end to end, from the architecture to keeping it healthy in production.

A serial crawler could never fit that window, so the collection scales horizontally on Kubernetes — 100 GB+ machines, 150+ pods. Each collection stores a screenshot as evidence, which is what lets the client audit any offer, and an anti-detection layer keeps the crawlers from being blocked as bots.

It costs more infrastructure and permanent maintenance of the anti-bot evasion. That is a cost I chose knowingly, because the window and the proof are what the client is actually buying.

40k+ offers monitored across 100+ marketplaces, complete inside the hourly window, with visual proof per offer.