Gaming Parts Monitor

Buying a PC component in Morocco means opening a dozen browser tabs. Each Moroccan retailer names the same product differently, prices it independently, and restocks it on its own schedule. The global price trackers do not crawl local shops, and the local forums repeat whatever price a poster saw last week.

Gaming Parts Monitor (gamingparts.ma) is the thing I built to close that gap. Every morning at 04:00 Casablanca time it crawls 20 Moroccan retailers, merges their catalogues into one canonical product graph, and records one price per store per day. The live site currently tracks 16,359 components across 37,353 store listings. The same codebase also runs an Algerian edition, gamingpartsdz.com, covering 28 Algerian retailers and 10,533 components in dinars. Figures are from the live sites on 9 November 2026.

The product today

The two mechanisms that make it work

Most comparison sites answer “is this a good price?” against list price or against a single global retailer. Neither helps a buyer in Casablanca. The only number they can act on is what the other Moroccan stores charge for the identical product today.

So the platform rests on two mechanisms a generic comparison site could not honestly copy:

1. A daily price census. Every listing is snapshotted once per day, producing a per-store, per-day price series. That history is what makes “is now a good moment to buy” answerable rather than a guess.
2. Bargains measured against the cross-store median. A deal is a listing sitting materially below what the other stores charge for the same product.

Architecture

The backend is Python 3.11, FastAPI, SQLModel and Alembic on PostgreSQL 16. The frontend is Next.js 15 (App Router), React 19 and Tailwind 4, with Zustand and TanStack Query. Docker Compose runs the services: the API, a separate marketplace service, the frontend, a post-deploy cache revalidation job, Caddy for TLS, and a self-hosted Umami for analytics. The codebase is roughly 40,000 lines of Python and 47,000 lines of TypeScript.

Anti-bot scraping that treats rate limiting and challenge as different problems

This was the hard engineering. The store scrapers run on Scrapling, and the base Scraper class owns all transport behind a cost escalation ladder, so a request only gets more expensive when it needs to be:

1. A static fetch with TLS fingerprint impersonation. This handles most requests.
2. A retry with other browser fingerprints.
3. A pooled headless browser that solves Cloudflare Turnstile, bounded process-wide so a worker pool cannot launch dozens of browsers at once.

The bug that shaped the design

The instinct when a store blocks you is to escalate. That is wrong, and it cost me a rewrite. HTTP 429 is not a challenge. A 429 body is not a product page, so parsing it produces garbage, and escalating it to a browser turns one rate-limited store into thousands of browser launches. SetupGame was rate-limiting most requests, and the browser path made it worse.

So the two are separate paths. A challenge (403/503 or a Turnstile page) escalates through the ladder. A rate limit is discarded and the host is backed off by HostThrottle: exponential per host, honouring Retry-After, with a cap on requests in flight per host. A rate-limited host should be paced, not parked.

Since then the same principle has removed work. Where a store exposes a public product API (Osaka Gaming and SetupGame), the scraper takes products from the API instead of parsing HTML. Stores that Cloudflare blocks from the server are refreshed from a local machine and written to the production database through an SSH tunnel.

Protecting the data from a bad crawl

A scraper that silently returns zero products is worse than one that fails loudly. Two guards sit around every run:

• Self-healing selectors. An opt-in mode fingerprints each store’s catalogue grid and relocates it after a redesign, so a renamed CSS class does not empty a store.
• A sweep guard. Listings that disappear from a store are retired, but when a run loses more than 15% of a store’s listings, the retirement sweep is skipped and a warning is raised. A store that answers 404 to most of its pages is not retired either.

Every run also records per-store timings, requests and 429 counts, so a slow or throttled retailer is visible rather than inferred.

Catalogue merging, which is the actual product

Twenty retailers, zero shared data model. The same GPU appears under three different names in three stores, and without merging, cross-store comparison is impossible: you cannot compute a median across rows that are not the same product.

• Title and spec normalizers parse each category (CPU, GPU, motherboard, RAM, storage, PSU, case, cooler), extract colour and variant, normalize sockets, DDR generations and form factors, and refuse to file prebuilt PCs and laptops as components.
• Reference catalogues for CPUs, GPUs and motherboards supply specs the retailers do not publish, so product pages can show a full spec sheet.
• Variants are grouped. Box, tray and bulk packagings of the same model share one product page, and one rule (offer_policy) decides what counts as a buyable offer.

Bargain detection with a median, not an average

The first version of the deals page was wrong, and the bug was informative. The naive implementation compared each store’s price to the average of its peers. It produced nonsense, because of a defect in the data: prebuilt desktops get filed under the CPU named in their title. An 8,999 MAD “Ryzen 5 5600 + RTX 5060” prebuilt sat in a ~1,349 MAD peer group of standalone processors, dragged the average up, and made correctly priced CPUs look like 40% discounts. Sorted by discount, the corrupted products went straight to the top.

The fix has three parts:

• The reference is the median, which one outlier cannot move.
• A plausibility filter (MARKET_PLAUSIBILITY_FACTOR = 2.5) drops listings far above their peer median before the reference is computed. A listing at 3× its peers is a different product filed under the wrong name, not a bargain.
• A cross-store comparison needs at least three distinct stores (MARKET_MIN_STORES = 3), and a store listing the same product on four URLs counts once.

A second detector handles historical drops with the same defensive posture: a 90-day window, rejection of samples more than 2× the current price, and a minimum number of observations before a “drop” is trusted. One bogus observation must never be able to manufacture a discount.

A hardware compatibility engine

The builder is not a wishlist. evaluateSystemCompatibility() runs nine rules over the current selection and returns typed issues with a severity that maps to the interface:

#	Rule	Severity
1	CPU socket vs motherboard socket (LGA1700 / AM5 / …)	error
2	DDR generation vs motherboard memory standard	error
3	SO-DIMM laptop RAM in a desktop build	error
4	Motherboard form factor vs chassis supported form factors	error
5	CPU cooler bracket support for the selected socket	error
6	GPU length vs case maximum GPU clearance	error
7	CPU cooler tower height vs case side-panel clearance	error
8	AIO radiator size vs case mount points	warning
9	No iGPU and no discrete GPU, so no display output	warning

On top of the static rules sits a power model: component draw is summed, a buffer is added, and the recommended PSU is that total with headroom, rounded up. It distinguishes a critical deficit (rated capacity below actual draw, an error) from low headroom (a warning, because GPU transient spikes need the margin). A candidate part can be simulated into the current rig, so the part picker marks a component before you add it instead of telling you the build is broken afterwards.

Builds are shared as a link with no server involved. The rig is compressed with the browser’s native CompressionStream and stored in the URL hash, which never reaches the server, so it is immune to header-size limits. Compression cuts the payload by 91%, and the decoder accepts four formats so a link shared months ago still opens.

Price alerts that cannot double-send or be hijacked

Alerts are opt-in: a buyer sets a target price for one product and gets an email when a store reaches it. The email delivery runs in production over SMTP. A few decisions keep it trustworthy:

• Ownership is confirmed. A confirmation link is mailed to the address, and managing alerts requires a confirmed address and an HMAC-signed token, so nobody can read or delete someone else’s alerts by guessing an ID. The confirmation token is kept out of search indexes and analytics.
• One alert per buyer and product. Creation is an upsert against a partial unique index over active rows, not a read-then-write check. Two requests in flight, which is what a double-click produces, can both read “no active alert” before either writes, and the buyer would get the email twice.
• Only real offers trigger it. An alert never mails a price from an inactive store, and a steep drop seen in a single crawl is held back until a rescrape confirms it, instead of reacting to one odd reading.

A used-parts marketplace

A separate service lets individuals sell parts, lots and complete PCs, with every listing compared against the new prices tracked that day. Sign-in uses Clerk, and photos are stored in Cloudflare R2.

The storage design has a story. In October the production photo bucket was emptied while the database still listed the photos as ready. The likely cause was a second instance with the same credentials running its nightly cleanup. I could not prove it, so I made a repeat structurally hard instead:

• production and development never share a bucket or token;
• the cleanup only runs in deployed environments;
• a circuit breaker aborts any sweep that would delete most of what it examined, or whose database and bucket share no photos at all;
• a bucket owner marker stops a restored database clone from sweeping the wrong bucket;
• every removal is a move to a locked trash/ prefix, not a delete.

One codebase, one deployment per country

The Algerian edition is a second Compose project beside the Moroccan one on the same host. It has its own Postgres, backend and frontend and shares nothing at runtime, so a dinar price can never reach the Moroccan census. What the countries share is code: matching, normalization and the reference catalogues. Absolute price thresholds are written in MAD and converted per market, so the same guard means the same thing in dinars.

Search visibility and performance

• Server-rendered pages for search engines. Each of the 20 stores has its own page, with city landing pages, model pages such as “RTX 5070 prices in Morocco”, and a spec sheet rendered on the server for each product. Removed products answer 410 and merged products redirect with a 301.
• Fast catalogue pages. A server-side faceted catalogue endpoint returns one page per view, images go through the Next.js optimizer, and pages are cached and revalidated after each deploy.
• Privacy-respecting analytics. Self-hosted Umami, which sets no cookies, replaced Google Analytics, and it also collects Core Web Vitals.

Design

The interface follows a light, token-driven system: a Pure White theme by default and a Carbon Dark alternative, IBM Plex Sans for the interface and IBM Plex Mono with tabular figures for every number so price columns align. Two conventions separate measurement from marketing. Out-of-stock offers sit in their own group so an unbuyable price can never win a “cheapest today” claim, and every spread is stated against its base (“1,109 MAD below the Moroccan market median”). I also designed the identity: a pixel “gp” mark with a monospace wordmark, built around the idea that the median is a rule broken only by the store priced under it.

Security, deployment and quality

• Security. An SSRF guard on every outbound request, fail-closed admin authentication, operator routes behind basic auth, and settings validators that refuse to boot in production with an unsafe configuration. The public API also refuses automated harvesting clients.
• Deployment. Production runs on an Oracle Cloud VPS with Docker Compose behind Caddy. One script is the only path to production: it deploys origin/main, takes a host lock, detects drift between the server and the repository, rebuilds only the services whose inputs changed, checks health, and saves what it replaced so a rollback is a command, not a rebuild.
• Tests. Over 2,000 backend tests and about 850 frontend tests, run by GitHub Actions on every pull request, with an uptime check workflow on the live site. The backend tests are adversarial rather than coverage-shaped: selector relocation, rate limit versus challenge classification, duplicate price snapshots, alert token ownership and an end-to-end field-completeness check on a real scrape.

What I’d do next

• Add a visual-regression pass that renders every surface in both themes.
• Build the local-market features still on the roadmap: a cheapest one- or two-store cart that accounts for each retailer’s shipping, and a “prebuilt versus DIY” markup calculator.