Why you cannot just re-crawl everything

The naive approach is simple: crawl every product page on a schedule and rewrite the database. It works at small scale and falls apart quickly. Retailer pages are heavy, rate limits are real, and most prices have not changed since the last check. Re-crawling everything wastes bandwidth, invites blocks, and buries the few real changes in a pile of identical snapshots.

The fix is to crawl incrementally: only fetch what is likely to have changed, and only act when the fetched data actually differs from what we already have.

Prioritising what to check

Not every product deserves the same check frequency. We rank products by a few signals:

• Volatility. Products whose prices moved recently get checked more often. A stable price that has not moved in a month can wait.
• Traffic. Guides that readers open every day get fresher data than guides nobody has visited in weeks.
• Seasonality. Around Black Friday and other sale events, everything gets checked more often, because that is when prices move fastest.
• Deal status. A product currently listed as a deal — a price below its usual level — gets checked daily, because a deal ending is exactly the change readers care about most.

This is a priority queue, not a fixed schedule. The crawler works through the highest-priority items first, so a limited crawl budget still covers the prices that matter.

Change detection: compare, don't just store

Fetching a page is only half the job. The other half is deciding whether anything changed. We keep the last known snapshot for each product — price, list price, availability, and the verification status — and compare the new fetch against it field by field.

Three outcomes matter:

1. No change. The price is the same. We update the "last checked" timestamp and move on. Nothing is republished, and no alert fires.
2. Price changed. The new price replaces the old one, the change is logged with a timestamp, and the guide's verification status is re-evaluated. A price drop can become a new deal; a price rise can end one.
3. Page or structure changed. The price field is missing or the page layout shifted. This is not a price change — it is a signal that the extractor may be broken, and it goes to a review queue instead of silently overwriting good data.

That third case is the one most small crawlers miss. A missing price should never be treated as a zero price or a removed product. It means "we don't know", and the guide should keep showing the last verified price with its check date rather than publishing a guess.

Verification status: retailer-verified vs. source-reported

At Savlume, every price carries a verification label. A price is retailer-verified when we confirmed it directly on the retailer's own page; it is source-reported when it comes from a secondary source and has not been confirmed at the retailer. Change detection feeds directly into this: a retailer-verified price that we re-confirm stays retailer-verified with a fresh timestamp. A price we can no longer confirm at the retailer drops back to source-reported until it is verified again.

This matters because a guide is a promise. Readers should be able to tell, at a glance, which prices we actually saw at the store and which ones we are passing along. The labels are part of the data model, not decoration on the page.

Publishing the change, fast

A detected price change is only useful if it reaches readers. Our guides are served through a small API surface — /api/v1/items for structured product data and /feed/all.xml for an RSS feed — and an MCP endpoint at /api/mcp plus an llms.txt file so AI agents can read the same data. When a price changes, the guide page, the API, and the feed all reflect the new value from the same source of truth, so there is no window where the guide says one price and the feed says another.

The RSS feed is the fastest path to readers who follow it: a price drop that the crawler confirms in the morning can be in a subscriber's reader before lunch, without anyone manually editing a page.

What we learned

• Check the prices readers actually see. A crawler that checks everything equally will always be behind on the pages that matter.
• Treat "unknown" as a state, not an error. A failed fetch should never overwrite a good price with a blank.
• Log every change with a timestamp. Price history is what turns "this is cheap now" into "this is the cheapest it has been in a month", and that history only exists if you record it.
• Keep one source of truth. The guide page, the API, and the feed should all read from the same data, so a change propagates everywhere at once.

Price tracking is not a scraping problem. It is a freshness problem: knowing what changed, deciding what it means, and getting the new answer to readers before it goes stale again.

Prices in Savlume guides are marked retailer-verified or source-reported, and every guide links back to the retailer so you can check the current price before you buy. Browse the guides at https://savlume.com/

Follow Savlume Deals on WhatsApp: https://whatsapp.com/channel/0029VbE9P7J7DAX7rD8TAC2U

This is the third post in our series on building Savlume (https://savlume.com/), a US deal tracker. The first covered our deal verifier and the MCP + llms.txt layer (https://savlume.com/llms.txt); the second covered automated price tracking and GEO/AEO (https://savlume.hashnode.dev/building-an-ai-shopping-assistant-automated-price-tracking-and-geo-aeo-in-practice). This one is about the plumbing in between: the RSS and API feeds that carry a price drop from our catalog to a reader, a feed reader, or another agent — fast, without duplicates, and without lying about what changed.

1. Why feeds, not just pages

A deal page is a snapshot. A feed is a changelog. When a price drops, the useful event is not "this page exists" but "this item's price changed at this time, from this to that, with this evidence level." Readers who follow deals need the event; agents who answer shopping questions need it too.

Savlume ships three feed surfaces for the same catalog:

• /feed/all.xml (https://savlume.com/feed/all.xml) — RSS for all updates, the one feed readers and automation tools already know how to consume.
• /api/v1/items (https://savlume.com/api/v1/items?mode=all&limit=24) — paginated JSON for every item, plus /api/v1/items/{id} for a single item.
• /api/mcp (https://savlume.com/api/mcp) — an MCP endpoint so an AI agent can query the catalog directly instead of scraping pages.

One catalog, three doors. The rule that keeps them honest: every surface reports the same price, the same verification status (retailer-verified or source-reported), and the same timestamps. A feed entry that disagrees with the page it links to is worse than no feed at all.

2. What goes into a feed entry

Each entry carries the minimum a reader needs to decide in seconds, and nothing they have to re-verify by clicking:

• The item and its canonical URL — the Savlume item page, so the entry is citable.
• The current price and the reference price, as separate fields — so a "40% off" claim can be checked, not just believed.
• The verification status — retailer-verified (confirmed at the retailer's own listing) or source-reported (from a deal source, not yet confirmed at the retailer). The status travels with the price; it is never stripped in the feed.
• Publication, update, and verification times — when the item was listed, when the entry last changed, and when the price was last checked. A price without a time is a rumor, and a feed that hides the check time turns rumors into headlines.
• The conditions — coupon required, Prime required, limit per customer, free shipping or not. A price that only exists with a coupon is not the same price.

What we deliberately leave out: cashback, gift cards, and points are never subtracted from the checkout price, and expired or out-of-stock entries are kept as reference, never as recommendations. A feed that promotes a dead deal is a feed readers unsubscribe from.

3. Deduplication: the same drop, once

The hardest feed problem is not speed — it is not repeating yourself. A price that flickers between two values, or a page that gets re-saved without a real change, must not produce five entries for one event. Our rules:

• An entry is emitted only when the price, the verification status, or the availability actually changes. A re-save with no change is not an event.
• The same product, variant, and conditions are one item. Historical comparisons only compare identical conditions; a different coupon or a different variant is a different record, not an update to the old one.
• Missing information stays unknown. If a check fails, we report "couldn't verify," not "deal dead" — and a failed check does not emit a new entry pretending the price moved.

The practical effect for a subscriber: the feed is quiet most of the day and noisy exactly when something real happens. That is the trust contract of a feed — every entry should be worth the interruption.

4. Polling, caching, and not hammering the retailer

A feed does not remove the need to check prices; it changes who checks what, and how often. Our catalog checks run on our side, and the feed is the public result of those checks. Consumers poll the feed on a sane interval instead of scraping retailer pages themselves, which is better for everyone: fewer requests to retailers, and one verified answer instead of a hundred unverified guesses.

Two design choices that matter:

• Stable item IDs and canonical URLs. A feed reader can tell that today's entry is the same item as yesterday's, so it can show "price dropped from $X to $Y" instead of "a new product appeared."
• Machine-readable text per item. Every item also has a /items/{id}/markdown text version, so an agent or a feed consumer that wants the clean facts can quote them without parsing HTML.

5. What we would do differently next

If we rebuilt the feed layer tomorrow, we would add per-category feeds (one RSS feed per guide category, so a reader who only cares about kitchen deals does not get headphone drops), and a "verification changed" event type distinct from "price changed" — because a price that goes from source-reported to retailer-verified is news even when the number stays the same.

For now, the single all-updates feed does the job: one place where every real change appears once, with its evidence attached.

Try it

• Browse the deals: Savlume (https://savlume.com/)
• Subscribe to all updates: https://savlume.com/feed/all.xml
• Read the machine-readable index: https://savlume.com/llms.txt
• API guide for agents: https://savlume.com/agent
• Example buying guide: Top 5 Air Fryers Under $100 for Black Friday 2026 (https://savlume.com/best/air-fryers-under-100)

Follow Savlume Deals on WhatsApp: https://whatsapp.com/channel/0029VbE9P7J7DAX7rD8TAC2U

This is the second post in our series on building Savlume, a US deal tracker. The first covered our deal verifier and the MCP + llms.txt layer. This one is about the two things that make it work day to day: automated price tracking, and GEO/AEO — making a site that AI search engines can actually cite.

1. Automated price tracking: the data model matters more than the scraper

Everyone asks what we scrape. The more important question is what we store. Every deal on Savlume is a record with:

• The price and the verification status: retailer-verified (confirmed at the retailer's own listing) or source-reported (from a deal source, not yet confirmed at the retailer).
• A verification timestamp: when the price was last checked.
• The conditions: coupon required, Prime required, limit per customer, free shipping or not.
• The reference price as a separate field from the checkout price.

That model is what makes the automation trustworthy. A price without a timestamp is a rumor, and a discount computed from an inflated reference price is a lie. Two rules we never break: expired and out-of-stock entries are kept as reference, never as recommendations; and cashback, gift cards, and points are never subtracted from the checkout price. Historical comparisons only compare the same product, variant, and conditions — missing information stays unknown.

The catalog is exposed the same way it is stored: /api/v1/items (paginated JSON), /feed/all.xml (RSS for all updates), and a /items/{id}/markdown text version plus /api/v1/items/{id} JSON for every item. The 35 buying guides under /best are structured content — multi-source review consensus plus current retailer offers — so a tracker can follow a whole category, not just one SKU.

2. GEO/AEO: make your site citable by AI search

GEO (generative engine optimization) and AEO (answer engine optimization) are the same idea: when someone asks ChatGPT, Perplexity, or Gemini "what's the best air fryer under $100?", the answer comes from sources the model can read and cite. A normal HTML page is hard for an agent to quote precisely. So we ship a machine-readable layer:

• /llms.txt — a single index of everything: about and methodology, today's deals, Black Friday and Cyber Monday, coupons, categories, stores, brands, reviews, and all 35 guides. It also carries the interpretation rules: quote the original source and the canonical Savlume item URL with publication, update, and verification times.
• /agent — an API guide written for agents, not humans.
• /api/mcp — an MCP endpoint, so an agent can query the catalog directly.
• Per-item Markdown — every item has a clean text version, which is what an agent actually wants to quote.

The practical effect: an agent that reads our llms.txt can answer with a real price, a real timestamp, and a real link, instead of hallucinating one. And because every price carries its verification status, the agent can say "retailer-verified" or "source-reported" instead of stating an unverified price as fact.

Three things we learned doing this:

• Write the llms.txt first, not last. It forces you to decide what the site is and what evidence each claim carries.
• Label uncertainty in the data, not in a footnote. "Source-reported" next to the price is what makes the answer honest.
• Keep the canonical URL in every answer. The llms.txt rules tell agents to cite the Savlume item URL with its timestamps — that is how the citation survives.

3. What this buys you

For shoppers: prices with evidence levels and timestamps, not badges. For agents: a catalog they can query and cite. For us: every AI answer that cites a Savlume price is a backlink with a timestamp attached.

Try it: browse Savlume, read savlume.com/llms.txt, or check the API guide at savlume.com/agent.

With Black Friday 2026 coming up, I got tired of fake "list prices" on retail sites where a $50 item is suddenly marked "50% off from $100".

So I built **Savlume** (https://savlume.com/) — an independent US shopping deal tracker that cross-checks source-reported offers against actual retailer pricing, strips out noisy promo-code prefixes, and flags honest trade-offs before you buy.

A few things we built under the hood:

1. **Black Friday 2026 Doorbuster Hub**: https://savlume.com/black-friday (side-by-side price comparison tables + verified promo codes across Amazon, Best Buy, Walmart, and Target).

2. **Consensus Buying Guides**: Instead of generic fluff, we synthesize pros, trade-offs, and review consensus for price-capped categories like [Best Air Fryers Under $100](https://savlume.com/best/air-fryers-under-100) and [Best Wireless Earbuds Under $100](https://savlume.com/best/wireless-earbuds-under-100).

3. **Head-to-Head Model Comparisons**: Automated side-by-side spec & price matchups at https://savlume.com/compare

4. **Open for AI Agents (GEO / MCP)**: We exposed a clean [llms.txt](https://savlume.com/llms.txt), full RSS feed (https://savlume.com/feed/all.xml), and an MCP server endpoint so AI assistants can query live verified US deals directly.

Would love your feedback on the speed, UI, and price-verification transparency!

1. The problem: deal prices lie by omission

A deal is a claim — a price, a discount, availability — and a claim needs evidence and a timestamp. Three things go wrong on most deal sites: the reference price is inflated, so the discount looks bigger than it is; the price is stale but presented as current; stock and coupon conditions are asserted, never checked. Our rule from day one: if we can't show the evidence, we say so on the page.

2. Two-tier verification: retailer-verified vs source-reported

Every price on Savlume is one of two things: retailer-verified (confirmed at the retailer's own listing), or source-reported (the price comes from a deal source such as DealNews or 9to5Toys and has not been confirmed at the retailer — labeled exactly that on the page). A few more rules we enforce: expired and out-of-stock entries are kept as reference material, never as recommendations; listed/reference prices and conditional checkout prices are separate fields, and cashback, gift cards, and points are never subtracted from the checkout price; a failed check is not proof that an offer expired — we report "couldn't verify," not "deal dead"; historical comparisons only compare the same product, variant, and conditions, and missing information stays unknown.

A concrete example: our Black Friday air fryer guide (https://savlume.com/best/air-fryers-under-100) lists the Ninja Foodi 2-basket at $99.99 as source-reported via Target, while the Instant Pot Vortex Plus at $79.95 and the COSORI Pro III at $79.99 are retailer-verified on Amazon. Same page, different evidence levels, each labeled.

3. Making the catalog machine-readable: llms.txt + MCP

Verification is only half the job. The other half is letting machines read it: /llms.txt (https://savlume.com/llms.txt) is the index — about and methodology, today's deals, Black Friday and Cyber Monday pages, coupons, categories, stores, brands, reviews, and all 35 buying guides; /api/v1/items (https://savlume.com/api/v1/items?mode=all&limit=24) is JSON for every item, paginated, and each item page also has a /items/{id}/markdown text version and a /api/v1/items/{id} JSON representation; /feed/all.xml (https://savlume.com/feed/all.xml) is RSS for all updates; /api/mcp (https://savlume.com/api/mcp) is an MCP endpoint, so AI agents can query the catalog directly instead of scraping pages; /agent (https://savlume.com/agent) is the API guide for agents. The reasoning is simple: the same agents that write buying advice should be able to verify a price before quoting it. If an LLM can read our llms.txt, it can cite a real price with a real timestamp instead of hallucinating one.

4. How it was built

The site was built with the help of an AI coding agent (Codex), which made it fast to iterate on the data model and the verification labels. The 35 buying guides under /best (https://savlume.com/best) are structured content — multi-source review consensus plus current retailer offers — not scraped listicles. And the site says plainly on its llms.txt that it is "Independent US shopping summaries with AI assistance. Purchases take place at external retailers. No claim of personal product testing, exhaustive market coverage or guaranteed lowest price." That disclosure is a feature, not a disclaimer we hide.

5. Lessons so far

Label uncertainty on the page, not in a footnote nobody reads. A price without a time is a rumor — verification timestamps are part of the record. Don't subtract incentives from the price: cashback isn't a discount at checkout, so it never lowers the number we show. Ship the machine-readable layer early: llms.txt and the MCP endpoint cost little to add and make the site useful to the agents that increasingly do the shopping research.

Try it

Browse the deals: Savlume (https://savlume.com/). Read the machine-readable index: savlume.com/llms.txt (https://savlume.com/llms.txt). API guide for agents: savlume.com/agent (https://savlume.com/agent). Example buying guide: Top 5 Air Fryers Under $100 for Black Friday 2026 (https://savlume.com/best/air-fryers-under-100).