Why Manual Competitor Monitoring Breaks Down

Checking competitor websites by hand is manageable until it isn't. You visit five pricing pages today, get pulled into other work, and three weeks later discover a competitor quietly launched a new free tier. By the time you notice, they've already started capturing the segment you were targeting.

The problem isn't awareness — it's the absence of a system. Competitive monitoring needs to be automatic and auditable. You need a log of what each competitor's page looked like last week, not a vague memory of what you think the price was.

This guide walks through a practical n8n workflow that uses Firecrawl to scrape competitor URLs on a schedule, extracts the page content, and stores a timestamped snapshot in Airtable. When something changes, you'll see it in the table without ever visiting the page yourself.

What You'll Need

To follow along, you need an active n8n instance (self-hosted or cloud), a Firecrawl account with an API key (firecrawl.dev offers a free tier), and an Airtable account.

Set up a base in Airtable with a table called "Competitor Snapshots." Add these fields: Competitor (single line text), URL (URL field), Snapshot Date (date), Content (long text), Pricing Detected (long text), and Changed (checkbox). This table becomes your permanent, searchable audit trail of competitor activity.

The Workflow Blueprint

The workflow runs on a schedule, loops through your competitor URLs, fetches a clean content snapshot for each using Firecrawl, compares the result to the previous snapshot already stored in Airtable, and writes a new record every time it runs — flagging the record if the content has changed.

Step 1: Schedule Trigger

Add a Schedule Trigger node and set it to run daily at 8:00 AM, or weekly if your competitors update infrequently. This is your workflow's entry point. Once active, no manual triggering is needed.

Step 2: Define Your Competitor List

Add a Code node immediately after the trigger. Write a short function that returns an array of competitor objects — each with a name property (the competitor's brand name) and a url property (the specific page you want to scrape, typically their pricing or features page). Keeping the list in this node means you can add or remove competitors in one place without restructuring the rest of the workflow.

The Code node outputs multiple items — one per competitor — which become the input for the next processing step.

Step 3: Process Each URL with Split In Batches

Add a Split In Batches node to process one competitor at a time. This prevents Firecrawl API rate limit issues and makes error handling cleaner: if one page fails to load, the remaining competitors in your list continue to be processed without interruption.

Step 4: Fetch Page Content with Firecrawl

Add an HTTP Request node and configure it to POST to https://api.firecrawl.dev/v1/scrape. Set the Authorization header to use Bearer token authentication with your Firecrawl API key.

In the request body, pass the competitor URL from the current batch item and set the formats parameter to ["markdown"]. Firecrawl renders the page (including JavaScript-loaded content), strips away navigation menus, footers, and ads, and returns a clean markdown representation of what's actually on the page. This is far more reliable than a raw HTML fetch, especially for modern single-page apps and dynamic pricing tables.

If you want to narrow the extraction to a specific section of the page, Firecrawl's includeTags parameter lets you target elements by CSS selector — useful if you only want the content inside a <section id="pricing"> container.

Step 5: Extract and Format the Data

Add another Code node to process the Firecrawl response. The API response returns extracted content under data.markdown and page metadata (title, description, published date) under data.metadata. Pull the markdown content into a variable, then run a basic keyword scan to detect pricing signals: currency symbols, phrases like "per month," "per seat," tier names like Starter, Pro, or Enterprise.

Output an object with the competitor name, URL, the full markdown content for archiving, the detected pricing fragment (if any), and the current timestamp.

Step 6: Retrieve the Previous Snapshot from Airtable

Add an Airtable node in Read (List Records) mode. Filter the Competitor Snapshots table by the current competitor's name and sort by Snapshot Date descending with a limit of one record. This returns the most recent snapshot already in your database, which you'll use as the baseline for comparison.

Step 7: Compare Old and New Content with an IF Node

Add an IF node to check whether the newly fetched content differs from the previous snapshot. Compare the first 500 characters of the freshly scraped markdown against the stored content from Airtable. If they don't match, the page has changed.

Route changed records to the true branch and unchanged records to the false branch. Both branches will write to Airtable — changed records just get the checkbox flagged.

Step 8: Write the Snapshot to Airtable

On the true branch, add an Airtable node in Create Record mode. Write the competitor name, URL, today's date, the full markdown content, any detected pricing text, and set the Changed field to true.

Connect an identical Airtable node to the false branch with Changed set to false. You still want a record for every run — the "no change" entries confirm you checked and establish the audit trail. Over time, comparing adjacent records for the same competitor shows you exactly how their page has evolved.

Example n8n Workflow in Action

Here is how the sequence plays out for a single competitor:

1. Schedule Trigger fires at 8 AM
2. Code node outputs three competitor objects from your list
3. Split In Batches picks up the first item — Competitor A's pricing page
4. HTTP Request calls Firecrawl and receives a clean markdown snapshot of the page
5. Code node extracts the content and scans for pricing keywords, finding "$49/month" has changed to "$59/month"
6. Airtable (Read) fetches last week's snapshot record for Competitor A
7. IF node detects the content has changed
8. Airtable (Create) logs the new snapshot with Changed set to true
9. Split In Batches advances to Competitor B and the process repeats

After the run, open Airtable and filter the Competitor Snapshots table by Changed equals true. You'll see exactly which competitors updated their pages since the last check. Compare the Content field across two adjacent rows for the same competitor to read the diff directly in Airtable.

Practical Tips for Reliable Results

Start with three to five URLs. Verify the Airtable records look correct after a couple of runs before scaling to a larger list. Firecrawl handles JavaScript rendering automatically, but some pages with aggressive bot detection may return incomplete results — flag those early and investigate.

Use meaningful Competitor names. If you're monitoring multiple pages per competitor (a pricing page and a features page, for example), include a suffix in the name field like "Competitor A – Pricing" and "Competitor A – Features." Airtable's grouping and filtering will keep each page's history separate.

Extend with a notification step. If you want to know immediately when something changes rather than checking Airtable each morning, add a notification node on the true branch — an email via your preferred provider, a message to a team channel, or a webhook to another tool. The IF node's true branch is the natural place for any alert logic without affecting the archiving flow.

Keep the full markdown. It can be tempting to only store the pricing fragment, but saving the full markdown gives you context. Competitors often change messaging, reorder tier features, or rebrand plan names — changes that a simple price comparison would miss entirely.

What You've Built

A self-running competitive intelligence pipeline that requires no manual effort after setup. Every morning, n8n calls Firecrawl, extracts clean content from your competitor pages, compares it against Airtable, and flags anything that changed. Your team gets a reliable, searchable log of competitor page history without anyone having to visit a URL.

Browse the template library at n8nresources.dev/templates for pre-built workflows covering this pattern and hundreds of other common automation use cases.