Automated Competitive Intelligence with n8n and Claude: Monitor Competitors Without Brittle Scrapers

Most competitive intelligence setups break within weeks. You scrape a competitor's pricing page with a CSS selector, they redesign the layout, and your monitor goes silent. You do not notice until someone asks why you missed the price change.

The LLM-based approach fixes this. Instead of targeting specific HTML elements, you feed the raw page content to Claude and ask it to extract the data you care about. The extraction logic is in the prompt, not in fragile selectors — and it adapts automatically when the page structure changes.

The Architecture

The pipeline has three stages:

Stage 1 — Fetch: An HTTP Request node pulls the competitor's page on a schedule. No authentication, no browser automation — just a GET request for the raw HTML.

Stage 2 — Extract: A Claude Haiku API call parses the HTML and returns structured JSON with the pricing tiers, blog post titles, or changelog entries you are tracking.

Stage 3 — Delta compare: A Code node compares the current extraction against the previous run's data (stored in $getWorkflowStaticData). If anything changed, a notification fires.

The Extraction Prompt

The prompt determines what you get out of Claude. For pricing monitoring:

`

You are extracting pricing information from a competitor's website.

Given this HTML content, return ONLY a JSON object with this structure:

{

"plans": [

{"name": "plan name", "price": "price string", "billing": "monthly|annual", "features": ["feature1", "feature2"]}

],

"extracted_at": "current timestamp"

}

If you cannot find pricing information, return {"plans": [], "error": "reason"}.

Return ONLY the JSON object, no explanation, no code fences.

HTML: [page content here]

`

The key instruction is "no code fences" — without it, Claude wraps the JSON in markdown formatting that breaks JSON.parse() in your Code node.

For blog/changelog monitoring, adjust the prompt to extract {"posts": [{"title": "...", "date": "...", "url": "..."}]}.

The Delta Detection Code Node

`javascript

const staticData = $getWorkflowStaticData('global');

const competitor = $json.competitor_name;

// Initialize storage for this competitor

if (!staticData[competitor]) {

staticData[competitor] = {};

}

const previous = staticData[competitor];

const current = $json.extracted_data;

// Compare serialized versions

const previousStr = JSON.stringify(previous);

const currentStr = JSON.stringify(current);

if (previousStr === currentStr) {

// No change

return [{ json: { changed: false, competitor } }];

}

// Something changed - identify what

const changes = [];

if (current.plans) {

for (const plan of current.plans) {

const prevPlan = (previous.plans || []).find(p => p.name === plan.name);

if (!prevPlan) {

changes.push(New plan: ${plan.name} at ${plan.price});

} else if (prevPlan.price !== plan.price) {

changes.push(Price change: ${plan.name} ${prevPlan.price} → ${plan.price});

}

}

}

// Save current state

staticData[competitor] = current;

return [{ json: { changed: true, competitor, changes, current, previous } }];

`

Scaling to 50+ Competitors

For a large competitor list, avoid running one workflow per competitor. Instead:

1. Store your competitor list in a Google Sheet or database

2. Use a Split in Batches node (batch size 5–10) to process them sequentially

3. Add a Wait node (2–3 seconds between batches) to avoid rate-limiting the Anthropic API

4. Store the static data keyed by competitor name as shown above

At 10 competitors checked daily with Claude Haiku, the cost is approximately $0.01/day — about $3/month.

What Claude Cannot Replace

Claude cannot tell you why a competitor changed their pricing or what the change signals strategically. The pipeline surfaces facts: prices changed, a new blog post appeared, a feature was removed. Interpreting those facts — whether to respond, how, and when — remains your judgment.

The pipeline also has inherent latency. It runs on a schedule, not in real time. If a competitor changes pricing on a Friday afternoon and your workflow runs at midnight, you get the alert Saturday morning. For time-sensitive competitive responses, you still need human monitoring of competitor channels.

For everything else — systematic tracking, trend identification across time, alert routing to the right person — the n8n pipeline handles it reliably and cheaply without a dedicated analyst.

Set it up once. Let it run.