How Dominate Online Used Screaming Frog + ChatGPT API (SEO Automation)
Project Details
Client
ALL PROJECTS
Project Type
SEO AUTOMATION
Live Date
FEBRUARY 2026
The Challenge
Internal linking is one of the highest-ROI SEO levers, but it rarely gets executed properly at scale.
Across multiple domains, we saw the same pattern:
Blog content existed, but service pages weren’t receiving enough internal authority.
Links were added manually (or not at all), causing inconsistent anchor text, missed opportunities, and slow iteration.
Large sites created a discovery problem: even when pages were indexable, important URLs were buried and poorly connected.
Teams avoided internal linking projects because the process was time-heavy and error-prone.
The result: weaker topical clustering, slower ranking improvements for commercial pages, and underperformance of content that already had impressions and traffic potential.
Our Solution
We built a repeatable internal-linking automation system using Screaming Frog as the crawler + data hub and ChatGPT API for semantic matching and anchor text generation. The goal was controlled scale: high-quality links, predictable outputs, and minimal manual time.
Here’s what we implemented.
Screaming Frog Crawl Setup (Clean Inputs):
We configured Screaming Frog to crawl or ingest only the pages we actually want to work with, avoiding noisy URLs that would degrade link recommendations.Key setup choices:
Enabled HTML storage where needed to extract meaningful on-page context.
Collected essential signals (title, H1, meta description, body text snippets) for stronger semantic matching.
URL List Mode Using Sitemap URLs (Indexable-Only Targeting):
Instead of letting the crawler roam into non-priority URLs, we used List Mode with sitemap URLs so the workflow focused on indexable, canonical pages. This prevented the model from suggesting links to:parameter URLs
tag/search pages
thin utility pages
non-canonical duplicates
Outcome: link suggestions were constrained to URLs that actually matter for rankings.
Semantic Similarity Detection (Blog ↔ Service Mapping):
We extracted page context and sent it to the ChatGPT API to:identify semantically related blog posts and service pages
map the best contextual linking opportunities
avoid irrelevant matches that happen with basic keyword rules
This enabled linking based on topic alignment and user intent, not just shared terms.
Anchor Text Generation (Controlled, Natural, Intent-Matched):
For each recommended internal link, we generated anchor text that was:contextually relevant within the source page
aligned to the destination page’s primary intent
varied enough to avoid templated repetition
safe (no forced exact-match stuffing)
We also structured outputs so anchors could be reviewed quickly and implemented consistently.
Operational Workflow (Batchable + Repeatable):
We turned internal linking into a system:crawl/import sitemap URLs
export page context fields
run API enrichment for similarity + anchor suggestions
output a deployment-ready sheet: Source URL → Target URL → Anchor → Placement guidance
This made internal linking deployable across multiple domains without reinventing the process.
current results
After rolling out this method across our domains, we saw consistent performance improvements that align with what strong internal linking typically drives:
Faster discovery and recrawling of important pages (improved crawl paths)
Stronger topical clusters (clearer entity + intent relationships)
Better performance for service pages as internal authority increased
Improved user journeys from informational intent to commercial intent
Common measurable wins observed during rollouts:
Growth in impressions and clicks for prioritised service pages
Better average positions on pages that previously lacked internal support
Cleaner site structure signals (more coherent content hubs)
This is the type of optimisation that compounds: each new blog post becomes an asset that strengthens commercial pages instead of living in isolation.
why this worked
Most internal linking fails for predictable reasons: wrong inputs, poor prioritisation, and manual execution limits.
This workflow worked because we engineered around those failure points:
Indexable-only URL set via sitemap-driven List Mode
Semantic matching instead of basic keyword overlap
Anchor text consistency generated with rules, not guesswork
A scalable pipeline that can be applied to any domain with the same standards
Screaming Frog provided the crawl precision and structured extraction. The ChatGPT API provided the semantic layer that makes link recommendations materially better than template-based rules.
NEXT PHASE
To push this system further, the next iteration focuses on:
prioritisation logic (link equity routing to the pages that move revenue)
automated placement detection (suggesting the best paragraph/section for insertion)
clustering dashboards (topic hub coverage gaps, orphan detection, crawl depth changes)
continuous internal link refresh cycles as new content ships
Dominate Online uses automation to do what manual SEO execution can’t: maintain quality while scaling across multiple sites.
The Results
In the last three months, we scaled semantic internal linking between blog and service pages using Screaming Frog + the ChatGPT API. By feeding sitemap URLs in List Mode, we focused only on indexable pages, improved crawl paths, reduced orphan pages, and increased organic visibility
Why Choose Dominate Online?
- Expertise in Competitive Markets:
We thrive in helping brands establish themselves in crowded spaces. - Targeted Campaigns That Convert:
We focus on bottom-line impact, ensuring every pound spent delivers measurable ROI. - Empathy for Business Owners:
As entrepreneurs ourselves, we understand that every lead counts.
When it comes to dominating your market, we ensure that every campaign is built for results, not just impressions.
Are you ready to dominate your market like Zapperty?
Contact us today to see how we can help you achieve measurable, impactful results.