What to check when Google knows a URL but has not crawled it yet, from server health and crawl traps to internal depth and oversized low-value inventories.
Discovery happened, crawling did not
This status usually means Google knows the URL but has postponed fetching it. On a new site, a short delay can be normal. Across thousands of important URLs, it may signal low crawl demand, overloaded infrastructure, weak architecture or an inventory that is larger than the site's demonstrated value.
Do not interpret crawl demand as a fixed quota. Search engines continuously adjust attention based on host health, update frequency, link signals, duplication and observed value.
Check server capacity before requesting more crawling
Review logs around crawler visits. Look for 429, 500, 502, 503 and timeout patterns. Measure response time at the application and CDN layers. A host that slows down under parallel requests teaches crawlers to reduce pressure.
Make sure robots.txt and sitemap endpoints are fast and stable. Avoid generating a massive sitemap through an expensive database query on every request without caching.
Find crawl traps
Faceted navigation, calendars, internal searches and tracking parameters can create near-infinite URL spaces. Crawlers may spend attention on combinations that have no search value while important pages wait.
Map query parameters and path patterns from logs. Consolidate or block true traps carefully, remove links to useless combinations, and ensure canonical pages are the dominant internal targets.
Reduce click depth
Important URLs should not depend on a sitemap alone. Create category hubs, related-content links and crawlable pagination. Verify that links are real anchors in the rendered document and do not require a user-only interaction.
For large catalogues, prioritize. New and updated products can live in a recent sitemap and receive links from active category pages. Old, unavailable inventory should not compete equally for crawl attention.
Audit inventory quality
A site with 100,000 generated pages and little differentiation may discover URLs faster than Google chooses to crawl them. Sample by template and measure uniqueness, completeness and user value. Remove empty placeholders and merge pages that answer the same query.
Publishing fewer complete pages can improve crawl efficiency more than submitting a larger list.
A seven-day action plan
Day 1: Technical baseline
Record server error rate, median response time, robots rules, sitemap counts and important template types.
Day 2: Log analysis
Identify which paths crawlers visit, where they encounter errors and which parameter patterns consume requests.
Day 3: Architecture
Lower the click depth of priority pages and repair orphan URLs.
Day 4: Inventory cleanup
Remove obsolete sitemap entries and consolidate duplicates.
Day 5: Template improvement
Fix missing content, canonicals and rendering problems on one high-impact template.
Day 6: Controlled submission
Submit a representative set of corrected priority URLs, not the complete backlog.
Day 7: Observe
Compare crawler activity, status changes and server health. Avoid changing the experiment before you can read the signal.
When waiting is reasonable
Fresh pages on a new domain may simply need time and stronger site-level signals. If the server is healthy, pages are unique, internal links are clear and the sitemap is accurate, repeated daily submission adds little. Continue publishing useful connected content and monitor trends.
The objective is to make crawling easy and worthwhile. Submission can announce the inventory, but crawl demand grows when the site consistently rewards the crawler with fast, distinct and well-connected pages.
Quantify crawl waste
From server logs, group crawler requests by response code, parameter pattern and template. Calculate how many requests reach redirects, errors, duplicate filters and useful canonical pages. This is more actionable than guessing at a “crawl budget” number.
If half of bot requests go to session parameters, fix the link source and parameter generation. If many requests hit slow API-rendered pages, improve cache and response behavior. If crawlers revisit old URLs because internal links still reference them, update the architecture.
Prioritize the publication pipeline
Large sites should not publish every generated combination immediately. Require minimum data quality before a URL becomes indexable and enters the sitemap. A product page might need a title, description, price state, image and category assignment. A location page might need unique service details and useful local information.
Measure changes with cohorts
Create cohorts by publication week and template. Track time from discovery to first crawl and from first crawl to index selection. Cohorts reveal whether new changes improve the pipeline, while a total excluded count may stay large because old inventory remains in the report.
Avoid accidental crawl spikes
After fixing a template, do not instantly submit every historical URL if the infrastructure is untested. Send a controlled high-priority group and watch latency, errors and crawler response. Expand gradually. A stable staged recovery is faster than triggering a new wave of 429 and 5xx failures.
Crawl demand improves when the site demonstrates that newly discovered URLs are consistently fast, distinct and worth processing.
This guide is written and reviewed by the Index Grab editorial team using publicly available search-engine documentation and practical technical SEO checks. Product capabilities and prices are verified against the current Index Grab interface before publication.
Primary references: Google Search Central · Sitemaps.org · Schema.org