Browse by Topic

Using 'Fetch as Google' to Test whether Google Can Crawl Specific Pages

You can use the Fetch as Google tool to see how Google crawls and renders a specific URL from your site. It simulates the the natural crawling process of Googlebot, and can help identify if a page is crawlable by Google, and if any resources (like images or scripts) are inaccessible.

 

Google cannot crawl stores prior to launch or in maintenance mode. When you first start a new BigCommerce store, it will be hidden until the store is actually launched. Stores with Down for Maintenance enabled are not also crawlable by Google.

Using the Fetch as Google tool requires a free Google account. If you don’t have an account already, you can sign-up for one.

Run a Fetch

1. Log in to Google Search Console (formerly Webmaster Tools).

2. Click the name of the domain you want to test. If you don’t see your domain, you’ll need to first verify your domain with Google.

3. In the menu on the left side of the screen, click Crawl, then Fetch as Google.

4. Enter the path of the page you want to fetch. The domain name is already provided, so you only need what comes after the trailing slash of the domain name.

For example, if you want to fetch http://www.mystore.com/cameras/nikon-d7000.html, you only need to paste cameras/nikon-d7000.html into the URL field. Leave the URL field blank to fetch the homepage.

5. Optionally, you can choose what type of Googlebot to perform the fetch as:

  • Desktop
  • Mobile: Smartphone
  • Mobile: cHTML (a subset of mostly Japanese feature phones). Rendering not supported.
  • Mobile: XHTML/WML (feature phones). Rendering not supported.

6. Click Fetch or Fetch and Render.

  • Fetch — Fetches a specified URL in your site and displays the HTTP response. Does not request or run any associated resources (such as images or scripts) on the page. This is a relatively quick operation that you can use to check or debug suspected network connectivity or security issues with your site, and see the success or failure of the request.
  • Fetch and render — Fetches a specified URL in your site, displays the HTTP response and also renders the page according to a specified platform (desktop or smartphone). This operation requests and runs all resources on the page (such as images and scripts). Use this to detect visual differences between how Googlebot sees your page and how a user sees your page.

7. The request will be added to the fetch history table below the URL field, with a "pending" status. When the request is complete, the row will show the success or failure of the request and some basic information. Click any non-failed fetch row in the table to get additional details about the request, including raw HTTP response headers and data, and (for Fetch and Render) a list of blocked resources and a view of the rendered page.

8. If the request succeeded and is less than four hours old, you can tell Google to re-crawl and possibly re-index the fetched page, and optionally any pages that the fetched page links to.

Request Fetch Status

The fetch history table on the main page shows the last 100 fetch requests. To see details for a completed fetch, click on the corresponding row in the fetch history table. The following request fetch statuses can be displayed:

  • Complete — Google successfully contacted your site and crawled your page, and can get all resources referenced by the page. Click the table row to see more details about the fetch results.
  • Partial — Google got a response from your site and fetched the URL, but could not reach all resources referenced by the page because they were blocked by robots.txt files. If this is a fetch only, do a fetch and render. Examine the rendered page to see if any significant resources were blocked that could prevent Google from properly analyzing the meaning of the page. If significant resources were blocked, unblock the resources on robots.txt files that you own. For resources blocked by robots.txt files that you don't own, reach out to the resource site owners and ask them to unblock those resources to Googlebot. See the list of resource fetch error descriptions.
  • Redirected — The server responded with a redirect. The Fetch as Google tool does not follow redirects. Although the actual Google crawler follows redirects, the Fetch as Google tool will not. You must follow a redirect manually:
    • If the redirect is to the same property, the tool displays a button that allows to quickly follow the redirect by populating the fetch box with the redirect URL.
    • If the URL redirects to another property that you own, you can click "Follow" to autopopulate the URL box, then copy the URL, switch views to the new site, and then paste the URL into the fetch box.

    Follow button in Google Search Console for a redirected URL

    You can inspect the HTTP response on the fetch details page to see the redirect details. Locate the HTTP error code to learn more. Redirects can be triggered by the server or by meta tags or JavaScript on the page itself.

  • Specific error type... — Any of the resource type fetch errors can also apply to a fetch request for the entire page and can be shown in the status column. For example: Not found or Unreachable.

Was this article helpful?