Shopify Robots.txt: Everything You Need to Know
By Sam Nguyen
Are you currently running an online business on the Shopify platform? If yes, I assume that you have to try every single day to improve your store and enhance customer experience as well. Shopify Robots.txt is a great tool for that. It aims to access and arrange content and bring that content to the users.
To learn more about details of this file, please take a look at the following article Shopify Robots.txt: Everything You Need to Know.
Let’s get started now!
Shopify Robots.txt: An Overview
Shopify Robots.txt has a significant influence on Search Engine Optimization (SEO). It is the file that helps merchants a lot in controlling search robots.
What is Robots.txt File?
Robots.txt is a text file created by webmasters to tell web robots (usually search engines) which pages on your site to crawl and which pages not to crawl. The robots.txt file is included in the robots exclusion protocol (REP), which is known as a group of web standards. These standards show how robots crawl the web, process and index content and bring that content to users. In the REP, there are also directives such as meta robots, page-, subdirectory-, or site-wide directions for how search engines should treat links, for example: “follow” or “nofollow”.
In practice, robots.txt files are used to tell if certain user agents can or cannot crawls the web. These crawl directions are designated by “disallowing” or “allowing’ the behavior of user agents.
Robots.txt file is located at the root directory of your Shopify store’s primary domain name. Hence, it is the first document that spiders or crawlers open when entering your site. Nevertheless, the file does not take control of crawling. It’s possible for you to integrate a link to your sitemap, which provides search engine spiders with an overview of all current URLs of your domain.
So how to generate a robots.txt file? Don’t worry. Shopify will automatically create a robots.txt file for you if you run an online store on this platform.
Why is Robots.txt File Necessary?
Robot.txt files aim to manage spiders access to certain areas of your website. Although it is not very safe if you unintentionally disallow Googlebot from crawling your whole site, you still can take advantage of robots.txt files in some situations:
-
Avoiding same content from appearing in SERPs (notice that meta robots are often a better option for this)
-
Maintaining entire sections of a website private
-
Keeping internal search results pages from appearing on a public SERP
-
Defining the location of sitemaps
-
Avoiding search engines from indexing certain files on your sites such as photos and PDFs
-
Designating a crawl delay to prevent your servers from being overloaded when spiders load a number of pieces of content at the same time
Where Can You Find Shopify Robots.txt?
The robots.txt file is stored at the root directory of your Shopify store’s domain name.
Can You Edit Shopify Robots.txt File?
Sadly, you can’t. However, you should notice that the content of your store’s robots.txt file blocks from search engines. For instance, the shopping cart page is blocked from search engines as you want customers to go to your site first rather than the cart page.
How to Hide a Page on Your Shopify Store from Search Engine
In order to hide a page that is not included in your robots.txt file, customize the <head> section of your store’s theme.liquid layout file. You also have to enter some code to noindex certain pages.
Step 1. Enter Themes section
In your Shopify admin, go to Online Store and then Themes.
Step 2. Edit code
Choose the theme you want to edit, click on Actions and then click Edit code.
Step 3. Choose the theme.liquid layout file
Click the theme.liquid button.
Step 4. Enter code to exclude the search template
Type the following code in the <head> section.
{% if template contains 'search' %}
<meta name="robots" content="noindex">
{% endif %}
Step 5. Enter code to exclude a specific code
Write the following code in the <head> section.
{% if handle contains 'page-handle-you-want-to-exclude' %}
<meta name="robots" content="noindex">
{% endif %}
Remember that you replace page-handle-you-want-to-exclude with the exact page handle.
Step 6. Save
Click on Save button.
For more detailed instructions, please check out at the article How to Hide a Page From Search Engines on Shopify.
Conclusion
That’s all. Through this article Shopify Robots.txt: Everything You Need to Know, I hope you can understand deeply about robots.txt file and what it can benefit your business on Shopify. In case you have any questions or concerns, feel free to leave a comment for us. We will not let you wait for our replies.
Thank you for reading!
