This question came from a customer. He saw a Q&A in our docs saying that to avoid a ranking penalty from using a mirror, it's best to upload a
robots.txt to the origin's root directory. So he asked why using a mirror could hurt a site's ranking. Before answering, let's look at what site authority is.
Site Authority
Site authority is the trust value a search engine gives a site, based on an overall evaluation. The higher it is, the more weight the site has in search results and the better it ranks. Improving site authority helps a site rank higher in search engines and increases its overall traffic and credibility, which is crucial for its growth.
Problem
To bring in more traffic, many sites started doing search engine optimization and used all kinds of tactics to boost their authority. One of these was mirroring. Put simply, a mirror is an exact copy of a site with a different domain and IP but identical content, so it gets indexed multiple times. Today, many search engines have their own systems for filtering mirror sites. Once they detect a mirror, they may remove both the origin and the mirror from their index, lowering the site's authority, reducing traffic, and hurting the business.
UPYUN Mirroring
UPYUN launched its mirror CDN to solve the problem of migrating data from an origin server. Once mirroring is enabled, it automatically fetches resources from the origin and stores them permanently on
UPYUN, greatly reducing the load on the origin. But this can also create a potential risk: the ranking penalty mentioned at the start of this post.
rebots
Since search engines filter out mirrors, the way to avoid a penalty is to keep them from discovering the mirror. The resources are publicly accessible, so how do you stop search engines from crawling them? This is where robots.txt comes in. For more on what robots.txt does, see
robots.txt. We can manually upload a robots.txt containing standard rules that tell search engines not to crawl the mirrored resources. Using UPYUN mirroring as an example, you can refer to this robots.txt template:
|
|