Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

How to stop pages being crawled from xml feed?

Intermediate & Advanced SEO

654

jazavide last edited by

We have a site that has an xml feed going out to many other sites.
The xml feed is behind a password protected page so cannot use a cannonical link to point back to original url.

How do we stop the pages being crawled on all of the sites using the xml feed? as with hundreds using it after launch it will cause instant duplicate content issues?

Thanks
1 Reply Last reply
Reply Quote 0
CoreyNorthcutt last edited by

You'll probably want to disallow spiders from crawling them with robots.txt:

http://www.robotstxt.org/robotstxt.html
Built some companies, sold some companies. Currently building https://roi.fyi.
1 Reply Last reply
Reply Quote 1

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

I have a metadata issue. My site crawl is coming back with missing descriptions, but all of the pages look like site tags (i.e. /blog/?_sft_tag=call-routing)

I have a metadata issue. My site crawl is coming back with missing descriptions, but all of the pages look like site tags (i.e. /blog/?_sft_tag=call-routing)
Intermediate & Advanced SEO | | amarieyoussef

0
New page not topping on results

Hi, We have created a new page on our website for same keyword in slug but the page is not showing up for same keyword even combined with website name: website.com/keyword is new page and not listing on top of results for exact search query "website keyword". This page is listing as 3rd result and other pages are making on top even they don't match with page title, h1 tags and URL. This new page is indexed. How long it'll take to Google to adopt this? I don't think it'll remain same forever. Is there anything we can do from our end?
Intermediate & Advanced SEO | | vtmoz

0
How to optimize count of interlinking by increasing Interlinking count of chosen landing pages and decreasing for less important pages within the site?

We have taken out our interlinking counts (Only Internal Links and not Outbound Links) through Google WebMaster tool and discovered that the count of interlinking of our most significant pages are less as compared to of less significant pages. Our objective is to reverse the existing behavior by increasing Interlinking count of important pages and reduce the count for less important pages so that maximum link juice could be transferred to right pages thereby increasing SEO traffic.
Intermediate & Advanced SEO | | vivekrathore

0
How to 301 Redirect /page.php to /page, after a RewriteRule has already made /page.php accessible by /page (Getting errors)

A site has its URLs with php extensions, like this: example.com/page.php I used the following rewrite to remove the extension so that the page can now be accessed from example.com/page RewriteCond %{REQUEST_FILENAME}.php -f
RewriteRule ^(.*)$ $1.php [L] It works great. I can access it via the example.com/page URL. However, the problem is the page can still be accessed from example.com/page.php. Because I have external links going to the page, I want to 301 redirect example.com/page.php to example.com/page. I've tried this a couple of ways but I get redirect loops or 500 internal server errors. Is there a way to have both? Remove the extension and 301 the .php to no extension? By the way, if it matters, page.php is an actual file in the root directory (not created through another rewrite or URI routing). I'm hoping I can do this, and not just throw a example.com/page canonical tag on the page. Thanks!
Intermediate & Advanced SEO | | rcseo

0
Date of page first indexed or age of a page?

Hi does anyone know any ways, tools to find when a page was first indexed/cached by Google? I remember a while back, around 2009 i had a firefox plugin which could check this, and gave you a exact date. Maybe this has changed since. I don't remember the plugin. Or any recommendations on finding the age of a page (not domain) for a website? This is for competitor research not my own website. Cheers, Paul
Intermediate & Advanced SEO | | MBASydney

0
An affiliate website uses datafeeds and around 65.000 products are deleted in the new feeds. What are the best practises to do with the product pages? 404 ALL pages, 301 Redirect to the upper catagory?

Note: All product pages are on INDEX FOLLOW. Right now this is happening with the deleted productpages: 1. When a product is removed from the new datafeed the pages stay online and are showing simliar products for 3 months. The productpages are removed from the categorie pages but not from the sitemap! 2. Pages receiving more than 3 hits after the first 3 months keep on existing and also in the sitemaps. These pages are not shown in the categories. 3. Pages from deleted datafeeds that receive 2 hits or less, are getting a 301 redirect to the upper categorie for again 3 months 4. Afther the last 3 months all 301 redirects are getting a customized 404 page with similar products. Any suggestions of Comments about this structure? 🙂 Issues to think about:
- The amount of 404 pages Google is warning about in GWT
- Right now all productpages are indexed
- Use as much value as possible in the right way from all pages
- Usability for the visitor Extra info about the near future: Beceause of the duplicate content issue with datafeeds we are going to put all product pages on NOINDEX, FOLLOW and focus only on category and subcategory pages.
Intermediate & Advanced SEO | | Zanox

0
Optimize the category page or a content page?

Hi, We wish to start ranking on a specific keyword ("log house prices" in italian). We have two options on what pages we should optimize for this keyword: A long content page (1000+ words with images) Log houses category page, optimized for the keyword (we have 50+ houses on this page, together with a short price summary). I would think that we have better chances with ranking with option nr.2 , but then we can't use that page for ranking with a more short-tail keyword (like "log houses"). What would you suggest? Is there maybe a third option for this?
Intermediate & Advanced SEO | | JohanMattisson

0
How Bad is it to Not Have a Home Page?

The site I'm currently developing is far different than any other project I've every worked on in that search traffic is likely to represent only a very small percentage of the total traffic. Because of this, I want to make sure I optimize the site for the people clicking from Facebook, Twitter, Reddit, etc more so than the BIG G. I can't for the life of me think of a reason to have a home page other than for SEO purposes. I'd much rather throw the user directly into the experience than have him be distracted by a home page. At the same time, I'd like to salvage any search engine traffic that I can. My plan is to 301 redirect chucklebot.com/ to /funny-memes/SOME_RANDOM_IMAGE and then put the content of the current home page at /about. Does that kill any possibility of the site ranking well? Or can the subpages (eg /meme-generator) still rank well if they are properly optimized? Thanks!
Intermediate & Advanced SEO | | PatrickGriffith

0