Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

"Url blocked by robots.txt." on my Video Sitemap

Technical SEO

950

Critical_Mass Subscriber last edited by

I'm getting a warning about "Url blocked by robots.txt." on my video sitemap - but just for youtube videos? Has anyone else encountered this issue, and how did you fix it if so?! Thanks, J
1 Reply Last reply
Reply Quote 0
sergeystefoglo last edited by

Hi there,

It's definitely possible that you are blocking a folder or subfolder in your robots.txt file that is preventing Google from crawling some of your videos. Could you possibly share a link to the robots.txt file so I could look at it specifically? Also, Google Search Console should tell you which URLs are being blocked specifically so that could be useful as well.

Happy to help — just please send some more specific details. Thanks!
1 Reply Last reply
Reply Quote 0

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Product Subdomain Outranking "Marketing" Domains

Hello, Moz community! I have been puzzling about what to do for a client. Here is the challenge. The client's "product"/welcome page lives at www.client.com this page allows the visitor to select the country/informational site they want OR to login to their subdomain/install of the product. Google is choosing this www.client.com url as the main result for client brand searches. In a perfect world, for searchers in the US, we would get served the client's US version of the information/marketing site which lives at https://client.com/us, and so on for other country level content (also living in a directory for that country) It's a brand new client, we've done geo-targeting within the search console, and I'm kind of scared to rock the boat by de-indexing this www.client.com welcome screen. Any thoughts, ideas, potential solutions are so appreciated. THANKS! Thank you!
Technical SEO | | SimpleSearch

0
Sitemaps:

Hello, doing an audit found in our sitemaps the tag which at the time was to say that the url was mobile. In our case the URL is the same for desktop and mobile.
Do you recommend leaving or removing it?
Thank you!
Technical SEO | | romaro

0
New SEO manager needs help! Currently only about 15% of our live sitemap (~4 million url e-commerce site) is actually indexed in Google. What are best practices sitemaps for big sites with a lot of changing content?

In Google Search console 4,218,017 URLs submitted 402,035 URLs indexed what is the best way to troubleshoot? What is best guidance for sitemap indexation of large sites with a lot of changing content? view?usp=sharing
Technical SEO | | Hamish_TM

1
Robots txt. in page with 301 redirect

We currently have a a series of help pages that we would like to disallow from our robots txt. The thing is that these help pages are located in our old website, which now has a 301 redirect to current site. Which is the proper way to go around? 1- Add the pages we want to disallow to the robots.txt of the new website? 2- Break the redirect momentarily and add the pages to the robots.txt of the old one? Thanks
Technical SEO | | Kilgray

0
SEO question: Need help on rel="alternate" hreflang="x"

Hi all, we have webcontent in 3 languages (official belgian yellow pages), we use a separate domain per language, these are also our brands.
ex. for the restaurant Wagamamahttp://www.goudengids.be/wagamama-antwerpen-2018/ corresponds to nl-be
http://www.pagesdor.be/wagamama-antwerpen-2018/ corresponds to fr-be
http://www.pagesdor.be/wagamama-antwerpen-2018/ corresponds to en-be The trouble is that sometimes I see the incorrect urls appearing when doing a search in google, ex. when searching on google.be (dutch=nederlands=nl-be) I see the www.pagesdor.be version appearing (french) I was trying to find a fix for this within https://support.google.com/webmasters/answer/189077?hl=nl , but this only seems to apply to websites which use SUBdomains for language purposes. I'm not sure if can work for DOMAINS. Can anyone help me out? Kind regards
Technical SEO | | TruvoDirectories

0
Looking for feedback about "look-ahead" navigation

Our company has been creating websites where the navigation is developed in such a way as to allow the visitor to get a preview of the image and/or content on that is on the page. Here are two websites that use this technology:
http://www.uniquepadprinting.com/
http://www.empathia.com/ (On this site, the previews are only available if you click on "Whole", "Productive" or "Safe" at the top of the page. I'm looking for feedback such as: What do you call this type of navigation (We call it look-ahead, but I can't find much info that term on the web) Have you experienced any issues with this type of navigation? Do you have any recommendations on it? Some of the things we've seen are: It adds the same content to every page of the website It creates a lot of internal links It can create a lot of code on pages It can slow page-load times
Technical SEO | | TopFloor

0
Robots.txt issue - site resubmission needed?

We recently had an issue when a load of new files were transferred from our dev server to the live site, which unfortunately included the dev site's robots.txt file which had a disallow:/ instruction. Bad! Luckily I spotted it quickly and the file has been replaced. The extent of the damage seems to be that some descriptions aren't displaying and we're getting a message about robots.txt in the SERPs for a few keywords. I've done a site: search and generally it seems to be OK for 99% of our pages. Our positions don't seem to be affected right now but obviously it's not great for the CTRs on those keywords affected. My question is whether there is anything I can do to bring the updated robots.txt file to Google's attention? Or should we just wait and sit it out? Thanks in advance for your answers!
Technical SEO | | GBC

0
OK to block /js/ folder using robots.txt?

I know Matt Cutts suggestions we allow bots to crawl css and javascript folders (http://www.youtube.com/watch?v=PNEipHjsEPU) But what if you have lots and lots of JS and you dont want to waste precious crawl resources? Also, as we update and improve the javascript on our site, we iterate the version number ?v=1.1... 1.2... 1.3... etc. And the legacy versions show up in Google Webmaster Tools as 404s. For example: http://www.discoverafrica.com/js/global_functions.js?v=1.1
http://www.discoverafrica.com/js/jquery.cookie.js?v=1.1
http://www.discoverafrica.com/js/global.js?v=1.2
http://www.discoverafrica.com/js/jquery.validate.min.js?v=1.1
http://www.discoverafrica.com/js/json2.js?v=1.1 Wouldn't it just be easier to prevent Googlebot from crawling the js folder altogether? Isn't that what robots.txt was made for? Just to be clear - we are NOT doing any sneaky redirects or other dodgy javascript hacks. We're just trying to power our content and UX elegantly with javascript. What do you guys say: Obey Matt? Or run the javascript gauntlet?
Technical SEO | | AndreVanKets

0