Welcome to the Q&A Forum

Browse the forum for helpful insights and fresh discussions about all things SEO.

Your browser does not seem to support JavaScript. As a result, your viewing experience will be diminished, and you have been placed in read-only mode.

Please download a browser that supports JavaScript, or enable it if it's disabled (i.e. NoScript).

Will Google Revisit a 403 Page

Intermediate & Advanced SEO

847

dbuckles last edited by

Hi,

We've got some pretty strict anti-scraping logic in our website, and it seems we accidentally snared a Googlebot with it.

About 100 URL requests were responded to with a 403 Forbidden error. The logic has since been updated, so this should not happen again.

I was just wondering if/when Googlebot will come back and try those URLs again. They are linked from other pages on the site, and they are also in our sitemap.

Thanks in advance for any assistance.
1 Reply Last reply
Reply Quote 0
dbuckles @AlanMosley last edited by

Thanks for the reply Alan. We'll wait for them to crawl the site again.
1 Reply Last reply
Reply Quote 0
AlanMosley last edited by

My belief is that they always visit a page a few times before they make a decision for this reason.
If you have links to the page than they will certainly revisit.
1 Reply Last reply
Reply Quote 1

Got a burning SEO question?

Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.

Start my free trial

Browse Questions

View

From

Sorted by

With category

Explore more categories

Related Questions

Validated pages on GSC displays 5x more pages than when performing site:domain.com?

Hi mozzers, When checking the coverage report on GSC I am seeing over 649,000 valid pages https://cl.ly/ae46ec25f494 but when performing site:domain.com I am only seeing 130,000 pages. Which one is more the source of truth especially I have checked some of these "valid" pages and noticed they're not even indexed?
Intermediate & Advanced SEO | | Ty1986

0
Client has an inexplicable jump in crawled pages being reported in Google Search Console

Recently a client of mine noticed an inexplicable jump in crawled pages being reported in Google Search Console. We researched the following culprits and found nothing: Rel=canonicals are put in place No SSL/non SSL duplication We used a tool to extrapolate search query page data from Google Search Insights; nothing unusual No dynamic pages being made on the website All necessary landing pages are in the XML sitemap Could this be a glitch in GSC? We are wondering what the heck is going on. 7eaeS
Intermediate & Advanced SEO | | BigChad2

0
Google slow to index pages

Hi We've recently had a product launch for one of our clients. Historically speaking Google has been quick to respond, i.e when the page for the product goes live it's indexed and performing for branded terms within 10 minutes (without 'Fetch and Render'). This time however, we found that it took Google over an hour to index the pages. we found initially that press coverage ranked until we were indexed. Nothing major had changed in terms of the page structure, content, internal linking etc; these were brand new pages, with new product content. Has anyone ever experienced Google having an 'off' day or being uncharacteristically slow with indexing? We do have a few ideas what could have caused this, but we were interested to see if anyone else had experienced this sort of change in Google's behaviour, either recently or previously? Thanks.
Intermediate & Advanced SEO | | punchseo

0
After adding a ssl certificate to my site I encountered problems with duplicate pages and page titles

Hey everyone! After adding a ssl certificate to my site it seems that every page on my site has duplicated it's self. I think that is because it has combined the www.domainname.com and domainname.com. I would really hate to add a rel canonical to every page to solve this issue. I am sure there is another way but I am not sure how to do it. Has anyone else ran into this problem and if so how did you solve it? Thanks and any and all ideas are very appreciated.
Intermediate & Advanced SEO | | LovingatYourBest

0
Duplicate page title at bottom of page - ok, or bad?

Can I get you experts opinion? A few years ago, we customized our pages to repeat the page title at the bottom of the page. So the page title is in the breadcrumbs at the top, and then it's also at the bottom of the page under all the contents. Here is a sample page: bit.ly/1pYyrUl I attached a screen shot and highlighted the second occurence of the page title. Am worried that this might be keyword stuffing, or over optimizing? Thoughts or advice on this? Thank you so much! ron ZH8xQX6
Intermediate & Advanced SEO | | yatesandcojewelers

0
Why will Google not remove a manual penalty against us?

Our site was placed under a manual penalty last year in June 2012 after penguin rolled out. We were advised by Google that we had unnatural links pointing to our site. We fought for months, running backlink checks and contacting webmasters where Google's WMT was showing the sites which had links. We have submitted numerous reconsideration requests with proof of our efforts in the form of huge well labeled spreadsheets, emails, and screen shots of online forms requesting link removal.When the disavow tool came out we thought it was a godsend and added all the sites who had either ignored us or refused to take down the links to the disavow.txt with the domain: tag. Then we submitted another reconsideration request, but to no avail.We have since had email correspondence with a member of the Google Quality Search Team who after reviewing the evidence of all our previous reconsideration requests and disavow.txt still advised us to make a genuine effort and listed sites which had inorganic links pointing to our site which were already included in the disavow.txt.Google has stated "In order for your site to have a successful reconsideration request, we will need to see a substantial, good-faith effort to remove the links, and this effort should result in a significant decrease in the number of bad links that we see."We have truly done everything we can and proven it too! Especially with all the sites in the disavow.txt there must be a decrease in links pointing to our site. What more can we do? Please help!
Intermediate & Advanced SEO | | Benbug

0
Google swapped our website's long standing ranking home page for a less authoritative product page?

Our website has ranked for two variations of a keyword, one singular & the other plural in Google at #1 & #2 (for over a year). Keep in mind both links in serps were pointed to our home page. This year we targeted both variations of the keyword in PPC to a products landing page(still relevant to the keywords) within our website. After about 6 weeks, Google swapped out the long standing ranked home page links (p.a. 55) rank #1,2 with the ppc directed product page links (p.a. 01) and dropped us to #2 & #8 respectively in search results for the singular and plural version of the keyword. Would you consider this swapping of pages temporary, if the volume of traffic slowed on our product page?
Intermediate & Advanced SEO | | JingShack

0
Additional page listings under main listing in google

Hi, I'm working on the SEO for a site and I'd love to get the additional page links under our main site link in google, any ideas as to how we can achieve this ? Thanks,
Intermediate & Advanced SEO | | Prongo

0