If a page isn't linked to or directly sumitted to a search engine can it get indexed?
-
Hey Guys,
I'm curious if there are ways a page can get indexed even if the page isn't linked to or hasn't been submitted to a search engine.
To my knowledge the following page on our website is not linked to and we definitely didn't submit it to Google - but it's currently indexed:
<cite>takelessons.com/admin.php/adminJobPosition/corp</cite>
Anyone have any ideas as to why or how this could have happened? Hopefully I'm missing something obvious
Thanks,
Jon
-
You're welcome Jon.
That's a good question. I don't know the official answer on that one, though suspect that Google does check to see if the page exists, mainly because often it will be a valid URL that somebody types in the search box instead of the address bar. If Google don't have that page in their index, they'd at least like to consider adding it.
http://www.google.com/addurl.html is the URL adding page for Google as you'll already know. As well as Google relying on people to use this form, they will also, I suspect, crawl URLs that are entered into a search box. Makes sense to me that Google would at least visit these pages searched on, though can't be sure.
Regards
Simon
-
Thanks for the responses guys!
Are either of you aware of whether or not Google would ever index a page if someone searched for it? For example, say someone did a Google search for the URL I specified above. Would Google ever get curious and then try and crawl it?
Thanks again for taking the time to help out
-
Hey there Jon.
It seems that it was last indexed almost a month ago on Oct. 21, 2011. I would suggest that you follow Simon's advice on the NoFollow tag.
Additionally, check your GWT to ensure that it is not indexed...If it is, then try to resubmit it to get indexed (I know, I know, it doesn't make sense), but it will send out a message to crawl it. NoFollow, NoIndex tells the spider NOT HERE....Anywho, good luck with that and let us know how it turned out!
Cheers!
P.S. At least it's not indexed in Bing
-
Hi Jon
This is a strange one, I too haven't found a link to your admin page. There could have been one at some point in the past. Google's bot is rather clever at finding pages on the so-called 'invisible web', so best to request non-indexing of pages that you don't want indexed (covered below).
You're right in thinking that search bots follow links to find and index pages, whether it be an external or internal site link or a sitemap link. They also find pages through actual submissions.
- I'd suggest modifying the Robots tag on your admin page, include a 'NoIndex' just before the NoFollow'.
- Also include a Disallow command in your Robots.txt file for this admin page, and perhaps all pages within the admin section.
- Then, request the URL be removed from Google's index via Google Webmaster Tools ("Site configuration", "Crawler access", "Remove URL").
Hope that helps,
Regards
Simon
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Page Indexing without content
Hello. I have a problem of page indexing without content. I have website in 3 different languages and 2 of the pages are indexing just fine, but one language page (the most important one) is indexing without content. When searching using site: page comes up, but when searching unique keywords for which I should rank 100% nothing comes up. This page was indexing just fine and the problem arose couple of days ago after google update finished. Looking further, the problem is language related and every page in the given language that is newly indexed has this problem, while pages that were last crawled around one week ago are just fine. Has anyone ran into this type of problem?
Technical SEO | | AtuliSulava1 -
Google indexes page elements
Hello We face this problem that Google indexes page elements from WordPress as single pages. How can we prevent these elements from being indexed separately and being displayed in the search results? For example this project: www.rovana.be When scrolling down the search results, there are a lot of elements that are indexed separately. When clicking on the link, this is wat we see (see attachements) Does anyone have experience with this way of indexing and how can we solve this problem? Thanks! LlAWG4w.png C7XDDYS.png gVroomx.png
Technical SEO | | conversal0 -
Any idea why pages are not being indexed?
Hi Everyone, One section on our website is not being indexed. The product pages are, but not some of the subcategories. These are very old pages, so thought it was strange. Here is an example one one: https://www.moregems.com/loose-cut-gemstones/prasiolite-loose-gemstones.html If you take a chunk of text, it is not found in Google. No issues in Bing/Yahoo, only Google. You think it takes a submission to Search Console? Jeff
Technical SEO | | vetofunk1 -
Why would Google not index all submitted pages?
On Google Search console we see that many of our submitted pages weren't indexed. What could be the reasons? | Web pages |
Technical SEO | | Leagoldberger
| 130,030 Submitted |
| 87,462 Indexed |0 -
Blog page won't get indexed
Hi Guys, I'm currently asked to work on a website. I noticed that the blog posts won't get indexed in Google. www.domain.com/blog does get indexed but the blogposts itself won't. They have been online for over 2 months now. I found this in the robots.txt file: Allow: / Disallow: /kitchenhandle/ Disallow: /blog/comments/ Disallow: /blog/author/ Disallow: /blog/homepage/feed/ I'm guessing that the last line causes this issue. Does anyone have an idea if this is the case and why they would include this in the robots.txt? Cheers!
Technical SEO | | Happy-SEO2 -
Why is Google Webmaster Tools showing 404 Page Not Found Errors for web pages that don't have anything to do with my site?
I am currently working on a small site with approx 50 web pages. In the crawl error section in WMT Google has highlighted over 10,000 page not found errors for pages that have nothing to do with my site. Anyone come across this before?
Technical SEO | | Pete40 -
How to Break Up a Page with Too Many Links
My client has a live page with 100+ links subdivided into 10 categories that each have great potential keyword targeting opportunities. I'd like to improve this page and my intuition is to split it into 11 pages, one page with links to all the others and a bit of content about each. Here's an example of the potential IA: Dog Rescue Groups
Technical SEO | | elenarox
Golden Retriever Rescue - description
Poodle Rescue - description
Cocker Spaniel Rescue - description
Poodle Rescue - description
Labrador Retriever Rescue - description
etc. --------- Golden Retriever Rescue
Link 1 - description
Link 2 - description
Link 3 - description Is this a good idea and will I see a big traffic drop overall at first? Also, these are all internal links, not external.0 -
Does Google Still Pass Anchor Text for Multiple Links to the Same Page When Using a Hashtag? What About Indexation?
Both of these seem a little counter-intuitive to me so I want to make sure I'm on the same page. I'm wondering if I need to add "#s to my internal links when the page I'm linking to is already: a.) in the site's navigation b.) in the sidebar More specifically, in your experience...do the search engines only give credit to (or mostly give credit to) the anchor text used in the navigation and ignore the anchor text used in the body of the article? I've found (in here) a couple of folks mentioning that content after a hashtagged link isn't indexed. Just so I understand this... a.) if I were use a hashtag at the end of a link as the first link in the body of a page, this means that the rest of the article won't be indexed? b.) if I use a table of contents at the top of a page and link to places within the document, then only the areas of the page up to the table of contents will be indexed/crawled? Thanks ahead of time! I really appreciate the help.
Technical SEO | | Spencer_LuminInteractive0