If a page isn't linked to or directly sumitted to a search engine can it get indexed?
-
Hey Guys,
I'm curious if there are ways a page can get indexed even if the page isn't linked to or hasn't been submitted to a search engine.
To my knowledge the following page on our website is not linked to and we definitely didn't submit it to Google - but it's currently indexed:
<cite>takelessons.com/admin.php/adminJobPosition/corp</cite>
Anyone have any ideas as to why or how this could have happened? Hopefully I'm missing something obvious
Thanks,
Jon
-
You're welcome Jon.
That's a good question. I don't know the official answer on that one, though suspect that Google does check to see if the page exists, mainly because often it will be a valid URL that somebody types in the search box instead of the address bar. If Google don't have that page in their index, they'd at least like to consider adding it.
http://www.google.com/addurl.html is the URL adding page for Google as you'll already know. As well as Google relying on people to use this form, they will also, I suspect, crawl URLs that are entered into a search box. Makes sense to me that Google would at least visit these pages searched on, though can't be sure.
Regards
Simon
-
Thanks for the responses guys!
Are either of you aware of whether or not Google would ever index a page if someone searched for it? For example, say someone did a Google search for the URL I specified above. Would Google ever get curious and then try and crawl it?
Thanks again for taking the time to help out
-
Hey there Jon.
It seems that it was last indexed almost a month ago on Oct. 21, 2011. I would suggest that you follow Simon's advice on the NoFollow tag.
Additionally, check your GWT to ensure that it is not indexed...If it is, then try to resubmit it to get indexed (I know, I know, it doesn't make sense), but it will send out a message to crawl it. NoFollow, NoIndex tells the spider NOT HERE....Anywho, good luck with that and let us know how it turned out!
Cheers!
P.S. At least it's not indexed in Bing
-
Hi Jon
This is a strange one, I too haven't found a link to your admin page. There could have been one at some point in the past. Google's bot is rather clever at finding pages on the so-called 'invisible web', so best to request non-indexing of pages that you don't want indexed (covered below).
You're right in thinking that search bots follow links to find and index pages, whether it be an external or internal site link or a sitemap link. They also find pages through actual submissions.
- I'd suggest modifying the Robots tag on your admin page, include a 'NoIndex' just before the NoFollow'.
- Also include a Disallow command in your Robots.txt file for this admin page, and perhaps all pages within the admin section.
- Then, request the URL be removed from Google's index via Google Webmaster Tools ("Site configuration", "Crawler access", "Remove URL").
Hope that helps,
Regards
Simon
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
How to use Google search console's 'Name change' tool?
Hi There, I'm having trouble performing a 'Name change' for a new website (rebrand and domain change) in Google Search console. Because the 301 redirects are in place (a requirement of the name change tool), Google can no longer verify the site, which means I can't complete the name change? To me, step two (301 redirect) conflicts with step there (site verification) - or is there a way to perform a 301 redirect and have the tool verify the old site? Any pointers in the right direction would be much appreciated. Cheers Ben
Technical SEO | | cmscss0 -
Purchased domain with links - redirect page by page or entire domain?
Hi, I purchased an old domain with a lot of links that I'm redirecting to my site. I want all of their links to redirect to the same page on my site so I can approach this two different ways: Entire site
Technical SEO | | ninel_P
1.) RedirectMatch 301 ^(.*)$ http://www.xyz.com or Page by page
2). Redirect 301 /retiredpage.html http://www.xyz.com/newpage.html Is there a better option I should go with in regards to SEO effectiveness? Thanks in advance!0 -
How do I direct users to site page when they search vanity URL?
My company runs a contest via a landing page on our website. The full URL to the landing page is rather long so we have a vanity URL that we use for advertising purposes. I have a 301 on the vanity URL to the landing page URL so people visiting it directly end up where they should just fine. But if a user goes to Google and types the vanity URL into the search bar, the landing page is nowhere to be found in the results. What do I need to do to get the landing page to show in results when people search the vanity URL?
Technical SEO | | jarjarjarvis0 -
Why are my webpages not getting indexed?
I want to figure out why a lot of my pages for my website are not getting indexed by google. I have installed the SEO plugin by Yoast to my wordpress website. Under the titles and meta section of the plugin options I have set categories and tags to noindex. In WMT, google is saying that all my category pages and most of my tag pages are not being indexed. I want to make sure that the reason these pages are not being indexed are because of the SEO plugin. I want to prevent duplicate content so that is the reason I have set my categories and tags to noindex. Please respond if you know the absolute answer, its very important that I have my website indexed the proper way I want it to.
Technical SEO | | Dino640 -
How can I get the most out of uploading a print magazine to my client's website?
Hi Mozers, My client is just about to launch a print magazine for her watch business. There is so much valuable content in the magazine and we want to feature it on the website both for SEO purposes and also for those who prefer to read articles online instead of reading a physical magazine. My question is: what is the best method of displaying the magazine to get the most from search rankings and also to capitalise on the beautiful imagery from the magazine. The best option that I can think of is to upload the magazine as a flipbook and create a separate page on the website to display each article so that search engine crawlers can index the content. I do understand that this could be problematic if users are only spending time reading the flipbook and not so much time on the article pages. Do you guys have any suggestions about how to get the most out of this opportunity for my client? THANK YOU IN ADVANCE. Meaghan
Technical SEO | | StoryScout0 -
What is meant by to many on page links
I have just done the report for my site http://www.in2town.co.uk and it says i have 246 on page links but i am not sure how come i have got that many. I know i have a large number of links and in the old days it says that you should keep the links under 100 but now with website speed and the net, people are saying this is no longer listened to. A report i read said that the links should not confuse the reader or put them off, so i am just wondering what your thoughts are on a site with over a 100 links on the home page and also if my site does have to many links what should i do about it. I cannot understand why it is showing 246 when i do not see that many on the page, any advice would be great
Technical SEO | | ClaireH-1848860 -
Noindex Pages indexed
I'm having problem that gogole is index my search results pages even though i have added the "noindex" metatag. Is the best thing to block the robot from crawling that file using robots.txt?
Technical SEO | | Tedred0 -
No index directory pages?
All, I have a site built on WordPress with directory software (edirectory) on the backend that houses a directory of members. The Wordpress portion of the site is full of content and drives traffic through to the directory. Like most directories, the results pages are thin on content and mainly contain links to member profiles. Is it best to simply no index the search results for the directory portion of the site?
Technical SEO | | JSOC0