If a page isn't linked to or directly sumitted to a search engine can it get indexed?
-
Hey Guys,
I'm curious if there are ways a page can get indexed even if the page isn't linked to or hasn't been submitted to a search engine.
To my knowledge the following page on our website is not linked to and we definitely didn't submit it to Google - but it's currently indexed:
<cite>takelessons.com/admin.php/adminJobPosition/corp</cite>
Anyone have any ideas as to why or how this could have happened? Hopefully I'm missing something obvious
Thanks,
Jon
-
You're welcome Jon.
That's a good question. I don't know the official answer on that one, though suspect that Google does check to see if the page exists, mainly because often it will be a valid URL that somebody types in the search box instead of the address bar. If Google don't have that page in their index, they'd at least like to consider adding it.
http://www.google.com/addurl.html is the URL adding page for Google as you'll already know. As well as Google relying on people to use this form, they will also, I suspect, crawl URLs that are entered into a search box. Makes sense to me that Google would at least visit these pages searched on, though can't be sure.
Regards
Simon
-
Thanks for the responses guys!
Are either of you aware of whether or not Google would ever index a page if someone searched for it? For example, say someone did a Google search for the URL I specified above. Would Google ever get curious and then try and crawl it?
Thanks again for taking the time to help out
-
Hey there Jon.
It seems that it was last indexed almost a month ago on Oct. 21, 2011. I would suggest that you follow Simon's advice on the NoFollow tag.
Additionally, check your GWT to ensure that it is not indexed...If it is, then try to resubmit it to get indexed (I know, I know, it doesn't make sense), but it will send out a message to crawl it. NoFollow, NoIndex tells the spider NOT HERE....Anywho, good luck with that and let us know how it turned out!
Cheers!
P.S. At least it's not indexed in Bing
-
Hi Jon
This is a strange one, I too haven't found a link to your admin page. There could have been one at some point in the past. Google's bot is rather clever at finding pages on the so-called 'invisible web', so best to request non-indexing of pages that you don't want indexed (covered below).
You're right in thinking that search bots follow links to find and index pages, whether it be an external or internal site link or a sitemap link. They also find pages through actual submissions.
- I'd suggest modifying the Robots tag on your admin page, include a 'NoIndex' just before the NoFollow'.
- Also include a Disallow command in your Robots.txt file for this admin page, and perhaps all pages within the admin section.
- Then, request the URL be removed from Google's index via Google Webmaster Tools ("Site configuration", "Crawler access", "Remove URL").
Hope that helps,
Regards
Simon
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
I'm struggling to understand (and fix) why I'm getting a 404 error. The URL includes this "%5Bnull%20id=43484%5D" but I cannot find that anywhere in the referring URL. Does anyone know why please? Thanks
Can you help with how to fix this 404 error please? It appears that I have a redirect from one page to the other, although the referring page URL works, but it appears to be linking to another URL with this code at the end of the the URL - %5Bnull%20id=43484%5D that I'm struggling to find and fix. Thanks
Technical SEO | | Nichole.wynter20200 -
What to do about spam links I didn't create?
I have dropped in rankings 3-5 points over the past 6 months and have been trying to figure out why. One thing I found was a ton of my pictures on a image net ring. I obviously didn't put those photos there or give permission to use them. It looks like an offshore website. How do we deal with these type of bad links?
Technical SEO | | CalicoKitty20000 -
Any idea why pages are not being indexed?
Hi Everyone, One section on our website is not being indexed. The product pages are, but not some of the subcategories. These are very old pages, so thought it was strange. Here is an example one one: https://www.moregems.com/loose-cut-gemstones/prasiolite-loose-gemstones.html If you take a chunk of text, it is not found in Google. No issues in Bing/Yahoo, only Google. You think it takes a submission to Search Console? Jeff
Technical SEO | | vetofunk1 -
How can I make sure a desktoppage is shown in the (desktop) search results instead of the mobile page?
When I search for my brandname, the mobile version of the customer support page is shown in the (desktop) results. We use a m.example.nl mobile webpage. To try to solve the problem, we’ve adjusted the following: Made sure the homepage is marked according to schema.org Homepage expanded with textual content and headings containing our brandname Removed all the textual content from the mobile customer support page Added the mobile customer support page to the mobile sitemap What can we change more in settings/marking/sitemap, to make sure our desktop homepage is shown in the brandname results?
Technical SEO | | WillieBV0 -
Is there a way to index important pages manually or to make sure a certain page will get indexed in a short period of time??
Hi There! The problem I'm having is that certain pages are waiting already three months to be indexed. They even have several backlinks. Is it normal to have to wait more than three months before these pages get an indexation? Is there anything i can do to make sure these page will get an indexation soon? Greetings Bob
Technical SEO | | rijwielcashencarry0400 -
404 error - but I can't find any broken links on the referrer pages
Hi, My crawl has diagnosed a client's site with eight 404 errors. In my CSV download of the crawl, I have checked the source code of the 'referrer' pages, but can't find where the link to the 404 error page is. Could there be another reason for getting 404 errors? Thanks for your help. Katharine.
Technical SEO | | PooleyK0 -
Any idea why our sitemap images aren't indexed?
Here's our sitemap: http://www.driftworks.com/shop/sitemap/dw_sitemap.xml In google webmaster tools, I can see the sitemap report and it says: Items:Web Submitted:2,798 Indexed:2,910 Items:Images Submitted:3,178 Indexed:0 Do you have any idea why our images are not being indexed according to webmaster tools? I checked a few of the image URLs and they worked nicely. Thanks in advance, J
Technical SEO | | DWJames0