Please help to identify the following bots and spiders
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Googlebot and other spiders are searching for odd links in our website trying to understand why, and what to do about it.
I recently began work on an existing Wordpress website that was revamped about 3 months ago. https://thedoctorwithin.com. I'm a bit new to Wordpress, so I thought I should reach out to some of the experts in the community.Checking ‘Not found’ Crawl Errors in Google Search Console, I notice many irrelevant links that are not present in the website, nor the database, as near as I can tell. When checking the source of these irrelevant links, I notice they’re all generated from various pages in the site, as well as non-existing pages, allegedly in the site, even though these pages have never existed. For instance: https://thedoctorwithin.com/category/seminars/newsletters/page/7/newsletters/page/3/feedback-and-testimonials/ allegedly linked from: https://thedoctorwithin.com/category/seminars/newsletters/page/7/newsletters/page/3/ (doesn’t exist) In other cases, these goofy URLs are even linked from the sitemap. BTW - all the URLs in the sitemap are valid URLs. Currently, the site has a flat structure. Nearly all the content is merely URL/content/ without further breakdown (or subdirectories). Previous site versions had a more varied page organization, but what I'm seeing doesn't seem to reflect the current page organization, nor the previous page organization. Had a similar issue, due to use of Divi's search feature. Ended up with some pretty deep non-existent links branching off of /search/, such as: https://thedoctorwithin.com/search/newsletters/page/2/feedback-and-testimonials/feedback-and-testimonials/online-continuing-education/consultations/ allegedly linked from: https://thedoctorwithin.com/search/newsletters/page/2/feedback-and-testimonials/feedback-and-testimonials/online-continuing-education/ (doesn't exist). I blocked the /search/ branches via robots.txt. No real loss, since neither /search/ nor any of its subdirectories are valid. There are numerous pre-existing categories and tags on the site. The categories and tags aren't used as pages. I suspect Google, (and other engines,) might be creating arbitrary paths from these. Looking through the site’s 404 errors, I’m seeing the same behavior from Bing, Moz and other spiders, as well. I suppose I could use Search Console to remove URL/category/ and URL/tag/. I suppose I could do the same, in regards to other legitimate spiders / search engines. Perhaps it would be better to use Mod Rewrite to lead spiders to pages that actually do exist. Looking forward to suggestions about best way to deal with these errant searches. Also curious to learn about why these are occurring. Thank you.
Technical SEO | | linkjuiced0 -
How to identify orphan pages?
I've read that you can use Screaming Frog to identify orphan pages on your site, but I can't figure out how to do it. Can anyone help? I know that Xenu Link Sleuth works but I'm on a Mac so that's not an option for me. Or are there other ways to identify orphan pages?
Technical SEO | | MarieHaynes0 -
Creating in-text links with ' 'target=_blank' - helping/hurting SEO!?!
Good Morning Mozzers, I have a question regarding a new linking strategy I'm trying to implement at my organization. We publish 'digital news magazines' that oftentimes have in-text links that point to external sites. More recently, the editorial department and me (SEO) conferred on some ways to reduce our bounce rate and increase time on page. One of the suggestions I offered is to add the 'target=_blank" attribute to all the links so that site visitors don't necessarily have to leave the site in order to view the link. It has, however, come to my attention that this can have some very negative effects on my SEO program, most notably, (fake or inaccurate) time(s) on-page. Is this an advisable way to create in-text links? Are there any other negative effects that I can expect from implementing such a strategy?
Technical SEO | | NiallSmith0 -
Getting a link removed from brand search - please help!
Hello all you mozzers! Ive just come into work with an established company who have one major problem when you google "palicomp" the second link that comes up is to consumeractiongroup with a thread that has been damaging the business for over 2 years, this thread is absolutely not representative of the business today. Strangely stronger links in search have better authority but google has ranked this post as being highly relevant to the business, does anybody know of any strategies we can do to get this removed, we have contacted consumeractiongroup directly but they are not prepared to move it. Does anyone have any idea of removal ideas or what we can do its crippling our business, we cant work out as to why its ranking better! Chris
Technical SEO | | palicomp0 -
SEO Yoast Help Needed
Anyone familar with SEO Yoast and interested in being hired to check out my settings for SEO. Thinking about 30 minute screen sharing session an helping me figure out what I am am doing wrong? Just cleaned up duplicates because of tags and now I see the images are getting duplicated as well as some of the titles. So new to Wordpress here I shine. Message me if you can help. Much Appreciated!!
Technical SEO | | Force70 -
Need Help writing 301 redirects in .htaccess file
SEOmoz tool shows me 2 errors for duplicate content pages (www.abc.com and www.abc.com/index.html). I believe, the solution to this is writing 301 redirects I need two 301 redirects 1. abc.com to www.abc.com 2. /index.html to / (which is www.abc.com/index.html to www.abc.com) The code that I currently have is ................................................... RewriteEngine On
Technical SEO | | WebsiteEditor
RewriteCond %{HTTP_HOST} ^abc.com
RewriteRule (.*) http://www.abc.com/$1 [R=301,L] Redirect 301 http://www.abc.com/index.html http://www.abc.com ...................................................... but this does not redirect /index.html to abc.com. What is wrong here? Please help.0 -
Please help....
Hi Guys! Ok a bit of a funny one here which is causing a confusion between us and a web designer and I was wondering if anyone on here might be able to help. Just a bit of back ground for you, the website has been built on Concrete 5 and when we tried to building a sitemap we found over 110,000 pages. When we spoke to the web designer they have told us that within Google webmaster tools, Google has only indexed 58. But.... (and this is where things get a little confusing, so bare with me.) I thought that cant be right so into the Google search bar I put in site:www.sitename.co.uk and had 217 results appear. So google cant have just 58 pages indexed, right? So after speaking to the designer he then posted on the Concrete 5 help forum, to try and help figure it out. I have posted his exact forum post below that the web designer has asked: I'm having some issues where a site we are working on seems to be making multiple pages going to the same page. An SEO specialist has run a report and found a number of duplicate pages created by C5. We are concerned that this is going to dilute or worse penalise the way google sees the site. http://www.sitename.co.uk/
Technical SEO | | NoisyLittleMonkey
[http://www.sitename.co.uk/index.php?cID=?akID[155]atSelectOptionID...
[http://www.sitename.co.uk/index.php?cID=?akID[155]atSelectOptionID...
[http://www.sitename.co.uk/index.php?cID=?akID[155]atSelectOptionID... Is there a way of stopping google from accessing these duplicate 'cID' pages and stop them being made? Also is there a way of getting rid of the ones that are there? We've done a number of sites in C5 and are beginning to get concerned about this... So I guess my question is: If I can access the same content via 4-5 different cID's is that classed as duplicate content? Thanks in advance guys, and any help would greatly appreciated. 🙂0