Strange 404s in Screaming Frog
-
I just ran a website (Drupal) through screaming frog and the only 404s I found related to web pages which were the same as URLs already used on the website plus the company phone number so... www.company.com/[their phone number] - www.company.com/services[their phone number] - any ideas what might be causing this problem?
-
Hi Luke,
As the guys above replied with, sounds like an a href with a phone number
If you check the 'inlinks' (via the lower window tab), you'll be able to see the source of these errors (the pages they are located). Obviously you can then view the source code & find the exact link, and what might be the issue.
Hope that helps!
Feel free to pop through any further questions directly to our support btw (http://www.screamingfrog.co.uk/seo-spider/support/), I only spotted this via a Google alert.
(We try and reply super quick & will always look into any problems!)
Cheers.
Dan
-
This is typically caused by a link on the page that is not formed correctly.
-
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Google Search Operators Acting Strange
Hi Mozers, I'm using search operators for a count of how many pages have been indexed for each section of the site. I was able to download the first 1000 pages from Google Search Console but there are more than 1000 pages indexed, so I'm using operators for a count (even if I can't get the complete list of indexed URLs). [Although, if there is a better way, PLEASE let me know!] Anyway, in terms of search operators: from my understanding, the more general the URL, the more results should come up. However, when I put in the domain site:www.XXX it gives me FEWER results than when I put in site:www.XXX/. When I add the backslash to the end of the domain, it gives me MORE results. And when I put in site:www.AAA/BBB/CC it gives me MORE results than when I put in site:www.AAA/BBB. What's with this? Yael
Intermediate & Advanced SEO | | yaelslater1 -
Strange: page no longer present in SERPS and I'm not sure why
I indexed a new page last week and it ranked 1st The page is still live, still registering sessions in analytics, registering activity in search console Why is it no longer present for the keyword in ranked first for on Friday?
Intermediate & Advanced SEO | | Jacksons_Fencing0 -
Setting A Custom User Agent in Screaming Frog
Hi all, Probably a dumb question, but I wanted to make sure I get this right. How do we set a custom user agent in Screaming Frog? I know its in the configuration settings, but what do I have to do to create a custom user agent specifically for a website? Thanks much! Malika
Intermediate & Advanced SEO | | Malika10 -
Soft 404s for unpublished & 301'd content
Hi, One site I work with unpublished a lot of thin content. Great idea, right? These unpublished pages were then 301'd up to the main category page that they previously existed in. Now Google Webmaster Tools calls them out as soft 404 errors. This seems unexpected since the pages were 301'd. Here is my question; Is this a serious problem that may affect the site's overall organic results and if so what should I do about it? Thanks... Darcy
Intermediate & Advanced SEO | | 945010 -
Ranking strangeness
Just wondering if anyone had come across this kind of scenario which I had/have with a client. Rebuilt/revamped site that has got problems with thin content we are looking to resolve. Rankings for the main keywords were ok on page 2 or 3 and being worked on. Two weeks ago ALL main keywords disappeared from ranking (category pages). The homepage took over and ranked ALL those keywords at #1! Bang -straight to top spot. However happy I was I find this disconcerting as there is a reasonable level of competition. Today homepage rankings in the main go and directories come back into the rankings at some good positions on page one. We still need to fix thin and duplicated content! I can justify the problems and swing with the category pages but the massive cross the board top position for the homepage has been pretty bizarre. Has anyone else had the experience where the homepage suddenly 'out-performs' in a big way?
Intermediate & Advanced SEO | | MickEdwards0 -
Can't crawl website with Screaming frog... what is wrong?
Hello all - I've just been trying to crawl a site with Screaming Frog and can't get beyond the homepage - have done the usual stuff (turn off JS and so on) and no problems there with nav and so on- the site's other pages have indexed in Google btw. Now I'm wondering whether there's a problem with this robots.txt file, which I think may be auto-generated by Joomla (I'm not familiar with Joomla...) - are there any issues here? [just checked... and there isn't!] If the Joomla site is installed within a folder such as at e.g. www.example.com/joomla/ the robots.txt file MUST be moved to the site root at e.g. www.example.com/robots.txt AND the joomla folder name MUST be prefixed to the disallowed path, e.g. the Disallow rule for the /administrator/ folder MUST be changed to read Disallow: /joomla/administrator/ For more information about the robots.txt standard, see: http://www.robotstxt.org/orig.html For syntax checking, see: http://tool.motoricerca.info/robots-checker.phtml User-agent: *
Intermediate & Advanced SEO | | McTaggart
Disallow: /administrator/
Disallow: /bin/
Disallow: /cache/
Disallow: /cli/
Disallow: /components/
Disallow: /includes/
Disallow: /installation/
Disallow: /language/
Disallow: /layouts/
Disallow: /libraries/
Disallow: /logs/
Disallow: /modules/
Disallow: /plugins/
Disallow: /tmp/0 -
Recent ranking drop followed by strange behavior in SERPS
Recently on the evening of August 5th almost all of the keywords our pages ranked highly for dropped by anywhere from 5 to 8 pages. The only activity during this time was an article that had been picked up by a major news outlet and then was apparently copied onto other sources with links to our domain and article. More puzzling though is rather than simply having the same page show up lower for a keyword, in a number of instances a different page is now shown for the result, often with less or no relevance to the keyword. In some cases, for a single keyword phrase we've seen as many as 10 different pages rotated throughout the day when performing a search. Prior to our rankings falling, we've never seen this behavior.
Intermediate & Advanced SEO | | BrianQuinn0 -
Strange affiliate links in web master tools
Hi All I have been looking in the Google webmasters tools account for a client and from out of nowhere they suddenly have 42,000 links last time I checked it was just over 4,000. It looks to me like some of our affiliates have links showing, they are formatted so it goes via a php script that looks like accomtracking.php?estid=1234&ref=123 this link appears 3 times in the source code of the page that is linking to my client but Google webmaster tools thinks it is linking 12,000 times. I assume this is a problem with the database where the link points to, issuing a unique link for each visit or something similar but I am not sure how to explain it to my client and how to correct it / stop it happening again. There are around 37,000 links like this to the site from just 6 referring sites and the client have had an unnatural link warning so I am anxious to get this fixed. Any help gratefully received. Sean
Intermediate & Advanced SEO | | SKE0