Setting A Custom User Agent in Screaming Frog
-
Hi all,
Probably a dumb question, but I wanted to make sure I get this right.
How do we set a custom user agent in Screaming Frog? I know its in the configuration settings, but what do I have to do to create a custom user agent specifically for a website?
Thanks much!
- Malika
-
Setting a custom user agent determines things like HTTP/2 so there can be a big difference if you change it to something that might not take advantage of something like HTTP/2
Apparently, it is coming to Pingdom very soon just like it is to Googlebot
http://royal.pingdom.com/2015/06/11/http2-new-protocol/
This Is an excellent example of a user agent's ability to modify the way your site is crawled as well as how efficient it is.
https://www.keycdn.com/blog/https-performance-overhead/
It is important to note that we didn’t use Pingdom in any of our tests because they use Chrome 39, which doesn’t support the new HTTP/2 protocol. HTTP/2 in Chrome isn’t supported until Chrome 43. You can tell this by looking at the
User-Agentin the request headers of your test results.
Pingdom user-agent
Note: WebPageTest uses Chrome 47 which does support HTTP/2.
Hope that clears things up,
Tom
-
Hi Malika,
Think about screaming frog and what it has to detect in order to do that correctly it needs the correct user agent syntax for it will not be able to make a crawl that would satisfy people.
Using a proper syntax for a user agent is essential and I have tried to be non-technical in this explanation I hope it works.
the reason screaming frog needs the user agent because the user-agent was added to HTTP to help web application developers deliver a better user experience. By respecting the syntax and semantics of the header, we make it easier and faster for header parsers to extract useful information from the headers that we can then act on.
Browser vendors are motivated to make web sites work no matter what specification violations are made. When the developers building web applications don’t care about following the rules, the browser vendors work to accommodate that. It is only by us application developers developing a healthy respect
When the developers building web applications don’t care about following the rules, the browser vendors work to accommodate that. It is only by us application developers developing a healthy respect
It is only by us application developers developing a healthy respect for the standards of the web, that the browser vendors will be able to start tightening up their codebase knowing that they don’t need to account for non-conformances.
For client libraries that do not enforce the syntax rules, you run the risk of using invalid characters that many server side frameworks will not detect. It is possible that only certain users, in particular, environments would identify the syntax violation. This can lead to difficult to track down bugs.
I hope this is a good explanation I've tried to keep it very to the point.
Respectfully,
Thomas
-
Hi Thomas,
would you have a simpler tutorial for me to understand? I am struggling a bit.
Thanks heaps in advance
-
I think I want something that is dumbed down to my level for me to understand. The above tutorials are great but not being a full time coder, I get lost while reading those.
-
Hi Matt,
I havent had a luck with this one yet.
-
Hi Malika! How'd it go? Did everything work out?
-
happy I could be of help let me know if there's any issue and I will try to be of help with it. All the best
-
Hi Thomas,
That's a lot of useful information there. I will have a go on it and let you know how it went.
Thanks heaps!
-
please let me know if I did not answer the question or you have any other questions
-
this gives you a very clear breakdown of user agents and their set of syntax rules. The following is valid example of user-agent that is full of special characters,
read this please http://www.bizcoder.com/the-much-maligned-user-agent-header
user-agent: foo&bar-product!/1.0a$*+ (a;comment,full=of/delimitersreferences but you want to pay attention to the first URL
https://developer.mozilla.org/en-US/docs/Web/HTTP/Gecko_user_agent_string_reference
| Mozilla/5.0 (X11; Linux i686; rv:10.0) Gecko/20100101 Firefox/10.0 |
http://stackoverflow.com/questions/15069533/http-request-header-useragent-variable
-
if you formatted it correctly see below
User-Agent = product *( RWS ( product / comment ) )and it was received by your headers yes you could fill in the blanks and test it.
https://mobiforge.com/research-analysis/webviews-and-user-agent-strings
http://mobiforge.com/news-comment/standards-and-browser-compatibility
-
No, you Cannot just put anything in there. The site has to recognize it and ask why you are doing this?
I have listed how to build and already built in addition to what your browser will create by using useragentstring.com
Must be formatted correctly and have it work with a header it is not as easy as it sometimes seems but not that hard either.
You can make & use this to make your own from your Mac or PC
http://www.useragentstring.com/
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2747.0 Safari/537.36
how to build a user agent
- https://developer.mozilla.org/en-US/docs/Web/HTTP/Gecko_user_agent_string_reference
- https://developer.mozilla.org/en-US/docs/Setting_HTTP_request_headers
- https://msdn.microsoft.com/en-us/library/ms537503(VS.85).aspx
Lists of user agents
https://support.google.com/webmasters/answer/1061943?hl=en
https://msdn.microsoft.com/en-us/library/ms537503(v=vs.85).aspx
-
Hi Thomas,
Thanks for responding, much appreciated!
Does that mean, if I type in something like -
HTTP request user agent -
Crawler access V2
&
Robots user agent
Crawler access V2
This will work too?
-
To crawl using a different user agent, select ‘User Agent’ in the ‘Configuration’ menu, then select a search bot from the drop-down or type in your desired user agent strings.
http://i.imgur.com/qPbmxnk.png
&
Video http://cl.ly/gH7p/Screen Recording 2016-05-25 at 08.27 PM.mov
Or
Also see
http://www.seerinteractive.com/blog/screaming-frog-guide/
https://www.screamingfrog.co.uk/seo-spider/user-guide/general/#user-agent
https://www.screamingfrog.co.uk/seo-spider/user-guide/
https://www.screamingfrog.co.uk/seo-spider/faq/
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Setting country specific top level domain as alias - will site benefit from TLDs authority?
I have a host of sites that follow a top level domain strategy. For each local site they will be on the top level domain but with their country-languages prefix as the subdirectory. Such as below: example.com
Intermediate & Advanced SEO | | gracejo
example.com/uk-en
example.com/sg-en
example.com/de-de Each local site being on the TLD will benefit them in terms of SEO and it makes it easier to have one strategy. My question however, if the Netherlands comes on board, they would generally have example.com/nl-en. However they want their primary domain as examplenetherlands.nl and the TLD (example.com/nl-en) set as an alias/secondary domain that redirects to the primary. Will they benefit from any SEO if the TLD is not the primary address?0 -
How to manage user profiles in your website?
We have a real estate website in which agents and builders can create their profiles. My question is shall we use h1 or h2 tags in business profile pages or make them according to web 2.0 standards? In case header tags are used, if two agents have the same name and we have used h2 tag for them, then search result page will end up having two same h2's. Can someone please tell me the right way to manage business profiles in a website? Thanks
Intermediate & Advanced SEO | | dailynaukri1 -
SEO Best Practices for Customer Portals
We have a customer portal which is used to display customer's serial numbers, the knowledgebase, support ticketing information and forum. This information is behind a wall as a user must have support in order to view. The question is what are the best practices for SEO with the customer portal? Should we block these sections from bots? Is there a way to take advantage of the number of pages that are within the portal?
Intermediate & Advanced SEO | | ASCI-Marketing0 -
Feefo customer reviews for SEO
Hey Guys, I'm not overly techy, but, I need to advise on Feefo integration. I have been reading and it seems that https://support.feefo.com/support/solutions/articles/8000042161-full-integration-javascript-on-page this is the best integration method. Do you guys have any pointers with using Feefo? Are we still good to use intermediaries like this to handle our reviews for us? Thanks in advance for any help. Kay
Intermediate & Advanced SEO | | eLab_London0 -
User profile page optimisation - tips required
Hello, we have developed a network of medical professionals and our main goal on SEO is to rank on user names. I would like to use a profile [h**p://goo.gl/bUwFWW] i build in corporation with my client as sample and request any tips to increase ranking position of users profile page while searching for his name. Right now we list on 2nd page of google page. I would to know any specific tips / advices i miss out on page optimisation. Thanks in advance, C
Intermediate & Advanced SEO | | HaCos0 -
Can I use the same set of social media accounts on two sites?
I have a client who's company name / main site name is not his name. All his social accounts connected to his site are in his name. The site is verified with Google Places, etc. Now he asked for me to create a site for him in his own name with some similar info and a lot of new info. The million dollar question is do I use the same social media accounts on the new site? Facebook, twitter, youtube, etc? Will that hurt the organic rankings of the main site? I've seen this similar situation before. You may have someone who works at a large corporation who is mentioned on the main site has their own personal profile site just about them. Where they can go more in depth about things they are doing. If the other set of social account where in the company name it would be a no brainer to create new social accounts in his name. The issue is the main companies social accounts are already in his name.
Intermediate & Advanced SEO | | markpine3600 -
I need help with setting the preferred domain; www. or not??
Hi! I'm kinda new to the SEO game and struggling with this site I'm working on: http://www.moondoggieinc.com I set the preferred domain to www. in GWT but I'm not seeing it reroute to that? I can't seem to get any of my internal pages to rank, and I was thinking it's possiblly b/c of a duplicate content issue cause by this problem. Any help or guidance on the right way to set preferred domain for this site and whiy I can't get my internal pages to rank? THANKS! KristyO
Intermediate & Advanced SEO | | KristyO0 -
Setting up a Blog - Guest Authors
I am planning on setting up a blog in the next couple months. We would like to have 10 different categories that professionals can write about and submit articles. Any suggestions on which blogging software to put on the site. So far I have heard of WordPress and Joomla. (Not sure which is the best version) We want it to maybe have the bloggers logo or pictures, date, etc. None of us have had any experience with this so we needed some input. I have been reading everywhere that blogging is huge for SEO and so I just wanted to improve ours plus maybe drive more traffic with a high level blog for professionals. Any ideas or even books that I should go purchase to study up. When we do this it would be great to get it set up properly from the get go. 🙂 Boo
Intermediate & Advanced SEO | | Boodreaux0