No index
-
Screaming frog spider does index pages on our website like:
- wp-content/plugins/woocommerce/assets/js/frontend/jquery-ui-touch-punch.min.js?ver=2.3.9
- wp-content/plugins/mailchimp-for-wp/assets/css/checkbox.min.css?ver=2.3.2
Is it a bad/good idea to set my parameters in Webmastertools and tell Google not to crawl pages that begin with wp/content?
Thanks!
-
Hi Dirk,
Good explanation, that makes sense.
Thanks!
-
Hi,
If these files are used to render the content on screen - it's not a good idea to block these files from indexing.
Official Google recommendation is to allow these files to be indexed:
"If resources like JavaScript or CSS in separate files are blocked (say, with robots.txt) so that Googlebot can’t retrieve them, our indexing systems won’t be able to see your site like an average user. We recommend allowing Googlebot to retrieve JavaScript and CSS so that your content can be indexed better. "Source: http://googlewebmastercentral.blogspot.be/2014/05/understanding-web-pages-better.html
Rgds,
Dirk
-
I will not suggest you to apply parameters in your robots.txt/WMT unless you're quite sure that your "wp-content" does not contain any content that you want to get indexed.
If you have any problem with Screaming Frog, you may restrict it to not crawl your site. If you're on WordPress try out this plugin "WP Ban",
https://lesterchan.net/portfolio/programming/php/#wordpress-wp-banHope this helps!
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
My Website stopped being in the Google Index
Hi there, So My website is two weeks old, and I published it and it was ranking at about page 10 or 11 for a week maybe a bit longer. The last few days it dropped off the rankings, which I assumed was the google algorithm doing its thing but when I checked Google Search Console it says my domain is not in the index. 'This page is not in the index, but not because of an error. See the details below to learn why it wasn't indexed.' I click request indexing, then after a bit, it goes green saying it was successfully indexed. Then when I refresh the website it gives me the same message 'This page is not in the index, but not because of an error. See the details below to learn why it wasn't indexed.' Not sure why it says this, any ideas or help is appreciated cheers.
Technical SEO | | sydneygardening0 -
IP Redirect causing Indexing Issue
Hi, I am trying to redirect any IP from outside India that comes to Store site (https://store.nirogam.com/) to Global Store site (https://global.nirogam.com/) using this methodThis is causing various indexing issues for Store site as Googlebot from US also gets redirected!- Very few pages for "store.nirogam.com/products/" are being indexed. Even after submission of sitemap it indexed ~50 pages and then went back to 1 page etc. Only ~20 pages indexed for now.- After this I tried manually indexing via "Crawl -> Fetch as Google" - but then it showed me a redirect to global.nirogam.com. All have their "status -> Redirected" - This is why bots are not able to index the site.What are possible solutions for this? How can we tell bots to index these pages and not get redirected?Will a popup method where we ask user if they are outside India help in solving this issue?All approaches/suggestions will be highly appreciated.
Technical SEO | | pks3330 -
Noindex Pages indexed
I'm having problem that gogole is index my search results pages even though i have added the "noindex" metatag. Is the best thing to block the robot from crawling that file using robots.txt?
Technical SEO | | Tedred0 -
Getting querystring indexed?
Hi everybody! I work with tags a lot on my photo blog but I haven't gotten Google to index one tag so far. Any tips on how to do this? Thanks / Niklas
Technical SEO | | KAN-Malmo0 -
IP addresses indexed?
I've met with a potential client who has a site with 1,000's of very specific part #'s which don't show in the SERP's on Google. They definitely have the issue of dynamic URL's - but the URL for the part # searches is an IP address rather than their domain name - example: 188.888.888.888/partssearch.php?pnum='1233445' I've not seen the IP address used like this for an external website - is this acceptable for SEO purposes? Thanks, Mark
Technical SEO | | DenverKelly0 -
Bing indexing
Hello, people~ I want to discuss about Bing indexation. I have a new web site which opened about 3 months ago. Google has no problem to index my site and all pages within the site indexed by Google. However, Bing and Yahoo is different story. I used manual submission, Bing webmaster tool to let Bing know about the site. However, Bing is not indexing my site yet. I researched about it and found that my site should have some external links before I get index by Bing. I check external links of my site with Google webmaster tool, SEOmoz tool and "link:" on Google. All tools show different number as below. Google webmaster Tool : more than 50 SEMoz site explorer : 5 link: on Google: none Why all method of checking links are different and which on should most depend on? Also how many links should I have in order to get index by Bing? Could you people please share your opinion?
Technical SEO | | Artience0 -
How to get user genreated reviews indexed properly?
We are currently working to improve the deployment of a review widget on our website. The widget was deployed about 18 months ago and all reviews are behind Java navigation. I have been working with our IT staff to get the reviews into an HTML page which will either live on the product page as a tab or will be a link from the product page. Our IT staff has suggested leaving the Java navigation for users and creating separate HTML pages specifically for search engines. Based on my experience, this sounds like a bad idea, basically creating pages just for search engines that will not be use by site visitors, although the visitors will have access to the same content via the Java navigation. Anyone care to comment on this? Is creating HTML pages specifically for search engines a bad idea? An acceptable idea?
Technical SEO | | seorunner0
