How to fix these unwanted URLs?
-
Right now i have wordpress, one page website, but google also show wp-content. KIndly check below in google.
site:http://baltimoreelite.com/
How I can fix this issue?
-
Great job, Mark! I can see from this end that nearly all of those unwanted URLs have already dropped out of the results. That's far quicker than even I expected! And the ones that aren't gone are leading to a 403 Forbidden page, which is great.
One last thing you can do if you want. Because you are on HostGator, they are displaying their custom 403 error page, which has their branding all over it (nasty, kinda ugly) You could create your own simple 403 error page, add your own basic branding to it, and for instance add a line that says something like "You don't have permission to view this page or it is blocked for security reasons. Drop by the home page [link to home page] to find what you're looking for, or to conduct a search."
This basic page can be used to replace the one that HostGator provides by default so any visitors that hit it by accident will still feel like they are on your site, and will have a suggestion for what to do next. Your hosting control panel will have instructions for how & where to provide your own custom error pages.
Hope that last little tweak's useful.
Paul
-
Thank You Paul. All is well now.
-
If I were you, Mark, I'd add it right at the top of your htaccess file. I'd also add in a descriptive comment to make the reason for the directive clear. So:
BEGIN Remove ability to read directory indexes
Options -Indexes
END Remove ability to read directory indexes
These lines would be inserted right at the top of the htaccess file. I would also warn though, that I've had situations where caching plugins have overwritten such directives when they update the htaccess themselves. If that happens, you man need to try inserting it after #END WPSuperCache and before # BEGIN WordPress.
Hope that works for you?
Paul
-
Andy and Nishada - don't forget... Adding robots.txt disallows will do nothing to get already indexed URLs out of the search index after the fact.
Paul
-
You have a much bigger problem than what can be solved just with a robots.txt file, Mark.
All of those URLs are showing up because of a misconfiguration of your theme installation (likely caused by the theme developer) is allowing full display of all of the content of each of those directories. In addition to polluting your search results, as you've noticed, it's a also a pretty major security risk. You can see this in action by going to http://baltimoreelite.com/wp-content/themes/sintia/wpv_theme/assets/css/ What should happen when you go to that URL is you see a blank page, or receive a 403-Forbidden warning. Instead, you're seeing a full listing of the directory contents - bad news.
Since I don't know your hosting configuration, the easiest way to fix this issue is to add a line to your .htaccess file at the root of your site. This should correct for all such instances, You need to add this line:
<code>Options -Indexes</code>
If you're not familiar, the .htaccess file is a text file which you can edit with any text editor. You'll need to use an FTP program or the file manager in your hosting control panel to access it at the root of your site. (You may also need to enable "show hidden files" in your program). I always recommend backing up the existing file before editing just in case. You'll add this new line on its own line with a blank line between it and any other lines in the file. It can go near or at the top of your htaccess file.
Now getting those URLs out of the search index is going to take a bit more work. You'll want to implement the robots.txt exclusions like Andy suggested, and then you'll need to go into Google Webmaster Tools and use the Remove URLs tool to specifically request removal of the directories you blocked with the robots.txt and that you also want removed from the search results. (The robots.txt is a critical part of this process, as Google requires it be in place in order to process the removal requests.)
This, combined with the htaccess edit mentioned above, should keep those URLs from showing up again in teh future.
Hope that all makes sense. If not, be sure to ask!
Paul
-
Hi Mark,
I block the same on my site (which is also a single page). Here is the content of my Robots.txt file.
User-agent: * Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/ Sitemap: http://www.inetseo.co.uk/sitemap.xml.gz
-Andy
-
Hi Mark,
You need to tell crawlers to not to index those content by modifying the robots.txt file. Below is a good link with some examples and instructions
http://stackoverflow.com/questions/17029811/how-to-set-up-robots-txt-file-for-wordpress
Got a burning SEO question?
Subscribe to Moz Pro to gain full access to Q&A, answer questions, and ask your own.
Browse Questions
Explore more categories
-
Moz Tools
Chat with the community about the Moz tools.
-
SEO Tactics
Discuss the SEO process with fellow marketers
-
Community
Discuss industry events, jobs, and news!
-
Digital Marketing
Chat about tactics outside of SEO
-
Research & Trends
Dive into research and trends in the search industry.
-
Support
Connect on product support and feature requests.
Related Questions
-
Unsolved Orphaned unwanted urls from the cms
Hi
Technical SEO | | MattHopkins
I am working on quite an old cms, and there are bunch of urls that don't make any sense.
https://www.trentfurniture.co.uk/products/all-outdoor-furniture/all-outdoor-furniture/1
https://www.trentfurniture.co.uk/products/all-chairs/all-chairs/1
https://www.trentfurniture.co.uk/products/all-industries/all-chairs/1
https://www.trentfurniture.co.uk/products/all-chairs/all-industries/1
https://www.trentfurniture.co.uk/products/all-chairs/banqueting-furniture/1
https://www.trentfurniture.co.uk/products/all-chairs/bar-furniture/1
https://www.trentfurniture.co.uk/products/all-chairs/bentwood-furniture/1
For example there are no internal links. And fortunately not much traffic at all. But I can't see in the cms why they are generating? I've tried to check the html code to check why, what's the reason? But all I can think of is the structure....? something odd the cms writes?
Anyone have any ideas please? And would I redirect all these? Just thinking there could be a better solution/fix, rather than redirects since there are no links or traffic.....Like the devs solve why they are generating.....Unfortunately I get very slow responses from the devs as a 3rd pty company, hence on here ;0). (Some of those are indexed too)... :0) Thanks in advance....0 -
Folders in url structure?
Hello, Revamping an out-of-date website and am wondering if I need to include the folders (categories) in the url structure? The proposed structure has 8 main folders. I've been reading that Google is ok if the folder is not included in the url, but is it really? The hesitation I have is that the urls are getting long and the main folder only has only a sub folder beneath it. So, /folder-name/facility-name/treatment-overview. This looks too long, doesn't it? Thanks!
Technical SEO | | lfrazer1230 -
Which URL is better?
Hi everyone, Could you please help me with picking out the right URL for my company's website? We are MoonCreate and we make beautiful clothes. Unfortunately, the domain mooncreate.com is not available and I have to choose between mooncreatebrand.com or mooncreatewear.com Which one is better, in your opinion? Look forward to receive your suggestions! Thank you! 🙂
Technical SEO | | kirupa0 -
Strange URL redirecting to my new site
Hi all, I recently relaunched a site on a brand new URL - www.boardwarehouse.co.uk. I've spent the last couple of weeks building some backlinks as well as developing a basic content strategy. We've started ranking for a few of our less competitive keywords which is great, however there's a strange site which either redirects or is mirroring our content. I'm at a complete loss as to what's causing this to happen and what i can do to stop it. On the attachment - my content is top and second. The fourth result is the offending site. Any help/ advice would be most helpful! Thanks in advance, Alick 0BSyNn6
Technical SEO | | Alick3000 -
Remove unwanted map in SERP
My company is based in Brighton. We run courses in London. If you search 'london business writing' in Google UK, you get this: http://i39.tinypic.com/35me3qs.jpg Lolwut. Google is placing a link for a map to our Brighton offices beneath the second result. For a London-related keyword that links to a page for our London courses that contains an address for our London venue. We are registered on Google maps as being based in Brighton; we also have a map of our Brighton office on our contact page. But obviously, this is not relevant to this search. How do I get rid of this map for this keyword?
Technical SEO | | JacobFunnell0 -
How to change URL of RSS Feed?
Hi, There are some websites that keeps on scraping my content. I have blocked them already from accessing my website using .htaccess but they still get my content via RSS feed. I have tried delaying the RSS feed but I think this affected google rankings. My question is, is there a way to change the URL of my RSS Feed? From: http://www.mysite.com/feed to http://www.mysite.com/feed2
Technical SEO | | Trigun0 -
301 an old URL with a ? in the URL?
I am redoing a site and the URL's are changing structure. The client's site was in magento and in the store they would get two URLs, for example: /store/categoryname/productname and /store/categoryname/productname?SID=dslkajsfdoiu947598whouieht983hg98 Do I have to 301 redirect both of these URL's to their new counterpart? Both go to the same content but magento seemed to add these SIDs into the navigation and Google has both versions in the index.
Technical SEO | | DanDeceuster0 -
Canonical for non-exist URL ?
Hi I have a website what has parameter URL. For example www.example.com/index.php?page_id=1&no=2 I want that search engine see my page URL as; www.example.com/toys/cars But this URL is not exist in my website. And when i externally enter this page it goes to 404 page. If i add canonical url as www.example.com/toys/cars to the page www.example.com/index.php?page_id=1&no=2, what happened ? Is the url at the serp change as www.example.com/toys/cars ?
Technical SEO | | SEMTurkey0