Welcome to the Q&A Forum

Hi, everyone: problem solved.

Here is what I did: I created a seperate sitemap-xml and linked to all the new pdfs.

I updated the general sitemap.xml and linked to the new sitemap as well.

I (re)submitted both sitempas via the Webmaster Tools.

Within a few hours, most of pdfs got indexed and the overall quality of search has improved dramatically. Thanks for all your help.

Try this, I hope it helps: http://ready.mobi/launch.jsp?locale=en_EN

Hi! With this file, you can create a Google-friendly sitemap for any given folder almost automatically. No limits on the number of files. Please note that the code is the courtesy of @frkandris who generously helped me out when I had a similair problem. I hope it will be as helpful to you as it was to me

• Copy / paste the code below into a text editor.
• Edit the beginning of the file: where you see seomoz.com, put your own domain name there
• Save the file as getsitemap.php and ftp it to the appropriate folder.
• Write the full URL in your browser: http://www.yourdomain.com/getsitemap.php
• The moment you do it, a sitemap.xml will be generated in your folder
• Refresh your ftp client and download the sitemap. Make further changes to it if you wish.

===  CODE STARTS HERE ===

define(DIRBASE, './');define(URLBASE, 'http://www.seomoz.com/'); $isoLastModifiedSite = "";$newLine = "\n";$indent = " ";if (!$rootUrl) $rootUrl = "http://www.seomoz.com"; $xmlHeader = "$newLine"; $urlsetOpen = "<urlset xmlns="&quot;http://www.google.com/schemas/sitemap/0.84&quot;"  =""   <="" span="">xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.google.com/schemas/sitemap/0.84 http://www.google.com/schemas/sitemap/0.84/sitemap.xsd">$newLine";$urlsetValue = "";$urlsetClose = "</urlset>$newLine"; function makeUrlString ($urlString) {   return htmlentities($urlString, ENT_QUOTES, 'UTF-8');} function makeIso8601TimeStamp ($dateTime) {   if (!$dateTime) {       $dateTime = date('Y-m-d H:i:s');   }   if (is_numeric(substr($dateTime, 11, 1))) {       $isoTS = substr($dateTime, 0, 10) ."T"                .substr($dateTime, 11, ."+00:00";   }   else {       $isoTS = substr($dateTime, 0, 10);   }   return $isoTS;} function makeUrlTag ($url, $modifiedDateTime, $changeFrequency, $priority) {   GLOBAL $newLine;   GLOBAL $indent;   GLOBAL $isoLastModifiedSite;   $urlOpen = "$indent<url>$newLine";</url>   $urlValue = "";   $urlClose = "$indent$newLine";   $locOpen = "$indent$indent<loc>";</loc>   $locValue = "";   $locClose = "$newLine";   $lastmodOpen = "$indent$indent<lastmod>";</lastmod>   $lastmodValue = "";   $lastmodClose = "$newLine";   $changefreqOpen = "$indent$indent<changefreq>";</changefreq>   $changefreqValue = "";   $changefreqClose = "$newLine";   $priorityOpen = "$indent$indent<priority>";</priority>   $priorityValue = "";   $priorityClose = "$newLine";    $urlTag      = $urlOpen;   $urlValue     = $locOpen .makeUrlString("$url") .$locClose;   if ($modifiedDateTime) {    $urlValue .= $lastmodOpen .makeIso8601TimeStamp($modifiedDateTime) .$lastmodClose;    if (!$isoLastModifiedSite) { // last modification of web site        $isoLastModifiedSite = makeIso8601TimeStamp($modifiedDateTime);    }   }   if ($changeFrequency) {    $urlValue .= $changefreqOpen .$changeFrequency .$changefreqClose;   }   if ($priority) {    $urlValue .= $priorityOpen .$priority .$priorityClose;   }   $urlTag .= $urlValue;   $urlTag .= $urlClose;   return $urlTag;} function rscandir($base='', &$data=array()) {  $array = array_diff(scandir($base), array('.', '..')); # remove ' and .. from the array /  foreach($array as $value) : / loop through the array at the level of the supplied $base /    if (is_dir($base.$value)) : / if this is a directory /     $data[] = $base.$value.'/'; / add it to the $data array /     $data = rscandir($base.$value.'/', $data); / then make a recursive call with the     current $value as the $base supplying the $data array to carry into the recursion /    elseif (is_file($base.$value)) : / else if the current $value is a file /     $data[] = $base.$value; / just add the current $value to the $data array */    endif;  endforeach;  return $data; // return the $data array } function kill_base($t) {   return(URLBASE.substr($t, strlen(DIRBASE)));} $dir = rscandir(DIRBASE);$a = array_map("kill_base", $dir); foreach ($a as $key => $pageUrl) {       $pageLastModified = date ("Y-m-d", filemtime($dir[$key]));       $pageChangeFrequency = "monthly";       $pagePriority = 0.8;        $urlsetValue .= makeUrlTag ($pageUrl, $pageLastModified, $pageChangeFrequency, $pagePriority); } $current = "$xmlHeader$urlsetOpen$urlsetValue$urlsetClose"; file_put_contents('sitemap.xml', $current); ?>

=== CODE ENDS HERE ===

It may be a good idea to include all the pdf files on the sitemap, even if it is a troublesome process.

Otherwise it just takes too long for Google to index them.

What still surprises me is that even for a site search, you need to win the 'indexing battle'. I thought that Google indexes everythig within the map for the 'sake of the site search' and displays the results when a visitor is searching within the site. Less fancy softwares are actually doing the job. I thought a Google Site Search provides something even better.

Last crawl - thanks, great info.

yes, all new pdfs are linked from the html files.

This the summary page of one article: http://www.ejmh.eu/5archives_ppr_jaggle_061.html

In the middle of the page, you see 'download full text' - this is from where the individual papers (pdf) are linked.

You are great, thanks for your time. Yeah, I did check things out with this google command: there are pdf's listed but these are all old pdfs I have put up a long time ago. None of the pdfs I have put up recently are among those indexed.

Do you think that only those urls come up through a customized site search that are indexed by Google? Does Google not crawl the site and make a list of urls for the sake of the search purely? (Zoom search does it, for example) In theory, there could be two different type of 'crawls': one for the site search and one for the larger world, searching in the browser.

As for the settings...can you plase help me further: what exactly would you change?

Thanks for the tip, it's a good one. But they are all 100% texts.

The site in question is http://www.ejmh.eu

I am pretty unsatisfied with the results I am getting from the Site Search provided by Google.

We have over 160 pdf files in this subfolder: http://www.ejmh.eu/mellekletek

The files are the digital versions of articles. When I search for content in those pdf files, Google does not show results. It does show results from older pages, dating back 1-2 years but it is certainly not showing anything from pdf files that I have just put up 3 weeks ago.

My questions:

If I place a Google Search on a site, does it not automatically display results from ALL the content in the root domain?

Is there any correlation between how the Site Search is indexing the files and how Google is indexing the urls in general?

Should I just wait and see whether site search performance improves or should I switch to another Search software like Zoom Search?

It is vital to have a proper, high-quality search functioning on that site in the very near future.

What are your experiences? Any tips are greatly appreciated.