Nutch Re-crawling

Nutch allows you to crawl the web or a filesystem in order to build up index store of all its content. If your objective is to simply crawl the content once, it is fairly easy. But if you want to continuously monitor a site and crawl updates, it can be harder. Harder because the Nutch documentation does not have many details about it.

Nutch will update any previously indexed urls/files, delete any inactive ones and add any new ones it encounters while re-crawling. There are a few very simple settings that need changed if you wish Nutch to do this

Nutch stores a record of all the files/urls it has encountered whilst doing its crawl and is called the crawldb. Initially this is built from the list of urls/files provided by the user using the inject command which will be normally be taken from your seed.txt file, in our case, the nutch.txt file

Nutch uses a generate/fetch/update process:
generate: This command looks at the crawldb for all the urls/files that are due for fetching and regroups them in a segment. A url/file is due for fetch if if it is new or the time has expired for that url/file and is now due for recrawling (default is 30 days).
fetch: This command will go an fetch all the urls/files specified in the segment.
update: This command will add the results of the crawling, which have been stored in the segment, into the crawldb and each url/file will be updated to indicate the time it wad fetched and when its next scheduled fetch is. If any urls/files have been discovered, they will be added and marked as not fetched.

How does Nutch can detect if a page has changed or not? Each time a page is fetched, Nutch computes a signature for the page. At the next fetch, if the signature is the same (or if a 304 is returned by the web server because of the If-Modified-Since header), Nutch can tell if the page was modified or not (It is not just the content, if the http headers or metatags have changed it will be marked as modified). If a document no longer exists it returns a 404 and will be marked DB_GONE. During the update cycle Nutch has the ability to purge all the those urls/files that have been marked DB_GONE.

The linkdb stores the finalised indexes that Nutch has generated from the crawl and this is the data that Nutch passes to the Solr Server during the solrindex process.

Set-Up

The first thing is, you need to allow Nutch to re-crawl. You may have noticed that if you try to run a crawl on the same source using the same folder name as you used before that it will tell you there are no more urls/files to crawl. There is a default setting for a url/file that states it may not be re-crawled for 30 days. So edit your nutch-site.xml and edit the value to your liking:

<property>
  <name>db.fetch.interval.default</name>
  <value>43200</value>
  <description>The default number of seconds between re-fetches of a page (30 days).
  </description>
</property>

If you set the number too low you may get into an infinite loop. During a crawl when it finishes one of its cycles, it will notice that it is time for that url to be re-crawled, and at the end of that cycle it notices again that it needs re-crawled. So ensure that the number of seconds you select will be longer than the nutch crawler will take to complete it crawl of everything

Adding the following code will delete all urls/files marked with DB_GONE and thus ensuring all urls/files are active and up-to-date.

<property>
  <name>db.update.purge.404</name>
  <value>true</value>
  <description>If true, updatedb will add purge records with status DB_GONE
  from the CrawlDB.
  </description>
</property>

You are now all set to re-crawl! Just issue the same command to crawl normally but ensure the folder containing the Nutch indexes is the one you used before, so if you performed the crawl previously and your folder was named "Test"

./nutch crawl urls -dir Test -solr http://localhost:8080/solr/ -depth 3 -topN 100

Then you will want to issue the exact same command again to recrawl, although you are free to change the depth and topN to your own liking

./nutch crawl urls -dir Test -solr http://localhost:8080/solr/ -depth 3 -topN 100

Issue

There is an issue when it comes to indexing this to Solr, it passes all the indexes over and updates the ones that require updating. If a url/file has been deleted, it has been purged from Nutch's indexes and when it passes them over to Solr, Solr does not know that it has been deleted so still stores a record of the old ur/filel. The issue can be combatted by deleting the Solr indexes, and then passing the Nutch indexes over to Solr, and it comes with very little additional performance cost

Deleting Dead URLS & Files That No Longer Exist Nutch

Nutch allows you to crawl the web or a filesystem in order to build up index store of all its content. If your objective is to simply crawl the content once, it is fairly easy. But if you want to continuously monitor a site and crawl updates, it can be harder. Harder because the Nutch documentation does not have many details about that.

When you recrawl your source there can be a previously active URLs or Files that now no longer exist and you would like Nutch to remove them from its Indexes.  Nutch would update any changes made to documents it had indexed in the past but any files that had been deleted still remained in the indexes.

If you wish to skip straight to the solution then just go to the end of the post, but if you wish to understand what is happening then read on.

Nutch stores a record of all the files/urls it has encountered whilst doing its crawl and is called the crawldb.  Initially this is built from the list of urls/files provided by the user using the inject command which will be normally be taken from your seed.txt file.

Nutch uses a generate/fetch/update process:
generate:  This command looks at the crawldb for all the urls/files that are due for fetching and regroups them in a segment. A url/file is due for fetch if if it is new or the time has expired for that url/file and is now due for recrawling (default is 30 days).

fetch:  This command will go an fetch all the urls/files specified in the segment.

update:  This command will add the results of the crawling, which have been stored in the segment, into the crawldb and each url/file will be updated to indicate the time it wad fetched and when its next scheduled fetch is.  If any urls/files have been discovered, they will be added and marked as not fetched.

How does Nutch can detect if a page has changed or not? Each time a page is fetched, Nutch computes a signature for the page. At the next fetch, if the signature is the same (or if a 304 is returned by the web server because of the If-Modified-Since header), Nutch can tell if the page was modified or not (It is not just the content, if the http headers or metatags have changed it will be marked as modified).  If a document no longer exists it returns a 404 and will be marked DB_GONE. During the update cycle Nutch has the ability to purge all the those urls/files that have been marked DB_GONE.

The linkdb stores the finalised indexes that Nutch has generated from the crawl and this is the data that Nutch passes to the Solr Server during the solrindex process.

To tell Nutch that you would like all the urls/files that have been deleted you need to add the following code to your nutch-site.xml:

<property>
  <name>db.update.purge.404</name>
  <value>true</value>
  <description>If true, updatedb will add purge records with status DB_GONE
  from the CrawlDB.
  </description>
</property>

I hope this is of some help to you. 

Nutch Apostrophe/Single Quotes Issue (Solved) - Java Program

Nutch was designed to run on Unix and the filepath rules on Unix differ to that of Windows.  Considering we are running Nutch on Windows, through cygwin, it has a problem when it tries to get a filename with an apostrophe in it and this can mean many files on a system can go un-crawled.  To combat the issue, you need to run a crawler over the fileshare first and any file containing an apostrophe must be named in the nutch.txt file. For some reason, when it is given the exact filename it has no problem with apostrophes so as long as all filenames containing apostrophes are declared in the nutch.txt file then Nutch will fetch the files without a problem

I have written a short concurrent Java program that does all the work for you, so all that you are required to do is specify the fileshare (Args[1]) and the location of your seed.txt file, in my case nutch.txt (Args[0]).

This is the code for fileCrawler.java:

package FileName;
import java.io.File;
import java.io.PrintWriter;
import java.util.ArrayList;
 
public class fileCrawler {
 
  private WorkQueue workQ;
  static int i = 0;
 public static PrintWriter out;
 
 private class Worker implements Runnable {
 
  private WorkQueue queue;
 
  public Worker(WorkQueue q) {
   queue = q;
  }
 
//  since main thread has placed all directories into the workQ, we
//  know that all of them are legal directories; therefore, do not need
//  to try ... catch in the while loop below
 
  public void run() {
   String name;
   while ((name = queue.remove()) != null) {
    File file = new File(name);
    String entries[] = file.list();
    if (entries == null)
     continue;
    for (String entry : entries) {
     if (entry.compareTo(".") == 0)
      continue;
     if (entry.compareTo("..") == 0)
      continue;
     String fn = name + "/" + entry;
     if (fn.contains("'")){
      if (fn.startsWith("//")){
       out.println("file://" + fn.replaceAll(" ", "%20")); //Get rid of all spaces
       System.out.println("file://" + fn.replaceAll(" ", "%20"));
      }
      else{
       out.println("file:/" + fn.replaceAll(" ", "%20")); //Get rid of all spaces
       System.out.println("file:/" + fn.replaceAll(" ", "%20"));
      }
     }
    }
   }
  }
 }
 
 public fileCrawler() {
  workQ = new WorkQueue();
 }
 
 public Worker createWorker() {
  return new Worker(workQ);
 }
 
 
// need try ... catch below in case the directory is not legal
 
 public void processDirectory(String dir) {
  try {
 
   File file = new File(dir);
   if (file.isDirectory()) {
    String entries[] = file.list();
    if (entries != null)
     workQ.add(dir);
 
    for (String entry : entries) {
     String subdir;
     if (entry.compareTo(".") == 0)
      continue;
     if (entry.compareTo("..") == 0)
      continue;
     if (dir.endsWith("/"))
      subdir = dir+entry;
     else
      subdir = dir+"/"+entry;
     processDirectory(subdir);
    }
   }
  } catch (Exception e) {};
 }
 
 public static void main(String Args[]) {
 
  fileCrawler fc = new fileCrawler();
 
  //now start all of the worker threads
  System.out.println("Starting new File Crawler on " + Args[1]);
  int N = 5;
  ArrayList<Thread> thread = new ArrayList<Thread>(N);
  for (int i = 0; i < N; i++) {
   Thread t = new Thread(fc.createWorker());
   thread.add(t);
   t.start();
  }
 
  //File to be written to
  //@throws FileNotFoundException 
  try {
   out = new PrintWriter(Args[0]);
  }
  catch(Exception e){
   System.err.println("File Not Found: " + Args[0]);
  }
 
  //Directory to be crawled
  String a = Args[1];
  fc.processDirectory(a);
 
  //indicate that there are no more directories to add
 
  fc.workQ.finish();
 
  //Finally add the directory so that it can be crawled
  if (a.startsWith("//")){
   System.out.println("Adding: file://"+a);
   out.println("file://"+a);
  }
  else{
   System.out.println("Adding: file:/"+a);
   out.println("file:/"+a);
  }
 
  System.out.println("Closing File");
  out.close();
 
  //Kill the final threads
  for (int i = 0; i < N; i++){
   try {
    thread.get(i).join();
   } catch (Exception e) {};
  }
  System.out.println("Completed");
 }
}

This is the code for WorkQueue.java:

package FileName;
import java.util.*;
 
public class WorkQueue {
 
//
// since we are providing the concurrency control, can use non-thread-safe
// linked list
//
  private LinkedList<String> workQ;
 private boolean done;  // no more directories to be added
 private int size;  // number of directories in the queue
 
 public WorkQueue() {
  workQ = new LinkedList<String>();
  done = false;
  size = 0;
 }
 
 public synchronized void add(String s) {
  workQ.add(s);
  size++;
  notifyAll();
 }
 
 public synchronized String remove() {
  String s;
  while (!done && size == 0) {
   try {
    wait();
   } catch (Exception e) {};
  }
  if (size > 0) {
   s = workQ.remove();
   size--;
   notifyAll();
  } else
   s = null;
  return s;
 }
 
 public synchronized void finish() {
  done = true;
  notifyAll();
 }
}

Installation Guide To Set Up Apache Nutch On Windows

Nutch is coded entirely in the Java programming language and is a crawler with a wide variety of features. Some of these features are:

• highly scalable and feature rich crawler
• features like politeness which obeys robots.txt rules
• robust and scalable - Nutch can run on a cluster of up to 100 machines
• quality - crawling can be biassed to fetch "important" pages first

For our purposes, it will allow us to crawl a source and will automatically index it over to our Solr server.

Preparation

You should have set-up Solr on Tomcat along with Tika's extracting request handler as shown in the previous two guides:    http://amac4.blogspot.co.uk/2013/07/setting-up-solr-with-apache-tomcat-be.html       http://amac4.blogspot.co.uk/2013/07/setting-up-tika-extracting-request.html    

Download the binary Nutch from the Apache website. Some releases of Nutch were designed specifically to work with certain versions of Solr so be aware that the version of Nutch you try to integrate with Solr is important. For this example I am using Solr4.3 and Nutch1.4.

Downloads

Most of these will be set-up prior to this so you do not need to download them again

• Download Java jre7 and jdk7
• Download Tomcat 7
• Download Solr 4.3
• Download Cygwin - Run Set-up.exe and install all packages (Default may be sufficient)
• Download Nutch 1.4 bin

Set-up

• Cygwin should be installed to C:/cygwin or similar. Copy your Nutch download to the cygwin/home folder. This Nutch installation will be referred to as $NUTCH_HOME.

• Set-up a system environment varibale called JAVA_HOME and set the location as the location of your JDK (e.g C:/Java/jdk1.7).

• If cygwin does not recognise that you have set up an environment variable then you can issue the following instruction. Note that you will be required to type this instruction every time you wish to issue any commands that use the jdk .

export JAVA_HOME=[JDK Location]

• In cygwin change directory into the $NUTCH_HOME/runtime/local/bin directory. If your environment variable has been set up correctly then when you run the command ./nutch the output should be Usage: nutch [-core] COMMAND

cd $NUTCH_HOME/runtime/local/bin
./nutch
 
#Output Should Be
Usage: nutch [-core] COMMAND

• NOTE: cygpath cant convert empty path is not an error and will be displayed each time you run any Nutch command.

• In the $NUTCH_HOME/runtime/local/bin folder create a new folder and give it any name, say urls. This folder will contain a text file that will be used to determine what sites will get crawled.

• In the new folder create a text document say (nutch.txt) and add a list of the urls you wish to crawl (e.g http://amac4.blogspot.co.uk)

• In $NUTCH_HOME/runtime/local/conf open the regex-urlfilter.txt and where it says # accept anything else add the line +^http://amac4.blogspot.co.uk/ if you wish to search anything that comes under the amac4.blogspot.co.uk domain.

• Nutch can only extract data from certain types of file and cannot extract data from images or various other binary file types. You as a user may not want data extraction to occur when using certain file types so you have the option to ignore these files by adding their tags to the list in the same fashion as it shows (|xml|XML|jpeg|asp|.. etc)

• Now open nutch-site.xml and add the following code between the <configuration> headers

<name>http.agent.name</name>
<value>My Nutch Spider</value> #(You can add any name here)

• In nutch-default.xml there should be a line under the <property> tag which has <name>http.agent.name</name>. The <value>field below should be empty so you should add the name of your crawler that you specified before, so in this case it would be:

<value>My Nutch Spider</value>

• You can test the crawl is working by navigating to the $NUTCH_HOME/runtime/local/bin folder and executing:

cd $NUTCH_HOME/runtime/local/bin
./nutch crawl urls -dir [dir name] -depth [depth] -topN [files]

• The directory you will supply (dir name) will store the indexes from the crawl. Running the crawl again will also cause the files to be re-indexed if they are found. The depth is asking how deep down a hierarchy do you wish to go and the topN is asking how many pages on each level you wish to index.

• To get it linking to Solr copy the schema.xml from $NUTCH_HOME/runtime/local/conf into the $SOLR_HOME/collection1/conf/folder which should overwrite the previous schema.xml file but only if you are using Solr 1/2/3. If it is Solr4 you are using, which we are in this case then copy the schema-solr4.xml from the $NUTCH_HOME/runtime/local/conf directory into the$SOLR_HOME/collection1/conf/ and then rename it in the Solr directory back to schema.xml which should overwrite the old one. Changes made to the schema.xml made in the Solr set-up may need to be re-done.

• Add this line to the schema.xml in the Solr installation and also to the schema-solr4.xml in the Nutch installation.

<field name="_version_" type="long" stored="true" indexed="true" multiValued="false"/>  

• If you are NOT using the Solr4 schema file then edit the schema file you copied over and comment out the line like this:

<!--<filter class="solr.EnglishPorterFilterFactory" protected="protwords.txt"/>-->

• To test the crawl is indexing to Solr type into cygwin:

./nutch crawl urls -dir newCrawl -solr http://localhost:8080/solr/ -depth 3 -topN 4

• Note Nutch is now configured to crawl over the web but there are issues that have turned up so changes need to be made and these changes can be found at:

Optimising Nutch Performance

You may notice if you try and run Nutch that it works its way through the crawl very slowly, that is because by default Nutch is set-up to use using one thread and doesn't take advantage of the Multi-threaded implementation. Nutch will use multi-threading to crawl various hosts simultaneously and therefore the settings must be changed in order for search times to be kept to a reasonable level.

In the nutch-site.xml add the code between the <configuration> tags

  <property>
  <name>fetcher.threads.per.queue</name>
     <value>10</value>
     <description></description>
  </property>
 
  <property>
  <name>fetcher.threads.per.host</name>
     <value>10</value>
     <description></description>
  </property> 

Parsing Errors

You may find errors popping up every so often that look similar to

Error parsing: 192.168.0.42/test/AGENDA.doc: failed(2,0): Unable to read 512 bytes from 65536 in stream of length 65421

Fix By default there is a limit to how much data that Solr will parse so to take that limit away you need to set the content limit to -1. This has repercussions in terms of performance as large files will take longer to parse but no file should fail because of its length.

Add this code to the nutch-site.xml

  <property> 
  <name>http.content.limit</name> 
  <value>-1</value> 
  <description>The length limit for downloaded content, in bytes. 
               If this value is nonnegative (>=0), content longer than it 
  will be 
               truncated;otherwise, no truncation at all. 
  </description> 
  </property>

Failing to parse documents with a space in the title

URLs are not allowed to contain white-space and Nutch was not replacing the space character with %20 which meant that filename became invalid.

Fix

Add the following text to regex-normalise.xml:

  <regex> 
     <pattern>&#x20;</pattern> 
     <substitution>%20</substitution> 
  </regex> 

Next Steps

You may now want to set-up Nutch to crawl a local filesystem, the guide can be found at:     
http://amac4.blogspot.co.uk/2013/07/setting-up-nutch-to-crawl-filesystem.html    
http://amac4.blogspot.co.uk/2013/07/web-service-to-query-solr-rest.html    

Otherwise you may wish to check out some tweaks that can be made to Solr including deduplication and highlighting:
http://amac4.blogspot.co.uk/2013/08/setting-up-highlighting-for-solr-4.html 
http://amac4.blogspot.co.uk/2013/08/deleting-dead-urls-files-that-no-longer.html    

http://amac4.blogspot.co.uk/2013/08/solr-deduplication.html