Nutch crawling slowly - Multi-Threaded Solution

Expecting Nutch to be flying through all the websites you want it to crawl but in reality it is sequentially, one by one going through each page. Well you have not configured Nutch to use multiple threads! Multiple threads allows many threads to work in "parallel" and speed up the job significantly.

Preparation

At this point you should have Solr and Tika set-up as well as having Nutch ready to crawl . The guides can be found here:

http://amac4.blogspot.co.uk/2013/07/setting-up-solr-with-apache-tomcat-be.html
http://amac4.blogspot.co.uk/2013/07/setting-up-tika-extracting-request.html
http://amac4.blogspot.co.uk/2013/07/configuring-nutch-to-crawl-urls.htmlhttp://amac4.blogspot.co.uk/2013/07/setting-up-nutch-to-crawl-filesystem.html

The required changes that need to be done to make Nutch crawl a file-system require that Nutch is set-up to crawl the web so please ensure you have read the post.

To configure your multi-threaded Nutch crawler is easy:

In the nutch-site.xml add the code between the <configuration> tags add:

<property>
<name>fetcher.threads.per.queue</name>
   <value>10</value>
   <description></description>
</property>

<property>
<name>fetcher.threads.per.host</name>
   <value>10</value>
   <description></description>
</property> 

10 threads was the most efficient to use on the crawling data set I chose and would advise you to start with the number of threads around here, having too many threads will slow the computer down and you may be left working at a similar speed to that of the sequential crawler. 

Installation Guide To Set Up Nutch To Crawl A Filesystem Or Intranet (Windows)

Preparation

At this point you should have Solr and Tika set-up as well as having Nutch ready to crawl the web. The guides can be found here:    http://amac4.blogspot.co.uk/2013/07/setting-up-solr-with-apache-tomcat-be.html         http://amac4.blogspot.co.uk/2013/07/setting-up-tika-extracting-request.html       http://amac4.blogspot.co.uk/2013/07/configuring-nutch-to-crawl-urls.html     

The required changes that need to be done to make Nutch crawl a file-system require that Nutch is set-up to crawl the web so please ensure you have read the post.

URLs/nutch.txt Format

This is where you specify where you would like to crawl, so add your locations with a new line after each one. If you wish to crawl a fileshare then it must be in a similar format to :

file:////189.189.1.42/test

Or if you wish to crawl the local filesystem it must be in this format:

file:/C:/Users/alamil/test/

Required Changes

Change 1 (regex-urlfilter.txt)

The default setting for Nutch is to skip anything that begins with "file" so our first change is to get Nutch to parse files. Nutch is currently set-up to parse anything that it finds via http so we need to change this line:

# skip file: ftp: and mailto: urls
-^(file|ftp|mailto):

to say

# skip http: ftp: and mailto: urls
-^(http|ftp|mailto):

Change 2 (regex-urlfilter.txt)

This change is not necessary but may make your life easier.  Any file types you do not want to index need to be added to the list otherwise Nutch will often try to parse them and fail in doing so as it doesnt know how to deal with a lot of binary file types.:

# skip image and other suffixes we can't yet parse
# for a more extensive coverage use the urlfilter-suffix plugin

-\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|CSS|sit|SIT|eps|EPS
|wmf|WMF|zip|ZIP|ppt|PPT|mpg|MPG|gz|GZ|rpm|RPM|tgz|TGZ|mov
|MOV|exe|EXE|jpeg|JPEG|bmp|BMP|js|JS|asp|ASP|xxx|XXX|yyy|YYY
|cs|CS|dll|DLL|refresh|REFRESH)$

Change 3 (regex-urlfilter.txt)

Our final change to Nutch is to filter out unwanted files from being parsed.  Often you will only want the subdirectories and files of the initial directory you supplied to be indexed. For example, if I supply Nutch with the file:////189.189.1.42/test In the nutch.txt file then I only want the files within any of its subdirectories to be indexed so I would filter by:

# accept anything else
+^file:////189.189.1.42/test

Change 4 (nutch-site.xml)

To allow Nutch to recognise that it is dealing with files it needs a plugin to be activated. So adding the protocol-file plugin to the list of plugins that are used allows Nutch to know how to deal with files:

<property>
<name>plugin.includes</name>
<value>protocol-file|protocol-httpclient|urlfilter-regex|parse-(html|tika)|index-(basic|anchor)|
  scoring-opic|urlnormalizer-(pass|regex|basic)|index-more</value>
  <description>Regular expression naming plugin directory names to 
    include.  Any plugin not matching this expression is excluded. 
    In any case you need at least include the nutch-extensionpoints plugin.
  </description>
</property>

Change 5 (nutch-site.xml)

We encountered the problem of file size earlier when doing the web crawler, if you set a limit on the content size it can cause the read to fail and the same applies here, so not matter how big the file is we want it to be parsed.

<property>
 <name>file.content.limit</name>
 <value>-1</value>
 <description> Needed to stop buffer overflow errors - Unable to read.....</description>
</property>

Change 6 (regex-normalise.xml)

We do not want duplicate slashes to be replaced by single slashes other the filenames we specify will not be valid. Therefore we need to comment out the code that does the replacement:

<!-- removes duplicate slashes 
<regex>
  <pattern>(?&lt;!:)/{2,}</pattern>
  <substitution>/</substitution>
</regex>
-->

Running the Crawl

Ensure you add the location you wish to crawl to the nutch.txt file 

To run the crawl, like before, you issue the command:

./nutch crawl urls -dir [dir_name] -solr http://localhost:8080/solr/ -depth [depth] -topN [topN]

Be sure to set a depth and a topN number of files to index otherwise nutch will use its default setting of depth 3 which will mean that you may have a lot of files that are untouched.

Next Steps

An issue I have noticed that occurs is with files that contain an apostrophe in the filepath.  Nutch was designed to run on Unix, which is why in this tutorial you are required to download cygwin, but this also has its complications.  Unix by default does not allow any filepaths to contain an apostrophe, but Windows filepaths do.  This means that when Nutch comes across an apostrophe, it thinks that is the end of the file.  I have written a Java program that will solve the problem and it can be found here:

http://amac4.blogspot.co.uk/2013/07/nutch-apostrophesingle-quotes-issue_23.html 

You should now have everything set-up to work now and the next step is to integrate your crawler and search server into your web application.  I have written a C# Web Service that makes calls to the Solr server and returns the xml response, have a look, copy and tweak it to your own liking to create your own search engine:

http://amac4.blogspot.co.uk/2013/07/web-service-to-query-solr-rest.html

Installation Guide To Set Up Apache Nutch On Windows

Nutch is coded entirely in the Java programming language and is a crawler with a wide variety of features. Some of these features are:

• highly scalable and feature rich crawler
• features like politeness which obeys robots.txt rules
• robust and scalable - Nutch can run on a cluster of up to 100 machines
• quality - crawling can be biassed to fetch "important" pages first

For our purposes, it will allow us to crawl a source and will automatically index it over to our Solr server.

Preparation

You should have set-up Solr on Tomcat along with Tika's extracting request handler as shown in the previous two guides:    http://amac4.blogspot.co.uk/2013/07/setting-up-solr-with-apache-tomcat-be.html       http://amac4.blogspot.co.uk/2013/07/setting-up-tika-extracting-request.html    

Download the binary Nutch from the Apache website. Some releases of Nutch were designed specifically to work with certain versions of Solr so be aware that the version of Nutch you try to integrate with Solr is important. For this example I am using Solr4.3 and Nutch1.4.

Downloads

Most of these will be set-up prior to this so you do not need to download them again

• Download Java jre7 and jdk7
• Download Tomcat 7
• Download Solr 4.3
• Download Cygwin - Run Set-up.exe and install all packages (Default may be sufficient)
• Download Nutch 1.4 bin

Set-up

• Cygwin should be installed to C:/cygwin or similar. Copy your Nutch download to the cygwin/home folder. This Nutch installation will be referred to as $NUTCH_HOME.

• Set-up a system environment varibale called JAVA_HOME and set the location as the location of your JDK (e.g C:/Java/jdk1.7).

• If cygwin does not recognise that you have set up an environment variable then you can issue the following instruction. Note that you will be required to type this instruction every time you wish to issue any commands that use the jdk .

export JAVA_HOME=[JDK Location]

• In cygwin change directory into the $NUTCH_HOME/runtime/local/bin directory. If your environment variable has been set up correctly then when you run the command ./nutch the output should be Usage: nutch [-core] COMMAND

cd $NUTCH_HOME/runtime/local/bin
./nutch
 
#Output Should Be
Usage: nutch [-core] COMMAND

• NOTE: cygpath cant convert empty path is not an error and will be displayed each time you run any Nutch command.

• In the $NUTCH_HOME/runtime/local/bin folder create a new folder and give it any name, say urls. This folder will contain a text file that will be used to determine what sites will get crawled.

• In the new folder create a text document say (nutch.txt) and add a list of the urls you wish to crawl (e.g http://amac4.blogspot.co.uk)

• In $NUTCH_HOME/runtime/local/conf open the regex-urlfilter.txt and where it says # accept anything else add the line +^http://amac4.blogspot.co.uk/ if you wish to search anything that comes under the amac4.blogspot.co.uk domain.

• Nutch can only extract data from certain types of file and cannot extract data from images or various other binary file types. You as a user may not want data extraction to occur when using certain file types so you have the option to ignore these files by adding their tags to the list in the same fashion as it shows (|xml|XML|jpeg|asp|.. etc)

• Now open nutch-site.xml and add the following code between the <configuration> headers

<name>http.agent.name</name>
<value>My Nutch Spider</value> #(You can add any name here)

• In nutch-default.xml there should be a line under the <property> tag which has <name>http.agent.name</name>. The <value>field below should be empty so you should add the name of your crawler that you specified before, so in this case it would be:

<value>My Nutch Spider</value>

• You can test the crawl is working by navigating to the $NUTCH_HOME/runtime/local/bin folder and executing:

cd $NUTCH_HOME/runtime/local/bin
./nutch crawl urls -dir [dir name] -depth [depth] -topN [files]

• The directory you will supply (dir name) will store the indexes from the crawl. Running the crawl again will also cause the files to be re-indexed if they are found. The depth is asking how deep down a hierarchy do you wish to go and the topN is asking how many pages on each level you wish to index.

• To get it linking to Solr copy the schema.xml from $NUTCH_HOME/runtime/local/conf into the $SOLR_HOME/collection1/conf/folder which should overwrite the previous schema.xml file but only if you are using Solr 1/2/3. If it is Solr4 you are using, which we are in this case then copy the schema-solr4.xml from the $NUTCH_HOME/runtime/local/conf directory into the$SOLR_HOME/collection1/conf/ and then rename it in the Solr directory back to schema.xml which should overwrite the old one. Changes made to the schema.xml made in the Solr set-up may need to be re-done.

• Add this line to the schema.xml in the Solr installation and also to the schema-solr4.xml in the Nutch installation.

<field name="_version_" type="long" stored="true" indexed="true" multiValued="false"/>  

• If you are NOT using the Solr4 schema file then edit the schema file you copied over and comment out the line like this:

<!--<filter class="solr.EnglishPorterFilterFactory" protected="protwords.txt"/>-->

• To test the crawl is indexing to Solr type into cygwin:

./nutch crawl urls -dir newCrawl -solr http://localhost:8080/solr/ -depth 3 -topN 4

• Note Nutch is now configured to crawl over the web but there are issues that have turned up so changes need to be made and these changes can be found at:

Optimising Nutch Performance

You may notice if you try and run Nutch that it works its way through the crawl very slowly, that is because by default Nutch is set-up to use using one thread and doesn't take advantage of the Multi-threaded implementation. Nutch will use multi-threading to crawl various hosts simultaneously and therefore the settings must be changed in order for search times to be kept to a reasonable level.

In the nutch-site.xml add the code between the <configuration> tags

  <property>
  <name>fetcher.threads.per.queue</name>
     <value>10</value>
     <description></description>
  </property>
 
  <property>
  <name>fetcher.threads.per.host</name>
     <value>10</value>
     <description></description>
  </property> 

Parsing Errors

You may find errors popping up every so often that look similar to

Error parsing: 192.168.0.42/test/AGENDA.doc: failed(2,0): Unable to read 512 bytes from 65536 in stream of length 65421

Fix By default there is a limit to how much data that Solr will parse so to take that limit away you need to set the content limit to -1. This has repercussions in terms of performance as large files will take longer to parse but no file should fail because of its length.

Add this code to the nutch-site.xml

  <property> 
  <name>http.content.limit</name> 
  <value>-1</value> 
  <description>The length limit for downloaded content, in bytes. 
               If this value is nonnegative (>=0), content longer than it 
  will be 
               truncated;otherwise, no truncation at all. 
  </description> 
  </property>

Failing to parse documents with a space in the title

URLs are not allowed to contain white-space and Nutch was not replacing the space character with %20 which meant that filename became invalid.

Fix

Add the following text to regex-normalise.xml:

  <regex> 
     <pattern>&#x20;</pattern> 
     <substitution>%20</substitution> 
  </regex> 

Next Steps

You may now want to set-up Nutch to crawl a local filesystem, the guide can be found at:     
http://amac4.blogspot.co.uk/2013/07/setting-up-nutch-to-crawl-filesystem.html    
http://amac4.blogspot.co.uk/2013/07/web-service-to-query-solr-rest.html    

Otherwise you may wish to check out some tweaks that can be made to Solr including deduplication and highlighting:
http://amac4.blogspot.co.uk/2013/08/setting-up-highlighting-for-solr-4.html 
http://amac4.blogspot.co.uk/2013/08/deleting-dead-urls-files-that-no-longer.html    

http://amac4.blogspot.co.uk/2013/08/solr-deduplication.html    

Installation Guide To Set Up Solr 4.3 With Apache Tomcat (Windows & Cygwin)

Installation Guide to Set Up Solr 4 With Apache Tomcat On Windows

This is a short guide on how to set up Apache Solr on Windows 7 running on Apache Tomcat. Be aware that Solr-4.3.0 and Apache-tomcat-7.0.40. Older versions of Solr use different folder structure and therefore this guide will not adequately cover its requirements.

If you wish to have a fully functioning file or web crawler using Nutch that Indexes to Solr then follow the next steps of the guide at:
http://amac4.blogspot.co.uk/2013/07/setting-up-tika-extracting-request.html   
http://amac4.blogspot.co.uk/2013/07/configuring-nutch-to-crawl-urls.html   
http://amac4.blogspot.co.uk/2013/07/setting-up-nutch-to-crawl-filesystem.html   
http://amac4.blogspot.co.uk/2013/07/web-service-to-query-solr-rest.html  

Preparation

First of all we need an Apache Tomcat servlet container. It can be found at the Apache Tomcat website:Apache Website Any version beyond 7.0.0 is considered stable with sufficient support and documentation available. Please Download Tomcat & Solr in preparation to starting this guide. You will also require the Java Runtime Environment to be downloaded and set up. Its location will be referred to as $JAVA.

• Solr 4.3.0
• Tomcat 7 or Tomcat 8
• Cygwin
• JRE 7

Set-up

• Firstly, choose the location you wish to install Tomcat and copy the files from your downloads to your chosen directory. This will be referred to as $TOMCAT_HOME. Do the same for your Solr download, copy the files to your chosen location and this will be referred to as which will be referred to as $SOLR.

• Edit the server.xml file in $TOMCAT_HOME/conf to look like the following code snippet:

<Connector port="8080" protocol="HTTP/1.1"
connectionTimeout="20000"
redirectPort="8443"
URIEncoding="UTF-8" />

• Now copy the [$SOLR]/example/solr directory somewhere else and rename it solr_home. This will be your main Solr installation to run your Solr server and will be referred to as $SOLR_HOME.

• The next thing is setting up the Solr deployment. To do that we need the apache-solr-4.0.0.war (found in $SOLR/example/webapps) file that contains the necessary files and libraries to run Solr to be copied to the $TOMCAT_HOME/webapps directory and renamed solr.war.

• Create a proper context file by creating a solr.xml file in the $TOMCAT_HOME/conf/Catalina/localhost directory. If the Catalina/localhost/ directory does not exist then you will need to make it yourself. The contents of the file should look like the following code:

<Context docBase="/solr.war"debug="0" crossContext="true">
<Environment name="solr/home" type="java.lang.String" value="$SOLR_HOME" override="true"/>
</Context>

• Modify solr_home/collection1/conf/solrconfig.xml to set the absolute directory:

<lib dir="$SOLR/contrib/extraction/lib" regex=".*\.jar" />

• Also set the data directory like:

<dataDir>${solr.data.dir:$SOLR_HOME/data}</dataDir>

• Now we can start the servlet container. Navigate to the $TOMCAT_HOME/bin and double click: startup.bat. Using command prompt:

cd $TOMCAT_HOME/bin
startup.bat run

• Copy the jar files from $SOLR\example\lib\ext into the $TOMCAT_HOME/lib folder and also copy the $SOLR/example/solr/example/resources/log4j.properties into the same location.
• Runtimes may need to be set in the startup.bat file. Add set JAVA_HOME=$JAVA or equivalent location to the second line of the file catalina.bat if it fails to run.

• Your Solr server is now running and to ensure that Solr is running properly, you can open a browser and navigate to:http://localhost:8080/solr/. The Solr administration screen should appear and there are a variety of options to choose from. 

Next Steps

The next step is to ensure that Tika's extracting request handler is configured correctly so that it can deal with pdf, doc, xls etc files:
http://amac4.blogspot.co.uk/2013/07/setting-up-tika-extracting-request.html

You will also want to set-up de-duplication and potentially Solr's Highlighting functionality:http://amac4.blogspot.co.uk/2013/08/setting-up-highlighting-for-solr-4.htmlhttp://amac4.blogspot.co.uk/2013/08/solr-deduplication.html

Tomcat Options

When running Tomcat you have various options which are availible to you here:

• stop: This stops Apache Tomcat
• restart: This restarts Apache Tomcat
• debug: This start Apache Tomcat in debug mode
• run: This runs Apache Tomcat in the current window, so you can see the output on the console from which you run Tomcat.