Configuring Solr 4 Data Import Handler with JDBC

Preparation

You should have set-up Solr on Tomcat along with Tika's extracting request handler as shown in the previous two guides:
   http://amac4.blogspot.co.uk/2013/07/setting-up-solr-with-apache-tomcat-be.html       http://amac4.blogspot.co.uk/2013/07/setting-up-tika-extracting-request.html    

Set-Up

• Firstly we need the libraries that are required to use Data Import Handler. Create a folder and name it dih (preferably in your $SOLR_HOME), and place solr-dataimporthandler-4.0.0.jar and solr-dataimporthandler-extras-4.0.0.jar from $SOLR/dist directory in the dih folder. 
  
  Add this to the solrconfig.xml file:

<lib dir="../../dih" regex=".*\.jar" />

• Now we modify the solrconfig.xml file. Add the following :

<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">
<lst name="defaults">
<str name="config">db-data-config.xml</str>
</lst>
</requestHandler>

• Create a db-data-config.xml. This is for the Data Import Handler configuration. It should look similar to:

<dataConfig>
<dataSource driver="org.postgresql.Driver"
url="jdbc:postgresql://localhost:1234/users" user="users"
password="secret" />
<document>
<entity name="user" query="SELECT id, name from users">
<field column="id" name="id" />
<field column="name" name="name" />
<entity name="birthday" query="select birthday from table where id=${user.id}">
<field column="description" name="description" />
</entity>
</entity>
</document>
</dataConfig></dataConfig>

• For other database engines enter the specific driver, url, user and password attributes.

• We now need to modify the fields section of the schema.xml file to something like the following snippet:

<field name="id" type="string" indexed="true" stored="true"
required="true"/>
<field name="name" type="text" indexed="true" stored="true" />
<field name="birthday" type="text" indexed="true" stored="true"/>
<field name="description" type="" indexed="true"
stored="true"/>

Solr may not like type="text", if so then change it to text_general

One more thing before the indexing – you should copy an appropriate JDBC driver to the $SOLR_HOME/lib directory of your Solr installation or the dih directory we created before. You can get the libraries from the databases websites

To index, run the following query: http://localhost:8080/solr/dataimport?command=full-import .The HTTP protocol is asynchronous so you won't be updated on the status of the indexing process. To check the status of the indexing process, you can run the command once again.

Installation Guide To Set Up Nutch To Crawl A Filesystem Or Intranet (Windows)

Preparation

At this point you should have Solr and Tika set-up as well as having Nutch ready to crawl the web. The guides can be found here:    http://amac4.blogspot.co.uk/2013/07/setting-up-solr-with-apache-tomcat-be.html         http://amac4.blogspot.co.uk/2013/07/setting-up-tika-extracting-request.html       http://amac4.blogspot.co.uk/2013/07/configuring-nutch-to-crawl-urls.html     

The required changes that need to be done to make Nutch crawl a file-system require that Nutch is set-up to crawl the web so please ensure you have read the post.

URLs/nutch.txt Format

This is where you specify where you would like to crawl, so add your locations with a new line after each one. If you wish to crawl a fileshare then it must be in a similar format to :

file:////189.189.1.42/test

Or if you wish to crawl the local filesystem it must be in this format:

file:/C:/Users/alamil/test/

Required Changes

Change 1 (regex-urlfilter.txt)

The default setting for Nutch is to skip anything that begins with "file" so our first change is to get Nutch to parse files. Nutch is currently set-up to parse anything that it finds via http so we need to change this line:

# skip file: ftp: and mailto: urls
-^(file|ftp|mailto):

to say

# skip http: ftp: and mailto: urls
-^(http|ftp|mailto):

Change 2 (regex-urlfilter.txt)

This change is not necessary but may make your life easier.  Any file types you do not want to index need to be added to the list otherwise Nutch will often try to parse them and fail in doing so as it doesnt know how to deal with a lot of binary file types.:

# skip image and other suffixes we can't yet parse
# for a more extensive coverage use the urlfilter-suffix plugin

-\.(gif|GIF|jpg|JPG|png|PNG|ico|ICO|css|CSS|sit|SIT|eps|EPS
|wmf|WMF|zip|ZIP|ppt|PPT|mpg|MPG|gz|GZ|rpm|RPM|tgz|TGZ|mov
|MOV|exe|EXE|jpeg|JPEG|bmp|BMP|js|JS|asp|ASP|xxx|XXX|yyy|YYY
|cs|CS|dll|DLL|refresh|REFRESH)$

Change 3 (regex-urlfilter.txt)

Our final change to Nutch is to filter out unwanted files from being parsed.  Often you will only want the subdirectories and files of the initial directory you supplied to be indexed. For example, if I supply Nutch with the file:////189.189.1.42/test In the nutch.txt file then I only want the files within any of its subdirectories to be indexed so I would filter by:

# accept anything else
+^file:////189.189.1.42/test

Change 4 (nutch-site.xml)

To allow Nutch to recognise that it is dealing with files it needs a plugin to be activated. So adding the protocol-file plugin to the list of plugins that are used allows Nutch to know how to deal with files:

<property>
<name>plugin.includes</name>
<value>protocol-file|protocol-httpclient|urlfilter-regex|parse-(html|tika)|index-(basic|anchor)|
  scoring-opic|urlnormalizer-(pass|regex|basic)|index-more</value>
  <description>Regular expression naming plugin directory names to 
    include.  Any plugin not matching this expression is excluded. 
    In any case you need at least include the nutch-extensionpoints plugin.
  </description>
</property>

Change 5 (nutch-site.xml)

We encountered the problem of file size earlier when doing the web crawler, if you set a limit on the content size it can cause the read to fail and the same applies here, so not matter how big the file is we want it to be parsed.

<property>
 <name>file.content.limit</name>
 <value>-1</value>
 <description> Needed to stop buffer overflow errors - Unable to read.....</description>
</property>

Change 6 (regex-normalise.xml)

We do not want duplicate slashes to be replaced by single slashes other the filenames we specify will not be valid. Therefore we need to comment out the code that does the replacement:

<!-- removes duplicate slashes 
<regex>
  <pattern>(?&lt;!:)/{2,}</pattern>
  <substitution>/</substitution>
</regex>
-->

Running the Crawl

Ensure you add the location you wish to crawl to the nutch.txt file 

To run the crawl, like before, you issue the command:

./nutch crawl urls -dir [dir_name] -solr http://localhost:8080/solr/ -depth [depth] -topN [topN]

Be sure to set a depth and a topN number of files to index otherwise nutch will use its default setting of depth 3 which will mean that you may have a lot of files that are untouched.

Next Steps

An issue I have noticed that occurs is with files that contain an apostrophe in the filepath.  Nutch was designed to run on Unix, which is why in this tutorial you are required to download cygwin, but this also has its complications.  Unix by default does not allow any filepaths to contain an apostrophe, but Windows filepaths do.  This means that when Nutch comes across an apostrophe, it thinks that is the end of the file.  I have written a Java program that will solve the problem and it can be found here:

http://amac4.blogspot.co.uk/2013/07/nutch-apostrophesingle-quotes-issue_23.html 

You should now have everything set-up to work now and the next step is to integrate your crawler and search server into your web application.  I have written a C# Web Service that makes calls to the Solr server and returns the xml response, have a look, copy and tweak it to your own liking to create your own search engine:

http://amac4.blogspot.co.uk/2013/07/web-service-to-query-solr-rest.html