Pages

Showing posts with label web. Show all posts
Showing posts with label web. Show all posts

Tuesday, 23 July 2013

RESTful Java Web Service For Solr


You have your Solr server set-up, now what?  You want people to be able to perform queries and have some control over what can be input, well you need a Java Web Service!  This is a short guide on how to create your Java Web Service which may not be tailored to your particular needs but you can tweak it as you please.

Set-Up

To set up the Java Web Service you will need:
  • Netbeans 7 with GlassFish
  • Solr Set-up and Running


Steps Involved

Project Set-Up

  • Start Netbeans and Go to FileNew ProjectJava WebWeb Applications and then hit Next.
  • Give your project a name and then hit Next
  • Ensure that Glassfish is the selected server and hit Finish

HelloResource.java

  • Right click the Default Package and create a new Java Class and call it HelloResource.java
  • Enter the following code:

    import java.net.URL;
    import javax.ws.rs.GET;
    import javax.ws.rs.Path;
    import javax.ws.rs.Produces;
    import javax.ws.rs.QueryParam;
    import java.io.BufferedReader;
    import java.io.InputStreamReader;
    import java.net.URLConnection;
     
    /**
     *
     * @author alamil
     */
     
    //Looks for hello in the pathname
    @Path("hello")
    public class HelloResource {
     
        /**
         *
         * @param arg
         * @param rows
         * @return
         */
        @GET
        @Path("/query")     //Looks for /query in the pathname
        @Produces("text/xml")       //Returns xml
        public String hello(@QueryParam("q") String arg, @QueryParam("rows") String rows, @QueryParam("filter") String filter){
     
            //Variables
            String xmldoc = "";
            String inputLine;
     
            //If the user has not selected a number of rows to display then 50 is set to defualt
            if (rows == null) 
                rows = "50";
     
            //If the user has not selected fields to filter on then it uses the default
            if (filter == null)
                filter = "id,title";
     
            try{
     
                //Trys to connect to the Solr Server with the query
                URL solr = new URL("http://localhost:8080/solr/select?q=url:(" + arg.replaceAll(" ","%20") 
                        + ")^25%20text:(" + arg.replaceAll(" ","%20") + ")&fl=" + filter + "&rows=" + rows);
     
                URLConnection yc = solr.openConnection();
     
                //Reads the returned xml file from the server
                BufferedReader in = new BufferedReader(new InputStreamReader(yc.getInputStream()));
     
                while ((inputLine = in.readLine()) != null){ 
                        xmldoc = xmldoc + inputLine;
                }
     
                //Close the file
                in.close();
     
                //Return the xml document (Replacing the %20's with spaces)
                return xmldoc.replaceAll("%20", " ");
            }catch (Exception e){
                    return "Exception";
            }
        }
    }

RESTConfig.java

  • Right click the Default Package and create a new Java Class and call it RESTConfig.java
  • Enter the following code:

    import javax.ws.rs.core.Application;
    import javax.ws.rs.ApplicationPath;
    /**
     *
     * @author alamil
     */
    @ApplicationPath("SearchInt")
    public class RESTConfig extends Application {
     
    }

Testing

  • Go to RunRun Project and your server should start
  • A browser window will open to the index.html page that is there which you can tweak and edit to your own liking
  • Start your solr server
  • In your browser, Navigate to:
    localhost:8080/HelloRest/SearchInt/hello/query?q=[query term]&filter=[filter term]&rows=[rows]
  • The input parameters set up are:
    1. q=… : representing the keywords in the query
    2. rows=… : representing the number of rows you wish to have returned
    3. filter=… :representing which fields are displayed to the user
Your server and Solr server may be running from the same port which can cause problems

Monday, 22 July 2013

C# Web Service To Query Solr (REST)

Preparation

You should have a Solr Server set-up on port 8080, configured to use Tika's extracting reuqest handler as well as having the highlighting functionality: 
http://amac4.blogspot.co.uk/2013/08/setting-up-highlighting-for-solr-4.html
http://amac4.blogspot.co.uk/2013/07/setting-up-tika-extracting-request.html
http://amac4.blogspot.co.uk/2013/07/setting-up-solr-with-apache-tomcat-be.html
Now that you have your Solr server up and running, now what? How do you query? How to you make use of the server? Well, you need to create a web service that will query Solr when the method is called from an ASP page. If you want your search service to be used on the web then you should most certainly consider creating a web service to provide you with a clean solution to the problem.
I am a novice when it comes to C# and this was my first involvement with the language, but due to the similarities it holds with Java, it wasn't difficult to understand the basics and get the web service up and running. It will return an xml file based on the input parameters and has the capability of highlighting too

You will require:

  • Microsoft Visual Studio with .NET framework of 3.5 or higher

Running The Web Service

  • Open Microsoft Visual Studio
  • Go to File→New Project and ensure that .NET Framework 3.5 is selected from the pull-down menu. Then select ASP.NET Web Service Application using Visual C#. Give your Web Service a relevant name and hit OK
  • Input the source code:

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Web;
    using System.Web.Services;
    using System.Net;
    using System.Xml;
    using System.Web.UI;
     
    namespace Query
    {
        /// <summary>
        /// Queries the Solr Server based on input parameters and returns the XML response
        /// </summary>
        [WebService(Namespace = "http://tempuri.org/")]
        [WebServiceBinding(ConformsTo = WsiProfiles.BasicProfile1_1)]
        [System.ComponentModel.ToolboxItem(false)]
        // To allow this Web Service to be called from script, using ASP.NET AJAX, uncomment the following line. 
        // [System.Web.Script.Services.ScriptService]
        public class Service1 : System.Web.Services.WebService
        {
     
            [WebMethod(Description = "Query Solr server and return XML response")]
            public XmlDocument QuerySolr(string keyword, string notKeyword, string exact, string rows, string filter,
                string fuzzy, string highlight)
            {
     
     
                /*  Default values should the user not enter anything */
                if (rows.Equals(""))
                    rows = "50";
     
                if (keyword.Equals("") && notKeyword.Equals("") && exact.Equals(""))
                    keyword = "*";
     
                if (filter.Equals(""))
                    filter = "id,title";
     
                if (!exact.Equals(""))
                    exact = " \"" + exact + "\"";
     
                if(!highlight.Equals(""))
                    highlight = "&hl=true&hl.fl=content";
     
                /* Should the user require that a certain word or words are not present */
                if (!notKeyword.Equals(""))
                {
     
                    /* Split each word */
                    string[] nwords = notKeyword.Split(null);
                    notKeyword = "";
     
                    /* Loops and adds the hyphen to each word */
                    for (int i = 0; i < nwords.Length; i++)
                    {
                        notKeyword = notKeyword + "-" + nwords[i] + " ";
                    }
                    notKeyword = notKeyword.Substring(0, notKeyword.Length - 1);
     
                }
     
                /* Should the user enable a fuzzy search a tilda must be added on to the end of each word */
                if (fuzzy.Equals("true") && !keyword.Equals(""))
                {
     
                    /* Splits the words by whitespace */
                    string[] words = keyword.Split(null);
                    keyword = "";
     
                    /* Loops and adds the tilda to each word */
                    for (int i = 0; i < words.Length; i++)
                    {
                        keyword = keyword + words[i] + "~ ";
                    }
                    keyword = keyword.Substring(0, keyword.Length - 1);
     
                }    
     
                /* Will throw an error should it have problems connecting to the Solr server */
                try
                {
     
                    /* Connecting to the Solr server and querying it with the search criteria that the user has provided*/
                    WebClient client = new WebClient();
                    string text = client.DownloadString(
                        "http://localhost:8080/solr/select?q=url:("+ keyword + " " + notKeyword + exact + ")^20%20text:("+ keyword
                        + " " + notKeyword + exact + ")&fl=" + filter + "&rows=" + rows + highlight);
     
                    /* The response is returned as a string so it must turn it into an XML document */
                    XmlDocument xml = new XmlDocument();
                    xml.LoadXml(text.Replace("%20", " ").Replace("\\", "/"));   //Changes to windows style filepath
     
                    return xml;
     
                }
                catch (Exception)
                {
                    /* Should an exception occur it returns an XML document with details of the problem */
                    XmlDocument xmle = new XmlDocument();
                    xmle.LoadXml("<Problem>Solr server cannot be reached</Problem>");
     
                    return xmle;
                }
            }
     
        }
    }
  • To test the Web Service hit the Green Triangle in the Top Menu
  • A Browser Window should open and display the Web Services that are currently being offered, click on the Query Solr Web Service
  • You should be able to enter the search criteria beneath and hit invoke to test your results

Installation Guide To Set Up Apache Nutch On Windows

Nutch is coded entirely in the Java programming language and is a crawler with a wide variety of features. Some of these features are:

  • highly scalable and feature rich crawler
  • features like politeness which obeys robots.txt rules
  • robust and scalable - Nutch can run on a cluster of up to 100 machines
  • quality - crawling can be biassed to fetch "important" pages first
For our purposes, it will allow us to crawl a source and will automatically index it over to our Solr server.

Preparation

You should have set-up Solr on Tomcat along with Tika's extracting request handler as shown in the previous two guides:    http://amac4.blogspot.co.uk/2013/07/setting-up-solr-with-apache-tomcat-be.html       http://amac4.blogspot.co.uk/2013/07/setting-up-tika-extracting-request.html    
Download the binary Nutch from the Apache website. Some releases of Nutch were designed specifically to work with certain versions of Solr so be aware that the version of Nutch you try to integrate with Solr is important. For this example I am using Solr4.3 and Nutch1.4.

Downloads

Most of these will be set-up prior to this so you do not need to download them again
  • Download Java jre7 and jdk7
  • Download Tomcat 7
  • Download Solr 4.3
  • Download Cygwin - Run Set-up.exe and install all packages (Default may be sufficient)
  • Download Nutch 1.4 bin

Set-up

  • Cygwin should be installed to C:/cygwin or similar. Copy your Nutch download to the cygwin/home folder. This Nutch installation will be referred to as $NUTCH_HOME.
  • Set-up a system environment varibale called JAVA_HOME and set the location as the location of your JDK (e.g C:/Java/jdk1.7).
  • If cygwin does not recognise that you have set up an environment variable then you can issue the following instruction. Note that you will be required to type this instruction every time you wish to issue any commands that use the jdk .
export JAVA_HOME=[JDK Location]
  • In cygwin change directory into the $NUTCH_HOME/runtime/local/bin directory. If your environment variable has been set up correctly then when you run the command ./nutch the output should be Usage: nutch [-core] COMMAND
cd $NUTCH_HOME/runtime/local/bin
./nutch
 
#Output Should Be
Usage: nutch [-core] COMMAND
  • NOTE: cygpath cant convert empty path is not an error and will be displayed each time you run any Nutch command.
  • In the $NUTCH_HOME/runtime/local/bin folder create a new folder and give it any name, say urls. This folder will contain a text file that will be used to determine what sites will get crawled.
  • In the new folder create a text document say (nutch.txt) and add a list of the urls you wish to crawl (e.g http://amac4.blogspot.co.uk)
  • In $NUTCH_HOME/runtime/local/conf open the regex-urlfilter.txt and where it says # accept anything else add the line +^http://amac4.blogspot.co.uk/ if you wish to search anything that comes under the amac4.blogspot.co.uk domain.
  • Nutch can only extract data from certain types of file and cannot extract data from images or various other binary file types. You as a user may not want data extraction to occur when using certain file types so you have the option to ignore these files by adding their tags to the list in the same fashion as it shows (|xml|XML|jpeg|asp|.. etc)
  • Now open nutch-site.xml and add the following code between the <configuration> headers
<name>http.agent.name</name>
<value>My Nutch Spider</value> #(You can add any name here)
  • In nutch-default.xml there should be a line under the <property> tag which has <name>http.agent.name</name>. The <value>field below should be empty so you should add the name of your crawler that you specified before, so in this case it would be:
<value>My Nutch Spider</value>
  • You can test the crawl is working by navigating to the $NUTCH_HOME/runtime/local/bin folder and executing:
cd $NUTCH_HOME/runtime/local/bin
./nutch crawl urls -dir [dir name] -depth [depth] -topN [files]
  • The directory you will supply (dir name) will store the indexes from the crawl. Running the crawl again will also cause the files to be re-indexed if they are found. The depth is asking how deep down a hierarchy do you wish to go and the topN is asking how many pages on each level you wish to index.
  • To get it linking to Solr copy the schema.xml from $NUTCH_HOME/runtime/local/conf into the $SOLR_HOME/collection1/conf/folder which should overwrite the previous schema.xml file but only if you are using Solr 1/2/3. If it is Solr4 you are using, which we are in this case then copy the schema-solr4.xml from the $NUTCH_HOME/runtime/local/conf directory into the$SOLR_HOME/collection1/conf/ and then rename it in the Solr directory back to schema.xml which should overwrite the old one. Changes made to the schema.xml made in the Solr set-up may need to be re-done.
  • Add this line to the schema.xml in the Solr installation and also to the schema-solr4.xml in the Nutch installation.
<field name="_version_" type="long" stored="true" indexed="true" multiValued="false"/>  
  • If you are NOT using the Solr4 schema file then edit the schema file you copied over and comment out the line like this:
<!--<filter class="solr.EnglishPorterFilterFactory" protected="protwords.txt"/>-->
  • To test the crawl is indexing to Solr type into cygwin:
./nutch crawl urls -dir newCrawl -solr http://localhost:8080/solr/ -depth 3 -topN 4
  • Note Nutch is now configured to crawl over the web but there are issues that have turned up so changes need to be made and these changes can be found at:

Optimising Nutch Performance

You may notice if you try and run Nutch that it works its way through the crawl very slowly, that is because by default Nutch is set-up to use using one thread and doesn't take advantage of the Multi-threaded implementation. Nutch will use multi-threading to crawl various hosts simultaneously and therefore the settings must be changed in order for search times to be kept to a reasonable level.
In the nutch-site.xml add the code between the <configuration> tags
  <property>
  <name>fetcher.threads.per.queue</name>
     <value>10</value>
     <description></description>
  </property>
 
  <property>
  <name>fetcher.threads.per.host</name>
     <value>10</value>
     <description></description>
  </property> 

Parsing Errors

You may find errors popping up every so often that look similar to
Error parsing: 192.168.0.42/test/AGENDA.doc: failed(2,0): Unable to read 512 bytes from 65536 in stream of length 65421
Fix By default there is a limit to how much data that Solr will parse so to take that limit away you need to set the content limit to -1. This has repercussions in terms of performance as large files will take longer to parse but no file should fail because of its length.
Add this code to the nutch-site.xml
  <property> 
  <name>http.content.limit</name> 
  <value>-1</value> 
  <description>The length limit for downloaded content, in bytes. 
               If this value is nonnegative (>=0), content longer than it 
  will be 
               truncated;otherwise, no truncation at all. 
  </description> 
  </property>

Failing to parse documents with a space in the title

URLs are not allowed to contain white-space and Nutch was not replacing the space character with %20 which meant that filename became invalid.
Fix
Add the following text to regex-normalise.xml:
  <regex> 
     <pattern>&#x20;</pattern> 
     <substitution>%20</substitution> 
  </regex> 

Next Steps

You may now want to set-up Nutch to crawl a local filesystem, the guide can be found at:     
http://amac4.blogspot.co.uk/2013/07/setting-up-nutch-to-crawl-filesystem.html    
http://amac4.blogspot.co.uk/2013/07/web-service-to-query-solr-rest.html    

Otherwise you may wish to check out some tweaks that can be made to Solr including deduplication and highlighting:
http://amac4.blogspot.co.uk/2013/08/setting-up-highlighting-for-solr-4.html 
http://amac4.blogspot.co.uk/2013/08/deleting-dead-urls-files-that-no-longer.html