{"id":69,"date":"2013-02-26T14:15:35","date_gmt":"2013-02-26T13:15:35","guid":{"rendered":"http:\/\/www.francelabs.com\/blog\/?p=69"},"modified":"2016-09-21T14:37:00","modified_gmt":"2016-09-21T13:37:00","slug":"tutorial-for-combining-manifoldcf-and-solr-for-files-search","status":"publish","type":"post","link":"https:\/\/www.francelabs.com\/blog\/tutorial-for-combining-manifoldcf-and-solr-for-files-search\/","title":{"rendered":"Tutorial for combining ManifoldCF and Solr for files search"},"content":{"rendered":"

NOTE: If you are interested in using ManifoldCF with Solr, you may want to look at our Datafari software<\/a>, which combines Apache ManifoldCF with Solr, so it eases this kind of integration. The code is available on google code: https:\/\/github.com\/francelabs\/datafari<\/a><\/p>\n

With the arrival of Manifold CF 1.0 (now already in v2.5), the open source community is looking for tutorials to combine it with Solr 4. That’s the intent of this tutorial, which will drive you through the different steps required to make it work.<\/p>\n

First, we’ll recap the installation process of Manifold CF (we’ll call it MCF later on), and of Solr. Second, we’ll configure both tools so that they can interact with each other. Third, we’ll configure MCF so that it crawls a windows file share. In this tutorial, when I specify installation directory such as solr-4.1.0, you have to complete with the absolute path of the installation directory.<\/p>\n

Installing Solr:<\/strong><\/p>\n

That’s a rather simple one. Download the last version of Solr (tested with Solr 4.1) and unzip it. You also have to download a servlet container in order to deploy Solr and MCF. Download the lastest version of tomcat (tested with apache-tomcat-7.0.37) and unzip it.<\/p>\n

Some libs are missing in Solr 4.1 : some ClassNotFound Exception can occur because of the missing jars while parsing some files (such as signed files) and these exceptions, because there are not Tika exceptions, lead to a HTTP response error code that is not handled correctly by MCF. To fix this problem, create a directory lib in<\/p>\n

solr-4.1.0\\example\\solr<\/pre>\n
Edit the<\/p>\n
solr-4.1.0\\example\\solr\\solr.xml<\/pre>\n
file and add sharedLib=\u201dlib\u201d to the solr tag:<\/p>\n
<solr persistent="true" sharedLib="lib"><\/pre>\n
Download asm-3.1.jar and aspectjrt-1.7.1.jar and copy\/paste it into<\/p>\n
solr-4.1.0\\example\\solr\\lib<\/pre>\n
.<\/p>\n
Edit the<\/p>\n
solr-4.1.0\\example\\solr\\collection1\\conf\\solrconfig.xml<\/pre>\n
file, set ignoreTikaException to true in the update\/extract request handler:<\/p>\n
<requestHandler name="\/update\/extract"\r\n                  startup="lazy"\r\n                  class="solr.extraction.ExtractingRequestHandler" >\r\n    <lst name="defaults">\r\n      <bool name="ignoreTikaException">true<\/bool>\r\n       \u2026\r\n    <\/lst>\r\n<\/requestHandler><\/pre>\n
In directory<\/p>\n
apache-tomcat-7.0.37\\conf\\Catalina\\localhost<\/pre>\n
(create Catalina and localhost if they don\u2019t exist), create a file called solr.xml.<\/p>\n
<?xml version="1.0" encoding="utf-8"?>\r\n<Context docBase=" solr-4.1.0\\dist\\solr-4.1.0.war" crossContext="true">\r\n  <Environment name="solr\/home" type="java.lang.String" value=" solr-4.1.0\\example\\solr" override="true"\/>\r\n<\/Context><\/pre>\n
You can now start your Solr in command line with<\/p>\n
apache-tomcat-7.0.37\\bin\\startup.bat<\/pre>\n
Have a look at the logs to be sure that everything went well, and test Solr with your browser :
\nhttp:\/\/localhost:8080\/solr\/#\/<\/a><\/p>\n
Installing ManifoldCF:<\/strong><\/p>\n
We will install and deploy Manifold CF on tomcat with PostGre SQL DB, which is the recommanded DB for Manifold CF deployment in production environnement.<\/p>\n
– First download the last released version of MCF (tested with Apache Manifold CF 1.1.1) and unzip it.<\/p>\n
– You also have to download the jcifs library to crawl Samba share (tested with jcifs-1.3.17). Download and copy\/paste the jcifs-1.3.17.jar into:<\/p>\n
apache-manifoldcf-1.1.1\\connector-lib-proprietary<\/pre>\n
.<\/p>\n
– Download version 9.1 of PostgreSQL. Install it with the installer.<\/p>\n
In<\/p>\n
apache-tomcat-7.0.37\\conf\\Catalina\\localhost<\/pre>\n
create 3 xml files: mcf-api-service.xml, mcf-authority-service.xml and mcf-crawler-ui.xml.<\/p>\n
In mcf-api-service.xml, add the following code:<\/p>\n
<?xml version="1.0" encoding="utf-8"?>\r\n<Context docBase=" apache-manifoldcf-1.1.1\\web\\war\\mcf-api-service.war" crossContext="true" >\r\n<\/Context><\/pre>\n
Do the same operation for mcf-authority-service.xml and mcf-crawler-ui.xml.<\/p>\n
Edit<\/p>\n
apache-manifoldcf-1.1.1\\connectors.xml<\/pre>\n
in this file you need to uncomment:<\/p>\n
<repositoryconnector name="Windows shares" class="org.apache.manifoldcf.crawler.connectors.sharedrive.SharedDriveConnector" \/><\/pre>\n
Edit<\/p>\n
apache-manifoldcf-1.1.1\\multiprocess-example\\properties.xml<\/pre>\n
The first five properties can be commented because they are only useful for a jetty server.<\/p>\n
Modify the database implementation class in order to use postgre instead of derby:<\/p>\n
<property name="org.apache.manifoldcf.databaseimplementationclass" value="org.apache.manifoldcf.core.database.DBInterfacePostgreSQL"\/><\/pre>\n
Add a line to specify the database name, the specified database shoudn\u2019t exist for the moment in the postgresql server.<\/p>\n
<property name="org.apache.manifoldcf.database.name" value="manifoldcf"\/><\/pre>\n
Modify the username and password:<\/p>\n
<property name="org.apache.manifoldcf.dbsuperusername" value="user"\/>\r\n<property name="org.apache.manifoldcf.dbsuperuserpassword" value="password"\/><\/pre>\n
You can now initialize your database with the following script. This script will automatically create the database and tables that will be used by the crawler:<\/p>\n
apache-manifoldcf-1.1.1\\multiprocess-example\\initialize.bat<\/pre>\n
Now set the classpath values with the following script:<\/p>\n
apache-manifoldcf-1.1.1\\multiprocess-example\\setclasspath.bat<\/pre>\n
Start the agents that are in charge of crawling the data:<\/p>\n
apache-manifoldcf-1.1.1\\multiprocess-example\\start-agents.bat<\/pre>\n
Edit<\/p>\n
apache-tomcat-7.0.37\\bin\\startup.bat<\/pre>\n
Add the following line at the beginning of the file to point the configuration file for MCF :<\/p>\n
set "CATALINA_OPTS=-Dorg.apache.manifoldcf.configfile= apache-manifoldcf-1.1.1\\multiprocess-example\\properties.xml\\properties.xml"<\/pre>\n
Configuring Manifold CF and crawling:<\/strong><\/p>\n
Start Tomcat. Go to the admin interface : http:\/\/localhost:8080\/mcf-crawler-ui\/<\/a><\/p>\n
\"adminUI\"<\/a><\/p>\n
In Output -> List Output Connections, select Add a new output connection<\/p>\n
Type a name : \u201cSolr\u201d then select Solr for type and continue.<\/p>\n
In the server tab, add collection1 for Core\/Collection name and change the port to 8080 :<\/p>\n
\"configSolr\"<\/a><\/p>\n
Click on Save, you should have Connection status:Connection working.<\/p>\n
In Repositories -> List Repository Connections, select Add a new connection<\/p>\n
Type a name : \u201cFileShare\u201d then select Windows Share for type and continue.<\/p>\n
In the Server tab, enter the information of your windows server, (hostname, domain, username and password).<\/p>\n
Click on Save, you should have Connection status:Connection working.<\/p>\n
In Jobs -> List all Jobs, select Add a new job,<\/p>\n
Select a name, \u201cCrawlJob\u201d, in Connection tab, select Solr for the Output connection and FileShare for the repository connection.<\/p>\n
Click on continue and go to Paths.<\/p>\n
Select your folder and Click on +. Then click on Add. You should have:<\/p>\n
\"crawlJob\"<\/a><\/p>\n
Click on Save and go to Jobs -> Status and Job Management<\/p>\n
Start your job. When the job is finished, Manifold CF automatically performs a commit on Solr.<\/p>\n
\"jobfinished\"<\/a><\/p>\n
You can now perform a search with Solr:<\/p>\n
http:\/\/localhost:8080\/solr\/collection1\/select?q=*%3A*&fl=id&wt=xml&indent=true<\/a><\/p>\n
\"solrResult\"<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
NOTE: If you are interested in using ManifoldCF with Solr, you may want to look at our Datafari software, which combines Apache ManifoldCF with Solr, so it eases this kind of integration. The code is available on google code: https:\/\/github.com\/francelabs\/datafari … Continue reading →<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[19,1,12],"tags":[20,22,60,21,59],"class_list":["post-69","post","type-post","status-publish","format-standard","hentry","category-manifoldcf","category-search","category-solr","tag-files","tag-manifold-cf","tag-manifoldcf","tag-share","tag-solr"],"_links":{"self":[{"href":"https:\/\/www.francelabs.com\/blog\/wp-json\/wp\/v2\/posts\/69","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.francelabs.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.francelabs.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.francelabs.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.francelabs.com\/blog\/wp-json\/wp\/v2\/comments?post=69"}],"version-history":[{"count":55,"href":"https:\/\/www.francelabs.com\/blog\/wp-json\/wp\/v2\/posts\/69\/revisions"}],"predecessor-version":[{"id":395,"href":"https:\/\/www.francelabs.com\/blog\/wp-json\/wp\/v2\/posts\/69\/revisions\/395"}],"wp:attachment":[{"href":"https:\/\/www.francelabs.com\/blog\/wp-json\/wp\/v2\/media?parent=69"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.francelabs.com\/blog\/wp-json\/wp\/v2\/categories?post=69"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.francelabs.com\/blog\/wp-json\/wp\/v2\/tags?post=69"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}