Xavier Briand @xavierbriand
Senior web developer at experiencePoint
Senior web developer at experiencePoint
How to
search
easily&
efficiently
?
Solr is a standalone enterprise search server with a REST-like API. You put documents in it [...] over HTTP. You query it via HTTP GET and receive [...] results.
Schema
- Defines the field types and fields of documents
- Can drive more intelligent processing
- Declarative Lucene Analyzer specification
- Dynamic Fields enables on-the-fly addition of new fields
- CopyField functionality allows indexing a single field multiple ways, or combining multiple fields into a single searchable field
- Explicit types eliminates the need for guessing types of fields
- External file-based configuration of stopword lists, synonym lists, and protected word lists
- Many additional text analysis components including word splitting, regex and sounds-like filters
Query
- HTTP interface with configurable response formats (XML/XSLT, JSON, Python, Ruby, PHP, Velocity, binary)
- Sort by any number of fields, and by complex functions of numeric fields
- Advanced DisMax query parser for high relevancy results from user-entered queries
- Highlighted context snippets
- Faceted Searching based on unique field values, explicit queries, date ranges, and numeric ranges
- Multi-Select Faceting by tagging and selectively excluding filters
- Spelling suggestions for user queries
- More Like This suggestions for given document
- Function Query - influence the score by user specified complex functions of numeric fields or query relevancy scores.
- Range filter over Function Query results
- Date Math - specify dates relative to "NOW" in queries and updates
- Dynamic search results clustering using Carrot2
- Numeric field statistics such as min, max, average, standard deviation
- Combine queries derived from different syntaxes
- Auto-suggest functionality for completing user queries
- Allow configuration of top results for a query, overriding normal scoring and sorting
- Performance Optimizations
Core
- Dynamically create and delete document collections without restarting
- Pluggable query handlers and extensible XML data format
- Pluggable user functions for Function Query
- Customizable component based request handler with distributed search support
- Document uniqueness enforcement based on unique key field
- Duplicate document detection, including fuzzy near duplicates
- Custom index processing chains, allowing document manipulation before indexing
- User configurable commands triggered on index changes
- Ability to control where docs with the sort field missing will be placed
- "Luke" request handler for corpus information
Caching
- Configurable Query Result, Filter, and Document cache instances
- Pluggable Cache implementations, including a lock free, high concurrency implementation
- Cache warming in background
- When a new searcher is opened, configurable searches are run against it in order to warm it up to avoid slow first hits. During warming, the current searcher handles live requests.
- Autowarming in background
- The most recently accessed items in the caches of the current searcher are re-populated in the new searcher, enabling high cache hit rates across index/searcher changes.
- Fast/small filter implementation
- User level caching with autowarming support
Replication
- Efficient distribution of index parts that have changed
- Pull strategy allows for easy addition of searchers
- Configurable distribution interval allows tradeoff between timeliness and cache utilization
- Replication and automatic reloading of configuration files
Admin Interface
- Comprehensive statistics on cache utilization, updates, and queries
- Interactive schema browser that includes index statistics
- Replication monitoring
- Full logging control
- Text analysis debugger, showing result of every stage in an analyzer
- Web Query Interface w/ debugging output
- Parsed query output
- Lucene explain() document score detailing
- explain score for documents outside of the requested range to debug why a given document wasn't ranked higher.
Twitter, Apple, wikipedia, LinkedIn, ...
Etsy, AOL yellow pages, Sears, SourceForge, Nasa, digg, buy.com, OverBlog, ...
$ wget http://(...)/3.6.0/apache-solr-3.6.0.tgz $ tar -xfz apache-solr-3.6.0.tgz $ cd apache-solr-3.6.0/example/ $ java -jar start.jar (a lot of java log...)
Solarium
http://www.solarium-project.org/
<fields>
<field name="id" type="string" indexed="true" stored="true" required="true" />
<field name="description" type="text_general" indexed="true" stored="true"/>
<field name="path" type="string" indexed="false" stored="true"/>
</fields>
<types>
<fieldType name="text_general" class="solr.TextField" positionIncrementGap="100">
<analyzer>
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StopFilterFactory" ignoreCase="true" words="stopwords.txt" />
<filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" />
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.PorterStemFilterFactory"/>
</analyzer>
</fieldType>
<!-- ... -->
</types>
I'm looking for Wi-Fi
I | m | looking | for | Wi-Fi
looking | Wi-fi
looking | WiFi
looking | wifi
look | wifi
JaMes looks aT the wIfi signal
JaMes | looks | aT | the | wIfi | signal
JaMes | looks | wIfi | signal
JaMes | looks | wIfi | signal
james | looks | wifi | signal
james | look | wifi | signal
// Acme/BlogBundle/Entity/BlogPost.php
class BlogPost
{
//...
public function toSolrDocument(\Solarium_Document_ReadWrite $doc)
{
$doc->id = $this->getId();
$doc->description = $this->getTitle().' '.$this->getBody();
$doc->path = $this->getSlug();
return $doc;
}
//...
}
// Acme/BlogBundle/Command/SolrIndexCommand.php
$client = $this->getContainer()->get('solarium');
// get an update query instance
$update = $client->createUpdate();
// create documents
$documents = array();
foreach ($blogPosts as $blogPost) {
$documents[] = $blogPost->toSolrDocument($update->createDocument());
}
// add the documents and a commit command to the update query
$update->addDocuments($documents);
$update->addCommit();
// this executes the query and returns the result
$result = $client->update($update);
$client = $this->get('solarium');
$query = $client->createSelect();
$query->setQuery('description:"Foo Bar"');
$query->setStart(1)->setRows(10);
$resultset = $client->select($query);
<ul>
{% for doc in resultset %}
<li> {{ doc.description }} </li>
{% endfor %}
</ul>
Relevency
TF -
IDFCoord
lengthNorm
Boost
<field name="title" type="text_general" indexed="true" stored="true"/>
$query->setQuery('"Foo Bar"');
$query->getDisMax()->setQueryParser('edismax');
$query->getDisMax()->setQueryFields('title^2 description');
Faceting
$query = $client->createSelect();
$facetSet = $query->getFacetSet();
$facetSet->createFacetField('cat')->setField('category');
{% for facet, value in resultset.facetSet.facet('category') %}
{{ facet }} ({{ value }})
{% endfor %}
X
$query->setQuery('Chocolat category:book');
$query->setQuery('Chocolat');
$query->createFilterQuery('cat')->setQuery('category:book');
highlight
$query->getHighlighting()->setFields('description');
$query->getHighlighting()->setSimplePrefix('<em>');
$query->getHighlighting()->setSimplePostfix('</em>');
<ul>
{% for doc in resultset %}
<li>
{{ doc.title }}<br />
{% for text in resultset.highlighting.result(doc.id).field('description') %}
{{ text|raw }}
{% endfor %}
</li>
{% endfor %}
</ul>
<!-- request handler ./config/solrconfig.xml -->
<!-- ... -->
<arr name="last-components">
<str>spellcheck</str>
</arr>
</requestHandler>
$query->getSpellcheck()->setQuery('misplelign');
$query->getSpellcheck()->setCollateExtendedResults(true);
{% if resultset.spellcheck.correctlySpelled %}
{% for suggestion in resultset.spellcheck %}
{{ suggestion.word }}
{% endfor %}
{% for collation in resultset.spellcheck.collation.corrections %}
{{ collation }}
{% endfor %}
Auto..
Autopsy
Automatic
Autosuggest
Automobile
Automatic
Autosuggest
Automobile
???
Serv...=
Service Con...
=
Service Container
<!-- ./config/schema.xml -->
<fieldType name="suggester" class="solr.TextField">
<analyzer type="index">
<tokenizer class="solr.KeywordTokenizerFactory"/>
<filter class="solr.EdgeNGramFilterFactory" minGramSize="3"
maxGramSize="15" side="front"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.KeywordTokenizerFactory"/>
</analyzer>
</fieldType>
Serv
Servi
Servic
Service
Service
Service C
Service Co
...
/
I don't speak english
multi-index (schema.xml)
multi-config (solrconfig.xml)
<!-- ./solr.xml -->
<!-- from -->
<solr>
<cores adminPath="/admin/cores">
<core name="collection1" instanceDir="." />
</cores>
</solr>
<!-- to -->
<solr>
<cores adminPath="/admin/cores">
<core name="english" instanceDir="core_en/"/>
<core name="french" instanceDir="core_fr/"/>
</cores>
</solr>
▾ solr/
▾ conf/
schema.xml
solrconfig.xml
▸ data/
README.txt
solr.xml
▾ solr/
▾ core_en/
▾ conf/
schema.xml
solrconfig.xml
▸ data/
▾ core_fr/
▾ conf/
schema.xml
solrconfig.xml
▸ data/
README.txt
solr.xml
$client->getAdapter()->setCore('french');
http://localhost:8983/solr/admin/cores?action=UNLOAD&core=collection1
move / duplicate / setup config
http://local:8983/solr/admin/cores?action=CREATE&name=english&instanceDir=core_en/
http://local:8983/solr/admin/cores?action=CREATE&name=french&instanceDir=core_fr/
STATUS / RELOAD / RENAME / ALIAS / SWAP
=
replication load balancing
sharding
Unlimited Power
SUM UP
$client = $this->get('solarium');
$query = $client->createSelect();
$query->setQuery('text:"Foo Bar"');
$query->setStart(1)->setRows(10);
$query->getDisMax()->setQueryParser('edismax');
$query->getDisMax()->setQueryFields('title^2 description');
$query->getFacetSet()->createFacetField('cat')->setField('category');
// + query filtering
$query->getHighlighting()->setFields('description');
$query->getHighlighting()->setSimplePrefix('<em>');
$query->getHighlighting()->setSimplePostfix('</em>');
$query->getSpellcheck()->setQuery('misplelign');
$query->getSpellcheck()->setCollateExtendedResults(true);
$resultset = $client->select($query);
Solarium Plugin
abstract class Solarium_Plugin_Abstract
{
protected function _initPlugin()
public function preCreateRequest($query)
public function postCreateRequest($query, $request)
public function preExecuteRequest($request)
public function postExecuteRequest($request, $response)
public function preCreateResult($query, $response)
public function postCreateResult($query, $response, $result)
public function preExecute($query)
public function postExecute($query, $result)
public function preCreateQuery($type, $options)
public function postCreateQuery($type, $options, $query)
}
<service id="solarium.autosuggest" parent="solarium">
<call method="registerPlugin">
<argument>autosuggest</argument>
<argument type="service" id="solarium.plugin.autosuggest" />
</call>
</service>
<service id="solarium.plugin.autosuggest" class="%class%" public="false" >
<argument type="service" id="router" />
</service>
// Sensio\Bundle\SearchBundle\Solarium\Plugin\Autosuggest
class Autosuggest extends \Solarium_Plugin_Abstract
{
public function __construct(Router $router)
protected function _initPlugin()
{
$this->_client->registerQueryType(
\Solarium_Client::QUERYTYPE_SELECT,
'Sensio\Bundle\SearchBundle\Solarium\Query\Autosuggest',
'Solarium_Client_RequestBuilder_Select',
'Solarium_Client_ResponseParser_Select'
);
}
public function preCreateResult($query, $response)
{
// generate routes for your jquery autosuggest response
}
// (...)
// Sensio\Bundle\SearchBundle\Solarium\Query\Autosuggest
class Autosuggest extends \Solarium_Query_Select
{
protected function _init()
{
$this->createFilterQuery('enabled')->setQuery('is_active:true');
$this->getDisMax()->setQueryFields('autosuggest');
$this->getGrouping()->addField('type');
$this->setFields('id, name, link');
}
}
▾ Solarium/
▾ Plugin/
Autosuggest.php
Search.php
▾ Query/
Autosuggest.php
Search.php
=
serch terms, page, filter values, ....
