| 词条 | Query understanding |
| 释义 |
MethodsTokenizationTokenization is the process of breaking up a text string into words or other meaningful elements called tokens. Typically, tokenization occurs at the word level. However, it is sometimes difficult to define what is meant by a "word". Often a tokenizer relies on simple heuristics, such as splitting the string on punctuation and whitespace characters. Tokenization is more challenging in languages without spaces between words, such as Chinese and Japanese. Tokenizing text in these languages requires the use of word segmentation algorithms.[7]Spelling correctionSpelling correction is the process of automatically detecting and correcting spelling errors in search queries. Most spelling correction algorithms are based on a language model, which determines the a priori probability of an intended query, and an error model (typically a noisy channel model), which determines the probability of a particular misspelling, given an intended query.[8]Stemming and lemmatizationMany, but not all, language inflect words to reflect their role in the utterance they appear in: a word such as *care* may appear as, besides the base form. as *cares*, *cared*, *caring*, and others. The variation between various forms of a word is likely to be of little importance for the relatively coarse-grained model of meaning involved in a retrieval system, and for this reason the task of conflating the various forms of a word is a potentially useful technique to increase recall of a retrieval system.[9] The languages of the world vary in how much morphological variation they exhibit, and for some languages there are simple methods to reduce a word in query to its lemma or root form or its stem. For some other languages, this operation involves non-trivial string processing. A noun in English typically appears in four variants: *cat* *cat's* *cats* *cats'* or *child* *child´s* *children* *children's*. Other languages have more variation. Finnish, e.g., potentially exhibits about 5000 forms for a noun,[10] and for many languages the inflectional forms are not limited to affixes but change the core of the word itself. Stemming algorithms, also known as stemmers, typically use a collection of simple rules to remove suffixes intended to model the language’s inflection rules.[11] More advanced methods, lemmatisation methods, group together the inflected forms of a word through more complex rule sets based on a word’s part of speech or its record in a lexical database, transforming an inflected word through lookup or a series of transformations to its lemma. For a long time, it was taken to be proven that morphological normalisation by and large did not help retrieval performance.[12] Once the attention of the information retrieval field moved to languages other than English, it was found that for some languages there were obvious gains to be found.[13] Entity recognitionEntity recognition is the process of locating and classifying entities within a text string. Named-entity recognition specifically focuses on named entities, such as names of people, places, and organizations. In addition, entity recognition includes identifying concepts in queries that may be represented by multi-word phrases. Entity recognition systems typically use grammar-based linguistic techniques or statistical machine learning models.[14] Query rewritingQuery rewriting is the process of automatically reformulating a search query to more accurately capture its intent. Query expansion adds additional query terms, such as synonyms, in order to retrieve more documents and thereby increase recall. Query relaxation removes query terms to reduce the requirements for a document to match the query, thereby also increasing recall. Other forms of query rewriting, such as automatically converting consecutive query terms into phrases and restricting query terms to specific fields, aim to increase precision. Apache Lucene search engine [15] uses query rewrite to transform complex queries to more primitive queries, such as expressions with wildcards (e.g. quer*) into a boolean query of the matching terms from the index (such as query OR queries).[16] See also
References1. ^{{cite web|url=http://www.sigir.org/files/forum/2010D/sigirwksp/2010d_sigirforum_croft.pdf|title=Association of Computing Machinery (ACM) Special Interest Group on Information Retrieval (SIGIR) 2010 Workshop on Query Representation and Understanding}} 2. ^{{cite web|url=https://aws.amazon.com/amazon-ai/|title=Amazon AI - Artificial Intelligence}} 3. ^{{cite web|url=https://www.apple.com/ios/siri/|title=iOS - Siri - Apple}} 4. ^{{cite web|url=http://searchengineland.com/google-uses-machine-learning-search-algorithms-261158|title=How Google uses machine learning in its search algorithms}} 5. ^{{cite web|url=https://venturebeat.com/2014/07/16/when-watson-met-siri-apples-ibm-deal-could-make-siri-a-lot-smarter/|title=When Watson met Siri: Apple's IBM deal could make Siri a lot smarter}} 6. ^{{cite web|url=https://www.theverge.com/2014/4/2/5570866/cortana-windows-phone-8-1-digital-assistant|title=The story of Cortana, Microsoft's Siri killer}} 7. ^{{cite web|url=http://nlp.stanford.edu/IR-book/html/htmledition/tokenization-1.html|title=Tokenization|publisher=}} 8. ^{{cite web|url=http://norvig.com/spell-correct.html|title=How to Write a Spelling Corrector|publisher=}} 9. ^{{cite book |last1= Lowe|first1= Thomas|last2 =Roberts|first2=David|last3=Kurtz|first3=Peterdate=1973 |title=Additional Text Processing for On-Line Retrieval (The RADCOL System). Volume 1 |publisher= DTIC Document}}{{cite journal | last1 = Lennon | first1 = Martin | last2= Peirce | first2 = David | last3 = Tarry | first3 = Brian D | last4 = Willett | first4 = Peter | author-link = | title = An evaluation of some conflation algorithms for information retrieval | journal = Information Scientist | volume = 3 | issue = 4 | publisher = SAGE | date = 1981}} 10. ^{{cite book |last= Karlsson |first= Fred |date= 2008|title= Finnish: an essential grammar|publisher= Routledge}} 11. ^{{cite book |last1= Lovins|first1= Julie|date=1968 |title=Development of a stemming algorithm |publisher=MIT Information Processing Group}} 12. ^{{cite journal | last = Harman | first = Donna | date = 1991 | title = How Effective is Suffixing? | journal = Journal of the American Society for Information Science | volume = 42 | issue = 1}} 13. ^{{cite journal | last1 = Popovic | first1 = Mirkoc | last2= Willett | first2 = Peter | title = The effectiveness of stemming for natural-language access to Slovene textual data | journal = Information Scientist | volume = 3 | issue = 4 | publisher = SAGE | date = 1981}} 14. ^{{cite web|url=http://nlp.cs.nyu.edu/sekine/papers/li07.pdf|title=A Survey of Named Entity Recognition and Classification}} 15. ^{{cite web|url=https://lucene.apache.org|title=Apache Lucene|publisher=}} 16. ^{{cite web|url=https://lucene.apache.org/core/6_4_1/core/org/apache/lucene/search/Query.html#rewrite-org.apache.lucene.index.IndexReader-|title=Query in Lucene 6.4.1 API documentation|publisher=}} 1 : Information retrieval techniques |
| 随便看 |
开放百科全书收录14589846条英语、德语、日语等多语种百科知识，基本涵盖了大多数领域的百科知识，是一部内容自由、开放的电子版国际百科全书。