| 词条 | Crawl frontier |
| 释义 |
A crawl frontier is a data structure used for storage of URLs eligible for crawling and supporting such operations as adding URLs and selecting for crawl. Sometimes it can be seen as priority queue. [1] OverviewA crawl frontier is one of the components that make up the architecture of a web crawler. The crawl frontier contains the logic and policies that a crawler follows when visiting websites. This activity is known as [https://www.google.com/insidesearch/howsearchworks/crawling-indexing.html crawling]. The policies can include such things as what pages should be visited next, the priorities for each page to be searched, and how often the page is to be visited.[2] The efficiency of the crawl frontier is especially important since one of the characteristics of the Web that make web crawling a challenge; is that it contains such a large volume of data and it is constantly changing.[3][4] ArchitectureThe initial list of URLs contained in the crawler frontier are known as seeds. The web crawler will constantly ask the frontier what pages to visit. As the crawler visits each of those pages, it will inform the frontier with the response of each page. The crawler will also update the crawler frontier with any new hyperlinks contained in those pages it has visited. These hyperlinks are added to the frontier and will visit those new web pages based on the policies of the crawler frontier.[3][4] This process continues recursively until all URLs in the crawl frontier are visited. The policies used to determine what pages to visit are commonly based on a score. This score is typically computed from a number of different attributes. Such as the freshness of a page, the time the page was updated and the relevance of the content with respect to certain terms. ComponentsFrontier API/ManagerThe Frontier Manager is the component that the web crawler will use to communicate with the crawl frontier. The frontier API can also be used to communicate with the crawl frontier.[3] MiddlewaresThe frontier middlewares sit between the manager and the backend. The purpose of middlewares is to manage the communication between frontier and the backend. Middlewares are an ideal way to add or extend additional functionality simply by plugging additional code.[5] BackendThe backend component contains all the logic and policies that are used in a search. The function of the backend is to identify the pages to be crawled.[5] Known implementations
References1. ^{{Cite journal|url=http://infolab.stanford.edu/~olston/publications/crawling_survey.pdf|title=Web Crawling|journal=Foundations and Trends in Information Retrieval|first=Christopher|last=Olston|first2=Marc|last2=Najork}} 2. ^{{Cite journal|last=Patil|first=Yugandhara|last2=Patil|first2=Sonal|year=2016|title=Review of Web Crawlers with Specification and Working|url=http://www.ijarcce.com/upload/2016/january-16/IJARCCE%2052.pdf|journal=International Journal of Advanced Research in Computer and Communication Engineering|volume=5|pages=4|via=}} 3. ^1 2 {{Cite web|url=https://media.readthedocs.org/pdf/frontera/v0.2.0/frontera.pdf|title=crawlfrontier Documentation|last=|first=|date=April 15, 2015|website=|archive-url=|archive-date=|dead-url=|access-date=}} 4. ^1 {{Cite journal|last=Dhenakaran|first=S.S.|last2=Sambanthan|first2=K. Thirugnana|year=2011|title=Web Crawler - An Overview|url=http://csjournals.com/IJCSC/PDF2-1/Article_49.pdf|journal=International Journal of Computer Science and Communication|volume=2|pages=3|via=}} 5. ^1 {{Cite web|url=https://frontera.readthedocs.io/en/v0.7.1/topics/architecture.html#middlewares|title=Frontera Architecture|last=|first=|date=2017|website=|archive-url=|archive-date=|dead-url=|access-date=}} 1 : Search engine software |
| 随便看 |
|
开放百科全书收录14589846条英语、德语、日语等多语种百科知识，基本涵盖了大多数领域的百科知识，是一部内容自由、开放的电子版国际百科全书。