| 词条 | Common Crawl | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 释义 |
| name = Common Crawl | company_type = 501(c)(3) non-profit | traded_as = | foundation = | dissolved = | location = San Francisco, California, USA; Los Angeles, California, USA | incorporated = | founder = Gil Elbaz | chairman = | president = | CEO = | key_people = Peter Norvig, Nova Spivack, Carl Malamud, Kurt Bollacker Joi Ito | industry = | products = | services = | assets = | equity = | owner = | num_employees = | parent = | divisions = | subsid = | company_slogan = | url = commoncrawl.org | ipv6 = | alexa = | website_type = | registration = | num_users = | language = English | launch_date = | current_status = | screenshot = | caption = | footnotes = | intl = }}Common Crawl is a nonprofit 501(c)(3) organization that crawls the web and freely provides its archives and datasets to the public.[1][2] Common Crawl's web archive consists of petabytes of data collected since 2011.[3] It completes crawls generally every month.[4] Common Crawl was founded by Gil Elbaz.[5] Advisors to the non-profit include Peter Norvig and Joi Ito.[6] The organization's crawlers respect nofollow and robots.txt policies. Open source code for processing Common Crawl's data set is publicly available. HistoryAmazon Web Services began hosting Common Crawl's archive through its Public Data Sets program in 2012.[7]The organization began releasing metadata files and the text output of the crawlers alongside .arc files in July of that year.[8] Common Crawl's archives had only included .arc files previously.[8] In December 2012, blekko donated to Common Crawl search engine metadata blekko gathered from crawls it conducted from February to October 2012.[9] The donated data helped Common Crawl "improve its crawl while avoiding spam, porn and the influence of excessive SEO."[9] In 2013, Common Crawl began using Apache Software Foundation's Nutch webcrawler instead of a custom crawler.[10] Common Crawl switched from using .arc files to .warc files with its November 2013 crawl.[11] History of Common Crawl dataThe following data have been collected from the official Common Crawl Blog
Norvig Web Data Science AwardIn corroboration with SURFsara, Common Crawl sponsors the Norvig Web Data Science Award, a competition open to students and researchers in Benelux.[12][13] The award is named for Peter Norvig who also chairs the judging committee for the award.[12] References1. ^{{cite news |title=Tech entrepreneur Gil Elbaz made it big in L.A.|author=Rosanna Xia|publisher=Los Angeles Times|date=February 5, 2012|accessdate=July 31, 2014|url=http://articles.latimes.com/2012/feb/05/business/la-fi-himi-elbaz-20120205}} 2. ^{{cite news |title=Gil Elbaz and Common Crawl|publisher=NBC News|date=April 4, 2013|accessdate=July 31, 2014|url=http://www.pressheretv.com/gil-elbaz-and-common-crawl/}} 3. ^{{cite web |url=https://commoncrawl.org/the-data/get-started/ |title=So you're ready to get started |accessdate=2018-06-02 }} 4. ^{{cite news |title=Winter 2013 Crawl Data Now Available|author=Lisa Green|date=January 8, 2014|accessdate=June 2, 2018|url=https://commoncrawl.org/2014/01/winter-2013-crawl-data-now-available/}} 5. ^{{cite news |title=Startups - Gil Elbaz and Nova Spivack of Common Crawl - TWiST #222|publisher=This Week In Startups|date=January 10, 2012}} 6. ^{{cite news |title=A Free Database of the Entire Web May Spawn the Next Google|author=Tom Simonite|publisher=MIT Technology Review|date=January 23, 2013|accessdate=July 31, 2014|url=http://www.technologyreview.com/news/509931/a-free-database-of-the-entire-web-may-spawn-the-next-google/}} 7. ^{{cite news |title=Common Crawl To Add New Data In Amazon Web Services Bucket|author=Jennifer Zaino|publisher=Semantic Web|date=March 13, 2012|accessdate=July 31, 2014|url=http://semanticweb.com/common-crawl-to-add-new-data-in-amazon-web-services-bucket_b27341}} 8. ^1 {{cite news |title=Common Crawl Corpus Update Makes Web Crawl Data More Efficient, Approachable For Users To Explore|author=Jennifer Zaino|publisher=Semantic Web|date=July 16, 2012|accessdate=July 31, 2014|url=http://semanticweb.com/common-crawl-corpus-update-makes-web-crawl-data-more-efficient-approachable-for-users-to-explore_b30771}} 9. ^1 {{cite news |title=Blekko Data Donation Is A Big Benefit To Common Crawl|author=Jennifer Zaino|publisher=Semantic Web|date=December 18, 2012|accessdate=July 31, 2014|url=http://semanticweb.com/blekko-data-donation-is-a-big-benefit-to-common-crawl_b34177}} 10. ^{{cite web |title=Common Crawl's Move to Nutch|author=Jordan Mendelson|publisher=Common Crawl|date=February 20, 2014|accessdate=July 31, 2014|url=http://commoncrawl.org/common-crawl-move-to-nutch/}} 11. ^{{cite web |title=New Crawl Data Available!|author=Jordan Mendelson|publisher=Common Crawl|date=November 27, 2013|accessdate=July 31, 2014|url=http://commoncrawl.org/new-crawl-data-available/}} 12. ^1 {{cite web |title=The Norvig Web Data Science Award|author=Lisa Green|publisher=Common Crawl|date=November 15, 2012|accessdate=July 31, 2014|url=http://commoncrawl.org/the-norvig-web-data-science-award/}} 13. ^{{cite web |title=Norvig Web Data Science Award 2014|publisher=Dutch Techcentre for Life Sciences|accessdate=July 31, 2014|url=http://www.dtls.nl/dtl/news/norvig-web-data-science-award-2014.html}} External links
3 : Internet companies|Web archiving|Web archiving initiatives |
|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| 随便看 |
|
开放百科全书收录14589846条英语、德语、日语等多语种百科知识，基本涵盖了大多数领域的百科知识，是一部内容自由、开放的电子版国际百科全书。