Search engine Like google,yahoo,bing and more use the method called crawling and indexing to get the webpage for the search result of search engine.these crawling and indexing are the computer algorithm and make change in the program to show the quality content in the search result
Crawling is the process to find or discover new website or new page in the internet to help the Search engine to get index the web page the search engine’s search result
Search Engine like Google use method crawling with web crawler called Googlebot and also called spider fetch the content on the billion of web page in the internet.Googlebot is a computer program with algorithm process to show the way to how and when the content of the website will fetched.
Web Crawler like Googlebot start work with url and fetch the url then follow the url in the web page on the internet then fetch the content.If a website is still not found google search it will find the website later or a website can be submitted to search engine with directly or with google webmaster tool by submitting the sitemap of the website
Indexing is also the process when the crawler fetch the web page in the internet then the indexing process is started to research the word and the position of the word in the content and also in url,title and meta description to provide the search result of the query for the users
Indexing do research with the crawled content from the internet but there are many factor plays to serve the search result so they serve result by the method called Serving
Google is the best search engine to provide the quality search result for the user’s search queries and you can block the search appearance of the restricted content from the website by using robot.txt .Google provide the search result according to relevancy of the website.Major one of the relevancy is page rank and this is based on the backlink of the website but google improve the search result they use some black mark to website contain spam link or paid backlink.If the website have high pagerank the website will appear high on the search results of the search engine,Page Rank is based on the quality of the quality of the backlink like link on educational website or government website not on the quantity