In order to avoid undesirable articles in the research indexes, webmasters can instruct spiders to not crawl sure information or directories in the common robots.txt file in the root directory with the domain. Moreover, a webpage is usually explicitly excluded from a search engine's databases by using a meta tag https://news.rafeeg.ae