Men and women use DA to find out which Web sites are reliable in their area and to determine exactly where to acquire fantastic back links. txt file is then parsed and will instruct the robot concerning which webpages are usually not for being crawled. To be a internet search https://www.youtube.com/watch?v=ZPZ0MrJuEfM