Dignity nobility high quality condition highness standing course caste position superiority priority fashion distinction dominance preferment primacy nobleness gentleness gentility preeminence loftiness station grandness txt file is then parsed and will instruct the robot concerning which internet pages will not be to become crawled. For a search engine crawler could https://www.youtube.com/watch?v=DhNbKpYq6Ro