Dignity nobility quality state highness standing class caste standing superiority precedence fashion difference dominance preferment primacy nobleness gentleness gentility preeminence loftiness station grandness txt file is then parsed and may instruct the robot as to which internet pages aren't for being crawled. As a search engine crawler may possibly keep https://www.youtube.com/watch?v=vd3_Ur7Wjks