With that, Google expanded how Google products are affected by each crawler and gave robots.txt examples for each crawler. Google made a series of updates to its crawlers and user-triggered fetchers ...