Important Note guys, Google is sending an email to publishers:
"As part of Google’s efforts to increase brand safety for advertisers, Google is making changes to the way it buys impressions on URLs that are uncrawled. More specifically, starting on December 15th, AdWords and DoubleClick Bid Manager will adopt more restrictive bidding on ad requests coming from URLs that are uncrawled. This is necessary to avoid the risk of ads running on sensitive content.
We detected that your AdX /Adsense account is sending a significant number of ad requests from URLs that are not crawled. To avoid a potential revenue impact from this change, please consider the following best practices for ensuring URLs can be properly crawled:
AdSense Publishers, see:
About the AdSense crawler
How to fix AdSense crawler errors
Display ads on login-protected pages
Give access to our crawler in your robots.txt file
DFP and AdX Publishers, see Crawler Access.
take a note @Hitesh_Bhasin @abdsam @Arshi @Pulkit_Sharma and others.