https://alexschreyer.net/web-programming/using-robots-txt-to-your-advantage-to-reduce-server-load/