Create default robots.txt for Angular UI

[tdonohue]

May be loosely based on the existing DSpace 6.x robots.txt files:

• XMLUI: https://github.com/DSpace/DSpace/blob/dspace-6_x/dspace-xmlui/src/main/webapp/static/robots.txt
• JSPUI: https://github.com/DSpace/DSpace/blob/dspace-6_x/dspace-jspui/src/main/webapp/robots.txt

Will require a webpack step to build the robots.txt (to insert in the sitemaps URL dynamically, if possible).

Possible useful resources:

• https://www.thecodecampus.de/blog/angular-cli-sitemap-robots-txt/
• https://github.com/valor-software/angular2-webpack-starter/tree/master/src/assets

[heathergreerklein]

Estimate from release plan: 4 hours Expressed interest: Atmire