Dynamic robots.txt

This will allow to create dynamic robot.txt file with TWIG.

This is our robotstxt CMS page with twig variables:

title = "robots"
url = "/robots"
is_hidden = 0
==
<?php
function onStart()
{
    $this->setResponseHeader('Content-Type', 'text/plain');
}
==
Modified: 9-27-2019
User-agent: *
Allow: /
Disallow: /api/*
Disallow: /payment/*

{% set baseUrl = url('/') %}
Disallow: {{ 'myCustomPage'|page(false)|replace({ (baseUrl): '' }) }}

Sitemap: {{ url('/') }}/sitemap.xml