# All robots will spider the domain User-agent: * Disallow: # Disallow directory /clients/ User-agent: * Disallow: /clients/ # Disallow directory /public/ User-agent: * Disallow: /public/