Skip to content

Crawler

View as Markdown

BelfortBot reads financial statements that Thai listed companies publish on their own investor-relations websites. It feeds the Thai company figures. It does not read SET, Settrade or SETSMART.

Every request carries this User-Agent:

BelfortBot/0.1 (+https://docs.belfortagent.xyz/crawler/; support@belfortagent.xyz)
  • It follows robots.txt. It obeys the rules for BelfortBot, and the rules for * when there is no entry for it. A path that is disallowed is not fetched.
  • It follows one redirect of robots.txt, and no more. If your robots.txt redirects to https://<host>/robots.txt on your own company domain, or on the same site as the host that redirected, it reads that file and applies its rules to the page it was reading. A redirect to anywhere else, a second redirect, and a robots.txt it cannot read are all taken to mean “do not fetch”.
  • It is slow. At most one request every one and a half seconds to any one host, one at a time, with a twenty second timeout.
  • It stays on the company’s own pages. Every request, the first and each redirect, goes over https to the company’s own domain or its investor-relations platform’s domains, and only to public internet addresses. A link to any other host is not followed.
  • It reads public files only. No login, no form, no script. A page that needs one is skipped.
  • It reads the statements and nothing else. It keeps each file as retrieved, with a hash, so that a figure can be checked against the file it came from.

If you run one of these sites and want us to stop, or want a company’s figures removed:

  1. Email support@belfortagent.xyz from an address on the company’s domain, or tell us which company you speak for. Say which company and what you want: stop reading the site, remove the figures, or both.
  2. We remove the figures and stop reading the site within two business days, and we reply when it is done.

You can also add a rule for BelfortBot to your robots.txt. We read it before each visit, so it takes effect on the next one.

Each Thai figure the API serves links to the company’s own file, so a company can always see what we read and where. See Thai company figures.