Crawler
BelfortBot reads financial statements that Thai listed companies publish on their own investor-relations websites. It feeds the Thai company figures. It does not read SET, Settrade or SETSMART.
How to tell it is us
Section titled “How to tell it is us”Every request carries this User-Agent:
BelfortBot/0.1 (+https://docs.belfortagent.xyz/crawler/; support@belfortagent.xyz)How it behaves
Section titled “How it behaves”- It follows robots.txt. It obeys the rules for
BelfortBot, and the rules for*when there is no entry for it. A path that is disallowed is not fetched. - It follows one redirect of robots.txt, and no more. If your robots.txt redirects to
https://<host>/robots.txton your own company domain, or on the same site as the host that redirected, it reads that file and applies its rules to the page it was reading. A redirect to anywhere else, a second redirect, and a robots.txt it cannot read are all taken to mean “do not fetch”. - It is slow. At most one request every one and a half seconds to any one host, one at a time, with a twenty second timeout.
- It stays on the company’s own pages. Every request, the first and each redirect, goes over https to the company’s own domain or its investor-relations platform’s domains, and only to public internet addresses. A link to any other host is not followed.
- It reads public files only. No login, no form, no script. A page that needs one is skipped.
- It reads the statements and nothing else. It keeps each file as retrieved, with a hash, so that a figure can be checked against the file it came from.
Asking us to stop
Section titled “Asking us to stop”If you run one of these sites and want us to stop, or want a company’s figures removed:
- Email support@belfortagent.xyz from an address on the company’s domain, or tell us which company you speak for. Say which company and what you want: stop reading the site, remove the figures, or both.
- We remove the figures and stop reading the site within two business days, and we reply when it is done.
You can also add a rule for BelfortBot to your robots.txt. We read it before each visit, so it takes effect on the next one.
What we publish
Section titled “What we publish”Each Thai figure the API serves links to the company’s own file, so a company can always see what we read and where. See Thai company figures.