# Crawler

> BelfortBot, the crawler that reads Thai company statements from investor-relations sites. How it behaves and how a company asks us to stop.

BelfortBot reads financial statements that Thai listed companies publish on their own investor-relations websites. It feeds the [Thai company figures](/datasets/th-financials/). It does not read SET, Settrade or SETSMART.

## How to tell it is us

Every request carries this User-Agent:

```text
BelfortBot/0.1 (+https://docs.belfortagent.xyz/crawler/; support@belfortagent.xyz)
```

## How it behaves

- **It follows robots.txt.** It obeys the rules for `BelfortBot`, and the rules for `*` when there is no entry for it. A path that is disallowed is not fetched.
- **It follows one redirect of robots.txt, and no more.** If your robots.txt redirects to `https://<host>/robots.txt` on your own company domain, or on the same site as the host that redirected, it reads that file and applies its rules to the page it was reading. A redirect to anywhere else, a second redirect, and a robots.txt it cannot read are all taken to mean "do not fetch".
- **It is slow.** At most one request every one and a half seconds to any one host, one at a time, with a twenty second timeout.
- **It stays on the company's own pages.** Every request, the first and each redirect, goes over https to the company's own domain or its investor-relations platform's domains, and only to public internet addresses. A link to any other host is not followed.
- **It reads public files only.** No login, no form, no script. A page that needs one is skipped.
- **It reads the statements and nothing else.** It keeps each file as retrieved, with a hash, so that a figure can be checked against the file it came from.

## Asking us to stop

If you run one of these sites and want us to stop, or want a company's figures removed:

1. Email **support@belfortagent.xyz** from an address on the company's domain, or tell us which company you speak for. Say which company and what you want: stop reading the site, remove the figures, or both.
2. We remove the figures and stop reading the site within two business days, and we reply when it is done.

You can also add a rule for `BelfortBot` to your `robots.txt`. We read it before each visit, so it takes effect on the next one.

## What we publish

Each Thai figure the API serves links to the company's own file, so a company can always see what we read and where. See [Thai company figures](/datasets/th-financials/).

---
Source: https://docs.belfortagent.xyz/crawler/
