JobwaechterBot – our crawler
This page is for anyone who sees JobwaechterBot in their server logs: what it reads, how often it visits and how to block it.
- User agent
- JobwaechterBot
- Rate
- at most 1 request per second per domain
- robots.txt
- respected before every request
Who we are
Jobwächter is a service for job seekers in Germany. Users describe the job they want, and we send them matching new jobs by email or Telegram. We take the jobs directly from company career pages, not from job boards. Every match links to the original listing.
What the crawler reads
- the domain's robots.txt
- the career page and the job listings linked from it
- public job interfaces of common applicant tracking systems, e.g. Personio, Greenhouse or Lever
- structured job data (schema.org JobPosting) in the pages
The crawler only fetches publicly accessible pages. It does not run JavaScript, never logs in and reads at most 50 listing pages per run.
We store title, location, work mode, employment type, link and the listing text. The text is only used to match search alerts; it is never published and is deleted 30 days after the listing disappears.
Going easy on the server
- At most one request per second per domain.
- A Crawl-delay in robots.txt is respected, up to 60 seconds between two requests.
- Conditional requests with ETag and Last-Modified: unchanged pages are not transferred again.
- Each company is usually checked once a day.
- If robots.txt returns a server error, the crawler does not fetch the domain. If it is missing (404), the domain is treated as allowed.
Blocking the crawler
The crawler follows the rules for the name JobwaechterBot in robots.txt.
User-agent: JobwaechterBot
Disallow: /User-agent: JobwaechterBot
Disallow: /intern/The block applies from the next run. The crawler no longer fetches pages excluded by robots.txt.
Questions or requests
Should a career page not be read at all, or read differently, or did you spot a mistake? Write to us, with the domain and ideally an excerpt from your log.
[KONTAKT-E-MAIL]