Skip to content
Jobwächter

JobwaechterBot – our crawler

This page is for anyone who sees JobwaechterBot in their server logs: what it reads, how often it visits and how to block it.

User agent
JobwaechterBot
Rate
at most 1 request per second per domain
robots.txt
respected before every request

Who we are

Jobwächter is a service for job seekers in Germany. Users describe the job they want, and we send them matching new jobs by email or Telegram. We take the jobs directly from company career pages, not from job boards. Every match links to the original listing.

What the crawler reads

  • the domain's robots.txt
  • the career page and the job listings linked from it
  • public job interfaces of common applicant tracking systems, e.g. Personio, Greenhouse or Lever
  • structured job data (schema.org JobPosting) in the pages

The crawler only fetches publicly accessible pages. It does not run JavaScript, never logs in and reads at most 50 listing pages per run.

We store title, location, work mode, employment type, link and the listing text. The text is only used to match search alerts; it is never published and is deleted 30 days after the listing disappears.

Going easy on the server

  • At most one request per second per domain.
  • A Crawl-delay in robots.txt is respected, up to 60 seconds between two requests.
  • Conditional requests with ETag and Last-Modified: unchanged pages are not transferred again.
  • Each company is usually checked once a day.
  • If robots.txt returns a server error, the crawler does not fetch the domain. If it is missing (404), the domain is treated as allowed.

Blocking the crawler

The crawler follows the rules for the name JobwaechterBot in robots.txt.

Block the whole domain:
User-agent: JobwaechterBot
Disallow: /
Block only one section:
User-agent: JobwaechterBot
Disallow: /intern/

The block applies from the next run. The crawler no longer fetches pages excluded by robots.txt.

Questions or requests

Should a career page not be read at all, or read differently, or did you spot a mistake? Write to us, with the domain and ideally an excerpt from your log.

[KONTAKT-E-MAIL]