The Civiceye collection robot

Civiceye makes the decisions, projects and finances of Swiss municipalities easier to follow. The information comes from the municipalities' official websites, collected by a robot that identifies itself as:

Civiceye/0.1 (+https://www.civiceye.ch/bot/; crawler@civiceye.ch)

What it reads

• The public pages of municipalities that list their proposals and reports, and the documents published on them (PDF). No protected pages, no forms, no logins.
• The federal geodata of geo.admin.ch (cadastral survey, addresses, place names), to locate on a map the places mentioned in the documents.
• The list of municipalities of the Federal Statistical Office and the municipalities' official websites recorded in Wikidata: one request to each of these two open data interfaces, the municipal boundaries published by the FSO on data.geo.admin.ch and its municipal statistics on stats.swiss (population, housing, vehicles…), for the All municipalities page and its map.
• Each municipality's home page, once, to find out what its website is built with and where the council's pages are: this is how Civiceye chooses the municipalities to add next. The websites of one hosting company are read one after the other, never at the same time.

How it behaves

• It respects each website's robots.txt, except for the exceptions described below.
• It makes one request at a time, at most one per second; at the pace the website's robots.txt asks for, if it gives one (Crawl-delay); and at most one every 10 seconds for what robots.txt forbids, in the exceptions described below.
• Each document is downloaded only once and kept. Documents already known are checked only about once a month, old documents about once a year, with conditional requests (If-None-Match, If-Modified-Since): an unchanged document is not downloaded again. When a page of the municipality changes, its documents are checked straight away.
• Collections are started by hand, one municipality at a time.

Exceptions to robots.txt

The rules written for all robots (User-agent: *) are not followed in the cases below, each for a specific reason. Even then, rules naming Civiceye are always followed, and the robot stops as soon as the server refuses a request (codes 401, 403 or 429): it does not insist and never changes the way it identifies itself.

• Orbe (orbe.ch): its robots.txt forbids all robots the /images/ folder, where all the official documents are. These documents are public and not protected by copyright (art. 5 of the Copyright Act). The robot downloads them as a person would: one document at a time, at most one every 10 seconds, each document only once. If the municipality objects, an e-mail to the address below or the lines User-agent: Civiceye / Disallow: /images/ are enough: the collection stops and the documents collected are removed.
• Pohlern (pohlern.ch), Sigriswil (sigriswil.ch), Rheinwald (rheinwald.ch) and Le Locle (lelocle.ch): their robots.txt forbids the whole website to all robots (Le Locle lets in search engines only). The robot reads only their homepage, to know what the website is made with, once, 10 seconds after robots.txt. If the municipality objects, an e-mail to the address below or the lines User-agent: Civiceye / Disallow: / are enough: the homepage is not read again and the copy kept is removed.
• Open data interfaces of the Federal Statistical Office (/api) and of Wikidata (/sparql): their robots.txt keeps search engines out, but they are published to be queried by programs. One request to each, respecting their delay.

What happens to the documents

The documents are archived as they are. Their text is extracted (with character recognition for scanned documents); summaries, amounts and places are drawn from them automatically, then published on this website. Each item says whether it was found word for word in the document, and each page links to the original document on the municipality's website.

Blocking or limiting the robot

To exclude it from all or part of a website, add to its robots.txt:

User-agent: Civiceye
Disallow: /

For any question, a request for correction or removal, or if the robot causes your server a problem, write to crawler@civiceye.ch.