wohnbot/README.md

# wohn-bot

A Python bot that monitors Berlin's public housing portal (inberlinwohnen.de) and WG rooms (wgcompany.de). Sends Telegram notifications when new listings appear and can automatically apply to some listings.

## What it does

- **Monitors** inberlinwohnen.de for new apartment listings from 6 housing companies (HOWOGE, Gewobag, Degewo, Gesobau, Stadt und Land, WBM)
- **Monitors** wgcompany.de for WG room listings with configurable filters
- **Notifies** via Telegram with rich listing details and application status
- **Logs** listing times to CSV for pattern analysis and visualization
- **Auto-applies** to new listings when autopilot is enabled (all 6 companies supported)
- **Generates** weekly listing pattern plots and autopilot performance analytics
- **Autocleans** debug files older than 48 hours to manage disk space
- **Tracks** application history with success/failure reasons in JSON

## Auto-Apply Support

All six housing companies monitored by this bot now support the autopilot (automatic application) feature. Use autopilot with care — automatic form submission is destructive and may send many requests if configured incorrectly.

| Company | Status | Notes |
|---------|--------|-------|
| HOWOGE | Working | Fully automated and tested |
| Degewo | Working | Uses Wohnungshelden portal; automated |
| Stadt und Land | Working | Embedded form handled automatically |
| Gewobag | Working | Wohnungshelden iframe handled automatically |
| Gesobau | Working | Automated form submission implemented |
| WBM | Working | Automated form submission implemented |
| WGcompany | Monitoring only | WGcompany monitoring only (no autopilot) |

Recommended precautions:

- Run with `/autopilot off` while testing new selectors or after changing config.
- Inspect `data/applications.json` and saved screenshots in `data/` after enabling autopilot.
- Respect site terms of use and rate limits; set `CHECK_INTERVAL` appropriately.

## Setup

### Docker (recommended)

```bash
cp .env.example .env
# Edit .env with your credentials
docker compose up -d
```

### Local development

```bash
pip install -r requirements.txt
playwright install chromium

export TELEGRAM_BOT_TOKEN=your_token
export TELEGRAM_CHAT_ID=your_chat_id
# ... other env vars (see .env.example)

python main.py
```

### Helper Scripts

The `helper_functions/` directory contains utilities for merging data from multiple machines:

- `merge_listing_times.py` - Merge listing_times.csv files
- `merge_applications.py` - Merge applications.json files
- `merge_dict_json.py` - Merge listings.json and wgcompany_listings.json
- `merge_wgcompany_times.py` - Merge wgcompany_times.csv files

All scripts deduplicate by key and timestamp.

## Configuration

### Required environment variables

- `TELEGRAM_BOT_TOKEN` - Bot token from @BotFather
- `TELEGRAM_CHAT_ID` - Your Telegram chat ID

### InBerlin login (required for auto-apply)

- `INBERLIN_EMAIL` - Your inberlinwohnen.de email
- `INBERLIN_PASSWORD` - Your inberlinwohnen.de password

### Form data (for auto-apply)

- `FORM_ANREDE` - Salutation (Herr/Frau)
- `FORM_VORNAME` - First name
- `FORM_NACHNAME` - Last name
- `FORM_EMAIL` - Email address
- `FORM_PHONE` - Phone number
- `FORM_STRASSE` - Street name
- `FORM_HAUSNUMMER` - House number
- `FORM_PLZ` - Postal code
- `FORM_ORT` - City
- `FORM_PERSONS` - Number of persons in household
- `FORM_ADULTS` - Number of adults (for GEWOBAG forms, defaults to 1)
- `FORM_CHILDREN` - Number of children (defaults to 0)
- `FORM_INCOME` - Monthly net income

### WGcompany filters

- `WGCOMPANY_ENABLED` - Enable WGcompany monitoring (true/false)
- `WGCOMPANY_MIN_SIZE` - Minimum room size in sqm
- `WGCOMPANY_MAX_SIZE` - Maximum room size in sqm
- `WGCOMPANY_MIN_PRICE` - Minimum price in EUR
- `WGCOMPANY_MAX_PRICE` - Maximum price in EUR
- `WGCOMPANY_BEZIRK` - District filter (optional)

## Telegram Commands

- `/autopilot on|off` - Enable or disable automatic applications
- `/status` - Show current status and statistics (autopilot state, application counts by company)
- `/plot` - Generate and send a weekly listing-patterns plot with heatmap and charts (high-res, seaborn-styled)
- `/errorrate` - Generate and send an autopilot performance analysis with success/failure rates by company (high-res, seaborn-styled)
- `/retryfailed` - Retry all previously failed applications
- `/resetlistings` - Reset seen listings (marks all current as failed to avoid spam)
- `/help` - Show available commands and usage information

**Important:** The bot only processes commands from the configured `TELEGRAM_CHAT_ID`. Use `/autopilot off` while testing selector changes or after modifying configuration to avoid accidental submissions.

**Plot Features:** All plots are generated at 300 DPI with seaborn styling for publication-quality output.

## Data files

All data is stored in the `data/` directory:

**Persistent State:**

- `listings.json` - Previously seen inberlinwohnen listings (deduplicated by hash)
- `wgcompany_listings.json` - Previously seen WGcompany listings (deduplicated by hash)
- `applications.json` - Application history with timestamps, success/failure status, and error messages
- `listing_times.csv` - Time series data for inberlinwohnen listings (for pattern analysis)
- `wgcompany_times.csv` - Time series data for WGcompany listings
- `state.json` - Runtime state (autopilot toggle, persistent across restarts)
- `monitor.log` - Rotating application logs (max 5MB, 5 backups)

**Generated Plots:**

- `weekly_plot.png` - Weekly listing patterns (heatmap + charts, 300 DPI)
- `error_rate.png` - Autopilot performance analysis (3-panel chart, 300 DPI)

**Debug Files (auto-cleaned after 48 hours):**

- `data/<company>/*.png` - Screenshots from failed applications
- `data/<company>/*.html` - Page HTML snapshots for debugging
- `data/debug_page.html` - InBerlin page snapshot
- `data/wgcompany_debug.html` - WGcompany page snapshot

**Note:** Debug files (screenshots, HTML) are automatically deleted after 48 hours to save disk space. Listing data, applications, and logs are never deleted.

## Debugging

When applications fail, the bot saves debug material to help diagnose issues:

**Company-specific folders:**

- `data/howoge/` - Howoge screenshots and HTML
- `data/gewobag/` - Gewobag screenshots and HTML
- `data/degewo/` - Degewo screenshots and HTML
- `data/gesobau/` - Gesobau screenshots and HTML
- `data/stadtundland/` - Stadt und Land screenshots and HTML
- `data/wbm/` - WBM screenshots and HTML

**General debug files:**

- `data/debug_page.html` - InBerlin page snapshot
- `data/wgcompany_debug.html` - WGcompany page snapshot

Check `applications.json` for error messages and timestamps. Debug files are automatically cleaned after 48 hours but can be manually inspected while fresh.

## Code Structure

The bot has been modularized for better maintainability. The main components are:

**Core:**

- `main.py` - Entry point, orchestrates monitoring loop and autoclean
- `application_handler.py` - Delegates applications to company handlers, generates plots
- `telegram_bot.py` - Async Telegram bot with httpx for commands and notifications
- `state_manager.py` - Manages persistent state (autopilot toggle)
- `autoclean_debug.py` - Deletes debug files older than 48 hours

**Handlers:**

- `handlers/base_handler.py` - Abstract base class with shared functionality (cookie handling, consent, logging)
- `handlers/howoge_handler.py` - HOWOGE application automation
- `handlers/gewobag_handler.py` - Gewobag application automation
- `handlers/degewo_handler.py` - Degewo application automation (Wohnungshelden)
- `handlers/gesobau_handler.py` - Gesobau application automation
- `handlers/stadtundland_handler.py` - Stadt und Land application automation
- `handlers/wbm_handler.py` - WBM application automation
- `handlers/wgcompany_notifier.py` - WGcompany monitoring (notification only, no autopilot)

**Utilities:**

- `helper_functions/` - Data merge utilities for combining stats from multiple sources
  - `merge_listing_times.py`
  - `merge_applications.py`
  - `merge_dict_json.py`
  - `merge_wgcompany_times.py`

**Tests:**

- `tests/` - Comprehensive unit tests (48 tests total)
  - `test_telegram_bot.py` - Telegram bot commands and messaging
  - `test_error_rate_plot.py` - Plot generation
  - `test_wgcompany_notifier.py` - WGcompany monitoring
  - `test_handlers.py` - Handler initialization
  - `test_application_handler.py` - Application orchestration
  - `test_helper_functions.py` - Merge utilities
  - `test_autoclean.py` - Autoclean script validation

## Unit Tests

The project includes comprehensive unit tests (48 tests total) to ensure functionality and reliability:

- `test_telegram_bot.py` - Telegram bot commands and messaging (13 tests)
- `test_error_rate_plot.py` - Plot generation and data analysis (2 tests)
- `test_wgcompany_notifier.py` - WGcompany monitoring (7 tests)
- `test_handlers.py` - Handler initialization and structure (6 tests)
- `test_application_handler.py` - Application orchestration (10 tests)
- `test_company_detection.py` - Company detection from URLs (6 tests)
- `test_state_manager.py` - State persistence (2 tests)
- `test_helper_functions.py` - Merge utilities (2 tests)
- `test_autoclean.py` - Autoclean script validation (1 test)

### Running Tests

```bash
pytest tests/ -v
```

All tests use mocking to avoid external dependencies and can run offline.

## Workflow Diagram

```mermaid
flowchart TD
    Start([Start Bot]) --> Init[Initialize Browser & Telegram Bot]
    Init --> Loop{Main Loop}
    
    %% InBerlin Monitoring
    Loop --> InBerlin[Fetch InBerlin Listings]
    InBerlin --> ParseIB[Parse & Hash Listings]
    ParseIB --> LoadIB[Load Previous InBerlin Listings]
    LoadIB --> DedupeIB{New InBerlin Listings?}
    
    DedupeIB -- Yes --> LogIB[Log to listing_times.csv]
    LogIB --> SaveIB[Save to listings.json]
    DedupeIB -- No --> WG
    
    %% WGcompany Monitoring
    SaveIB --> WG[Fetch WGcompany Listings]
    WG --> ParseWG[Parse & Hash Listings]
    ParseWG --> LoadWG[Load Previous WGcompany Listings]
    LoadWG --> DedupeWG{New WGcompany Listings?}
    
    DedupeWG -- Yes --> LogWG[Log to wgcompany_times.csv]
    LogWG --> SaveWG[Save to wgcompany_listings.json]
    DedupeWG -- No --> CheckAutopilot
    
    %% Autopilot Decision
    SaveWG --> CheckAutopilot{Autopilot Enabled?}
    SaveIB --> CheckAutopilot
    
    CheckAutopilot -- Off --> NotifyOnly[Send Telegram Notifications]
    NotifyOnly --> CheckClean
    
    CheckAutopilot -- On --> CheckApplied{Already Applied?}
    CheckApplied -- Yes --> Skip[Skip Listing]
    CheckApplied -- No --> DetectCompany[Detect Company]
    
    %% Application Flow
    DetectCompany --> SelectHandler[Select Handler]
    SelectHandler --> OpenPage[Open Listing Page]
    OpenPage --> Check404{404 or Deactivated?}
    
    Check404 -- Yes --> MarkPermanent[Mark deactivated]
    MarkPermanent --> SaveFail[Save to applications.json]
    SaveFail --> NotifyFail[Notify: Application Failed]
    
    Check404 -- No --> HandleCookies[Handle Cookie Banners]
    HandleCookies --> FindButton[Find Application Button]
    FindButton --> ButtonFound{Button Found?}
    
    ButtonFound -- No --> Screenshot1[Save Screenshot & HTML]
    Screenshot1 --> SaveFail
    
    ButtonFound -- Yes --> ClickButton[Click Application Button]
    ClickButton --> MultiStep{Multi-Step Form?}
    
    MultiStep -- Yes --> NavigateSteps[Navigate Form Steps]
    NavigateSteps --> FillForm
    MultiStep -- No --> FillForm[Fill Form Fields]
    
    FillForm --> SubmitForm[Submit Application]
    SubmitForm --> CheckConfirm{Confirmation Detected?}
    
    CheckConfirm -- Yes --> SaveSuccess[Save success to applications.json]
    SaveSuccess --> NotifySuccess[Notify: Application Success]
    
    CheckConfirm -- No --> Screenshot2[Save Screenshot & HTML]
    Screenshot2 --> SaveFail
    
    NotifySuccess --> CheckClean
    NotifyFail --> CheckClean
    Skip --> CheckClean
    
    %% Autoclean
    CheckClean{Time for Autoclean?}
    CheckClean -- Yes --> RunClean[Delete Debug Files >48h]
    RunClean --> Sleep
    CheckClean -- No --> Sleep[Sleep CHECK_INTERVAL]
    
    Sleep --> TelegramCmd{Telegram Command?}
    TelegramCmd -- /autopilot --> ToggleAutopilot[Toggle Autopilot State]
    TelegramCmd -- /status --> ShowStatus[Show Status & Stats]
    TelegramCmd -- /plot --> GenPlot[Generate Weekly Plot]
    TelegramCmd -- /errorrate --> GenError[Generate Error Rate Plot]
    TelegramCmd -- /retryfailed --> RetryFailed[Retry Failed Applications]
    TelegramCmd -- /resetlistings --> ResetListings[Reset Seen Listings]
    TelegramCmd -- /help --> ShowHelp[Show Help]
    TelegramCmd -- None --> Loop
    
    ToggleAutopilot --> Loop
    ShowStatus --> Loop
    GenPlot --> Loop
    GenError --> Loop
    RetryFailed --> Loop
    ResetListings --> Loop
    ShowHelp --> Loop
    
    style Start fill:#90EE90
    style SaveSuccess fill:#90EE90
    style SaveFail fill:#FFB6C1
    style MarkPermanent fill:#FFB6C1
    style RunClean fill:#87CEEB
    style CheckAutopilot fill:#FFD700
    style Check404 fill:#FFD700
    style ButtonFound fill:#FFD700
    style CheckConfirm fill:#FFD700
```

**Key Features:**

- **Dual Monitoring**: Tracks both InBerlin (6 companies) and WGcompany listings
- **Smart Deduplication**: MD5 hashing prevents duplicate notifications
- **Autopilot**: Automated application with company-specific handlers
- **Error Handling**: 404 detection, permanent fail tracking, debug screenshots
- **Autoclean**: Automatic cleanup of debug files every 48 hours
- **Rich Commands**: Status, plots, retry failed, reset listings
- **High-Res Analytics**: 300 DPI seaborn-styled plots for pattern analysis

## License

This project is licensed under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) License.

You are free to:

- **Share** — copy and redistribute the material in any medium or format
- **Adapt** — remix, transform, and build upon the material

Under the following terms:

- **Attribution** — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
- **NonCommercial** — You may not use the material for commercial purposes.

For more details, see the [full license text](https://creativecommons.org/licenses/by-nc/4.0/).