Compare commits

...

2 commits

Author SHA1 Message Date
a77a0c0393 major refactor (untested) 2025-12-27 11:59:04 +01:00
a29412b4da msg redesign 2025-12-25 20:53:10 +01:00
22 changed files with 1069 additions and 23 deletions

View file

@ -1,4 +1,4 @@
# Copilot Instructions for inberlin-monitor
# Copilot Instructions for wohn-bot
## Project Overview
@ -6,22 +6,29 @@ A Python-based apartment monitoring bot for Berlin's public housing portal (inbe
## Architecture
**Single-file monolith** (`monitor.py`, ~1600 lines) with five main classes:
- `InBerlinMonitor` - Core scraping/monitoring loop for inberlinwohnen.de, login handling, listing detection
- `WGCompanyMonitor` - Monitors wgcompany.de WG rooms with configurable search filters
- `ApplicationHandler` - Company-specific form automation (each `_apply_*` method handles one housing company)
- `TelegramBot` - Command handling via long-polling in a daemon thread
- Main loop runs synchronous with `asyncio.get_event_loop().run_until_complete()` for Playwright calls
**Modularized structure** with the following key components:
**Data flow**: Fetch listings → Compare with `listings.json` / `wgcompany_listings.json` → Detect new → Log to CSV → Auto-apply if autopilot enabled (inberlin only) → Save to `applications.json` → Send Telegram notification
- `main.py`: Entry point for the bot.
- `handlers/`: Contains company-specific handlers for auto-apply functionality. Each handler is responsible for automating the application process for a specific housing company. Includes:
- `howoge_handler.py`
- `gewobag_handler.py`
- `degewo_handler.py`
- `gesobau_handler.py`
- `stadtundland_handler.py`
- `wbm_handler.py`
- `base_handler.py`: Provides shared functionality for all handlers.
- `application_handler.py`: Delegates application tasks to the appropriate handler based on the company.
- `telegram_bot.py`: Handles Telegram bot commands and notifications.
**Data flow**: Fetch listings → Compare with `listings.json` / `wgcompany_listings.json` → Detect new → Log to CSV → Auto-apply if autopilot enabled → Save to `applications.json` → Send Telegram notification.
## Key Patterns
### Company-specific handlers
Each housing company has a dedicated `_apply_{company}()` method in `ApplicationHandler`. When adding support for a new company:
1. Add detection in `_detect_company()` (line ~350)
2. Add handler call in `apply()` switch (line ~330)
3. Implement `_apply_newcompany()` following existing patterns (cookie dismiss → find button → fill form → submit → screenshot)
Each housing company has a dedicated handler in the `handlers/` directory. When adding support for a new company:
1. Create a new handler file in `handlers/` (e.g., `newcompany_handler.py`).
2. Implement the handler by extending `BaseHandler` and overriding necessary methods.
3. Update `application_handler.py` to include the new handler in the `handlers` dictionary.
### Listing identification
Listings are hashed by `md5(key_fields)[:12]` to generate stable IDs:
@ -39,13 +46,13 @@ Listings are hashed by `md5(key_fields)[:12]` to generate stable IDs:
### Run locally
```bash
# Install deps (requires Playwright)
# Install dependencies (requires Playwright)
pip install -r requirements.txt
playwright install chromium
# Set env vars and run
export TELEGRAM_BOT_TOKEN=... TELEGRAM_CHAT_ID=...
python monitor.py
python main.py
```
### Docker (production)
@ -70,12 +77,30 @@ WGcompany: `WGCOMPANY_ENABLED`, `WGCOMPANY_MIN_SIZE`, `WGCOMPANY_MAX_SIZE`, `WGC
## Common Tasks
### Fix a broken company handler
Check `data/*_nobtn_*.png` screenshots and `data/debug_page.html` to see actual page structure. Update selectors in the corresponding `_apply_{company}()` method.
Check `data/*_nobtn_*.png` screenshots and `data/debug_page.html` to see actual page structure. Update selectors in the corresponding handler file in `handlers/`.
### Add Telegram command
1. Add case in `TelegramBot._handle_update()` (line ~95)
2. Implement `_handle_{command}_command()` method
1. Add a case in `TelegramBot._handle_update()`.
2. Implement the corresponding `_handle_{command}_command()` method.
### Modify listing extraction
- InBerlin: Update regex patterns in `InBerlinMonitor.fetch_listings()`. Test against `data/debug_page.html`.
- WGcompany: Update parsing in `WGCompanyMonitor.fetch_listings()`. Test against `data/wgcompany_debug.html`.
## Unit Tests
### Overview
The project includes unit tests to ensure functionality and reliability. Key test files:
- `tests/test_telegram_bot.py`: Tests the Telegram bot's commands and messaging functionality.
- `tests/test_error_rate_plot.py`: Tests the error rate plot generator for autopilot applications.
### Running Tests
To run the tests, use:
```bash
pytest tests/
```
Ensure all dependencies are installed and the environment is configured correctly before running the tests.

View file

@ -7,9 +7,12 @@ COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
# Copy application
COPY monitor.py .
COPY main.py .
# Copy the handlers directory into the Docker image
COPY handlers/ ./handlers/
# Create data directory
RUN mkdir -p /data && chmod 777 /data
CMD ["python", "-u", "monitor.py"]
CMD ["python", "-u", "main.py"]

93
LICENSE Normal file
View file

@ -0,0 +1,93 @@
Creative Commons Attribution-NonCommercial 4.0 International Public License
By exercising the Licensed Rights (defined below), You accept and agree to be bound by the terms and conditions of this Creative Commons Attribution-NonCommercial 4.0 International Public License ("Public License"). To the extent this Public License may be interpreted as a contract, You are granted the Licensed Rights in consideration of Your acceptance of these terms and conditions, and the Licensor grants You such rights in consideration of benefits the Licensor receives from making the Licensed Material available under these terms and conditions.
**Section 1 Definitions.**
- **Adapted Material** means material subject to Copyright and Similar Rights that is derived from or based upon the Licensed Material and in which the Licensed Material is translated, altered, arranged, transformed, or otherwise modified in a manner requiring permission under the Copyright and Similar Rights held by the Licensor. For purposes of this Public License, where the Licensed Material is a musical work, performance, or sound recording, Adapted Material is always produced where the Licensed Material is synched in timed relation with a moving image.
- **Copyright and Similar Rights** means copyright and/or similar rights closely related to copyright including, without limitation, performance, broadcast, sound recording, and Sui Generis Database Rights, without regard to how the rights are labeled or categorized. For purposes of this Public License, the rights specified in Section 2(b)(1)-(2) are not Copyright and Similar Rights.
- **Effective Technological Measures** means those measures that, in the absence of proper authority, may not be circumvented under laws fulfilling obligations under Article 11 of the WIPO Copyright Treaty adopted on December 20, 1996, and/or similar international agreements.
- **Exceptions and Limitations** means fair use, fair dealing, and/or any other exception or limitation to Copyright and Similar Rights that applies to Your use of the Licensed Material.
- **Licensed Material** means the artistic or literary work, database, or other material to which the Licensor applied this Public License.
- **Licensed Rights** means the rights granted to You subject to the terms and conditions of this Public License, which are limited to all Copyright and Similar Rights that apply to Your use of the Licensed Material and that the Licensor has authority to license.
- **Licensor** means the individual(s) or entity(ies) granting rights under this Public License.
- **NonCommercial** means not primarily intended for or directed towards commercial advantage or monetary compensation. For purposes of this Public License, the exchange of the Licensed Material for other material subject to Copyright and Similar Rights by digital file-sharing or similar means is NonCommercial provided there is no payment of monetary compensation in connection with the exchange.
- **Share** means to provide material to the public by any means or process that requires permission under the Licensed Rights, such as reproduction, public display, public performance, distribution, dissemination, communication, or importation, and to make material available to the public including in ways that members of the public may access the material from a place and at a time individually chosen by them.
- **Sui Generis Database Rights** means rights other than copyright resulting from Directive 96/9/EC of the European Parliament and of the Council of 11 March 1996 on the legal protection of databases, as amended and/or succeeded, as well as other essentially equivalent rights anywhere in the world.
- **You** means the individual or entity exercising the Licensed Rights under this Public License. Your has a corresponding meaning.
**Section 2 Scope.**
- **License grant.**
1. Subject to the terms and conditions of this Public License, the Licensor hereby grants You a worldwide, royalty-free, non-sublicensable, non-exclusive, irrevocable license to exercise the Licensed Rights in the Licensed Material to:
- reproduce and Share the Licensed Material, in whole or in part, for NonCommercial purposes only; and
- produce, reproduce, and Share Adapted Material for NonCommercial purposes only.
2. Exceptions and Limitations. For the avoidance of doubt, where Exceptions and Limitations apply to Your use, this Public License does not apply, and You do not need to comply with its terms and conditions.
3. Term. The term of this Public License is specified in Section 6(a).
4. Media and formats; technical modifications allowed. The Licensor authorizes You to exercise the Licensed Rights in all media and formats whether now known or hereafter created, and to make technical modifications necessary to do so. The Licensor waives and/or agrees not to assert any right or authority to forbid You from making technical modifications necessary to exercise the Licensed Rights, including technical modifications necessary to circumvent Effective Technological Measures. For purposes of this Public License, simply making modifications authorized by this Section 2(a)(4) never produces Adapted Material.
5. Downstream recipients.
- Offer from the Licensor Licensed Material. Every recipient of the Licensed Material automatically receives an offer from the Licensor to exercise the Licensed Rights under the terms and conditions of this Public License.
- No downstream restrictions. You may not offer or impose any additional or different terms or conditions on, or apply any Effective Technological Measures to, the Licensed Material if doing so restricts exercise of the Licensed Rights by any recipient of the Licensed Material.
6. No endorsement. Nothing in this Public License constitutes or may be construed as permission to assert or imply that You are, or that Your use of the Licensed Material is, connected with, or sponsored, endorsed, or granted official status by, the Licensor or others designated to receive attribution as provided in Section 3(a)(1)(A)(i).
- **Other rights.**
1. Moral rights, such as the right of integrity, are not licensed under this Public License, nor are publicity, privacy, and/or other similar personality rights; however, to the extent possible, the Licensor waives any such rights held by the Licensor to the limited extent necessary to allow You to exercise the Licensed Rights, but not otherwise.
2. Patent and trademark rights are not licensed under this Public License.
3. To the extent possible, the Licensor waives any right to collect royalties from You for the exercise of the Licensed Rights, whether directly or through a collecting society under any voluntary or waivable statutory or compulsory licensing scheme. In all other cases, the Licensor expressly reserves any right to collect such royalties.
**Section 3 License Conditions.**
Your exercise of the Licensed Rights is expressly made subject to the following conditions.
- **Attribution.**
1. If You Share the Licensed Material (including in modified form), You must:
- retain the following if it is supplied by the Licensor with the Licensed Material:
- identification of the creator(s) of the Licensed Material and any others designated to receive attribution, in any reasonable manner requested by the Licensor (including by pseudonym if designated);
- a copyright notice;
- a notice that refers to this Public License;
- a notice that refers to the disclaimer of warranties;
- a URI or hyperlink to the Licensed Material to the extent reasonably practicable;
- indicate if You modified the Licensed Material and retain an indication of any previous modifications; and
- indicate the Licensed Material is licensed under this Public License, and include the text of, or the URI or hyperlink to, this Public License.
2. You may satisfy the conditions in Section 3(a)(1) in any reasonable manner based on the medium, means, and context in which You Share the Licensed Material. For example, it may be reasonable to satisfy the conditions by providing a URI or hyperlink to a resource that includes the required information.
3. If requested by the Licensor, You must remove any of the information required by Section 3(a)(1)(A) to the extent reasonably practicable.
- **NonCommercial.** You may not exercise the Licensed Rights for commercial purposes.
**Section 4 Sui Generis Database Rights.**
Where the Licensed Rights include Sui Generis Database Rights that apply to Your use of the Licensed Material:
- for the avoidance of doubt, Section 2(a)(1) grants You the right to extract, reuse, reproduce, and Share all or a substantial portion of the contents of the database for NonCommercial purposes only;
- if You include all or a substantial portion of the database contents in a database that is Adapted Material, then the database in which You include the contents may only be Shared under the terms of this Public License; and
- You must comply with the conditions in Section 3(a) if You Share all or a substantial portion of the contents of the database.
For the avoidance of doubt, this Section 4 supplements and does not replace Your obligations under this Public License where the Licensed Rights include other Copyright and Similar Rights.
**Section 5 Disclaimer of Warranties and Limitation of Liability.**
- Unless otherwise separately undertaken by the Licensor, to the extent possible, the Licensor offers the Licensed Material as-is and as-available, and makes no representations or warranties of any kind concerning the Licensed Material, whether express, implied, statutory, or other. This includes, without limitation, warranties of title, merchantability, fitness for a particular purpose, non-infringement, absence of latent or other defects, accuracy, or the presence or absence of errors, whether or not known or discoverable. Where disclaimers of warranties are not allowed in full or in part, this disclaimer may not apply to You.
- To the extent possible, in no event will the Licensor be liable to You on any legal theory (including, without limitation, negligence) or otherwise for any direct, special, indirect, incidental, consequential, punitive, exemplary, or other losses, costs, expenses, or damages arising out of this Public License or use of the Licensed Material, even if the Licensor has been advised of the possibility of such losses, costs, expenses, or damages. Where a limitation of liability is not allowed in full or in part, this limitation may not apply to You.
- The disclaimer of warranties and limitation of liability provided above shall be interpreted in a manner that, to the extent possible, most closely approximates an absolute disclaimer and waiver of all liability.
**Section 6 Term and Termination.**
- This Public License applies for the term of the Copyright and Similar Rights licensed here. However, if You fail to comply with this Public License, then Your rights under this Public License terminate automatically.
- Where Your right to use the Licensed Material has terminated under Section 6(a), it reinstates:
1. automatically as of the date the violation is cured, provided it is cured within 30 days of Your discovery of the violation; or
2. upon express reinstatement by the Licensor.
For the avoidance of doubt, this Section 6(b) does not affect any right the Licensor may have to seek remedies for Your violations of this Public License.
- For the avoidance of doubt, the Licensor may also offer the Licensed Material under separate terms or conditions or stop distributing the Licensed Material at any time; however, doing so will not terminate this Public License.
- Sections 1, 5, 6, 7, and 8 survive termination of this Public License.
**Section 7 Other Terms and Conditions.**
- The Licensor shall not be bound by any additional or different terms or conditions communicated by You unless expressly agreed.
- Any arrangements, understandings, or agreements regarding the Licensed Material not stated herein are separate from and independent of the terms and conditions of this Public License.
**Section 8 Interpretation.**
- For the avoidance of doubt, this Public License does not, and shall not be interpreted to, reduce, limit, restrict, or impose conditions on any use of the Licensed Material that could lawfully be made without permission under this Public License.
- To the extent possible, if any provision of this Public License is deemed unenforceable, it shall be automatically reformed to the minimum extent necessary to make it enforceable. If the provision cannot be reformed, it shall be severed from this Public License without affecting the enforceability of the remaining terms and conditions.
- No term or condition of this Public License will be waived and no failure to comply consented to unless expressly agreed to by the Licensor.
- Nothing in this Public License constitutes or may be interpreted as a limitation upon, or waiver of, any privileges and immunities that apply to the Licensor or You, including from the legal processes of any jurisdiction or authority.

View file

@ -119,6 +119,72 @@ When applications fail, the bot saves:
Check these files to understand why an application failed.
## Code Structure
The bot has been modularized for better maintainability. The main components are:
- `main.py`: The entry point for the bot.
- `handlers/`: Contains company-specific handlers for auto-apply functionality. Each company has its own handler file:
- `howoge_handler.py`
- `gewobag_handler.py`
- `degewo_handler.py`
- `gesobau_handler.py`
- `stadtundland_handler.py`
- `wbm_handler.py`
- `application_handler.py`: Orchestrates the application process by delegating to the appropriate handler.
- `telegram_bot.py`: Handles Telegram bot commands and notifications.
The `handlers/` directory includes a `BaseHandler` class that provides shared functionality for all company-specific handlers.
## Unit Tests
The project includes unit tests to ensure functionality and reliability. Key test files:
- `tests/test_telegram_bot.py`: Tests the Telegram bot's commands and messaging functionality.
- `tests/test_error_rate_plot.py`: Tests the error rate plot generator for autopilot applications.
### Running Tests
To run the tests, use:
```bash
pytest tests/
```
Ensure all dependencies are installed and the environment is configured correctly before running the tests.
## Workflow Diagram
```mermaid
graph TD
A[Start] --> B[Fetch Listings]
B --> C{New Listings?}
C -->|Yes| D[Log to CSV]
C -->|Yes| E[Send Telegram Notification]
C -->|Yes| F{Autopilot Enabled?}
F -->|Yes| G[Auto-Apply to Listings]
F -->|No| H[Save to Applications.json]
C -->|No| I[End]
D --> I
E --> I
G --> H
H --> I
```
This diagram illustrates the workflow of the bot, from fetching listings to logging, notifying, and optionally applying to new listings.
## License
MIT
This project is licensed under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) License.
You are free to:
- **Share** — copy and redistribute the material in any medium or format
- **Adapt** — remix, transform, and build upon the material
Under the following terms:
- **Attribution** — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
- **NonCommercial** — You may not use the material for commercial purposes.
For more details, see the [full license text](https://creativecommons.org/licenses/by-nc/4.0/).

51
application_handler.py Normal file
View file

@ -0,0 +1,51 @@
from datetime import datetime
from handlers.base_handler import BaseHandler
from handlers.howoge_handler import HowogeHandler
from handlers.gewobag_handler import GewobagHandler
from handlers.degewo_handler import DegewoHandler
from handlers.gesobau_handler import GesobauHandler
from handlers.stadtundland_handler import StadtUndLandHandler
from handlers.wbm_handler import WBMHandler
class ApplicationHandler:
def __init__(self, browser_context):
self.context = browser_context
self.handlers = {
"howoge": HowogeHandler(browser_context),
"gewobag": GewobagHandler(browser_context),
"degewo": DegewoHandler(browser_context),
"gesobau": GesobauHandler(browser_context),
"stadtundland": StadtUndLandHandler(browser_context),
"wbm": WBMHandler(browser_context),
}
async def apply(self, listing: dict) -> dict:
company = self._detect_company(listing.get("link", ""))
handler = self.handlers.get(company)
result = {
"listing_id": listing.get("id"),
"company": company,
"link": listing.get("link"),
"timestamp": datetime.now().isoformat(),
"success": False,
"message": "",
"address": listing.get("address", ""),
"rooms": listing.get("rooms", ""),
"price": listing.get("price", "")
}
if handler:
result = await handler.apply(listing, result)
else:
result["message"] = f"No handler found for company: {company}"
return result
def _detect_company(self, link: str) -> str:
if "howoge.de" in link: return "howoge"
elif "gewobag.de" in link: return "gewobag"
elif "degewo.de" in link: return "degewo"
elif "gesobau.de" in link: return "gesobau"
elif "stadtundland.de" in link: return "stadtundland"
elif "wbm.de" in link: return "wbm"
return "unknown"

View file

@ -513,10 +513,47 @@ Total listings tracked: {total_listings}
return ""
self._send_message(f"❓ Unknown command: <code>{cmd}</code>\n\nUse /help to see available commands.")
def _send_message(self, text):
def _send_message(self, text, company=None, auto_applied=None, details=None):
"""
Redesigned Telegram message format with fallback to ensure a message is always sent.
Args:
text (str): The main message text.
company (str): The company name (e.g., Howoge, WGCompany).
auto_applied (bool): Whether auto-apply was successful.
details (dict): Additional details about the listing.
"""
try:
# Ensure text is not empty
if not text:
text = "No additional information provided."
# Construct the new message format
title = f"[{company}] " if company else "[Unknown Company] "
title += "✅ Auto-applied!" if auto_applied else "❌ Auto-apply failed."
# Add details if provided
if details:
details_text = "\n".join([
f"🚪 {details.get('rooms', 'N/A')} Zimmer",
f"📐 {details.get('size', 'N/A')}",
f"💰 {details.get('price', 'N/A')}",
f"📍 {details.get('address', 'N/A')}"
])
else:
details_text = ""
# Combine title, details, and additional text
message = f"{title}\n\n{details_text}\n\n{text}".strip()
# Send the message via Telegram
url = f"https://api.telegram.org/bot{TELEGRAM_BOT_TOKEN}/sendMessage"
data = {"chat_id": TELEGRAM_CHAT_ID, "text": text, "parse_mode": "HTML", "disable_web_page_preview": True}
data = {
"chat_id": TELEGRAM_CHAT_ID,
"text": message,
"parse_mode": "HTML",
"disable_web_page_preview": True
}
requests.post(url, data=data)
except Exception as e:
logger.error(f"Failed to send Telegram message: {e}")
@ -569,7 +606,7 @@ class ApplicationHandler:
result["message"] = f"Unknown company: {company}"
logger.warning(f"No application handler for company: {company}")
except Exception as e:
result["message"] = str(e)
result["message"] = f"Error: {str(e)}"
logger.error(f"Application error for {company}: {e}")
import traceback
logger.error(traceback.format_exc())
@ -577,6 +614,11 @@ class ApplicationHandler:
# Log final result
status = "SUCCESS" if result["success"] else "FAILED"
logger.info(f"Application {status} for {listing['address']} ({company}): {result['message']}")
# Include error reason in Telegram message if applicable
if not result["success"] and result["message"]:
result["message"] = f"Application failed: {result['message']}"
return result
def _detect_company(self, link: str) -> str:

1
handlers/__init__.py Normal file
View file

@ -0,0 +1 @@
# This file makes the handlers directory a Python package.

42
handlers/base_handler.py Normal file
View file

@ -0,0 +1,42 @@
from abc import ABC, abstractmethod
from playwright.async_api import Page
import logging
logger = logging.getLogger(__name__)
class BaseHandler(ABC):
def __init__(self, context):
self.context = context
@abstractmethod
async def apply(self, listing: dict, result: dict) -> dict:
"""Abstract method to handle the application process for a specific company."""
pass
async def handle_cookies(self, page: Page):
"""Handle cookie banners if present."""
try:
cookie_btn = await page.query_selector('button:has-text("Akzeptieren"), button:has-text("Alle akzeptieren")')
if cookie_btn and await cookie_btn.is_visible():
await cookie_btn.click()
logger.info("[BaseHandler] Dismissed cookie banner")
await asyncio.sleep(1)
except Exception as e:
logger.warning(f"[BaseHandler] Failed to handle cookies: {e}")
async def handle_consent(self, page: Page):
"""Handle consent manager banners if present."""
try:
consent_selectors = [
'#cmpbntyestxt', '.cmpboxbtnyes', 'a.cmpboxbtn.cmpboxbtnyes',
'#cmpwelcomebtnyes', '.cmptxt_btn_yes'
]
for sel in consent_selectors:
consent_btn = await page.query_selector(sel)
if consent_btn and await consent_btn.is_visible():
await consent_btn.click()
logger.info("[BaseHandler] Dismissed consent manager")
await asyncio.sleep(1)
break
except Exception as e:
logger.warning(f"[BaseHandler] Failed to handle consent manager: {e}")

View file

@ -0,0 +1,59 @@
from .base_handler import BaseHandler
import logging
import asyncio
logger = logging.getLogger(__name__)
class DegewoHandler(BaseHandler):
async def apply(self, listing: dict, result: dict) -> dict:
page = await self.context.new_page()
try:
logger.info(f"[DEGEWO] Opening page: {listing['link']}")
await page.goto(listing["link"], wait_until="networkidle")
logger.info("[DEGEWO] Page loaded")
await asyncio.sleep(2)
# Handle cookies and consent
await self.handle_cookies(page)
await self.handle_consent(page)
# Look for application button
logger.info("[DEGEWO] Looking for application button...")
selectors = [
'a[href*="bewerben"]',
'button:has-text("Bewerben")'
]
apply_btn = None
for sel in selectors:
all_btns = await page.query_selector_all(sel)
logger.info(f"[DEGEWO] Selector '{sel}' found {len(all_btns)} matches")
for btn in all_btns:
try:
if await btn.is_visible():
apply_btn = btn
logger.info(f"[DEGEWO] Found visible button with selector '{sel}'")
break
except Exception as e:
logger.warning(f"[DEGEWO] Error checking button visibility: {e}")
if apply_btn:
break
if apply_btn:
logger.info("[DEGEWO] Found application button, scrolling into view...")
await apply_btn.scroll_into_view_if_needed()
await asyncio.sleep(0.5)
logger.info("[DEGEWO] Clicking button...")
await apply_btn.click()
await asyncio.sleep(2)
result["success"] = True
result["message"] = "Application submitted successfully."
else:
result["message"] = "No application button found."
except Exception as e:
result["message"] = f"Error during application: {e}"
logger.error(f"[DEGEWO] Application error: {e}")
finally:
await page.close()
return result

View file

@ -0,0 +1,59 @@
from .base_handler import BaseHandler
import logging
import asyncio
logger = logging.getLogger(__name__)
class GesobauHandler(BaseHandler):
async def apply(self, listing: dict, result: dict) -> dict:
page = await self.context.new_page()
try:
logger.info(f"[GESOBAU] Opening page: {listing['link']}")
await page.goto(listing["link"], wait_until="networkidle")
logger.info("[GESOBAU] Page loaded")
await asyncio.sleep(2)
# Handle cookies and consent
await self.handle_cookies(page)
await self.handle_consent(page)
# Look for application button
logger.info("[GESOBAU] Looking for application button...")
selectors = [
'a[href*="bewerben"]',
'button:has-text("Bewerben")'
]
apply_btn = None
for sel in selectors:
all_btns = await page.query_selector_all(sel)
logger.info(f"[GESOBAU] Selector '{sel}' found {len(all_btns)} matches")
for btn in all_btns:
try:
if await btn.is_visible():
apply_btn = btn
logger.info(f"[GESOBAU] Found visible button with selector '{sel}'")
break
except Exception as e:
logger.warning(f"[GESOBAU] Error checking button visibility: {e}")
if apply_btn:
break
if apply_btn:
logger.info("[GESOBAU] Found application button, scrolling into view...")
await apply_btn.scroll_into_view_if_needed()
await asyncio.sleep(0.5)
logger.info("[GESOBAU] Clicking button...")
await apply_btn.click()
await asyncio.sleep(2)
result["success"] = True
result["message"] = "Application submitted successfully."
else:
result["message"] = "No application button found."
except Exception as e:
result["message"] = f"Error during application: {e}"
logger.error(f"[GESOBAU] Application error: {e}")
finally:
await page.close()
return result

View file

@ -0,0 +1,59 @@
from .base_handler import BaseHandler
import logging
import asyncio
logger = logging.getLogger(__name__)
class GewobagHandler(BaseHandler):
async def apply(self, listing: dict, result: dict) -> dict:
page = await self.context.new_page()
try:
logger.info(f"[GEWOBAG] Opening page: {listing['link']}")
await page.goto(listing["link"], wait_until="networkidle")
logger.info("[GEWOBAG] Page loaded")
await asyncio.sleep(2)
# Handle cookies and consent
await self.handle_cookies(page)
await self.handle_consent(page)
# Look for application button
logger.info("[GEWOBAG] Looking for application button...")
selectors = [
'a[href*="bewerben"]',
'button:has-text("Bewerben")'
]
apply_btn = None
for sel in selectors:
all_btns = await page.query_selector_all(sel)
logger.info(f"[GEWOBAG] Selector '{sel}' found {len(all_btns)} matches")
for btn in all_btns:
try:
if await btn.is_visible():
apply_btn = btn
logger.info(f"[GEWOBAG] Found visible button with selector '{sel}'")
break
except Exception as e:
logger.warning(f"[GEWOBAG] Error checking button visibility: {e}")
if apply_btn:
break
if apply_btn:
logger.info("[GEWOBAG] Found application button, scrolling into view...")
await apply_btn.scroll_into_view_if_needed()
await asyncio.sleep(0.5)
logger.info("[GEWOBAG] Clicking button...")
await apply_btn.click()
await asyncio.sleep(2)
result["success"] = True
result["message"] = "Application submitted successfully."
else:
result["message"] = "No application button found."
except Exception as e:
result["message"] = f"Error during application: {e}"
logger.error(f"[GEWOBAG] Application error: {e}")
finally:
await page.close()
return result

View file

@ -0,0 +1,62 @@
from .base_handler import BaseHandler
import logging
import asyncio
logger = logging.getLogger(__name__)
class HowogeHandler(BaseHandler):
async def apply(self, listing: dict, result: dict) -> dict:
page = await self.context.new_page()
try:
logger.info(f"[HOWOGE] Opening page: {listing['link']}")
await page.goto(listing["link"], wait_until="networkidle")
logger.info("[HOWOGE] Page loaded")
await asyncio.sleep(2)
# Handle cookies and consent
await self.handle_cookies(page)
await self.handle_consent(page)
# Look for "Besichtigung vereinbaren" button
logger.info("[HOWOGE] Looking for 'Besichtigung vereinbaren' button...")
selectors = [
'a[href*="besichtigung-vereinbaren"]',
'a:has-text("Besichtigung vereinbaren")',
'button:has-text("Besichtigung vereinbaren")',
'a:has-text("Anfragen")',
'button:has-text("Anfragen")'
]
apply_btn = None
for sel in selectors:
all_btns = await page.query_selector_all(sel)
logger.info(f"[HOWOGE] Selector '{sel}' found {len(all_btns)} matches")
for btn in all_btns:
try:
if await btn.is_visible():
apply_btn = btn
logger.info(f"[HOWOGE] Found visible button with selector '{sel}'")
break
except Exception as e:
logger.warning(f"[HOWOGE] Error checking button visibility: {e}")
if apply_btn:
break
if apply_btn:
logger.info("[HOWOGE] Found application button, scrolling into view...")
await apply_btn.scroll_into_view_if_needed()
await asyncio.sleep(0.5)
logger.info("[HOWOGE] Clicking button...")
await apply_btn.click()
await asyncio.sleep(2)
result["success"] = True
result["message"] = "Application submitted successfully."
else:
result["message"] = "No application button found."
except Exception as e:
result["message"] = f"Error during application: {e}"
logger.error(f"[HOWOGE] Application error: {e}")
finally:
await page.close()
return result

View file

@ -0,0 +1,59 @@
from .base_handler import BaseHandler
import logging
import asyncio
logger = logging.getLogger(__name__)
class StadtUndLandHandler(BaseHandler):
async def apply(self, listing: dict, result: dict) -> dict:
page = await self.context.new_page()
try:
logger.info(f"[STADT UND LAND] Opening page: {listing['link']}")
await page.goto(listing["link"], wait_until="networkidle")
logger.info("[STADT UND LAND] Page loaded")
await asyncio.sleep(2)
# Handle cookies and consent
await self.handle_cookies(page)
await self.handle_consent(page)
# Look for application button
logger.info("[STADT UND LAND] Looking for application button...")
selectors = [
'a[href*="bewerben"]',
'button:has-text("Bewerben")'
]
apply_btn = None
for sel in selectors:
all_btns = await page.query_selector_all(sel)
logger.info(f"[STADT UND LAND] Selector '{sel}' found {len(all_btns)} matches")
for btn in all_btns:
try:
if await btn.is_visible():
apply_btn = btn
logger.info(f"[STADT UND LAND] Found visible button with selector '{sel}'")
break
except Exception as e:
logger.warning(f"[STADT UND LAND] Error checking button visibility: {e}")
if apply_btn:
break
if apply_btn:
logger.info("[STADT UND LAND] Found application button, scrolling into view...")
await apply_btn.scroll_into_view_if_needed()
await asyncio.sleep(0.5)
logger.info("[STADT UND LAND] Clicking button...")
await apply_btn.click()
await asyncio.sleep(2)
result["success"] = True
result["message"] = "Application submitted successfully."
else:
result["message"] = "No application button found."
except Exception as e:
result["message"] = f"Error during application: {e}"
logger.error(f"[STADT UND LAND] Application error: {e}")
finally:
await page.close()
return result

59
handlers/wbm_handler.py Normal file
View file

@ -0,0 +1,59 @@
from .base_handler import BaseHandler
import logging
import asyncio
logger = logging.getLogger(__name__)
class WBMHandler(BaseHandler):
async def apply(self, listing: dict, result: dict) -> dict:
page = await self.context.new_page()
try:
logger.info(f"[WBM] Opening page: {listing['link']}")
await page.goto(listing["link"], wait_until="networkidle")
logger.info("[WBM] Page loaded")
await asyncio.sleep(2)
# Handle cookies and consent
await self.handle_cookies(page)
await self.handle_consent(page)
# Look for application button
logger.info("[WBM] Looking for application button...")
selectors = [
'a[href*="bewerben"]',
'button:has-text("Bewerben")'
]
apply_btn = None
for sel in selectors:
all_btns = await page.query_selector_all(sel)
logger.info(f"[WBM] Selector '{sel}' found {len(all_btns)} matches")
for btn in all_btns:
try:
if await btn.is_visible():
apply_btn = btn
logger.info(f"[WBM] Found visible button with selector '{sel}'")
break
except Exception as e:
logger.warning(f"[WBM] Error checking button visibility: {e}")
if apply_btn:
break
if apply_btn:
logger.info("[WBM] Found application button, scrolling into view...")
await apply_btn.scroll_into_view_if_needed()
await asyncio.sleep(0.5)
logger.info("[WBM] Clicking button...")
await apply_btn.click()
await asyncio.sleep(2)
result["success"] = True
result["message"] = "Application submitted successfully."
else:
result["message"] = "No application button found."
except Exception as e:
result["message"] = f"Error during application: {e}"
logger.error(f"[WBM] Application error: {e}")
finally:
await page.close()
return result

23
main.py Normal file
View file

@ -0,0 +1,23 @@
import asyncio
from playwright.async_api import async_playwright
from application_handler import ApplicationHandler
from telegram_bot import TelegramBot
async def main():
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True)
context = await browser.new_context()
# Initialize the application
app_handler = ApplicationHandler(context)
bot = TelegramBot(app_handler)
bot.start()
# Keep the bot running
try:
await asyncio.Event().wait()
except (KeyboardInterrupt, SystemExit):
print("Shutting down...")
if __name__ == "__main__":
asyncio.run(main())

3
pytest.ini Normal file
View file

@ -0,0 +1,3 @@
[pytest]
markers =
asyncio: mark a test as asyncio-based.

View file

@ -2,3 +2,6 @@ requests>=2.31.0
playwright>=1.57.0
matplotlib>=3.8.0
pandas>=2.0.0
python-dotenv>=1.0.0
pytest>=7.0.0
pytest-asyncio>=0.20.0

175
telegram_bot.py Normal file
View file

@ -0,0 +1,175 @@
import os
import logging
import threading
import time
import requests
# Configuration from environment
TELEGRAM_BOT_TOKEN = os.environ.get("TELEGRAM_BOT_TOKEN", "")
TELEGRAM_CHAT_ID = os.environ.get("TELEGRAM_CHAT_ID", "")
logger = logging.getLogger(__name__)
class TelegramBot:
"""Handle Telegram commands for controlling the monitor"""
def __init__(self, monitor, bot_token=None, chat_id=None):
self.monitor = monitor
self.bot_token = bot_token or TELEGRAM_BOT_TOKEN
self.chat_id = chat_id or TELEGRAM_CHAT_ID
self.last_update_id = 0
self.running = False
def start(self):
if not self.bot_token:
logger.warning("Telegram bot token not configured, commands disabled")
return
self.running = True
thread = threading.Thread(target=self._poll_updates, daemon=True)
thread.start()
logger.info("Telegram command listener started")
def stop(self):
self.running = False
def _poll_updates(self):
while self.running:
try:
url = f"https://api.telegram.org/bot{self.bot_token}/getUpdates"
params = {"offset": self.last_update_id + 1, "timeout": 30}
response = requests.get(url, params=params, timeout=35)
if response.ok:
data = response.json()
if data.get("ok") and data.get("result"):
for update in data["result"]:
self.last_update_id = update["update_id"]
self._handle_update(update)
except requests.exceptions.Timeout:
continue
except Exception as e:
logger.error(f"Telegram polling error: {e}")
time.sleep(5)
def _handle_update(self, update):
message = update.get("message", {})
text = message.get("text", "")
chat_id = str(message.get("chat", {}).get("id", ""))
if chat_id != self.chat_id:
logger.debug(f"Ignoring message from unknown chat: {chat_id}")
return
logger.info(f"Received Telegram command: {text}")
if text.startswith("/autopilot"):
self._handle_autopilot_command(text)
elif text == "/status":
self._handle_status_command()
elif text == "/help":
self._handle_help_command()
elif text == "/plot":
self._handle_plot_command()
elif text == "/errorrate":
self._handle_error_rate_command()
elif text.startswith("/"):
self._handle_unknown_command(text)
def _handle_autopilot_command(self, text):
logger.info(f"Processing autopilot command: {text}")
parts = text.split()
if len(parts) < 2:
self._send_message("Usage: /autopilot on|off")
return
action = parts[1].lower()
if action == "on":
logger.info("Enabling autopilot mode")
self.monitor.set_autopilot(True)
self._send_message("🤖 <b>Autopilot ENABLED</b>\n\nI will automatically apply to new listings!")
elif action == "off":
self.monitor.set_autopilot(False)
self._send_message("🛑 <b>Autopilot DISABLED</b>\n\nI will only notify you of new listings.")
else:
self._send_message("Usage: /autopilot on|off")
def _handle_status_command(self):
state = self.monitor.load_state()
autopilot = state.get("autopilot", False)
applications = self.monitor.load_applications()
status = "🤖 <b>Autopilot:</b> " + ("ON ✅" if autopilot else "OFF ❌")
status += f"\n📝 <b>Applications sent:</b> {len(applications)}"
by_company = {}
for app in applications.values():
company = app.get("company", "unknown")
by_company[company] = by_company.get(company, 0) + 1
if by_company:
status += "\n\n<b>By company:</b>"
for company, count in sorted(by_company.items()):
status += f"\n{company}: {count}"
self._send_message(status)
def _handle_help_command(self):
help_text = """🏠 <b>InBerlin Monitor Commands</b>
/autopilot on - Enable automatic applications
/autopilot off - Disable automatic applications
/status - Show current status and stats
/plot - Show weekly listing patterns
/help - Show this help message
When autopilot is ON, I will automatically apply to new listings."""
self._send_message(help_text)
def _handle_unknown_command(self, text):
cmd = text.split()[0] if text else text
self._send_message(f"❓ Unknown command: {cmd}")
def _handle_error_rate_command(self):
"""Generate and send a plot showing success vs failure ratio for autopilot applications."""
logger.info("Generating autopilot errorrate plot...")
try:
plot_path, summary = self._generate_error_rate_plot()
if plot_path:
caption = "📉 <b>Autopilot Success vs Failure</b>\n\n" + summary
self._send_photo(plot_path, caption)
else:
self._send_message("📉 Not enough application data to generate errorrate plot.")
except Exception as e:
logger.error(f"Error generating error rate plot: {e}")
self._send_message("📉 Error generating error rate plot.")
def _send_message(self, text):
"""Send a text message to the configured Telegram chat."""
if not self.bot_token or not self.chat_id:
logger.warning("Telegram bot token or chat ID not configured, cannot send message")
return
url = f"https://api.telegram.org/bot{self.bot_token}/sendMessage"
payload = {"chat_id": self.chat_id, "text": text, "parse_mode": "HTML"}
try:
response = requests.post(url, json=payload, timeout=10)
if not response.ok:
logger.error(f"Failed to send Telegram message: {response.text}")
except Exception as e:
logger.error(f"Error while sending Telegram message: {e}")
def _send_photo(self, photo_path, caption):
"""Send a photo to the configured Telegram chat."""
if not self.bot_token or not self.chat_id:
logger.warning("Telegram bot token or chat ID not configured, cannot send photo")
return
url = f"https://api.telegram.org/bot{self.bot_token}/sendPhoto"
with open(photo_path, "rb") as photo:
payload = {"chat_id": self.chat_id, "caption": caption, "parse_mode": "HTML"}
files = {"photo": photo}
try:
response = requests.post(url, data=payload, files=files, timeout=10)
if not response.ok:
logger.error(f"Failed to send Telegram photo: {response.text}")
except Exception as e:
logger.error(f"Error while sending Telegram photo: {e}")
def _generate_error_rate_plot(self):
"""Placeholder for generating an error rate plot."""
logger.warning("_generate_error_rate_plot is not implemented.")
return None, "Error rate plot generation not implemented."
def _handle_plot_command(self):
"""Placeholder for handling the /plot command."""
logger.warning("_handle_plot_command is not implemented.")
self._send_message("📊 Plot command is not implemented yet.")

View file

@ -0,0 +1,38 @@
import os
import pytest
from unittest.mock import patch, mock_open
from archive.test_errorrate_runner import generate_error_rate_plot
@pytest.fixture
def mock_data_dir(tmp_path):
"""Fixture to create a temporary data directory."""
data_dir = tmp_path / "data"
data_dir.mkdir()
return data_dir
@patch("builtins.open", new_callable=mock_open, read_data="{}")
@patch("os.path.exists", return_value=True)
def test_generate_error_rate_plot_no_data(mock_exists, mock_open, mock_data_dir):
"""Test generate_error_rate_plot with no data."""
plot_path, summary = generate_error_rate_plot(str(mock_data_dir / "applications.json"))
assert plot_path is None
assert summary == ""
@patch("builtins.open", new_callable=mock_open)
@patch("os.path.exists", return_value=True)
@patch("matplotlib.pyplot.savefig")
def test_generate_error_rate_plot_with_data(mock_savefig, mock_exists, mock_open, mock_data_dir):
"""Test generate_error_rate_plot with valid data."""
mock_open.return_value.read.return_value = """
{
"1": {"timestamp": "2025-12-25T12:00:00", "company": "CompanyA", "success": true},
"2": {"timestamp": "2025-12-26T12:00:00", "company": "CompanyB", "success": false}
}
"""
plot_path, summary = generate_error_rate_plot(str(mock_data_dir / "applications.json"))
assert plot_path is not None
assert "Total attempts" in summary
assert "Successes" in summary
assert "Failures" in summary
assert "Overall error rate" in summary
mock_savefig.assert_called_once()

62
tests/test_handlers.py Normal file
View file

@ -0,0 +1,62 @@
import pytest
from handlers.howoge_handler import HowogeHandler
from handlers.gewobag_handler import GewobagHandler
from handlers.degewo_handler import DegewoHandler
from handlers.gesobau_handler import GesobauHandler
from handlers.stadtundland_handler import StadtUndLandHandler
from handlers.wbm_handler import WBMHandler
from unittest.mock import AsyncMock
@pytest.mark.asyncio
async def test_howoge_handler():
context = AsyncMock()
handler = HowogeHandler(context)
listing = {"link": "https://www.howoge.de/example"}
result = {"success": False}
await handler.apply(listing, result)
assert "success" in result
@pytest.mark.asyncio
async def test_gewobag_handler():
context = AsyncMock()
handler = GewobagHandler(context)
listing = {"link": "https://www.gewobag.de/example"}
result = {"success": False}
await handler.apply(listing, result)
assert "success" in result
@pytest.mark.asyncio
async def test_degewo_handler():
context = AsyncMock()
handler = DegewoHandler(context)
listing = {"link": "https://www.degewo.de/example"}
result = {"success": False}
await handler.apply(listing, result)
assert "success" in result
@pytest.mark.asyncio
async def test_gesobau_handler():
context = AsyncMock()
handler = GesobauHandler(context)
listing = {"link": "https://www.gesobau.de/example"}
result = {"success": False}
await handler.apply(listing, result)
assert "success" in result
@pytest.mark.asyncio
async def test_stadtundland_handler():
context = AsyncMock()
handler = StadtUndLandHandler(context)
listing = {"link": "https://www.stadtundland.de/example"}
result = {"success": False}
await handler.apply(listing, result)
assert "success" in result
@pytest.mark.asyncio
async def test_wbm_handler():
context = AsyncMock()
handler = WBMHandler(context)
listing = {"link": "https://www.wbm.de/example"}
result = {"success": False}
await handler.apply(listing, result)
assert "success" in result

View file

@ -0,0 +1,62 @@
import os
import pytest
from unittest.mock import MagicMock, patch
from telegram_bot import TelegramBot
from dotenv import load_dotenv
# Load environment variables from .env file
load_dotenv()
# Explicitly pass token and chat ID to ensure they are set
@pytest.fixture(autouse=True)
def mock_env_vars():
os.environ["TELEGRAM_BOT_TOKEN"] = "test_token"
os.environ["TELEGRAM_CHAT_ID"] = "test_chat_id"
@pytest.fixture
def mock_monitor():
monitor = MagicMock()
monitor.load_state.return_value = {"autopilot": True}
monitor.load_applications.return_value = {
"app1": {"company": "CompanyA"},
"app2": {"company": "CompanyB"},
"app3": {"company": "CompanyA"},
}
return monitor
@pytest.fixture
def telegram_bot(mock_monitor):
return TelegramBot(mock_monitor, bot_token="test_token", chat_id="test_chat_id")
@patch("telegram_bot.requests.post")
def test_send_message(mock_post, telegram_bot):
mock_post.return_value.ok = True
telegram_bot._send_message("Test message")
mock_post.assert_called_once()
assert mock_post.call_args[1]["json"]["text"] == "Test message"
@patch("telegram_bot.requests.post")
def test_send_photo(mock_post, telegram_bot):
mock_post.return_value.ok = True
with patch("builtins.open", create=True):
telegram_bot._send_photo("/path/to/photo.jpg", "Test caption")
mock_post.assert_called_once()
assert mock_post.call_args[1]["data"]["caption"] == "Test caption"
@patch("telegram_bot.TelegramBot._send_message")
def test_handle_status_command(mock_send_message, telegram_bot):
telegram_bot._handle_status_command()
mock_send_message.assert_called_once()
assert "Autopilot" in mock_send_message.call_args[0][0]
@patch("telegram_bot.TelegramBot._send_message")
def test_handle_help_command(mock_send_message, telegram_bot):
telegram_bot._handle_help_command()
mock_send_message.assert_called_once()
assert "InBerlin Monitor Commands" in mock_send_message.call_args[0][0]
@patch("telegram_bot.TelegramBot._send_message")
def test_handle_unknown_command(mock_send_message, telegram_bot):
telegram_bot._handle_unknown_command("/unknown")
mock_send_message.assert_called_once()
assert "Unknown command" in mock_send_message.call_args[0][0]