aron/awebsite
aron/awebsite
1
0
Fork 0
Code Issues Pull requests Wiki Activity Actions

more translation

Browse source
This commit is contained in:
Aron Petau 2025-10-06 18:01:45 +02:00
parent 2ce9ca50b5
commit a41be821c1
997 changed files with 33247 additions and 32490 deletions
Download patch file Download diff file Expand all files Collapse all files

484
content/project/2024-01-30-airaspi-build-log/index.de.md
View file

@ -1,7 +1,7 @@
+++
title = "Übersetzung: AIRASPI Build Log"
title = "AIRASPI Build-Protokoll"
authors = ["Aron Petau"]
description = "Utilizing an edge TPU to build an edge device for image recognition and object detection"
description = "Nutzung einer Edge-TPU zum Bau eines Edge-Geräts für Bilderkennung und Objektdetektion"
date = 2024-01-30
[taxonomies]
tags = [
@ -20,78 +20,81 @@ show_copyright = true
show_shares = true
+++
## AI-Raspi Build Log
## AI-Raspi Build-Protokoll
This should document the rough steps to recreate airaspi as I go along.
Dieses Dokument dokumentiert den Prozess des Baus eines maßgeschneiderten Edge-Computing-Geräts für Echtzeit-Bilderkennung und Objektdetektion. Das Ziel war es, ein portables, eigenständiges System zu schaffen, das unabhängig von Cloud-Infrastruktur funktionieren kann.
Rough Idea: Build an edge device with image recognition and object detection capabilites.\
It should be realtime, aiming for 30fps at 720p.\
Portability and usage at installations is a priority, so it has to function without active internet connection and be as small as possible.\
It would be a real Edge Device, with no computation happening in the cloud.
**Projektziele:**
Inspo from: [pose2art](https://github.com/MauiJerry/Pose2Art)
Bau eines Edge-Geräts mit Bilderkennung und Objektdetektion, das Video in Echtzeit verarbeiten kann, mit dem Ziel von 30fps bei 720p-Auflösung. Portabilität und autonomer Betrieb sind kritische Anforderungen—das Gerät muss ohne aktive Internetverbindung funktionieren und einen kompakten Formfaktor für Installationsumgebungen beibehalten. Alle Berechnungen finden lokal auf dem Gerät selbst statt, was es zu einer echten Edge-Computing-Lösung ohne Cloud-Abhängigkeit macht.
Dieses Projekt wurde inspiriert von [pose2art](https://github.com/MauiJerry/Pose2Art), das das kreative Potenzial der Echtzeit-Posenerkennung für interaktive Installationen demonstrierte.
## Hardware
- [Raspberry Pi 5](https://www.raspberrypi.com/products/raspberry-pi-5/)
- [Raspberry Pi Camera Module v1.3](https://www.raspberrypi.com/documentation/accessories/camera.html)
- [Raspberry Pi GlobalShutter Camera](https://www.raspberrypi.com/documentation/accessories/camera.html)
- 2x CSI FPC Cable (needs one compact side to fit pi 5)
- 2x CSI FPC Kabel (benötigt eine kompakte Seite, um in Pi 5 zu passen)
- [Pineberry AI Hat (m.2 E key)](https://pineberrypi.com/products/hat-ai-for-raspberry-pi-5)
- [Coral Dual Edge TPU (m.2 E key)](https://www.coral.ai/products/m2-accelerator-dual-edgetpu)
- Raspi Official 5A Power Supply
- Raspi active cooler
- Raspi Official 5A Netzteil
- Raspi aktiver Kühler
## Setup
### Most important sources used
### Hauptressourcen
[coral.ai](https://www.coral.ai/docs/m2/get-started/#requirements)
[Jeff Geerling](https://www.jeffgeerling.com/blog/2023/pcie-coral-tpu-finally-works-on-raspberry-pi-5)
[Frigate NVR](https://docs.frigate.video)
Dieser Build wäre ohne die exzellente Dokumentation und Troubleshooting-Anleitungen aus der Community nicht möglich gewesen. Die Hauptquellen, auf die ich mich während dieses Projekts verlassen habe, waren:
### Raspberry Pi OS
- [coral.ai offizielle Dokumentation](https://www.coral.ai/docs/m2/get-started/#requirements) - Googles offizieller Setup-Leitfaden für die M.2 Edge TPU
- [Jeff Geerlings Blog](https://www.jeffgeerling.com/blog/2023/pcie-coral-tpu-finally-works-on-raspberry-pi-5) - Kritische PCIe-Konfigurationseinblicke für Raspberry Pi 5
- [Frigate NVR Dokumentation](https://docs.frigate.video) - Umfassender Leitfaden für die Network-Video-Recorder-Software
I used the Raspberry Pi Imager to flash the latest Raspberry Pi OS Lite to a SD Card.
### Raspberry Pi OS Installation
Needs to be Debian Bookworm.\
Needs to be the full arm64 image (with desktop), otherwise you will get into camera driver hell.
{: .notice}
Ich habe den Raspberry Pi Imager verwendet, um das neueste Raspberry Pi OS auf eine SD-Karte zu flashen. Die OS-Wahl ist kritisch für die Kamerakompatibilität.
Settings applied:
> [!IMPORTANT]
> Muss Debian Bookworm sein.
> Muss das vollständige arm64-Image (mit Desktop) sein, sonst gerät man in die
> Kameratreiber-Hölle.
- used the default arm64 image (with desktop)
- enable custom settings:
- enable ssh
- set wifi country
- set wifi ssid and password
- set locale
- set hostname: airaspi
**Initiale Konfigurationseinstellungen:**
### update
Mit den erweiterten Einstellungen des Raspberry Pi Imager habe ich vor dem Flashen Folgendes konfiguriert:
This is always good practice on a fresh install. It takes quite long with the full os image.
- Verwendung des Standard-arm64-Images (mit Desktop) - kritisch für Kameratreiber-Kompatibilität
- Aktivierung benutzerdefinierter Einstellungen für Headless-Betrieb
- Aktivierung von SSH für Fernzugriff
- Konfiguration des WiFi-Ländercodes für rechtliche Compliance
- Festlegung von WiFi-SSID und Passwort für automatische Netzwerkverbindung
- Konfiguration der Locale-Einstellungen für richtige Zeitzone und Tastaturlayout
- Festlegung eines benutzerdefinierten Hostnamens: `airaspi` für einfache Netzwerkidentifikation
### System-Update
Nach dem ersten Boot ist das Aktualisieren des Systems unerlässlich. Dieser Prozess kann mit dem vollständigen Desktop-Image beträchtliche Zeit in Anspruch nehmen, stellt aber sicher, dass alle Pakete aktuell sind und Sicherheitspatches angewendet werden.
```zsh
sudo apt update && sudo apt upgrade -y && sudo reboot
```
### prep system for coral
### Vorbereitung des Systems für Coral TPU
Thanks again @Jeff Geerling, this is completely out of my comfort zone, I rely on people writing solid tutorials like this one.
Die PCIe-Schnittstelle des Raspberry Pi 5 erfordert eine spezifische Konfiguration, um mit der Coral Edge TPU zu funktionieren. Dieser Abschnitt war technisch am anspruchsvollsten und umfasste Kernel-Modifikationen und Device-Tree-Änderungen. Ein großes Dankeschön an Jeff Geerling für die Dokumentation dieses Prozesses—ohne sein detailliertes Troubleshooting wäre dies nahezu unmöglich gewesen.
```zsh
# check kernel version
# Kernel-Version prüfen
uname -a
```
```zsh
# modify config.txt
# config.txt modifizieren
sudo nano /boot/firmware/config.txt
```
While in the file, add the following lines:
Während man in der Datei ist, folgende Zeilen hinzufügen:
```config
kernel=kernel8.img
@ -99,80 +102,99 @@ dtparam=pciex1
dtparam=pciex1_gen=2
```
Save and reboot:
Speichern und neu starten:
```zsh
sudo reboot
```
```zsh
# check kernel version again
# Kernel-Version erneut prüfen
uname -a
```
- should be different now, with a -v8 at the end
- sollte jetzt anders sein, mit einem -v8 am Ende
edit /boot/firmware/cmdline.txt
/boot/firmware/cmdline.txt bearbeiten
```zsh
sudo nano /boot/firmware/cmdline.txt
```
- add pcie_aspm=off before rootwait
- pcie_aspm=off vor rootwait hinzufügen
```zsh
sudo reboot
```
### change device tree
### Modifizierung des Device Tree
#### wrong device tree
#### Initialer Script-Versuch (Veraltet)
The script simply did not work for me.
Anfangs gab es ein automatisiertes Script, das die Device-Tree-Modifikationen handhaben sollte. Jedoch erwies sich dieses Script als problematisch und verursachte Probleme während meines Builds.
maybe this script is the issue?
i will try again without it
{: .notice}
> [!WARNING]
> vielleicht ist dieses Script das Problem?
> ich werde es ohne erneut versuchen
```zsh
curl https://gist.githubusercontent.com/dataslayermedia/714ec5a9601249d9ee754919dea49c7e/raw/32d21f73bd1ebb33854c2b059e94abe7767c3d7e/coral-ai-pcie-edge-tpu-raspberrypi-5-setup | sh
```
- Yes it was the issue, wrote a comment about it on the gist
[comment](https://gist.github.com/dataslayermedia/714ec5a9601249d9ee754919dea49c7e?permalink_comment_id=4860232#gistcomment-4860232)
Ja, es war das problematische Script. Ich habe einen Kommentar dokumentiert, der das Problem auf dem ursprünglichen Gist beschreibt:
[Mein Kommentar auf dem Gist](https://gist.github.com/dataslayermedia/714ec5a9601249d9ee754919dea49c7e?permalink_comment_id=4860232#gistcomment-4860232)
What to do instead?
#### Manuelle Device-Tree-Modifikation (Empfohlen)
Here, I followed Jeff Geerling down to the T. Please refer to his tutorial for more information.
Anstatt mich auf das automatisierte Script zu verlassen, folgte ich Jeff Geerlings manuellem Ansatz. Diese Methode gibt vollständige Kontrolle über den Prozess und hilft zu verstehen, was tatsächlich unter der Haube passiert.
In the meantime the Script got updated and it is now recommended again.
{: .notice}
> [!NOTE]
> In der Zwischenzeit wurde das Script aktualisiert und wird nun wieder empfohlen.
Der Device-Tree-Modifikationsprozess umfasst das Sichern des aktuellen Device-Tree-Blobs (DTB), das Dekompilieren in ein lesbares Format, das Bearbeiten der MSI-Parent-Referenz zur Behebung von PCIe-Kompatibilitätsproblemen und dann das Zurückkompilieren in Binärformat. Hier ist der schrittweise Prozess:
**1. Device Tree sichern und dekompilieren**
```zsh
# Back up the current dtb
# Aktuelles dtb sichern
sudo cp /boot/firmware/bcm2712-rpi-5-b.dtb /boot/firmware/bcm2712-rpi-5-b.dtb.bak
# Decompile the current dtb (ignore warnings)
# Aktuelles dtb dekompilieren (Warnungen ignorieren)
dtc -I dtb -O dts /boot/firmware/bcm2712-rpi-5-b.dtb -o ~/test.dts
# Edit the file
# Datei bearbeiten
nano ~/test.dts
# Change the line: msi-parent = <0x2f>; (under `pcie@110000`)
# To: msi-parent = <0x66>;
# Then save the file.
# Zeile ändern: msi-parent = <0x2f>; (unter `pcie@110000`)
# Zu: msi-parent = <0x66>;
# Dann Datei speichern.
# Recompile the dtb and move it back to the firmware directory
# dtb rekompilieren und zurück ins Firmware-Verzeichnis verschieben
dtc -I dts -O dtb ~/test.dts -o ~/test.dtb
sudo mv ~/test.dtb /boot/firmware/bcm2712-rpi-5-b.dtb
# Neustart für Änderungen
sudo reboot
```
Note: msi- parent sems to carry the value <0x2c> nowadays, cost me a few hours.
{: .notice}
> [!NOTE]
> Hinweis: msi-parent scheint heutzutage den Wert <0x2c> zu haben, hat mich ein paar Stunden gekostet.
### install apex driver
**2. Änderungen verifizieren**
following instructions from [coral.ai](https://coral.ai/docs/m2/get-started#2a-on-linux)
Nach dem Neustart prüfen, dass die Coral TPU vom System erkannt wird:
```zsh
lspci -nn | grep 089a
```
Die Ausgabe sollte ähnlich sein: `0000:01:00.0 System peripheral [0880]: Global Unichip Corp. Coral Edge TPU [1ac1:089a]`
### Installation des Apex-Treibers
Mit dem ordnungsgemäß konfigurierten Device Tree ist der nächste Schritt die Installation von Googles Apex-Treiber für die Coral Edge TPU. Dieser Treiber ermöglicht die Kommunikation zwischen Betriebssystem und TPU-Hardware.
Gemäß den offiziellen Anweisungen von [coral.ai](https://coral.ai/docs/m2/get-started#2a-on-linux):
```zsh
echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" | sudo tee /etc/apt/sources.list.d/coral-edgetpu.list
@ -188,61 +210,82 @@ sudo sh -c "echo 'SUBSYSTEM==\"apex\", MODE=\"0660\", GROUP=\"apex\"' >> /etc/ud
sudo groupadd apex
sudo adduser $USER apex
sudo reboot
```
Verify with
Diese Sequenz:
1. Fügt Googles Paket-Repository und GPG-Schlüssel hinzu
2. Installiert das gasket DKMS-Modul (Kernel-Treiber) und Edge-TPU-Runtime-Bibliothek
3. Erstellt udev-Regeln für Geräteberechtigungen
4. Erstellt eine `apex`-Gruppe und fügt den Benutzer hinzu
5. Neustart zum Laden des Treibers
Nach dem Neustart Installation verifizieren:
```zsh
lspci -nn | grep 089a
```
- should display the connected tpu
Dies sollte die verbundene Coral TPU als PCIe-Gerät anzeigen.
Als Nächstes bestätigen, dass der Device-Node mit entsprechenden Berechtigungen existiert:
```zsh
sudo reboot
ls -l /dev/apex_0
```
confirm with, if the output is not /dev/apex_0, something went wrong
Wenn die Ausgabe `/dev/apex_0` mit entsprechenden Gruppenberechtigungen zeigt, war die Installation erfolgreich. Falls nicht, udev-Regeln und Gruppenzugehörigkeit überprüfen.
### Testen mit Beispielmodellen
Um zu verifizieren, dass die TPU korrekt funktioniert, verwenden wir Googles Beispiel-Klassifizierungsskript mit einem vortrainierten MobileNet-Modell:
```zsh
ls /dev/apex_0
# Python-Pakete installieren
sudo apt-get install python3-pycoral
# Beispiel-Code und Modelle herunterladen
mkdir -p ~/coral && cd ~/coral
git clone https://github.com/google-coral/pycoral.git
cd pycoral
# Vogel-Klassifizierungsbeispiel ausführen
python3 examples/classify_image.py \
--model test_data/mobilenet_v2_1.0_224_inat_bird_quant_edgetpu.tflite \
--labels test_data/inat_bird_labels.txt \
--input test_data/parrot.jpg
```
### Docker
Die Ausgabe sollte Inferenz-Ergebnisse mit Konfidenzwerten zeigen, was bestätigt, dass die Edge TPU korrekt funktioniert.
Install docker, use the official instructions for debian.
### Docker-Installation
Docker bietet Containerisierung für die Anwendungen, die wir ausführen werden (Frigate, MediaMTX, etc.). Dies hält Abhängigkeiten isoliert und macht das Deployment wesentlich sauberer.
Docker mit dem offiziellen Convenience-Script von [docker.com](https://docs.docker.com/engine/install/debian/#install-using-the-convenience-script) installieren:
```zsh
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
```
```zsh
# add user to docker group
sudo groupadd docker
sudo usermod -aG docker $USER
```
Probably a source with source .bashrc would be enough, but I rebooted anyways
{: .notice}
Nach der Installation ab- und wieder anmelden, damit Änderungen der Gruppenzugehörigkeit wirksam werden.
```zsh
sudo reboot
```
```zsh
# verify with
docker run hello-world
```
### set docker to start on boot
Docker so konfigurieren, dass es automatisch beim Booten startet:
```zsh
sudo systemctl enable docker.service
sudo systemctl enable containerd.service
```
### Test the edge tpu
### Edge TPU testen (Optional)
Um zu verifizieren, dass die Edge TPU innerhalb eines Docker-Containers funktioniert, können wir ein Test-Image bauen. Dies ist besonders nützlich, wenn man plant, die TPU mit containerisierten Anwendungen zu nutzen.
Test-Verzeichnis und Dockerfile erstellen:
```zsh
mkdir coraltest
@ -250,7 +293,7 @@ cd coraltest
sudo nano Dockerfile
```
Into the new file, paste:
In die neue Datei einfügen:
```Dockerfile
FROM debian:10
@ -266,100 +309,126 @@ RUN echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" \
RUN curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
RUN apt-get update
RUN apt-get install -y edgetpu-examples
RUN apt-get install libedgetpu1-std
CMD /bin/bash
```
Test-Container bauen und ausführen, Coral-Gerät durchreichen:
```zsh
# build the docker container
# Docker-Container bauen
docker build -t "coral" .
```
```zsh
# run the docker container
# Docker-Container ausführen
docker run -it --device /dev/apex_0:/dev/apex_0 coral /bin/bash
```
Innerhalb des Containers ein Inferenz-Beispiel ausführen:
```zsh
# run an inference example from within the container
# Inferenz-Beispiel innerhalb des Containers ausführen
python3 /usr/share/edgetpu/examples/classify_image.py --model /usr/share/edgetpu/examples/models/mobilenet_v2_1.0_224_inat_bird_quant_edgetpu.tflite --label /usr/share/edgetpu/examples/models/inat_bird_labels.txt --image /usr/share/edgetpu/examples/images/bird.bmp
```
Here, you should see the inference results from the edge tpu with some confidence values.\
If it ain't so, safest bet is a clean restart
Man sollte Inferenz-Ergebnisse mit Konfidenzwerten von der Edge TPU sehen. Falls nicht, einen sauberen Neustart des Systems versuchen.
### Portainer
### Portainer (Optional)
This is optional, gives you a browser gui for your various docker containers
{: .notice}
Portainer bietet eine webbasierte GUI für die Verwaltung von Docker-Containern, Images und Volumes. Obwohl nicht erforderlich, macht es das Container-Management deutlich komfortabler.
Install portainer
> [!NOTE]
> Dies ist optional, gibt einem eine Browser-GUI für die verschiedenen Docker-Container.
Portainer installieren:
```zsh
docker volume create portainer_data
docker run -d -p 8000:8000 -p 9443:9443 --name portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce:latest
```
open portainer in browser and set admin password
Portainer im Browser aufrufen und Admin-Passwort setzen:
- should be available under <https://airaspi.local:9443>
- Navigieren zu: <https://airaspi.local:9443>
### vnc in raspi-config
### VNC-Setup (Optional)
optional, useful to test your cameras on your headless device.
You could of course also attach a monitor, but i find this more convenient.
{: .notice}
VNC bietet Remote-Desktop-Zugriff auf den Headless-Raspberry Pi. Dies ist besonders nützlich zum Testen von Kameras und Debuggen von visuellen Problemen, ohne einen physischen Monitor anzuschließen.
> [!NOTE]
> Dies ist optional, nützlich zum Testen der Kameras auf dem Headless-Gerät. Man könnte
> einen Monitor anschließen, aber ich finde VNC bequemer.
VNC über das Raspberry Pi Konfigurationstool aktivieren:
```zsh
sudo raspi-config
```
-- interface otions, enable vnc
Navigieren zu: **Interface Options** → **VNC** → **Enable**
### connect through vnc viewer
### Verbindung über VNC Viewer
Install vnc viewer on mac.\
Use airaspi.local:5900 as address.
[RealVNC Viewer](https://www.realvnc.com/en/connect/download/viewer/) auf dem Computer installieren (verfügbar für macOS, Windows und Linux).
### working docker-compose for frigate
Mit der Adresse verbinden: `airaspi.local:5900`
Start this as a custom template in portainer.
Man wird nach Benutzernamen und Passwort des Raspberry Pi gefragt. Nach der Verbindung hat man vollen Remote-Desktop-Zugriff zum Testen von Kameras und Debuggen.
Important: you need to change the paths to your own paths
{: .notice}
## Frigate NVR Setup
Frigate ist ein vollständiger Network Video Recorder (NVR) mit Echtzeit-Objektdetektion, angetrieben von der Coral Edge TPU. Es ist das Herzstück dieses Edge-AI-Systems.
### Docker Compose Konfiguration
Dieses Setup verwendet Docker Compose, um den Frigate-Container mit allen notwendigen Konfigurationen zu definieren. Wenn man Portainer verwendet, kann man dies als Custom Stack hinzufügen.
> [!IMPORTANT]
> Wichtig: Die Pfade müssen auf die eigenen Pfade angepasst werden.
```yaml
version: "3.9"
services:
frigate:
container_name: frigate
privileged: true # this may not be necessary for all setups
privileged: true # dies ist möglicherweise nicht für alle Setups notwendig
restart: unless-stopped
image: ghcr.io/blakeblackshear/frigate:stable
shm_size: "64mb" # update for your cameras based on calculation above
shm_size: "64mb" # für Kameras basierend auf obiger Berechnung aktualisieren
devices:
- /dev/apex_0:/dev/apex_0 # passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux
- /dev/apex_0:/dev/apex_0 # reicht PCIe Coral durch, Treiberanweisungen hier folgen https://coral.ai/docs/m2/get-started/#2a-on-linux
volumes:
- /etc/localtime:/etc/localtime:ro
- /home/aron/frigate/config.yml:/config/config.yml # replace with your config file
- /home/aron/frigate/storage:/media/frigate # replace with your storage directory
- type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear
- /home/aron/frigate/config.yml:/config/config.yml # durch eigene Config-Datei ersetzen
- /home/aron/frigate/storage:/media/frigate # durch eigenes Storage-Verzeichnis ersetzen
- type: tmpfs # Optional: 1GB Speicher, reduziert SSD/SD-Karten-Verschleiß
target: /tmp/cache
tmpfs:
size: 1000000000
ports:
- "5000:5000"
- "8554:8554" # RTSP feeds
- "8555:8555/tcp" # WebRTC over tcp
- "8555:8555/udp" # WebRTC over udp
- "8555:8555/tcp" # WebRTC über tcp
- "8555:8555/udp" # WebRTC über udp
environment:
FRIGATE_RTSP_PASSWORD: "******"
```
### Working frigate config file
Wichtige Konfigurationspunkte in dieser Docker-Compose-Datei:
Frigate wants this file wherever you specified earlier that it will be.\
This is necessary just once. Afterwards, you will be able to change the config in the gui.
{: .notice}
- **Privileged-Modus** und **Device-Mappings**: Erforderlich für Hardwarezugriff (TPU, Kameras)
- **Shared Memory Size**: Zugewiesen für effiziente Video-Frame-Verarbeitung
- **Port-Mappings**: Macht Frigate's Web-UI (5000) und RTSP-Streams (8554) zugänglich
- **Volume-Mounts**: Persistiert Aufnahmen, Konfiguration und Datenbank
### Frigate-Konfigurationsdatei
Frigate benötigt eine YAML-Konfigurationsdatei, um Kameras, Detektoren und Detektionszonen zu definieren. Diese Datei am Pfad erstellen, der in der docker-compose-Datei angegeben wurde (z.B. `/home/aron/frigate/config.yml`).
> [!NOTE]
> Dies ist nur einmal notwendig. Danach kann man die Konfiguration in der GUI ändern.
Hier ist eine funktionierende Konfiguration mit der Coral TPU:
```yaml
mqtt:
@ -374,14 +443,14 @@ detectors:
device: pci
cameras:
cam1: # <++++++ Name the camera
cam1: # <++++++ Kamera benennen
ffmpeg:
hwaccel_args: preset-rpi-64-h264
inputs:
- path: rtsp://192.168.1.58:8900/cam1
roles:
- detect
cam2: # <++++++ Name the camera
cam2: # <++++++ Kamera benennen
ffmpeg:
hwaccel_args: preset-rpi-64-h264
inputs:
@ -389,17 +458,31 @@ cameras:
roles:
- detect
detect:
enabled: True # <+++- disable detection until you have a working camera feed
width: 1280 # <+++- update for your camera's resolution
height: 720 # <+++- update for your camera's resolution
enabled: True # <+++- Detektion deaktivieren bis funktionierende Kamera-Feeds vorhanden
width: 1280 # <+++- für Kameraauflösung aktualisieren
height: 720 # <+++- für Kameraauflösung aktualisieren
```
### mediamtx
Diese Konfiguration:
install mediamtx, do not use the docker version, it will be painful
- **Deaktiviert MQTT**: Vereinfacht Setup für rein lokalen Betrieb
- **Definiert zwei Detektoren**: Einen Coral-TPU-Detektor (`coral`) und einen CPU-Fallback
- **Verwendet Standard-Detektionsmodell**: Frigate enthält ein vortrainiertes Modell
- **Konfiguriert zwei Kameras**: Beide auf 1280x720-Auflösung eingestellt
- **Verwendet Hardware-Beschleunigung**: `preset-rpi-64-h264` für Raspberry Pi 5
- **Detektionszonen**: Nur aktivieren, wenn Kamera-Feeds ordnungsgemäß funktionieren
double check the chip architecture here, caused me some headache
{: .notice}
## MediaMTX Setup
MediaMTX ist ein Echtzeit-Medienserver, der das Streaming von den Raspberry-Pi-Kameras zu Frigate handhabt. Es ist notwendig, weil Frigate `libcamera` (den modernen Raspberry Pi Kamera-Stack) nicht direkt unterstützt.
MediaMTX direkt auf dem System installieren (nicht via Docker - die Docker-Version hat Kompatibilitätsprobleme mit libcamera).
> [!WARNING]
> Chip-Architektur beim Download doppelt prüfen - dies verursachte mir erhebliche
> Kopfschmerzen beim Setup.
MediaMTX herunterladen und installieren:
```zsh
mkdir mediamtx
@ -409,9 +492,11 @@ wget https://github.com/bluenviron/mediamtx/releases/download/v1.5.0/mediamtx_v1
tar xzvf mediamtx_v1.5.0_linux_arm64v8.tar.gz && rm mediamtx_v1.5.0_linux_arm64v8.tar.gz
```
edit the mediamtx.yml file
### MediaMTX-Konfiguration
### working paths section in mediamtx.yml
Die `mediamtx.yml`-Datei bearbeiten, um Kamera-Streams zu konfigurieren. Die untenstehende Konfiguration verwendet `rpicam-vid` (Raspberry Pis modernes Kamera-Tool), das durch FFmpeg geleitet wird, um RTSP-Streams zu erstellen.
Folgendes zum `paths`-Abschnitt in `mediamtx.yml` hinzufügen:
```yaml
paths:
@ -423,45 +508,112 @@ paths:
runOnInitRestart: yes
```
also change rtspAddress: :8554\
to rtspAddress: :8900\
Otherwise there is a conflict with frigate.
Diese Konfiguration:
With this, you should be able to start mediamtx.
- **`cam1` und `cam2`**: Definieren zwei Kamerapfade
- **`rpicam-vid`**: Erfasst YUV420-Video von Raspberry-Pi-Kameras
- **`ffmpeg`**: Transkodiert das Rohvideo zu H.264-RTSP-Stream
- **`runOnInitRestart: yes`**: Startet Stream automatisch neu, falls er fehlschlägt
### Port-Konfiguration
Standard-RTSP-Port ändern, um Konflikte mit Frigate zu vermeiden:
In `mediamtx.yml` ändern:
```yaml
rtspAddress: :8554
```
Zu:
```yaml
rtspAddress: :8900
```
Sonst gibt es einen Port-Konflikt mit Frigate.
### MediaMTX starten
MediaMTX im Vordergrund ausführen, um zu verifizieren, dass es funktioniert:
```zsh
./mediamtx
```
If there is no error, you can verify your stream through vlc under rtsp://airaspi.local:8900/cam1 (default would be 8554, but we changed it in the config file)
Wenn keine Fehler auftreten, Streams mit VLC oder einem anderen RTSP-Client verifizieren:
### Current Status
- `rtsp://airaspi.local:8900/cam1`
- `rtsp://airaspi.local:8900/cam2`
I get working streams from both cameras, sending them out at 30fps at 720p.
frigate, however limits the display fps to 5, which is depressing to watch, especially since the tpu doesnt even break a little sweat.
Hinweis: Standard-RTSP-Port ist 8554, aber wir haben ihn in der Konfiguration auf 8900 geändert.
Frigate claime that the TPU is good for up to 10 cameras, so there is headroom.
## Aktueller Status und Performance
The stram is completely errant and drops frames left and right. I have sometimes seen detect fps of 0.2, but the TPU speed should definitely not be the bottleneck here. Maybe attach the cameras to a separate device and stream from there?
### Was funktioniert
The biggest issue here is that the google folx seems to have abandoned the coral, even though they just released a new piece of hardware for it.
Their most RECENT python build is 3.9.
Specifically, pycoral seems to be the problem there. without a decent update, I will be confined to debian 10, with python 3.7.3.
That sucks.
There are custom wheels, but nothing that seems plug and play.
Das System streamt erfolgreich von beiden Kameras mit 30fps und 720p-Auflösung. Die Coral Edge TPU führt Objektdetektion mit minimaler Latenz durch - die TPU selbst kommt nicht ins Schwitzen und behält durchgehend hohe Performance bei.
About the rest of this setup:
The decision to go for m.2 E key to save money, instead of spending more on the usb version was a huge mistake.
Please do yourself a favor and spend the extra 40 bucks.
Technically, its probably faster and better with continuous operation, but i have yet to feel the benefit of that.
Laut Frigate-Dokumentation kann die TPU bis zu 10 Kameras handhaben, es gibt also erheblichen Spielraum für Erweiterung.
### TODOs
### Aktuelle Probleme
- add images and screenshots to the build log
- Check whether vdo.ninja is a viable way to add mobile streams. then Smartphone stream evaluation would be on the horizon.
- Bother the mediamtx makers about the libcamera bump, so we can get rid of the rpicam-vid hack.
I suspect there is quirte a lot of performance lost there.
- tweak the frigate config to get snapshots and maybe build an image / video database to later train a custom model.
- worry about attaching an external ssd and saving the video files on it.
- find a way to export the landmark points from frigate. maybe send them via osc like in pose2art?
- find a different hat that lets me access the other TPU? I have the dual version, but can currently only acces 1 of the 2 TPUs due to hardware restrictions.
Es gibt jedoch mehrere signifikante Probleme, die das System behindern:
**1. Frigate Display-Limitierungen**
Frigate begrenzt die Display-FPS auf 5, was deprimierend anzusehen ist, besonders da die TPU nicht einmal ins Schwitzen kommt. Die Hardware ist eindeutig zu viel mehr fähig, aber Software-Limitierungen halten sie zurück.
**2. Stream-Stabilitätsprobleme**
Der Stream ist völlig unberechenbar und droppt ständig Frames. Ich habe manchmal Detektions-FPS von nur 0,2 beobachtet, aber die TPU-Geschwindigkeit sollte definitiv nicht der Flaschenhals sein. Eine mögliche Lösung könnte sein, die Kameras an ein separates Gerät anzuschließen und von dort zu streamen.
**3. Coral-Software-Aufgabe**
Das größte Problem ist, dass Google das Coral-Ökosystem scheinbar aufgegeben hat, obwohl sie gerade neue Hardware dafür veröffentlicht haben. Ihr aktuellster Python-Build unterstützt nur Python 3.9.
Speziell scheint `pycoral` das Problem zu sein - ohne ein ordentliches Update bin ich auf Debian 10 mit Python 3.7.3 beschränkt. Das ist mies. Es gibt Custom-Wheels, aber nichts, das plug-and-play zu sein scheint.
Dies schränkt die Fähigkeit, moderne Software und Bibliotheken mit dem System zu nutzen, erheblich ein.
## Reflexionen und Lessons Learned
### Hardware-Entscheidungen
**Die M.2 E Key-Wahl**
Die Entscheidung, die M.2 E Key-Version zu nehmen, um Geld zu sparen, anstatt mehr für die USB-Version auszugeben, war ein riesiger Fehler. Bitte tu dir selbst einen Gefallen und gib die zusätzlichen 40 Euro aus.
Technisch ist sie wahrscheinlich schneller und besser für Dauerbetrieb, aber ich habe den Vorteil davon noch nicht gespürt. Die USB-Version hätte wesentlich mehr Flexibilität und einfacheres Debugging geboten.
## Zukünftige Entwicklung
Mehrere Verbesserungen und Experimente sind geplant, um dieses System zu erweitern:
**Dokumentation und visuelle Hilfsmittel**
- Bilder und Screenshots zu diesem Build-Protokoll hinzufügen, um es einfacher nachzuvollziehen
**Mobile-Stream-Integration**
- Prüfen, ob [vdo.ninja](https://vdo.ninja) ein praktikabler Weg ist, mobile Streams hinzuzufügen, um Smartphone-Kamera-Integration und -Evaluierung zu ermöglichen
**MediaMTX libcamera-Unterstützung**
- Die MediaMTX-Entwickler*innen bezüglich libcamera-Unterstützung kontaktieren, was den aktuellen `rpicam-vid`-Workaround eliminieren würde. Ich vermute, dass in der aktuellen Pipeline einiges an Performance verloren geht.
**Frigate-Konfigurationsverfeinerung**
- Die Frigate-Konfiguration optimieren, um Snapshots zu aktivieren und möglicherweise eine Bild-/Videodatenbank zum späteren Training benutzerdefinierter Modelle aufzubauen
**Speichererweiterung**
- Sich um das Anbringen einer externen SSD kümmern und die Videodateien darauf für Langzeitspeicherung und -analyse speichern
**Datenexport-Fähigkeiten**
- Einen Weg finden, die Landmarkenpunkte von Frigate zu exportieren, möglicherweise via OSC (wie in meinem [pose2art](/project/pose2art/)-Projekt) für kreative Anwendungen zu senden
**Dual-TPU-Zugriff**
- Einen anderen HAT finden, der Zugriff auf die andere TPU ermöglicht - ich habe die Dual-Version, kann aber aufgrund von Hardware-Einschränkungen derzeit nur auf 1 der 2 TPUs zugreifen

462
content/project/2024-01-30-airaspi-build-log/index.md
View file

@ -22,15 +22,13 @@ show_shares = true
## AI-Raspi Build Log
This should document the rough steps to recreate airaspi as I go along.
This document chronicles the process of building a custom edge computing device for real-time image recognition and object detection. The goal was to create a portable, self-contained system that could operate independently of cloud infrastructure.
Rough Idea: Build an edge device with image recognition and object detection capabilites.\
It should be realtime, aiming for 30fps at 720p.\
Portability and usage at installations is a priority, so it has to function without active internet connection and be as small as possible.\
It would be a real Edge Device, with no computation happening in the cloud.
**Project Goals:**
Inspo from: [pose2art](https://github.com/MauiJerry/Pose2Art)
Build an edge device with image recognition and object detection capabilities that can process video in real-time, targeting 30fps at 720p resolution. Portability and autonomous operation are critical requirements—the device must function without an active internet connection and maintain a compact form factor suitable for installation environments. All computation happens locally on the device itself, making it a true edge computing solution with no cloud dependency.
This project was inspired by [pose2art](https://github.com/MauiJerry/Pose2Art), which demonstrated the creative potential of real-time pose detection for interactive installations.
## Hardware
@ -45,41 +43,47 @@ Inspo from: [pose2art](https://github.com/MauiJerry/Pose2Art)
## Setup
### Most important sources used
### Primary Resources
[coral.ai](https://www.coral.ai/docs/m2/get-started/#requirements)
[Jeff Geerling](https://www.jeffgeerling.com/blog/2023/pcie-coral-tpu-finally-works-on-raspberry-pi-5)
[Frigate NVR](https://docs.frigate.video)
This build wouldn't have been possible without the excellent documentation and troubleshooting guides from the community. The primary sources I relied on throughout this project were:
### Raspberry Pi OS
- [coral.ai official documentation](https://www.coral.ai/docs/m2/get-started/#requirements) - Google's official setup guide for the M.2 Edge TPU
- [Jeff Geerling's blog](https://www.jeffgeerling.com/blog/2023/pcie-coral-tpu-finally-works-on-raspberry-pi-5) - Critical PCIe configuration insights for Raspberry Pi 5
- [Frigate NVR documentation](https://docs.frigate.video) - Comprehensive guide for the network video recorder software
I used the Raspberry Pi Imager to flash the latest Raspberry Pi OS Lite to a SD Card.
### Raspberry Pi OS Installation
Needs to be Debian Bookworm.\
Needs to be the full arm64 image (with desktop), otherwise you will get into camera driver hell.
{: .notice}
I used the Raspberry Pi Imager to flash the latest Raspberry Pi OS to an SD card. The OS choice is critical for camera compatibility.
Settings applied:
> [!IMPORTANT]
> Needs to be Debian Bookworm.
> Needs to be the full arm64 image (with desktop), otherwise you will get into camera
> driver hell.
- used the default arm64 image (with desktop)
- enable custom settings:
- enable ssh
- set wifi country
- set wifi ssid and password
- set locale
- set hostname: airaspi
**Initial Configuration Settings:**
### update
Using the Raspberry Pi Imager's advanced settings, I configured the following before flashing:
This is always good practice on a fresh install. It takes quite long with the full os image.
- Used the default arm64 image (with desktop) - critical for camera driver compatibility
- Enabled custom settings for headless operation
- Enabled SSH for remote access
- Configured WiFi country code for legal compliance
- Set WiFi SSID and password for automatic network connection
- Configured locale settings for proper timezone and keyboard layout
- Set custom hostname: `airaspi` for easy network identification
### System Update
After the initial boot, updating the system is essential.
This process can take considerable time with the full desktop image, but ensures all packages are current and security patches are applied.
```zsh
sudo apt update && sudo apt upgrade -y && sudo reboot
```
### prep system for coral
### Preparing the System for Coral TPU
Thanks again @Jeff Geerling, this is completely out of my comfort zone, I rely on people writing solid tutorials like this one.
The Raspberry Pi 5's PCIe interface requires specific configuration to work with the Coral Edge TPU. This section was the most technically challenging, involving kernel modifications and device tree changes. A huge thanks to Jeff Geerling for documenting this process—without his detailed troubleshooting, this would have been nearly impossible.
```zsh
# check kernel version
@ -124,29 +128,38 @@ sudo nano /boot/firmware/cmdline.txt
sudo reboot
```
### change device tree
### Modifying the Device Tree
#### wrong device tree
#### Initial Script Attempt (Deprecated)
The script simply did not work for me.
Initially, there was an automated script available that was supposed to handle the device tree modifications. However, this script proved problematic and caused issues during my build.
maybe this script is the issue?
i will try again without it
{: .notice}
> [!WARNING]
> maybe this script is the issue?
> i will try again without it
```zsh
curl https://gist.githubusercontent.com/dataslayermedia/714ec5a9601249d9ee754919dea49c7e/raw/32d21f73bd1ebb33854c2b059e94abe7767c3d7e/coral-ai-pcie-edge-tpu-raspberrypi-5-setup | sh
```
- Yes it was the issue, wrote a comment about it on the gist
[comment](https://gist.github.com/dataslayermedia/714ec5a9601249d9ee754919dea49c7e?permalink_comment_id=4860232#gistcomment-4860232)
Yes, it was the problematic script. I left a comment documenting the issue on the original gist:
[My comment on the gist](https://gist.github.com/dataslayermedia/714ec5a9601249d9ee754919dea49c7e?permalink_comment_id=4860232#gistcomment-4860232)
What to do instead?
#### Manual Device Tree Modification (Recommended)
Here, I followed Jeff Geerling down to the T. Please refer to his tutorial for more information.
Instead of relying on the automated script, I followed Jeff Geerling's manual approach.
This method gives you complete control over the process and helps understand what's
actually happening under the hood.
In the meantime the Script got updated and it is now recommended again.
{: .notice}
> [!NOTE]
> In the meantime the Script got updated and it is now recommended again.
The device tree modification process involves backing up the current device tree blob
(DTB), decompiling it to a readable format, editing the MSI parent reference to fix
PCIe compatibility issues, and then recompiling it back to binary format. Here's the
step-by-step process:
**1. Back up and Decompile the Device Tree**
```zsh
# Back up the current dtb
@ -165,14 +178,31 @@ nano ~/test.dts
# Recompile the dtb and move it back to the firmware directory
dtc -I dts -O dtb ~/test.dts -o ~/test.dtb
sudo mv ~/test.dtb /boot/firmware/bcm2712-rpi-5-b.dtb
# Reboot for changes to take effect
sudo reboot
```
Note: msi- parent sems to carry the value <0x2c> nowadays, cost me a few hours.
{: .notice}
> [!NOTE]
> Note: msi-parent seems to carry the value <0x2c> nowadays, cost me a few hours.
### install apex driver
**2. Verify the Changes**
following instructions from [coral.ai](https://coral.ai/docs/m2/get-started#2a-on-linux)
After rebooting, check that the Coral TPU is recognized by the system:
```zsh
lspci -nn | grep 089a
```
You should see output similar to: `0000:01:00.0 System peripheral [0880]: Global Unichip Corp. Coral Edge TPU [1ac1:089a]`
### Installing the Apex Driver
With the device tree properly configured, the next step is installing Google's Apex
driver for the Coral Edge TPU. This driver enables communication between the operating
system and the TPU hardware.
Following the official instructions from [coral.ai](https://coral.ai/docs/m2/get-started#2a-on-linux):
```zsh
echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" | sudo tee /etc/apt/sources.list.d/coral-edgetpu.list
@ -188,61 +218,88 @@ sudo sh -c "echo 'SUBSYSTEM==\"apex\", MODE=\"0660\", GROUP=\"apex\"' >> /etc/ud
sudo groupadd apex
sudo adduser $USER apex
sudo reboot
```
Verify with
This sequence:
1. Adds Google's package repository and GPG key
2. Installs the gasket DKMS module (kernel driver) and Edge TPU runtime library
3. Creates udev rules for device permissions
4. Creates an `apex` group and adds your user to it
5. Reboots to load the driver
After the reboot, verify the installation:
```zsh
lspci -nn | grep 089a
```
- should display the connected tpu
This should display the connected Coral TPU as a PCIe device.
Next, confirm the device node exists with proper permissions:
```zsh
sudo reboot
ls -l /dev/apex_0
```
confirm with, if the output is not /dev/apex_0, something went wrong
If the output shows `/dev/apex_0` with appropriate group permissions, the installation
was successful. If not, review the udev rules and group membership.
### Testing with Example Models
To verify the TPU is functioning correctly, we'll use Google's example classification
script with a pre-trained MobileNet model:
```zsh
ls /dev/apex_0
# Install Python packages
sudo apt-get install python3-pycoral
# Download example code and models
mkdir -p ~/coral && cd ~/coral
git clone https://github.com/google-coral/pycoral.git
cd pycoral
# Run bird classification example
python3 examples/classify_image.py \
--model test_data/mobilenet_v2_1.0_224_inat_bird_quant_edgetpu.tflite \
--labels test_data/inat_bird_labels.txt \
--input test_data/parrot.jpg
```
### Docker
The output should show inference results with confidence scores, confirming the Edge
TPU is working correctly.
Install docker, use the official instructions for debian.
### Docker Installation
Docker provides containerization for the applications we'll be running (Frigate,
MediaMTX, etc.). This keeps dependencies isolated and makes deployment much cleaner.
Install Docker using the official convenience script from
[docker.com](https://docs.docker.com/engine/install/debian/#install-using-the-convenience-script):
```zsh
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
```
```zsh
# add user to docker group
sudo groupadd docker
sudo usermod -aG docker $USER
```
Probably a source with source .bashrc would be enough, but I rebooted anyways
{: .notice}
After installation, log out and back in for group membership changes to take effect.
```zsh
sudo reboot
```
```zsh
# verify with
docker run hello-world
```
### set docker to start on boot
Configure Docker to start automatically on boot:
```zsh
sudo systemctl enable docker.service
sudo systemctl enable containerd.service
```
### Test the edge tpu
### Test the Edge TPU (Optional)
To verify the Edge TPU works inside a Docker container, we can build a test image.
This is particularly useful if you plan to use the TPU with containerized applications.
Create a test directory and Dockerfile:
```zsh
mkdir coraltest
@ -266,65 +323,89 @@ RUN echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" \
RUN curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
RUN apt-get update
RUN apt-get install -y edgetpu-examples
RUN apt-get install libedgetpu1-std
CMD /bin/bash
```
Build and run the test container, passing through the Coral device:
```zsh
# build the docker container
docker build -t "coral" .
```
```zsh
# run the docker container
docker run -it --device /dev/apex_0:/dev/apex_0 coral /bin/bash
```
Inside the container, run an inference example:
```zsh
# run an inference example from within the container
python3 /usr/share/edgetpu/examples/classify_image.py --model /usr/share/edgetpu/examples/models/mobilenet_v2_1.0_224_inat_bird_quant_edgetpu.tflite --label /usr/share/edgetpu/examples/models/inat_bird_labels.txt --image /usr/share/edgetpu/examples/images/bird.bmp
```
Here, you should see the inference results from the edge tpu with some confidence values.\
If it ain't so, safest bet is a clean restart
You should see inference results with confidence values from the Edge TPU. If not, try
a clean restart of the system.
### Portainer
### Portainer (Optional)
This is optional, gives you a browser gui for your various docker containers
{: .notice}
Portainer provides a web-based GUI for managing Docker containers, images, and volumes.
While not required, it makes container management significantly more convenient.
Install portainer
> [!NOTE]
> This is optional, gives you a browser GUI for your various docker containers.
Install Portainer:
```zsh
docker volume create portainer_data
docker run -d -p 8000:8000 -p 9443:9443 --name portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce:latest
```
open portainer in browser and set admin password
Access Portainer in your browser and set an admin password:
- should be available under <https://airaspi.local:9443>
- Navigate to: <https://airaspi.local:9443>
### vnc in raspi-config
### VNC Setup (Optional)
optional, useful to test your cameras on your headless device.
You could of course also attach a monitor, but i find this more convenient.
{: .notice}
VNC provides remote desktop access to your headless Raspberry Pi. This is particularly
useful for testing cameras and debugging visual issues without connecting a physical
monitor.
> [!NOTE]
> This is optional, useful to test your cameras on your headless device. You could attach
> a monitor, but I find VNC more convenient.
Enable VNC through the Raspberry Pi configuration tool:
```zsh
sudo raspi-config
```
-- interface otions, enable vnc
Navigate to: **Interface Options** → **VNC** → **Enable**
### connect through vnc viewer
### Connecting through VNC Viewer
Install vnc viewer on mac.\
Use airaspi.local:5900 as address.
Install [RealVNC Viewer](https://www.realvnc.com/en/connect/download/viewer/) on your
computer (available for macOS, Windows, and Linux).
### working docker-compose for frigate
Connect using the address: `airaspi.local:5900`
Start this as a custom template in portainer.
You'll be prompted for your Raspberry Pi username and password. Once connected, you'll
have full remote desktop access for testing cameras and debugging.
Important: you need to change the paths to your own paths
{: .notice}
## Frigate NVR Setup
Frigate is a complete Network Video Recorder (NVR) with real-time object detection
powered by the Coral Edge TPU. It's the heart of this edge AI system.
### Docker Compose Configuration
This setup uses Docker Compose to define the Frigate container with all necessary
configurations. If you're using Portainer, you can add this as a custom stack.
> [!IMPORTANT]
> Important: you need to change the paths to your own paths.
```yaml
version: "3.9"
@ -355,11 +436,25 @@ services:
FRIGATE_RTSP_PASSWORD: "******"
```
### Working frigate config file
Key configuration points in this Docker Compose file:
Frigate wants this file wherever you specified earlier that it will be.\
This is necessary just once. Afterwards, you will be able to change the config in the gui.
{: .notice}
- **Privileged mode** and **device mappings**: Required for accessing hardware (TPU,
cameras)
- **Shared memory size**: Allocated for processing video frames efficiently
- **Port mappings**: Exposes Frigate's web UI (5000) and RTSP streams (8554)
- **Volume mounts**: Persists recordings, config, and database
### Frigate Configuration File
Frigate requires a YAML configuration file to define cameras, detectors, and detection
zones. Create this file at the path you specified in the docker-compose file (e.g.,
`/home/aron/frigate/config.yml`).
> [!NOTE]
> This is necessary just once. Afterwards, you will be able to change the config in the
> GUI.
Here's a working configuration using the Coral TPU:
```yaml
mqtt:
@ -394,12 +489,29 @@ cameras:
height: 720 # <+++- update for your camera's resolution
```
### mediamtx
This configuration:
install mediamtx, do not use the docker version, it will be painful
- **Disables MQTT**: Simplifies setup for local-only operation
- **Defines two detectors**: A Coral TPU detector (`coral`) and a CPU fallback
- **Uses default detection model**: Frigate includes a pre-trained model
- **Configures two cameras**: Both set to 1280x720 resolution
- **Uses hardware acceleration**: `preset-rpi-64-h264` for Raspberry Pi 5
- **Detection zones**: Enable only when camera feeds are working properly
double check the chip architecture here, caused me some headache
{: .notice}
## MediaMTX Setup
MediaMTX is a real-time media server that handles streaming from the Raspberry Pi
cameras to Frigate. It's necessary because Frigate doesn't directly support `libcamera`
(the modern Raspberry Pi camera stack).
Install MediaMTX directly on the system (not via Docker - the Docker version has
compatibility issues with libcamera).
> [!WARNING]
> Double-check the chip architecture when downloading - this caused me significant
> headaches during setup.
Download and install MediaMTX:
```zsh
mkdir mediamtx
@ -409,9 +521,13 @@ wget https://github.com/bluenviron/mediamtx/releases/download/v1.5.0/mediamtx_v1
tar xzvf mediamtx_v1.5.0_linux_arm64v8.tar.gz && rm mediamtx_v1.5.0_linux_arm64v8.tar.gz
```
edit the mediamtx.yml file
### MediaMTX Configuration
### working paths section in mediamtx.yml
Edit the `mediamtx.yml` file to configure camera streams. The configuration below uses
`rpicam-vid` (Raspberry Pi's modern camera tool) piped through FFmpeg to create RTSP
streams.
Add the following to the `paths` section in `mediamtx.yml`:
```yaml
paths:
@ -423,45 +539,135 @@ paths:
runOnInitRestart: yes
```
also change rtspAddress: :8554\
to rtspAddress: :8900\
Otherwise there is a conflict with frigate.
This configuration:
With this, you should be able to start mediamtx.
- **`cam1` and `cam2`**: Define two camera paths
- **`rpicam-vid`**: Captures YUV420 video from Raspberry Pi cameras
- **`ffmpeg`**: Transcodes the raw video to H.264 RTSP stream
- **`runOnInitRestart: yes`**: Automatically restarts the stream if it fails
### Port Configuration
Change the default RTSP port to avoid conflicts with Frigate:
In `mediamtx.yml`, change:
```yaml
rtspAddress: :8554
```
To:
```yaml
rtspAddress: :8900
```
Otherwise there will be a port conflict with Frigate.
### Start MediaMTX
Run MediaMTX in the foreground to verify it's working:
```zsh
./mediamtx
```
If there is no error, you can verify your stream through vlc under rtsp://airaspi.local:8900/cam1 (default would be 8554, but we changed it in the config file)
If there are no errors, verify your streams using VLC or another RTSP client:
### Current Status
- `rtsp://airaspi.local:8900/cam1`
- `rtsp://airaspi.local:8900/cam2`
I get working streams from both cameras, sending them out at 30fps at 720p.
frigate, however limits the display fps to 5, which is depressing to watch, especially since the tpu doesnt even break a little sweat.
Note: Default RTSP port is 8554, but we changed it to 8900 in the config.
Frigate claime that the TPU is good for up to 10 cameras, so there is headroom.
## Current Status and Performance
The stram is completely errant and drops frames left and right. I have sometimes seen detect fps of 0.2, but the TPU speed should definitely not be the bottleneck here. Maybe attach the cameras to a separate device and stream from there?
### What's Working
The biggest issue here is that the google folx seems to have abandoned the coral, even though they just released a new piece of hardware for it.
Their most RECENT python build is 3.9.
Specifically, pycoral seems to be the problem there. without a decent update, I will be confined to debian 10, with python 3.7.3.
That sucks.
There are custom wheels, but nothing that seems plug and play.
The system successfully streams from both cameras at 30fps and 720p resolution. The
Coral Edge TPU performs object detection with minimal latency - the TPU itself is not
breaking a sweat, maintaining consistently high performance.
About the rest of this setup:
The decision to go for m.2 E key to save money, instead of spending more on the usb version was a huge mistake.
Please do yourself a favor and spend the extra 40 bucks.
Technically, its probably faster and better with continuous operation, but i have yet to feel the benefit of that.
According to Frigate documentation, the TPU can handle up to 10 cameras, so there's
significant headroom for expansion.
### TODOs
### Current Issues
- add images and screenshots to the build log
- Check whether vdo.ninja is a viable way to add mobile streams. then Smartphone stream evaluation would be on the horizon.
- Bother the mediamtx makers about the libcamera bump, so we can get rid of the rpicam-vid hack.
I suspect there is quirte a lot of performance lost there.
- tweak the frigate config to get snapshots and maybe build an image / video database to later train a custom model.
- worry about attaching an external ssd and saving the video files on it.
- find a way to export the landmark points from frigate. maybe send them via osc like in pose2art?
- find a different hat that lets me access the other TPU? I have the dual version, but can currently only acces 1 of the 2 TPUs due to hardware restrictions.
However, there are several significant problems hampering the system:
**1. Frigate Display Limitations**
Frigate limits the display FPS to 5, which is depressing to watch, especially since the
TPU doesn't even break a sweat. The hardware is clearly capable of much more, but
software limitations hold it back.
**2. Stream Stability Problems**
The stream is completely errant and drops frames constantly. I've sometimes observed
detect FPS as low as 0.2, but the TPU speed should definitely not be the bottleneck
here. One potential solution might be to attach the cameras to a separate device and
stream from there.
**3. Coral Software Abandonment**
The biggest issue is that Google seems to have abandoned the Coral ecosystem, even
though they just released new hardware for it. Their most recent Python build supports
only Python 3.9.
Specifically, `pycoral` appears to be the problem - without a decent update, I'm
confined to Debian 10 with Python 3.7.3. That sucks. There are custom wheels available,
but nothing that seems plug-and-play.
This severely limits the ability to use modern software and libraries with the system.
## Reflections and Lessons Learned
### Hardware Decisions
**The M.2 E Key Choice**
The decision to go for the M.2 E key version to save money, instead of spending more on
the USB version, was a huge mistake. Please do yourself a favor and spend the extra 40
bucks.
Technically, it's probably faster and better for continuous operation, but I have yet to
feel the benefit of that. The USB version would have offered far more flexibility and
easier debugging.
## Future Development
Several improvements and experiments are planned to enhance this system:
**Documentation and Visual Aids**
- Add images and screenshots to this build log to make it easier to follow
**Mobile Stream Integration**
- Check whether [vdo.ninja](https://vdo.ninja) is a viable way to add mobile streams,
enabling smartphone camera integration and evaluation
**MediaMTX libcamera Support**
- Reach out to the MediaMTX developers about bumping libcamera support, which would
eliminate the current `rpicam-vid` workaround. I suspect there's quite a lot of
performance lost in the current pipeline.
**Frigate Configuration Refinement**
- Tweak the Frigate config to enable snapshots and potentially build an image/video
database for training custom models later
**Storage Expansion**
- Worry about attaching an external SSD and saving the video files on it for long-term
storage and analysis
**Data Export Capabilities**
- Find a way to export the landmark points from Frigate, potentially sending them via
OSC (like in my [pose2art](/project/pose2art/) project) for creative applications
**Dual TPU Access**
- Find a different HAT that lets me access the other TPU - I have the dual version, but
can currently only access 1 of the 2 TPUs due to hardware restrictions