aron/awebsite
aron/awebsite
1
0
Fork 0
Code Issues Pull requests Wiki Activity Actions

more translation

Browse source
This commit is contained in:
Aron Petau 2025-10-06 18:01:45 +02:00
parent 2ce9ca50b5
commit a41be821c1
997 changed files with 33247 additions and 32490 deletions
Download patch file Download diff file Expand all files Collapse all files

462
content/project/2024-01-30-airaspi-build-log/index.md
View file

@ -22,15 +22,13 @@ show_shares = true
## AI-Raspi Build Log
This should document the rough steps to recreate airaspi as I go along.
This document chronicles the process of building a custom edge computing device for real-time image recognition and object detection. The goal was to create a portable, self-contained system that could operate independently of cloud infrastructure.
Rough Idea: Build an edge device with image recognition and object detection capabilites.\
It should be realtime, aiming for 30fps at 720p.\
Portability and usage at installations is a priority, so it has to function without active internet connection and be as small as possible.\
It would be a real Edge Device, with no computation happening in the cloud.
**Project Goals:**
Inspo from: [pose2art](https://github.com/MauiJerry/Pose2Art)
Build an edge device with image recognition and object detection capabilities that can process video in real-time, targeting 30fps at 720p resolution. Portability and autonomous operation are critical requirements—the device must function without an active internet connection and maintain a compact form factor suitable for installation environments. All computation happens locally on the device itself, making it a true edge computing solution with no cloud dependency.
This project was inspired by [pose2art](https://github.com/MauiJerry/Pose2Art), which demonstrated the creative potential of real-time pose detection for interactive installations.
## Hardware
@ -45,41 +43,47 @@ Inspo from: [pose2art](https://github.com/MauiJerry/Pose2Art)
## Setup
### Most important sources used
### Primary Resources
[coral.ai](https://www.coral.ai/docs/m2/get-started/#requirements)
[Jeff Geerling](https://www.jeffgeerling.com/blog/2023/pcie-coral-tpu-finally-works-on-raspberry-pi-5)
[Frigate NVR](https://docs.frigate.video)
This build wouldn't have been possible without the excellent documentation and troubleshooting guides from the community. The primary sources I relied on throughout this project were:
### Raspberry Pi OS
- [coral.ai official documentation](https://www.coral.ai/docs/m2/get-started/#requirements) - Google's official setup guide for the M.2 Edge TPU
- [Jeff Geerling's blog](https://www.jeffgeerling.com/blog/2023/pcie-coral-tpu-finally-works-on-raspberry-pi-5) - Critical PCIe configuration insights for Raspberry Pi 5
- [Frigate NVR documentation](https://docs.frigate.video) - Comprehensive guide for the network video recorder software
I used the Raspberry Pi Imager to flash the latest Raspberry Pi OS Lite to a SD Card.
### Raspberry Pi OS Installation
Needs to be Debian Bookworm.\
Needs to be the full arm64 image (with desktop), otherwise you will get into camera driver hell.
{: .notice}
I used the Raspberry Pi Imager to flash the latest Raspberry Pi OS to an SD card. The OS choice is critical for camera compatibility.
Settings applied:
> [!IMPORTANT]
> Needs to be Debian Bookworm.
> Needs to be the full arm64 image (with desktop), otherwise you will get into camera
> driver hell.
- used the default arm64 image (with desktop)
- enable custom settings:
- enable ssh
- set wifi country
- set wifi ssid and password
- set locale
- set hostname: airaspi
**Initial Configuration Settings:**
### update
Using the Raspberry Pi Imager's advanced settings, I configured the following before flashing:
This is always good practice on a fresh install. It takes quite long with the full os image.
- Used the default arm64 image (with desktop) - critical for camera driver compatibility
- Enabled custom settings for headless operation
- Enabled SSH for remote access
- Configured WiFi country code for legal compliance
- Set WiFi SSID and password for automatic network connection
- Configured locale settings for proper timezone and keyboard layout
- Set custom hostname: `airaspi` for easy network identification
### System Update
After the initial boot, updating the system is essential.
This process can take considerable time with the full desktop image, but ensures all packages are current and security patches are applied.
```zsh
sudo apt update && sudo apt upgrade -y && sudo reboot
```
### prep system for coral
### Preparing the System for Coral TPU
Thanks again @Jeff Geerling, this is completely out of my comfort zone, I rely on people writing solid tutorials like this one.
The Raspberry Pi 5's PCIe interface requires specific configuration to work with the Coral Edge TPU. This section was the most technically challenging, involving kernel modifications and device tree changes. A huge thanks to Jeff Geerling for documenting this process—without his detailed troubleshooting, this would have been nearly impossible.
```zsh
# check kernel version
@ -124,29 +128,38 @@ sudo nano /boot/firmware/cmdline.txt
sudo reboot
```
### change device tree
### Modifying the Device Tree
#### wrong device tree
#### Initial Script Attempt (Deprecated)
The script simply did not work for me.
Initially, there was an automated script available that was supposed to handle the device tree modifications. However, this script proved problematic and caused issues during my build.
maybe this script is the issue?
i will try again without it
{: .notice}
> [!WARNING]
> maybe this script is the issue?
> i will try again without it
```zsh
curl https://gist.githubusercontent.com/dataslayermedia/714ec5a9601249d9ee754919dea49c7e/raw/32d21f73bd1ebb33854c2b059e94abe7767c3d7e/coral-ai-pcie-edge-tpu-raspberrypi-5-setup | sh
```
- Yes it was the issue, wrote a comment about it on the gist
[comment](https://gist.github.com/dataslayermedia/714ec5a9601249d9ee754919dea49c7e?permalink_comment_id=4860232#gistcomment-4860232)
Yes, it was the problematic script. I left a comment documenting the issue on the original gist:
[My comment on the gist](https://gist.github.com/dataslayermedia/714ec5a9601249d9ee754919dea49c7e?permalink_comment_id=4860232#gistcomment-4860232)
What to do instead?
#### Manual Device Tree Modification (Recommended)
Here, I followed Jeff Geerling down to the T. Please refer to his tutorial for more information.
Instead of relying on the automated script, I followed Jeff Geerling's manual approach.
This method gives you complete control over the process and helps understand what's
actually happening under the hood.
In the meantime the Script got updated and it is now recommended again.
{: .notice}
> [!NOTE]
> In the meantime the Script got updated and it is now recommended again.
The device tree modification process involves backing up the current device tree blob
(DTB), decompiling it to a readable format, editing the MSI parent reference to fix
PCIe compatibility issues, and then recompiling it back to binary format. Here's the
step-by-step process:
**1. Back up and Decompile the Device Tree**
```zsh
# Back up the current dtb
@ -165,14 +178,31 @@ nano ~/test.dts
# Recompile the dtb and move it back to the firmware directory
dtc -I dts -O dtb ~/test.dts -o ~/test.dtb
sudo mv ~/test.dtb /boot/firmware/bcm2712-rpi-5-b.dtb
# Reboot for changes to take effect
sudo reboot
```
Note: msi- parent sems to carry the value <0x2c> nowadays, cost me a few hours.
{: .notice}
> [!NOTE]
> Note: msi-parent seems to carry the value <0x2c> nowadays, cost me a few hours.
### install apex driver
**2. Verify the Changes**
following instructions from [coral.ai](https://coral.ai/docs/m2/get-started#2a-on-linux)
After rebooting, check that the Coral TPU is recognized by the system:
```zsh
lspci -nn | grep 089a
```
You should see output similar to: `0000:01:00.0 System peripheral [0880]: Global Unichip Corp. Coral Edge TPU [1ac1:089a]`
### Installing the Apex Driver
With the device tree properly configured, the next step is installing Google's Apex
driver for the Coral Edge TPU. This driver enables communication between the operating
system and the TPU hardware.
Following the official instructions from [coral.ai](https://coral.ai/docs/m2/get-started#2a-on-linux):
```zsh
echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" | sudo tee /etc/apt/sources.list.d/coral-edgetpu.list
@ -188,61 +218,88 @@ sudo sh -c "echo 'SUBSYSTEM==\"apex\", MODE=\"0660\", GROUP=\"apex\"' >> /etc/ud
sudo groupadd apex
sudo adduser $USER apex
sudo reboot
```
Verify with
This sequence:
1. Adds Google's package repository and GPG key
2. Installs the gasket DKMS module (kernel driver) and Edge TPU runtime library
3. Creates udev rules for device permissions
4. Creates an `apex` group and adds your user to it
5. Reboots to load the driver
After the reboot, verify the installation:
```zsh
lspci -nn | grep 089a
```
- should display the connected tpu
This should display the connected Coral TPU as a PCIe device.
Next, confirm the device node exists with proper permissions:
```zsh
sudo reboot
ls -l /dev/apex_0
```
confirm with, if the output is not /dev/apex_0, something went wrong
If the output shows `/dev/apex_0` with appropriate group permissions, the installation
was successful. If not, review the udev rules and group membership.
### Testing with Example Models
To verify the TPU is functioning correctly, we'll use Google's example classification
script with a pre-trained MobileNet model:
```zsh
ls /dev/apex_0
# Install Python packages
sudo apt-get install python3-pycoral
# Download example code and models
mkdir -p ~/coral && cd ~/coral
git clone https://github.com/google-coral/pycoral.git
cd pycoral
# Run bird classification example
python3 examples/classify_image.py \
--model test_data/mobilenet_v2_1.0_224_inat_bird_quant_edgetpu.tflite \
--labels test_data/inat_bird_labels.txt \
--input test_data/parrot.jpg
```
### Docker
The output should show inference results with confidence scores, confirming the Edge
TPU is working correctly.
Install docker, use the official instructions for debian.
### Docker Installation
Docker provides containerization for the applications we'll be running (Frigate,
MediaMTX, etc.). This keeps dependencies isolated and makes deployment much cleaner.
Install Docker using the official convenience script from
[docker.com](https://docs.docker.com/engine/install/debian/#install-using-the-convenience-script):
```zsh
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh
```
```zsh
# add user to docker group
sudo groupadd docker
sudo usermod -aG docker $USER
```
Probably a source with source .bashrc would be enough, but I rebooted anyways
{: .notice}
After installation, log out and back in for group membership changes to take effect.
```zsh
sudo reboot
```
```zsh
# verify with
docker run hello-world
```
### set docker to start on boot
Configure Docker to start automatically on boot:
```zsh
sudo systemctl enable docker.service
sudo systemctl enable containerd.service
```
### Test the edge tpu
### Test the Edge TPU (Optional)
To verify the Edge TPU works inside a Docker container, we can build a test image.
This is particularly useful if you plan to use the TPU with containerized applications.
Create a test directory and Dockerfile:
```zsh
mkdir coraltest
@ -266,65 +323,89 @@ RUN echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" \
RUN curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add -
RUN apt-get update
RUN apt-get install -y edgetpu-examples
RUN apt-get install libedgetpu1-std
CMD /bin/bash
```
Build and run the test container, passing through the Coral device:
```zsh
# build the docker container
docker build -t "coral" .
```
```zsh
# run the docker container
docker run -it --device /dev/apex_0:/dev/apex_0 coral /bin/bash
```
Inside the container, run an inference example:
```zsh
# run an inference example from within the container
python3 /usr/share/edgetpu/examples/classify_image.py --model /usr/share/edgetpu/examples/models/mobilenet_v2_1.0_224_inat_bird_quant_edgetpu.tflite --label /usr/share/edgetpu/examples/models/inat_bird_labels.txt --image /usr/share/edgetpu/examples/images/bird.bmp
```
Here, you should see the inference results from the edge tpu with some confidence values.\
If it ain't so, safest bet is a clean restart
You should see inference results with confidence values from the Edge TPU. If not, try
a clean restart of the system.
### Portainer
### Portainer (Optional)
This is optional, gives you a browser gui for your various docker containers
{: .notice}
Portainer provides a web-based GUI for managing Docker containers, images, and volumes.
While not required, it makes container management significantly more convenient.
Install portainer
> [!NOTE]
> This is optional, gives you a browser GUI for your various docker containers.
Install Portainer:
```zsh
docker volume create portainer_data
docker run -d -p 8000:8000 -p 9443:9443 --name portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce:latest
```
open portainer in browser and set admin password
Access Portainer in your browser and set an admin password:
- should be available under <https://airaspi.local:9443>
- Navigate to: <https://airaspi.local:9443>
### vnc in raspi-config
### VNC Setup (Optional)
optional, useful to test your cameras on your headless device.
You could of course also attach a monitor, but i find this more convenient.
{: .notice}
VNC provides remote desktop access to your headless Raspberry Pi. This is particularly
useful for testing cameras and debugging visual issues without connecting a physical
monitor.
> [!NOTE]
> This is optional, useful to test your cameras on your headless device. You could attach
> a monitor, but I find VNC more convenient.
Enable VNC through the Raspberry Pi configuration tool:
```zsh
sudo raspi-config
```
-- interface otions, enable vnc
Navigate to: **Interface Options** → **VNC** → **Enable**
### connect through vnc viewer
### Connecting through VNC Viewer
Install vnc viewer on mac.\
Use airaspi.local:5900 as address.
Install [RealVNC Viewer](https://www.realvnc.com/en/connect/download/viewer/) on your
computer (available for macOS, Windows, and Linux).
### working docker-compose for frigate
Connect using the address: `airaspi.local:5900`
Start this as a custom template in portainer.
You'll be prompted for your Raspberry Pi username and password. Once connected, you'll
have full remote desktop access for testing cameras and debugging.
Important: you need to change the paths to your own paths
{: .notice}
## Frigate NVR Setup
Frigate is a complete Network Video Recorder (NVR) with real-time object detection
powered by the Coral Edge TPU. It's the heart of this edge AI system.
### Docker Compose Configuration
This setup uses Docker Compose to define the Frigate container with all necessary
configurations. If you're using Portainer, you can add this as a custom stack.
> [!IMPORTANT]
> Important: you need to change the paths to your own paths.
```yaml
version: "3.9"
@ -355,11 +436,25 @@ services:
FRIGATE_RTSP_PASSWORD: "******"
```
### Working frigate config file
Key configuration points in this Docker Compose file:
Frigate wants this file wherever you specified earlier that it will be.\
This is necessary just once. Afterwards, you will be able to change the config in the gui.
{: .notice}
- **Privileged mode** and **device mappings**: Required for accessing hardware (TPU,
cameras)
- **Shared memory size**: Allocated for processing video frames efficiently
- **Port mappings**: Exposes Frigate's web UI (5000) and RTSP streams (8554)
- **Volume mounts**: Persists recordings, config, and database
### Frigate Configuration File
Frigate requires a YAML configuration file to define cameras, detectors, and detection
zones. Create this file at the path you specified in the docker-compose file (e.g.,
`/home/aron/frigate/config.yml`).
> [!NOTE]
> This is necessary just once. Afterwards, you will be able to change the config in the
> GUI.
Here's a working configuration using the Coral TPU:
```yaml
mqtt:
@ -394,12 +489,29 @@ cameras:
height: 720 # <+++- update for your camera's resolution
```
### mediamtx
This configuration:
install mediamtx, do not use the docker version, it will be painful
- **Disables MQTT**: Simplifies setup for local-only operation
- **Defines two detectors**: A Coral TPU detector (`coral`) and a CPU fallback
- **Uses default detection model**: Frigate includes a pre-trained model
- **Configures two cameras**: Both set to 1280x720 resolution
- **Uses hardware acceleration**: `preset-rpi-64-h264` for Raspberry Pi 5
- **Detection zones**: Enable only when camera feeds are working properly
double check the chip architecture here, caused me some headache
{: .notice}
## MediaMTX Setup
MediaMTX is a real-time media server that handles streaming from the Raspberry Pi
cameras to Frigate. It's necessary because Frigate doesn't directly support `libcamera`
(the modern Raspberry Pi camera stack).
Install MediaMTX directly on the system (not via Docker - the Docker version has
compatibility issues with libcamera).
> [!WARNING]
> Double-check the chip architecture when downloading - this caused me significant
> headaches during setup.
Download and install MediaMTX:
```zsh
mkdir mediamtx
@ -409,9 +521,13 @@ wget https://github.com/bluenviron/mediamtx/releases/download/v1.5.0/mediamtx_v1
tar xzvf mediamtx_v1.5.0_linux_arm64v8.tar.gz && rm mediamtx_v1.5.0_linux_arm64v8.tar.gz
```
edit the mediamtx.yml file
### MediaMTX Configuration
### working paths section in mediamtx.yml
Edit the `mediamtx.yml` file to configure camera streams. The configuration below uses
`rpicam-vid` (Raspberry Pi's modern camera tool) piped through FFmpeg to create RTSP
streams.
Add the following to the `paths` section in `mediamtx.yml`:
```yaml
paths:
@ -423,45 +539,135 @@ paths:
runOnInitRestart: yes
```
also change rtspAddress: :8554\
to rtspAddress: :8900\
Otherwise there is a conflict with frigate.
This configuration:
With this, you should be able to start mediamtx.
- **`cam1` and `cam2`**: Define two camera paths
- **`rpicam-vid`**: Captures YUV420 video from Raspberry Pi cameras
- **`ffmpeg`**: Transcodes the raw video to H.264 RTSP stream
- **`runOnInitRestart: yes`**: Automatically restarts the stream if it fails
### Port Configuration
Change the default RTSP port to avoid conflicts with Frigate:
In `mediamtx.yml`, change:
```yaml
rtspAddress: :8554
```
To:
```yaml
rtspAddress: :8900
```
Otherwise there will be a port conflict with Frigate.
### Start MediaMTX
Run MediaMTX in the foreground to verify it's working:
```zsh
./mediamtx
```
If there is no error, you can verify your stream through vlc under rtsp://airaspi.local:8900/cam1 (default would be 8554, but we changed it in the config file)
If there are no errors, verify your streams using VLC or another RTSP client:
### Current Status
- `rtsp://airaspi.local:8900/cam1`
- `rtsp://airaspi.local:8900/cam2`
I get working streams from both cameras, sending them out at 30fps at 720p.
frigate, however limits the display fps to 5, which is depressing to watch, especially since the tpu doesnt even break a little sweat.
Note: Default RTSP port is 8554, but we changed it to 8900 in the config.
Frigate claime that the TPU is good for up to 10 cameras, so there is headroom.
## Current Status and Performance
The stram is completely errant and drops frames left and right. I have sometimes seen detect fps of 0.2, but the TPU speed should definitely not be the bottleneck here. Maybe attach the cameras to a separate device and stream from there?
### What's Working
The biggest issue here is that the google folx seems to have abandoned the coral, even though they just released a new piece of hardware for it.
Their most RECENT python build is 3.9.
Specifically, pycoral seems to be the problem there. without a decent update, I will be confined to debian 10, with python 3.7.3.
That sucks.
There are custom wheels, but nothing that seems plug and play.
The system successfully streams from both cameras at 30fps and 720p resolution. The
Coral Edge TPU performs object detection with minimal latency - the TPU itself is not
breaking a sweat, maintaining consistently high performance.
About the rest of this setup:
The decision to go for m.2 E key to save money, instead of spending more on the usb version was a huge mistake.
Please do yourself a favor and spend the extra 40 bucks.
Technically, its probably faster and better with continuous operation, but i have yet to feel the benefit of that.
According to Frigate documentation, the TPU can handle up to 10 cameras, so there's
significant headroom for expansion.
### TODOs
### Current Issues
- add images and screenshots to the build log
- Check whether vdo.ninja is a viable way to add mobile streams. then Smartphone stream evaluation would be on the horizon.
- Bother the mediamtx makers about the libcamera bump, so we can get rid of the rpicam-vid hack.
I suspect there is quirte a lot of performance lost there.
- tweak the frigate config to get snapshots and maybe build an image / video database to later train a custom model.
- worry about attaching an external ssd and saving the video files on it.
- find a way to export the landmark points from frigate. maybe send them via osc like in pose2art?
- find a different hat that lets me access the other TPU? I have the dual version, but can currently only acces 1 of the 2 TPUs due to hardware restrictions.
However, there are several significant problems hampering the system:
**1. Frigate Display Limitations**
Frigate limits the display FPS to 5, which is depressing to watch, especially since the
TPU doesn't even break a sweat. The hardware is clearly capable of much more, but
software limitations hold it back.
**2. Stream Stability Problems**
The stream is completely errant and drops frames constantly. I've sometimes observed
detect FPS as low as 0.2, but the TPU speed should definitely not be the bottleneck
here. One potential solution might be to attach the cameras to a separate device and
stream from there.
**3. Coral Software Abandonment**
The biggest issue is that Google seems to have abandoned the Coral ecosystem, even
though they just released new hardware for it. Their most recent Python build supports
only Python 3.9.
Specifically, `pycoral` appears to be the problem - without a decent update, I'm
confined to Debian 10 with Python 3.7.3. That sucks. There are custom wheels available,
but nothing that seems plug-and-play.
This severely limits the ability to use modern software and libraries with the system.
## Reflections and Lessons Learned
### Hardware Decisions
**The M.2 E Key Choice**
The decision to go for the M.2 E key version to save money, instead of spending more on
the USB version, was a huge mistake. Please do yourself a favor and spend the extra 40
bucks.
Technically, it's probably faster and better for continuous operation, but I have yet to
feel the benefit of that. The USB version would have offered far more flexibility and
easier debugging.
## Future Development
Several improvements and experiments are planned to enhance this system:
**Documentation and Visual Aids**
- Add images and screenshots to this build log to make it easier to follow
**Mobile Stream Integration**
- Check whether [vdo.ninja](https://vdo.ninja) is a viable way to add mobile streams,
enabling smartphone camera integration and evaluation
**MediaMTX libcamera Support**
- Reach out to the MediaMTX developers about bumping libcamera support, which would
eliminate the current `rpicam-vid` workaround. I suspect there's quite a lot of
performance lost in the current pipeline.
**Frigate Configuration Refinement**
- Tweak the Frigate config to enable snapshots and potentially build an image/video
database for training custom models later
**Storage Expansion**
- Worry about attaching an external SSD and saving the video files on it for long-term
storage and analysis
**Data Export Capabilities**
- Find a way to export the landmark points from Frigate, potentially sending them via
OSC (like in my [pose2art](/project/pose2art/) project) for creative applications
**Dual TPU Access**
- Find a different HAT that lets me access the other TPU - I have the dual version, but
can currently only access 1 of the 2 TPUs due to hardware restrictions