feat: server-side PPTX conversion + screenshot dashboard system

PPTX Conversion: - Remove LibreOffice from client dependencies (server uses Gotenberg) - Update docs to reflect clients only display pre-rendered PDFs - Clarify server-side conversion workflow in README and copilot-instructions Screenshot System: - Add background screenshot capture in display_manager.py - Implement Wayland/X11 session detection with tool fallback - Add client-side image processing (downscale + JPEG compression) - Create timestamped files + latest.jpg symlink for easy access - Implement file rotation (max 20 screenshots by default) - Enhance simclient.py to transmit via MQTT dashboard topic - Add structured JSON payload with screenshot, timestamp, system info - New env vars: SCREENSHOT_CAPTURE_INTERVAL, SCREENSHOT_MAX_WIDTH, SCREENSHOT_JPEG_QUALITY, SCREENSHOT_MAX_FILES, SCREENSHOT_ALWAYS Architecture: Two-process design with shared screenshots/ volume
2025-11-30 13:49:27 +01:00
parent 65d7b99198
commit d021e21544
4 changed files with 623 additions and 100 deletions
--- a/README.md
+++ b/README.md
@@ -1,6 +1,6 @@
 # Infoscreen Client - Display Manager

-Digital signage system for Raspberry Pi that displays presentations, videos, and web content in kiosk mode. Centrally managed via MQTT with automatic client discovery and heartbeat monitoring.
+Digital signage system for Raspberry Pi that displays presentations, videos, and web content in kiosk mode. Centrally managed via MQTT with automatic client discovery, heartbeat monitoring, and screenshot-based dashboard monitoring.

 ## 🎯 Key Features

@@ -11,6 +11,7 @@ Digital signage system for Raspberry Pi that displays presentations, videos, and
 - **MQTT Integration** - Real-time event management from central server
 - **Group Management** - Organize clients into groups for targeted content
 - **Heartbeat Monitoring** - Regular status updates and screenshot dashboard
+- **Screenshot Dashboard** - Automatic screen capture with Wayland/X11 support, client-side compression
 - **Multi-Content Support** - Presentations, videos, and web pages
 - **Kiosk Mode** - Full-screen display with automatic startup

@@ -26,9 +27,9 @@ Digital signage system for Raspberry Pi that displays presentations, videos, and
 - Raspberry Pi OS (Bookworm or newer)
 - Python 3.x
 - Impressive (PDF presenter with auto-advance)
- (Optional legacy) LibreOffice ONLY if older workflow still sends raw PPTX; otherwise not required
 - Chromium browser (for web content)
 - VLC or MPV (for video playback)
+- Screenshot tools: `scrot` or ImageMagick (X11) OR `grim` or `gnome-screenshot` (Wayland)
 - CEC Utils (for HDMI-CEC TV control - optional)

 ## 🚀 Quick Start
@@ -40,14 +41,17 @@ Digital signage system for Raspberry Pi that displays presentations, videos, and
 cd ~/
 git clone <repository-url> infoscreen-dev
 cd infoscreen-dev
-
 # Install system dependencies
 sudo apt-get update
 sudo apt-get install -y \
    python3 python3-pip python3-venv \
-    libreoffice impressive \
+    impressive \
    chromium-browser vlc \
-    cec-utils
+    cec-utils \
+    scrot imagemagick
+
+# For Wayland systems, install screenshot tools:
+# sudo apt-get install grim gnome-screenshot

 # Create Python virtual environment
 python3 -m venv venv
@@ -67,10 +71,17 @@ ENV=production
 LOG_LEVEL=INFO

 # MQTT Configuration
-MQTT_BROKER=192.168.1.100
-MQTT_PORT=1883
-MQTT_BROKER_FALLBACKS=192.168.1.101,192.168.1.102
+# Timing (seconds)
+HEARTBEAT_INTERVAL=30
+SCREENSHOT_INTERVAL=60           # How often simclient transmits screenshots
+SCREENSHOT_CAPTURE_INTERVAL=60  # How often display_manager captures screenshots
+DISPLAY_CHECK_INTERVAL=5

+# Screenshot Configuration
+SCREENSHOT_MAX_WIDTH=800         # Downscale to this width (preserves aspect)
+SCREENSHOT_JPEG_QUALITY=70       # JPEG quality 1-95 (lower = smaller files)
+SCREENSHOT_MAX_FILES=20          # Keep this many screenshots (rotation)
+SCREENSHOT_ALWAYS=0              # Set to 1 to force capture even when no event active
 # Timing (seconds)
 HEARTBEAT_INTERVAL=30
 SCREENSHOT_INTERVAL=60
@@ -116,11 +127,12 @@ Or use the startup script:

 The system uses **Impressive** as the PDF presenter with native auto-advance and loop support:

-1. **PPTX files** are converted to PDF server-side (client receives ready-made PDFs)
-2. **PDF files** are displayed directly with Impressive
-3. **Auto-advance** uses Impressive's built-in `--auto` parameter
-4. **Loop mode** uses Impressive's `--wrap` parameter (infinite loop)
-5. **Auto-quit** uses Impressive's `--autoquit` parameter (exit after last slide)
+1. **Server-side rendering**: PPTX files are converted to PDF by the server using Gotenberg
+2. **Client receives PDFs**: Events contain pre-rendered PDF files ready for display
+3. **Direct display**: PDF files are displayed directly with Impressive (no client-side conversion needed)
+4. **Auto-advance** uses Impressive's built-in `--auto` parameter
+5. **Loop mode** uses Impressive's `--wrap` parameter (infinite loop)
+6. **Auto-quit** uses Impressive's `--autoquit` parameter (exit after last slide)

 ### Event JSON Format

@@ -277,13 +289,22 @@ Interactive menu for testing:

 **Loop mode (infinite):**
 ```bash
-./scripts/test-impressive-loop.sh
-```
-
-### Test MQTT Communication
+### Test Screenshot Capture

 ```bash
-./scripts/test-mqtt.sh
+./scripts/test-screenshot.sh
+```
+
+Captures test screenshot for dashboard monitoring.
+
+**Manual test:**
+```bash
+export SCREENSHOT_ALWAYS=1
+export SCREENSHOT_CAPTURE_INTERVAL=5
+python3 src/display_manager.py &
+sleep 15
+ls -lh src/screenshots/
+```
 ```

 Verifies MQTT broker connectivity and topics.
@@ -362,29 +383,18 @@ which impressive
 # If not found: sudo apt-get install impressive
 ```

-**Legacy LibreOffice (only if still receiving raw PPTX):**
-```bash
-which libreoffice
-# Optional install (legacy only): sudo apt-get install libreoffice
-```
-
 **Check logs:**
 ```bash
 tail -f logs/display_manager.log
 ```

-### Legacy PPTX Conversion (Deprecated)
-Server performs all PPTX→PDF rendering. Use these commands only if supporting an older workflow still sending PPTX:
-```bash
-libreoffice --headless --convert-to pdf --outdir /tmp presentation.pptx
-ls -l /tmp/*.pdf
-```
-
 **Check disk space:**
 ```bash
 df -h
 ```

+**Note:** PPTX conversion happens server-side via Gotenberg. The client only receives and displays pre-rendered PDF files.
+
 ### Slides don't auto-advance

 **Verify event JSON:**
@@ -460,18 +470,41 @@ This is the fastest workaround if hardware decode is not required or not availab
 ./scripts/test-mqtt.sh
 ```

-**Check broker status:**
+### Screenshots not uploading
+
+**Check which session type you're running:**
 ```bash
-# On server
-sudo systemctl status mosquitto
+echo $WAYLAND_DISPLAY  # Set if Wayland
+echo $DISPLAY          # Set if X11
 ```

-**Try fallback brokers:**
-Edit `.env` and add `MQTT_BROKER_FALLBACKS`
+**Install appropriate screenshot tool:**
+```bash
+# For X11:
+sudo apt-get install scrot imagemagick

-#### Client auto-reconnect and heartbeat behavior (Nov 2025)
- The MQTT client now uses Paho v2 callbacks and `loop_start()` for automatic reconnection with exponential backoff.
- All topic subscriptions are restored in `on_connect` and a discovery message is re-sent on reconnect to re-register the client.
+# For Wayland:
+sudo apt-get install grim gnome-screenshot
+```
+
+**Test screenshot capture:**
+```bash
+export SCREENSHOT_ALWAYS=1  # Force capture even without active event
+./scripts/test-screenshot.sh
+ls -lh src/screenshots/
+```
+
+**Check logs for session detection:**
+```bash
+tail -f logs/display_manager.log | grep -i screenshot
+# Should show: "Screenshot session=wayland" or "Screenshot session=x11"
+```
+
+**Verify simclient is reading screenshots:**
+```bash
+tail -f logs/simclient.log | grep -i screenshot
+# Should show: "Dashboard heartbeat sent with screenshot: latest.jpg"
+```ll topic subscriptions are restored in `on_connect` and a discovery message is re-sent on reconnect to re-register the client.
 - Heartbeats are sent only when connected; if publish occurs during a brief reconnect window, Paho may return rc=4 (NO_CONN). The client performs a short retry and logs the outcome.
 - Occasional `Heartbeat publish failed with code: 4` after broker restart or transient network hiccups is expected and not dangerous. It indicates "not connected at this instant"; the next heartbeat typically succeeds.
 - When to investigate: repeated rc=4 with no succeeding "Heartbeat sent" entries over multiple intervals.
@@ -622,13 +655,159 @@ CEC_POWER_OFF_WAIT=2
 ### Testing

 ```bash
-# Interactive test menu
-./scripts/test-hdmi-cec.sh
+## 📸 Screenshot System

-# Note: In development mode, option 5 (integration test) is skipped on purpose.
-# Use options 1–4 for manual commands, or set ENV=production to run the integration test.
+The system includes automatic screenshot capture for dashboard monitoring with support for both X11 and Wayland display servers.

-# Manual commands
+### Architecture
+
+**Two-process design:**
+1. **display_manager.py** - Captures screenshots on host OS (has access to display)
+2. **simclient.py** - Transmits screenshots via MQTT (runs in container)
+3. **Shared directory** - `src/screenshots/` volume-mounted between processes
+
+### Screenshot Capture (display_manager.py)
+
+## Recent changes (Nov 2025)
+
+The following notable changes were added after the previous release and are included in this branch:
+
+### Screenshot System Implementation
+- **Screenshot capture** added to `display_manager.py` with background thread
+- **Session detection**: Automatic Wayland vs X11 detection with appropriate tool selection
+- **Wayland support**: `grim`, `gnome-screenshot`, `spectacle` (in order)
+- **X11 support**: `scrot`, `import` (ImageMagick), `xwd`+`convert` (in order)
+- **Image processing**: Client-side downscaling and JPEG compression for bandwidth optimization
+- **File management**: Timestamped screenshots plus `latest.jpg` symlink, automatic rotation
+- **Transmission**: Enhanced `simclient.py` to prefer `latest.jpg`, added detailed logging
+- **Dashboard topic**: Structured JSON payload with screenshot, system info, and client status
+- **Configuration**: New environment variables for capture interval, quality, max width, file rotation
+- **Testing mode**: `SCREENSHOT_ALWAYS=1` forces capture even without active display
+
+### Previous Changes (Oct 2025)
+  - **Wayland**: `grim` → `gnome-screenshot` → `spectacle`
+  - **X11**: `scrot` → `import` (ImageMagick) → `xwd`+`convert`
+
+**Processing pipeline:**
+1. Capture full-resolution screenshot to PNG
+2. Downscale to configured max width (default 800px, preserves aspect ratio)
+3. Convert to JPEG with configured quality (default 70)
+4. Save timestamped file: `screenshot_YYYYMMDD_HHMMSS.jpg`
+5. Create/update `latest.jpg` symlink for easy access
+6. Rotate old screenshots (keeps max 20 by default)
+
+**Capture timing:**
+- Only captures when a display process is active (presentation/video/web)
+- Can be forced with `SCREENSHOT_ALWAYS=1` for testing
+- Default interval: 60 seconds
+
+### Screenshot Transmission (simclient.py)
+
+**Source selection:**
+- Prefers `latest.jpg` symlink (fastest, most recent)
+- Falls back to newest timestamped file if symlink missing
+
+**MQTT topic:**
+```
+infoscreen/{client_id}/dashboard
+```
+
+**Payload structure:**
+```json
+{
+  "timestamp": "2025-11-30T14:23:45.123456",
+  "client_id": "abc123-def456-789",
+  "status": "alive",
+  "screenshot": {
+    "filename": "latest.jpg",
+    "data": "<base64-encoded-image>",
+    "timestamp": "2025-11-30T14:23:40.000000",
+    "size": 45678
+  },
+  "system_info": {
+    "hostname": "infoscreen-pi-01",
+    "ip": "192.168.1.50",
+    "uptime": 1732977825.123456
+  }
+}
+```
+
+### Configuration
+
+```bash
+# Capture settings (display_manager.py)
+SCREENSHOT_CAPTURE_INTERVAL=60   # Seconds between captures
+SCREENSHOT_MAX_WIDTH=800         # Downscale width (0 = no downscaling)
+SCREENSHOT_JPEG_QUALITY=70       # JPEG quality 1-95
+SCREENSHOT_MAX_FILES=20          # Number to keep before rotation
+SCREENSHOT_ALWAYS=0              # Force capture even without active event
+
+# Transmission settings (simclient.py)
+SCREENSHOT_INTERVAL=60           # Seconds between MQTT publishes
+```
+
+### Scalability Recommendations
+
+**Small deployments (<10 clients):**
+- Default settings work well
+- `SCREENSHOT_CAPTURE_INTERVAL=30`, `SCREENSHOT_MAX_WIDTH=800`, `SCREENSHOT_JPEG_QUALITY=70`
+
+**Medium deployments (10-50 clients):**
+- Reduce capture frequency: `SCREENSHOT_CAPTURE_INTERVAL=60`
+- Lower quality: `SCREENSHOT_JPEG_QUALITY=60-65`
+- Ensure broker has adequate bandwidth
+
+**Large deployments (50+ clients):**
+- Further reduce frequency: `SCREENSHOT_CAPTURE_INTERVAL=120`
+- Aggressive compression: `SCREENSHOT_JPEG_QUALITY=50-60`, `SCREENSHOT_MAX_WIDTH=640`
+- Consider implementing hash-based deduplication (skip if unchanged)
+- Monitor MQTT broker load and consider retained message limits
+
+**Very large deployments (200+ clients):**
+- Consider HTTP storage + MQTT metadata pattern instead of base64-over-MQTT
+- Implement screenshot upload to file server, publish only URL via MQTT
+
+### Troubleshooting
+
+**No screenshots being captured:**
+```bash
+# Check session type
+echo "Wayland: $WAYLAND_DISPLAY"  # Set if Wayland
+echo "X11: $DISPLAY"              # Set if X11
+
+# Check logs for tool detection
+tail -f logs/display_manager.log | grep screenshot
+
+# Install appropriate tools
+sudo apt install scrot imagemagick  # X11
+sudo apt install grim              # Wayland
+```
+
+**Screenshots too large:**
+```bash
+# Reduce quality and size
+SCREENSHOT_MAX_WIDTH=640
+SCREENSHOT_JPEG_QUALITY=50
+```
+
+**Not transmitting over MQTT:**
+```bash
+# Check simclient logs
+tail -f logs/simclient.log | grep -i dashboard
+
+# Should see:
+# "Dashboard heartbeat sent with screenshot: latest.jpg (45678 bytes)"
+
+# If NO_CONN errors, check MQTT broker connectivity
+```
+
+---
+
+**Last Updated:** November 2025  
+**Status:** ✅ Production Ready  
+**Tested On:** Raspberry Pi 5, Raspberry Pi OS (Bookworm)
+
+## Recent changes (Nov 2025)
 echo "on 0" | cec-client -s -d 1     # Turn on
 echo "standby 0" | cec-client -s -d 1  # Turn off
 echo "pow 0" | cec-client -s -d 1    # Check status