feat: server-side PPTX conversion + screenshot dashboard system

PPTX Conversion: - Remove LibreOffice from client dependencies (server uses Gotenberg) - Update docs to reflect clients only display pre-rendered PDFs - Clarify server-side conversion workflow in README and copilot-instructions Screenshot System: - Add background screenshot capture in display_manager.py - Implement Wayland/X11 session detection with tool fallback - Add client-side image processing (downscale + JPEG compression) - Create timestamped files + latest.jpg symlink for easy access - Implement file rotation (max 20 screenshots by default) - Enhance simclient.py to transmit via MQTT dashboard topic - Add structured JSON payload with screenshot, timestamp, system info - New env vars: SCREENSHOT_CAPTURE_INTERVAL, SCREENSHOT_MAX_WIDTH, SCREENSHOT_JPEG_QUALITY, SCREENSHOT_MAX_FILES, SCREENSHOT_ALWAYS Architecture: Two-process design with shared screenshots/ volume
2025-11-30 13:49:27 +01:00
parent 65d7b99198
commit d021e21544
4 changed files with 623 additions and 100 deletions
--- a/.github/copilot-instructions.md
+++ b/.github/copilot-instructions.md
@@ -1,10 +1,40 @@
 # Copilot Instructions - Infoscreen Client

+## Quick Start for AI Assistants
+
+### Critical Rules
+- ✅ **ALWAYS use Impressive** for PDF presentations (has native auto-advance/loop)
+- ❌ **NEVER suggest xdotool** approaches (failed on Raspberry Pi due to focus issues)
+- ❌ **NEVER suggest video conversion** (adds complexity, had black screen issues)
+- ✅ **Virtual environment MUST have** pygame + pillow (required for Impressive)
+- ✅ **Client-side resize/compress** screenshots before MQTT transmission
+- ✅ **Server renders PPTX → PDF via Gotenberg** (client only displays PDFs, no LibreOffice needed)
+
+### Key Files & Locations
+- **Display logic**: `src/display_manager.py` (controls presentations/video/web)
+- **MQTT client**: `src/simclient.py` (event management, heartbeat, discovery)
+- **Runtime state**: `src/current_event.json` (current active event)
+- **Config**: `src/config/client_uuid.txt`, `src/config/last_group_id.txt`, `.env`
+- **Logs**: `logs/display_manager.log`, `logs/simclient.log`
+- **Screenshots**: `src/screenshots/` (shared volume between processes)
+
+### Common Tasks Quick Reference
+| Task | File | Key Method/Section |
+|------|------|-------------------|
+| Add event type | `display_manager.py` | `start_display_for_event()` |
+| Modify presentation | `display_manager.py` | `start_presentation()` |
+| Change MQTT topics | `simclient.py` | Topic constants/handlers |
+| Update screenshot | `display_manager.py` | `_capture_screenshot()` |
+| File downloads | `simclient.py` | `resolve_file_url()` |
+
+---
+
 ## Project Overview
-This is an **Infoscreen Client** system for Raspberry Pi that creates digital signage displays. The client communicates with a server via MQTT to display presentations, videos, and web content in kiosk mode. It's designed for educational/research environments where multiple displays need to be centrally managed.
- 
-### Server-Side PPTX Rendering (Update Nov 2025)
-The server now performs all PPTX → PDF conversion. The client only ever downloads and displays PDF files with Impressive. Any references below to local LibreOffice conversion are legacy and can be ignored for new deployments. Keep LibreOffice only if you must support older workflows that still send raw PPTX files.
+**Infoscreen Client** - Digital signage system for Raspberry Pi. Displays presentations, videos, and web content in kiosk mode. Server-managed via MQTT for educational/research environments with multiple displays.
+
+**Architecture**: Two-process design
+- `simclient.py` - MQTT communication (container/native)
+- `display_manager.py` - Display control (host OS with X11/Wayland access)

 ## Architecture & Technology Stack

@@ -12,7 +42,6 @@ The server now performs all PPTX → PDF conversion. The client only ever downlo
 - **Python 3.x** - Main application language
 - **MQTT (paho-mqtt)** - Real-time messaging with server
 - **Impressive** - PDF presenter with native auto-advance and loop support
- **LibreOffice** - PPTX to PDF conversion (headless mode)
 - **Environment Variables** - Configuration management via `.env` files
 - **JSON** - Data exchange format for events and configuration
 - **Base64** - Screenshot transmission encoding
@@ -24,7 +53,7 @@ The server now performs all PPTX → PDF conversion. The client only ever downlo
 - **Discovery System** - Automatic client registration with server
 - **Heartbeat Monitoring** - Regular status updates and keepalive
 - **Event Processing** - Handles presentation/content switching commands
- **Screenshot Service** - Dashboard monitoring via image capture
+- **Screenshot Service** - Dashboard monitoring via image capture (captured by display_manager.py, transmitted by simclient.py)
 - **File Management** - Downloads and manages presentation files
 - **Group Management** - Supports organizing clients into groups

@@ -38,7 +67,7 @@ The server now performs all PPTX → PDF conversion. The client only ever downlo
 - `on_connect` re-subscribes to all topics (`discovery_ack`, `config`, `group_id`, current group events) and re-sends discovery on reconnect to re-register with the server.
 - Heartbeats are gated by `client.is_connected()` and retry once on `NO_CONN` (rc=4). Occasional rc=4 warnings are normal right after broker restarts or brief network stalls and typically followed by a successful heartbeat.
 - Do not treat single rc=4 heartbeat warnings as failures. Investigate only if multiple consecutive heartbeats fail without recovery.
- **Dashboard**: Screenshot transmission via `infoscreen/{client_id}/dashboard`
+- **Dashboard**: Screenshot transmission via `infoscreen/{client_id}/dashboard` (includes base64-encoded screenshot, timestamp, client status, system info)
 - **Group Assignment**: Server sends group via `infoscreen/{client_id}/group_id`
 - **Events**: Content commands via `infoscreen/events/{group_id}`

@@ -64,8 +93,9 @@ The server now performs all PPTX → PDF conversion. The client only ever downlo
 ```

 ### Presentation System (Impressive-Based)
- **PDF files** are displayed natively with Impressive PDF presenter (no conversion needed)
- **PPTX files** are automatically converted to PDF using LibreOffice headless
+- **Server-side conversion**: PPTX files are converted to PDF by the server using Gotenberg
+- **Client receives PDFs**: All presentations arrive as pre-rendered PDF files
+- **Direct display**: PDF files are displayed natively with Impressive (no client-side conversion)
 - **Auto-advance**: Native Impressive `--auto` parameter (no xdotool needed)
 - **Loop mode**: Impressive `--wrap` parameter for infinite looping
 - **Auto-quit**: Impressive `--autoquit` parameter to exit after last slide
@@ -133,7 +163,14 @@ MQTT_BROKER_FALLBACKS=host1,host2  # Fallback brokers

 # Timing (seconds)
 HEARTBEAT_INTERVAL=10            # Status update frequency
-SCREENSHOT_INTERVAL=30           # Dashboard screenshot frequency
+SCREENSHOT_INTERVAL=30           # Dashboard screenshot transmission frequency (simclient.py)
+SCREENSHOT_CAPTURE_INTERVAL=30   # Screenshot capture frequency (display_manager.py)
+
+# Screenshot Configuration
+SCREENSHOT_MAX_WIDTH=800         # Downscale width (preserves aspect ratio)
+SCREENSHOT_JPEG_QUALITY=70       # JPEG compression quality (1-95)
+SCREENSHOT_MAX_FILES=20          # Number of screenshots to keep (rotation)
+SCREENSHOT_ALWAYS=0              # Force capture even when no display active (testing)
 
 # File/API Server (used to download presentation files)
 # Defaults to the same host as MQTT_BROKER, port 8000, scheme http.
@@ -200,24 +237,25 @@ FILE_SERVER_SCHEME=http          # http or https
 - Systemd service integration for auto-start
 - Resource limits and health checks configured
 - Persistent volume mounts for data
+### System Dependencies
+- Python 3.x runtime + virtual environment
+- MQTT broker connectivity
+- Display server: X11 or Wayland (for screenshots)
+- **Impressive** - PDF presenter (primary tool, requires pygame + pillow in venv)
+- **Chromium/Chrome** - Web kiosk mode
+- **VLC** - Video playback (python-vlc preferred, vlc binary fallback)
+- **Screenshot tools**: 
+  - X11: `scrot` or `import` (ImageMagick) or `xwd`+`convert`
+  - Wayland: `grim` or `gnome-screenshot` or `spectacle`

-## Code Style & Conventions
+**Note:** LibreOffice is NOT required on the client. PPTX→PDF conversion is handled server-side by Gotenberg.

-### Python Code Standards
- Environment variable parsing with fallback defaults
- Comprehensive docstrings for complex functions
- Logging at appropriate levels (DEBUG/INFO/WARNING/ERROR)
- Type hints where beneficial for clarity
- Exception handling with specific error types
-
-### Configuration Management
- Environment-first configuration (12-factor app principles)
- Support for inline comments in environment values
- Graceful handling of missing/invalid configuration
- Multiple environment file locations for flexibility
-
-### MQTT Message Handling
- JSON message format validation
+### Video Playback (python-vlc)
+- **Preferred**: python-vlc (programmatic control: autoplay, loop, volume)
+- **Fallback**: External vlc binary
+- **Fields**: `url`, `autoplay` (bool), `loop` (bool), `volume` (0.0-1.0 → 0-100)
+- **URL rewriting**: `server` host → configured file server
+- **HW decode errors**: `h264_v4l2m2m` failures are normal if V4L2 M2M unavailable; use software decode
 - Robust payload parsing with fallbacks
 - Topic-specific message handlers
 - Retained message support where appropriate
@@ -233,13 +271,14 @@ FILE_SERVER_SCHEME=http          # http or https
 ### System Dependencies
 - Python 3.x runtime
 - Network connectivity for MQTT
- Display server (X11) for screenshot capture
+- Display server (X11 or Wayland) for screenshot capture
 - **Impressive** - PDF presenter with auto-advance (primary presentation tool)
 - **pygame** - Required for Impressive (installed in venv)
 - **Pillow/PIL** - Required for Impressive PDF rendering (installed in venv)
- **LibreOffice** - PPTX to PDF conversion (headless mode)
 - Chromium/Chrome - Web content display (kiosk mode)
- VLC or MPV - Video playbook
+- VLC or MPV - Video playback
+
+**Note:** LibreOffice is NOT needed on the client. The server converts PPTX to PDF using Gotenberg.

 ### Video playback details (python-vlc)

@@ -266,10 +305,10 @@ FILE_SERVER_SCHEME=http          # http or https
 ## Presentation System Architecture

 ### How It Works
-1. **PDF File Received** → Event contains PDF file reference → Use directly (no conversion)
-2. **PPTX File Received** → Event contains PPTX file reference
-3. **Convert to PDF** → LibreOffice headless: `libreoffice --headless --convert-to pdf`
-4. **Cache PDF** → Converted PDF stored in `presentation/` directory
+1. **Server-side Conversion** → Server converts PPTX to PDF using Gotenberg
+2. **Event Received** → Client receives event with pre-rendered PDF file reference
+3. **Download PDF** → Client downloads PDF from file server
+4. **Cache PDF** → Downloaded PDF stored in `presentation/` directory
 5. **Display with Impressive** → Launch with venv environment and parameters:
   - `--fullscreen` - Full screen mode
   - `--nooverview` - No slide overview
@@ -295,8 +334,8 @@ FILE_SERVER_SCHEME=http          # http or https
 - **File**: `src/display_manager.py`
 - **Method**: `start_presentation()`
 - **Key Logic**: 
-  1. Check if PDF file (use directly) or PPTX (needs conversion)
-  2. Convert PPTX to PDF if needed (cached for reuse)
+  1. Receive event with PDF file reference (server already converted PPTX)
+  2. Download PDF file if not cached
  3. Set up virtual environment for Impressive (pygame + pillow)
  4. Build Impressive command with appropriate parameters
  5. Launch process and monitor
@@ -362,8 +401,8 @@ When working on this codebase:
 - **ALWAYS use Impressive** for PDF presentations (primary solution)
 - **DO NOT suggest xdotool approaches** - they failed on Raspberry Pi due to window focus issues
 - **DO NOT suggest video conversion** - adds complexity, had black screen issues
- **PDF files are supported natively** - no conversion needed
- **PPTX conversion is automatic** - LibreOffice headless handles PPTX→PDF
+- **All presentations are PDFs** - server converts PPTX to PDF using Gotenberg
+- **No client-side conversion** - client only displays pre-rendered PDFs
 - **Virtual environment is required** - pygame + pillow must be available for Impressive
 - **Loop mode uses `--wrap`** - not custom scripts or workarounds
 - **Auto-quit uses `--autoquit`** - native Impressive parameter
@@ -391,20 +430,103 @@ CEC testing notes:

 This system is designed for reliability and ease of maintenance in educational environments with multiple deployed clients. The Impressive-based presentation solution provides native auto-advance and loop support without complex window management hacks.

-## Recent changes (Oct 2025)
+## Screenshot System (Nov 2025)

-Summary of additions since the last Copilot instructions update:
+The screenshot capture and transmission system has been implemented with separation of concerns:

- Added `event_type` handling for scheduler events: `presentation`, `webuntis`, `webpage`, `website`. The display manager now uses `event_type` when available to pick the correct display path.
- Implemented auto-scroll for long websites. Two strategies are available:
-  - CDP injection via DevTools websocket using `websocket-client` and `requests`. The injector attempts to evaluate an auto-scroll script inside the page context. Default duration: 60s.
-  - Local Chrome extension fallback (`src/chrome_autoscroll`) loaded with `--load-extension` when CDP handshake is blocked (403), ensuring reliable autoscroll operation.
- New helper scripts for debugging DevTools handshake: `scripts/test_cdp.py` and `scripts/test_cdp_origins.py`.
- Updated `requirements.txt` to include `websocket-client`.
+### Architecture
+- **Capture**: `display_manager.py` captures screenshots in a background thread and writes to shared `screenshots/` directory
+- **Transmission**: `simclient.py` reads latest screenshot from shared directory and transmits via MQTT dashboard topic
+- **Sharing**: Volume-based sharing between display_manager (host OS) and simclient (container)

-Runtime behavior adjustments:
- HDMI-CEC is automatically disabled in development mode; both the Display Manager and the CEC integration test (option 5) honor this to protect developer setups from constant TV power cycling.
+### Capture Strategy (display_manager.py)
+- **Session Detection**: Automatically detects Wayland vs X11 session
+- **Wayland Tools**: Tries `grim`, `gnome-screenshot`, `spectacle` (in order)
+- **X11 Tools**: Tries `scrot`, `import` (ImageMagick), `xwd`+`convert` (in order)
+- **Processing**: Downscales to max width (default 800px), JPEG compresses (default quality 70)
+- **Output**: Creates timestamped files (`screenshot_YYYYMMDD_HHMMSS.jpg`) plus `latest.jpg` symlink
+- **Rotation**: Keeps max N files (default 20), deletes older
+- **Timing**: Only captures when display process is active (unless `SCREENSHOT_ALWAYS=1`)

-Notes for Copilot:
- When adding or modifying display logic, prefer Impressive for PDF presentations and avoid xdotool approaches for slideshow control.
- For web events, ensure autoscroll is only activated for `event_type: "website"` and keep the CDP injection optional/fallback-only when feasible.
+### Transmission Strategy (simclient.py)
+- **Source**: Prefers `screenshots/latest.jpg` if present, falls back to newest timestamped file
+- **Topic**: `infoscreen/{client_id}/dashboard`
+- **Format**: JSON with base64-encoded image data
+- **Payload Structure**:
+  ```json
+  {
+    "timestamp": "ISO datetime",
+    "client_id": "UUID",
+    "status": "alive",
+    "screenshot": {
+      "filename": "latest.jpg",
+      "data": "base64...",
+      "timestamp": "ISO datetime",
+      "size": 12345
+    },
+    "system_info": {
+      "hostname": "...",
+      "ip": "...",
+      "uptime": 123456.78
+    }
+  }
+  ```
+- **Logging**: Logs publish success/failure with file size for monitoring
+
+### Scalability Considerations
+- **Client-side resize/compress**: Reduces bandwidth and broker load (recommended for 50+ clients)
+- **Recommended production settings**: `SCREENSHOT_CAPTURE_INTERVAL=60`, `SCREENSHOT_MAX_WIDTH=800`, `SCREENSHOT_JPEG_QUALITY=60-70`
+- **Future optimization**: Hash-based deduplication to skip identical screenshots
+- **Alternative for large scale**: HTTP storage + MQTT metadata (200+ clients)
+
+### Testing
+- Install capture tools: `sudo apt install scrot imagemagick` (X11) or `sudo apt install grim gnome-screenshot` (Wayland)
+- Force capture for testing: `export SCREENSHOT_ALWAYS=1`
+- Check logs: `tail -f logs/display_manager.log logs/simclient.log`
+- Verify files: `ls -lh src/screenshots/`
+
+### Troubleshooting
+- **No screenshots**: Check session type in logs, install appropriate tools
+- **Large payloads**: Reduce `SCREENSHOT_MAX_WIDTH` or `SCREENSHOT_JPEG_QUALITY`
+- **Stale screenshots**: Check `latest.jpg` symlink, verify display_manager is running
+- **MQTT errors**: Check dashboard topic logs for publish return codes
+### Testing & Troubleshooting
+**Setup:**
+- X11: `sudo apt install scrot imagemagick`
+- Wayland: `sudo apt install grim gnome-screenshot`
+- Force capture: `export SCREENSHOT_ALWAYS=1`
+
+**Verify:**
+```bash
+tail -f logs/display_manager.log | grep screenshot  # Check capture
+tail -f logs/simclient.log | grep dashboard        # Check transmission
+ls -lh src/screenshots/                             # Check files
+```
+
+**Common Issues:**
+| Issue | Check | Fix |
+|-------|-------|-----|
+| No screenshots | Session type in logs | Install tools for X11/Wayland |
+| Large payloads | File sizes | Reduce `SCREENSHOT_MAX_WIDTH` or `SCREENSHOT_JPEG_QUALITY` |
+| Stale data | `latest.jpg` timestamp | Restart display_manager |
+| MQTT errors | Publish return codes | Check broker connectivity |
+
+---
+
+## Development Notes
+
+### Event Types (Scheduler Integration)
+- **Supported**: `presentation`, `webuntis`, `webpage`, `website`
+- **Auto-scroll**: Only for `event_type: "website"` (CDP injection + Chrome extension fallback)
+- **Display manager**: Uses `event_type` to select renderer when available
+
+### HDMI-CEC Behavior
+- **Development mode**: CEC auto-disabled (prevents TV cycling during testing)
+- **Test script**: `test-hdmi-cec.sh` option 5 skips integration test in dev mode
+- **Manual testing**: Options 1-4 work regardless of mode
+
+### Code Modification Guidelines
+- **Presentations**: Always use Impressive (`--auto`, `--wrap`, `--autoquit`)
+- **Web autoscroll**: Only for `event_type: "website"`, keep CDP optional
+- **Screenshot changes**: Remember two-process architecture (capture ≠ transmission)
+- **URL resolution**: Handle `server` host placeholder in file/video URLs