infoscreen-dev/.github/copilot-instructions.md

Copilot Instructions - Infoscreen Client

Quick Start for AI Assistants

Critical Rules

• ✅ ALWAYS use Impressive for PDF presentations (has native auto-advance/loop)
• ❌ NEVER suggest xdotool approaches (failed on Raspberry Pi due to focus issues)
• ❌ NEVER suggest video conversion (adds complexity, had black screen issues)
• ✅ Virtual environment MUST have pygame + pillow (required for Impressive)
• ✅ Client-side resize/compress screenshots before MQTT transmission
• ✅ Server renders PPTX → PDF via Gotenberg (client only displays PDFs, no LibreOffice needed)

Key Files & Locations

• Display logic: src/display_manager.py (controls presentations/video/web)
• MQTT client: src/simclient.py (event management, heartbeat, discovery)
• Runtime state: src/current_event.json (current active event)
• Config: src/config/client_uuid.txt, src/config/last_group_id.txt, .env
• Logs: logs/display_manager.log, logs/simclient.log
• Screenshots: src/screenshots/ (shared volume between processes)

Common Tasks Quick Reference

Task	File	Key Method/Section
Add event type	display_manager.py	start_display_for_event()
Modify presentation	display_manager.py	start_presentation()
Change MQTT topics	simclient.py	Topic constants/handlers
Update screenshot	display_manager.py	_capture_screenshot()
File downloads	simclient.py	resolve_file_url()

---

Project Overview

Infoscreen Client - Digital signage system for Raspberry Pi. Displays presentations, videos, and web content in kiosk mode. Server-managed via MQTT for educational/research environments with multiple displays.

Architecture: Two-process design

• simclient.py - MQTT communication (container/native)
• display_manager.py - Display control (host OS with X11/Wayland access)

Architecture & Technology Stack

Core Technologies

• Python 3.x - Main application language
• MQTT (paho-mqtt) - Real-time messaging with server
• Impressive - PDF presenter with native auto-advance and loop support
• Environment Variables - Configuration management via .env files
• JSON - Data exchange format for events and configuration
• Base64 - Screenshot transmission encoding
• Threading - Background services (screenshot monitoring)

System Components

• Main Client (simclient.py) - Core MQTT client and event processor
• Display Manager (display_manager.py) - Controls display applications (presentations, videos, web)
• Discovery System - Automatic client registration with server
• Heartbeat Monitoring - Regular status updates and keepalive
• Event Processing - Handles presentation/content switching commands
• Screenshot Service - Dashboard monitoring via image capture (captured by display_manager.py, transmitted by simclient.py)
• File Management - Downloads and manages presentation files
• Group Management - Supports organizing clients into groups

Key Features & Functionality

MQTT Communication Patterns

• Discovery: infoscreen/discovery → infoscreen/{client_id}/discovery_ack
• Heartbeat: Regular infoscreen/{client_id}/heartbeat messages

MQTT Reconnection & Heartbeat (Nov 2025)

• The client uses Paho MQTT v2 callback API with client.loop_start() and client.reconnect_delay_set() to handle automatic reconnection.
• on_connect re-subscribes to all topics (discovery_ack, config, group_id, current group events) and re-sends discovery on reconnect to re-register with the server.
• Heartbeats are gated by client.is_connected() and retry once on NO_CONN (rc=4). Occasional rc=4 warnings are normal right after broker restarts or brief network stalls and typically followed by a successful heartbeat.
• Do not treat single rc=4 heartbeat warnings as failures. Investigate only if multiple consecutive heartbeats fail without recovery.
• Dashboard: Screenshot transmission via infoscreen/{client_id}/dashboard (includes base64-encoded screenshot, timestamp, client status, system info)
• Group Assignment: Server sends group via infoscreen/{client_id}/group_id
• Events: Content commands via infoscreen/events/{group_id}

Event Types Supported

{
  "presentation": {
    "files": [{"url": "https://server/file.pptx", "filename": "file.pptx"}],
    "auto_advance": true,
    "slide_interval": 10,
    "loop": true
  },
  "web": {
    "url": "https://example.com"
  },
  "video": {
    "url": "https://server/video.mp4",
    "loop": false,
    "autoplay": true,
    "volume": 0.8
  }
}

Presentation System (Impressive-Based)

• Server-side conversion: PPTX files are converted to PDF by the server using Gotenberg
• Client receives PDFs: All presentations arrive as pre-rendered PDF files
• Direct display: PDF files are displayed natively with Impressive (no client-side conversion)
• Auto-advance: Native Impressive --auto parameter (no xdotool needed)
• Loop mode: Impressive --wrap parameter for infinite looping
• Auto-quit: Impressive --autoquit parameter to exit after last slide
• Virtual Environment: Uses venv with pygame + pillow for reliable operation
• Reliable: Works consistently on Raspberry Pi without window focus issues

Client Identification

• Hardware Token: SHA256 hash of serial number + MAC addresses
• Persistent UUID: Stored in config/client_uuid.txt
• Group Membership: Persistent group assignment in config/last_group_id.txt

Directory Structure

~/infoscreen-dev/
├── .env                    # Environment configuration
├── README.md               # Complete project documentation
├── IMPRESSIVE_INTEGRATION.md  # Presentation system details
├── QUICK_REFERENCE.md      # Quick command reference
├── .github/                # GitHub configuration
│   └── copilot-instructions.md
├── src/                    # Source code
│   ├── simclient.py       # MQTT client (event management)
│   ├── display_manager.py # Display controller (Impressive integration)
│   ├── current_event.json # Current active event (runtime)
│   ├── config/            # Persistent client data
│   │   ├── client_uuid.txt
│   │   └── last_group_id.txt
│   ├── presentation/      # Downloaded presentation files & PDFs
│   └── screenshots/       # Screenshot captures for monitoring
├── scripts/               # Production & testing utilities
│   ├── start-dev.sh       # Start development client
│   ├── start-display-manager.sh  # Start Display Manager
│   ├── test-display-manager.sh   # Interactive testing menu
│   ├── test-impressive.sh        # Test Impressive (auto-quit)
│   ├── test-impressive-loop.sh   # Test Impressive (loop mode)
│   ├── test-mqtt.sh       # MQTT connectivity test
│   ├── test-screenshot.sh # Screenshot capture test
│   └── present-pdf-auto-advance.sh  # PDF presentation wrapper
├── logs/                  # Application logs
│   ├── simclient.log
│   └── display_manager.log
└── venv/                  # Python virtual environment

Configuration & Environment Variables

Development vs Production

• Development: ENV=development, verbose logging, frequent heartbeats
• Production: ENV=production, minimal logging, longer intervals

HDMI-CEC behavior:

• In development mode (ENV=development) the Display Manager automatically disables HDMI-CEC to avoid constantly switching the TV during local testing. The test helper scripts/test-hdmi-cec.sh also respects this: option 5 (Display Manager CEC integration) detects dev mode and skips running CEC commands. Manual options (1–4) still work for direct cec-client checks.

Key Environment Variables

# Environment
ENV=development|production
DEBUG_MODE=1|0
LOG_LEVEL=DEBUG|INFO|WARNING|ERROR

# MQTT Configuration
MQTT_BROKER=192.168.1.100        # Primary MQTT broker
MQTT_PORT=1883                   # MQTT port
MQTT_BROKER_FALLBACKS=host1,host2  # Fallback brokers

# Timing (seconds)
HEARTBEAT_INTERVAL=10            # Status update frequency
SCREENSHOT_INTERVAL=30           # Dashboard screenshot transmission frequency (simclient.py)
SCREENSHOT_CAPTURE_INTERVAL=30   # Screenshot capture frequency (display_manager.py)

# Screenshot Configuration
SCREENSHOT_MAX_WIDTH=800         # Downscale width (preserves aspect ratio)
SCREENSHOT_JPEG_QUALITY=70       # JPEG compression quality (1-95)
SCREENSHOT_MAX_FILES=20          # Number of screenshots to keep (rotation)
SCREENSHOT_ALWAYS=0              # Force capture even when no display active (testing)
 
# File/API Server (used to download presentation files)
# Defaults to the same host as MQTT_BROKER, port 8000, scheme http.
# If incoming event URLs use host 'server' (or are host-less), simclient rewrites them to this server.
FILE_SERVER_HOST=                # optional; if empty, defaults to MQTT_BROKER
FILE_SERVER_PORT=8000            # default API port
FILE_SERVER_SCHEME=http          # http or https
# FILE_SERVER_BASE_URL=           # optional full override, e.g., http://192.168.1.100:8000

File Server URL Resolution

• The MQTT client (simclient.py) downloads presentation files listed in events.
• To avoid DNS issues when event URLs use http://server:8000/..., the client normalizes such URLs to the configured file server.
• By default, the file server host is the same as MQTT_BROKER, with port 8000 and scheme http.
• You can override behavior using .env variables above; FILE_SERVER_BASE_URL takes precedence over individual host/port/scheme.
• Inline comments in .env are supported; keep comments after a space and # so values stay clean.

Development Patterns & Best Practices

Error Handling

• Robust MQTT connection with fallbacks and retries
• Graceful degradation when services unavailable
• Comprehensive logging with rotating file handlers
• Exception handling for all external operations

State Management

• Event state persisted in current_event.json
• Client configuration persisted across restarts
• Group membership maintained with server synchronization
• Clean state transitions (delete old events on group changes)

Threading Architecture

• Main thread: MQTT communication and heartbeat
• Background thread: Screenshot monitoring service
• Thread-safe operations for shared resources

File Operations

• Automatic directory creation for all output paths
• Safe file operations with proper exception handling
• Atomic writes for configuration files
• Automatic cleanup of temporary/outdated files

Development Workflow

Local Development Setup

1. Clone repository to ~/infoscreen-dev
2. Create virtual environment: python3 -m venv venv
3. Install dependencies: pip install -r src/requirements.txt (includes pygame + pillow for PDF slideshows)
4. Configure .env file with MQTT broker settings
5. Use ./scripts/start-dev.sh for MQTT client or ./scripts/start-display-manager.sh for display manager
6. Important: Virtual environment must include pygame and pillow for PDF auto-advance to work

Testing Components

• ./scripts/test-mqtt.sh - MQTT connectivity
• ./scripts/test-screenshot.sh - Screenshot capture
• ./scripts/test-display-manager.sh - Interactive testing menu
• ./scripts/test-impressive.sh - Test auto-quit presentation mode
• ./scripts/test-impressive-loop.sh - Test loop presentation mode
• ./scripts/test-utc-timestamps.sh - Event timing validation
• Manual event testing via mosquitto_pub or test-display-manager.sh

Production Deployment

• Docker containerization available (docker-compose.production.yml)
• Systemd service integration for auto-start
• Resource limits and health checks configured
• Persistent volume mounts for data

System Dependencies

• Python 3.x runtime + virtual environment
• MQTT broker connectivity
• Display server: X11 or Wayland (for screenshots)
• Impressive - PDF presenter (primary tool, requires pygame + pillow in venv)
• Chromium/Chrome - Web kiosk mode
• VLC - Video playback (python-vlc preferred, vlc binary fallback)
• Screenshot tools:
  • X11: scrot or import (ImageMagick) or xwd+convert
  • Wayland: grim or gnome-screenshot or spectacle

Note: LibreOffice is NOT required on the client. PPTX→PDF conversion is handled server-side by Gotenberg.

Video Playback (python-vlc)

• Preferred: python-vlc (programmatic control: autoplay, loop, volume)
• Fallback: External vlc binary
• Fields: url, autoplay (bool), loop (bool), volume (0.0-1.0 → 0-100)
• URL rewriting: server host → configured file server
• HW decode errors: h264_v4l2m2m failures are normal if V4L2 M2M unavailable; use software decode
• Robust payload parsing with fallbacks
• Topic-specific message handlers
• Retained message support where appropriate

Hardware Considerations

Target Platform

• Primary: Raspberry Pi 4/5 with desktop environment
• Storage: SSD recommended for performance
• Display: HDMI output for presentation display
• Network: WiFi or Ethernet connectivity required

System Dependencies

• Python 3.x runtime
• Network connectivity for MQTT
• Display server (X11 or Wayland) for screenshot capture
• Impressive - PDF presenter with auto-advance (primary presentation tool)
• pygame - Required for Impressive (installed in venv)
• Pillow/PIL - Required for Impressive PDF rendering (installed in venv)
• Chromium/Chrome - Web content display (kiosk mode)
• VLC or MPV - Video playback

Note: LibreOffice is NOT needed on the client. The server converts PPTX to PDF using Gotenberg.

Video playback details (python-vlc)

• The Display Manager now prefers using python-vlc (libvlc) when available for video playback. This enables programmatic control (autoplay, loop, volume) and cleaner termination/cleanup. If python-vlc is not available, the external vlc binary is used as a fallback.
• Supported video event fields: url, autoplay (boolean), loop (boolean), volume (float 0.0-1.0). The manager converts volume to VLC's 0-100 scale.
• URLs using the placeholder host server (for example http://server:8000/...) are rewritten to the configured file server before playback. The resolution priority is: FILE_SERVER_BASE_URL > FILE_SERVER_HOST (or MQTT_BROKER) + FILE_SERVER_PORT + FILE_SERVER_SCHEME.
• Hardware-accelerated decoding errors (e.g., h264_v4l2m2m) may appear when the platform does not expose a V4L2 M2M device. To avoid these errors the Display Manager can be configured to disable hw-decoding (see README env var VLC_HW_ACCEL). By default the manager will attempt hw-acceleration when libvlc supports it.
• Fullscreen / kiosk: the manager will attempt to make libVLC windows fullscreen (remove decorations) when using python-vlc, and the README contains recommended system-level kiosk/session setup for a truly panel-free fullscreen experience.

Security & Privacy

Data Protection

• Hardware identification via cryptographic hash
• No sensitive data in plain text logs
• Local storage of minimal required data only
• Secure MQTT communication (configurable)

Network Security

• Configurable MQTT authentication (if broker requires)
• Firewall-friendly design (outbound connections only)
• Multiple broker fallback for reliability

Presentation System Architecture

How It Works

1. Server-side Conversion → Server converts PPTX to PDF using Gotenberg
2. Event Received → Client receives event with pre-rendered PDF file reference
3. Download PDF → Client downloads PDF from file server
4. Cache PDF → Downloaded PDF stored in presentation/ directory
5. Display with Impressive → Launch with venv environment and parameters:
   • --fullscreen - Full screen mode
   • --nooverview - No slide overview
   • --auto N - Auto-advance every N seconds
   • --wrap - Loop infinitely (if loop: true)
   • --autoquit - Exit after last slide (if loop: false)

Key Parameters

Parameter	Type	Default	Description
auto_advance	boolean	false	Enable automatic slide advancement
slide_interval	integer	10	Seconds between slides
loop	boolean	false	Loop presentation vs. quit after last slide

Why Impressive?

• ✅ Native auto-advance - No xdotool or window management hacks
• ✅ Built-in loop support - Reliable --wrap parameter
• ✅ Works on Raspberry Pi - No focus/window issues
• ✅ Simple integration - Clean command-line interface
• ✅ Maintainable - ~50 lines of code vs. 200+ with xdotool approaches

Implementation Location

• File: src/display_manager.py
• Method: start_presentation()
• Key Logic:
  1. Receive event with PDF file reference (server already converted PPTX)
  2. Download PDF file if not cached
  3. Set up virtual environment for Impressive (pygame + pillow)
  4. Build Impressive command with appropriate parameters
  5. Launch process and monitor

Common Development Tasks

When working on this codebase:

1. Adding new event types: Extend the event processing logic in display_manager.py → start_display_for_event()
2. Modifying presentation behavior: Update display_manager.py → start_presentation()
3. Configuration changes: Update environment variable parsing and validation
4. MQTT topics: Follow the established infoscreen/ namespace pattern
5. Error handling: Always include comprehensive logging and graceful fallbacks
6. State persistence: Use the established config/ directory pattern
7. Testing: Use ./scripts/test-display-manager.sh for interactive testing
8. Presentation testing: Use ./scripts/test-impressive*.sh scripts
9. File download host resolution: If the API server differs from the MQTT broker or uses HTTPS, set FILE_SERVER_* in .env or adjust resolve_file_url() in src/simclient.py.

Troubleshooting Guidelines

Common Issues

• MQTT Connection: Check broker reachability, try fallback brokers
• Screenshots: Verify display environment and permissions
• File Downloads: Check network connectivity and disk space
  • If event URLs use host server and DNS fails, the client rewrites to MQTT_BROKER by default.
  • Ensure MQTT_BROKER points to the correct server IP; if the API differs, set FILE_SERVER_HOST or FILE_SERVER_BASE_URL.
  • Match scheme/port via FILE_SERVER_SCHEME/FILE_SERVER_PORT for HTTPS or non-default ports.
• Group Changes: Monitor log for group assignment messages
• Service Startup: Check systemd logs and environment configuration

Debugging Tools

• Log files in logs/simclient.log and logs/display_manager.log with rotation
• MQTT message monitoring with mosquitto_sub
• Interactive testing menu: ./scripts/test-display-manager.sh
• Component test scripts: test-impressive*.sh, test-mqtt.sh, etc.
• Process monitoring: Check for impressive, libreoffice, chromium, vlc processes

File download URL troubleshooting

• Symptoms:
  • Failed to resolve 'server' or NameResolutionError when downloading files
  • Invalid URL 'http # http or https://...' in simclient.log
• What to check:
  • Look for lines like Lade Datei herunter von: in logs/simclient.log to see the effective URL used
  • Ensure the URL host is the MQTT broker IP (or your configured file server), not server
  • Verify .env values don’t include inline comments as part of the value (e.g., keep FILE_SERVER_SCHEME=http on its own line)
• Fixes:
  • If your API is on the same host as the broker: leave FILE_SERVER_HOST empty (defaults to MQTT_BROKER), keep FILE_SERVER_PORT=8000, and set FILE_SERVER_SCHEME=http or https
  • To override fully, set FILE_SERVER_BASE_URL (e.g., http://192.168.1.100:8000); this takes precedence
  • After changing .env, restart the simclient process
• Expected healthy log sequence:
  • Lade Datei herunter von: http://<broker-ip>:8000/...
  • Followed by "GET /... HTTP/1.1" 200 and Datei erfolgreich heruntergeladen:

Important Notes for AI Assistants

Virtual Environment Requirements (Critical)

• pygame and pillow MUST be installed in venv - Required for Impressive to work
• Display manager uses venv context - Ensures Impressive has access to dependencies
• Installation command: pip install pygame pillow (already in requirements.txt)
• If pygame missing: Impressive will fail with "No module named 'pygame'" error

Presentation System

• ALWAYS use Impressive for PDF presentations (primary solution)
• DO NOT suggest xdotool approaches - they failed on Raspberry Pi due to window focus issues
• DO NOT suggest video conversion - adds complexity, had black screen issues
• All presentations are PDFs - server converts PPTX to PDF using Gotenberg
• No client-side conversion - client only displays pre-rendered PDFs
• Virtual environment is required - pygame + pillow must be available for Impressive
• Loop mode uses --wrap - not custom scripts or workarounds
• Auto-quit uses --autoquit - native Impressive parameter

Testing Approach

• Use ./scripts/test-display-manager.sh for interactive testing
• Use ./scripts/test-impressive-loop.sh to verify loop functionality
• Test individual components with specific test scripts
• Always check logs in logs/ directory for debugging

CEC testing notes:

• In development mode, the CEC integration path is skipped on purpose. To test end-to-end, either set ENV=production temporarily or use the manual options (1–4) in scripts/test-hdmi-cec.sh.

Code Changes

• Display logic is in src/display_manager.py, not simclient.py
• MQTT client (simclient.py) writes events to current_event.json
• Display Manager reads current_event.json and launches appropriate applications
• Two separate processes: simclient.py (MQTT) + display_manager.py (display control)

Documentation

• README.md - Start here for comprehensive overview
• IMPRESSIVE_INTEGRATION.md - Deep dive into presentation system
• QUICK_REFERENCE.md - Quick commands and examples
• Source code has extensive comments and logging

This system is designed for reliability and ease of maintenance in educational environments with multiple deployed clients. The Impressive-based presentation solution provides native auto-advance and loop support without complex window management hacks.

Screenshot System (Nov 2025)

The screenshot capture and transmission system has been implemented with separation of concerns:

Architecture

• Capture: display_manager.py captures screenshots in a background thread and writes to shared screenshots/ directory
• Transmission: simclient.py reads latest screenshot from shared directory and transmits via MQTT dashboard topic
• Sharing: Volume-based sharing between display_manager (host OS) and simclient (container)

Capture Strategy (display_manager.py)

• Session Detection: Automatically detects Wayland vs X11 session
• Wayland Tools: Tries grim, gnome-screenshot, spectacle (in order)
• X11 Tools: Tries scrot, import (ImageMagick), xwd+convert (in order)
• Processing: Downscales to max width (default 800px), JPEG compresses (default quality 70)
• Output: Creates timestamped files (screenshot_YYYYMMDD_HHMMSS.jpg) plus latest.jpg symlink
• Rotation: Keeps max N files (default 20), deletes older
• Timing: Only captures when display process is active (unless SCREENSHOT_ALWAYS=1)

Transmission Strategy (simclient.py)

• Source: Prefers screenshots/latest.jpg if present, falls back to newest timestamped file
• Topic: infoscreen/{client_id}/dashboard
• Format: JSON with base64-encoded image data
• Payload Structure:

  {
    "timestamp": "ISO datetime",
    "client_id": "UUID",
    "status": "alive",
    "screenshot": {
      "filename": "latest.jpg",
      "data": "base64...",
      "timestamp": "ISO datetime",
      "size": 12345
    },
    "system_info": {
      "hostname": "...",
      "ip": "...",
      "uptime": 123456.78
    }
  }
  

• Logging: Logs publish success/failure with file size for monitoring

Scalability Considerations

• Client-side resize/compress: Reduces bandwidth and broker load (recommended for 50+ clients)
• Recommended production settings: SCREENSHOT_CAPTURE_INTERVAL=60, SCREENSHOT_MAX_WIDTH=800, SCREENSHOT_JPEG_QUALITY=60-70
• Future optimization: Hash-based deduplication to skip identical screenshots
• Alternative for large scale: HTTP storage + MQTT metadata (200+ clients)

Testing

• Install capture tools: sudo apt install scrot imagemagick (X11) or sudo apt install grim gnome-screenshot (Wayland)
• Force capture for testing: export SCREENSHOT_ALWAYS=1
• Check logs: tail -f logs/display_manager.log logs/simclient.log
• Verify files: ls -lh src/screenshots/

Troubleshooting

• No screenshots: Check session type in logs, install appropriate tools
• Large payloads: Reduce SCREENSHOT_MAX_WIDTH or SCREENSHOT_JPEG_QUALITY
• Stale screenshots: Check latest.jpg symlink, verify display_manager is running
• MQTT errors: Check dashboard topic logs for publish return codes

Testing & Troubleshooting

Setup:

• X11: sudo apt install scrot imagemagick
• Wayland: sudo apt install grim gnome-screenshot
• Force capture: export SCREENSHOT_ALWAYS=1

Verify:

tail -f logs/display_manager.log | grep screenshot  # Check capture
tail -f logs/simclient.log | grep dashboard        # Check transmission
ls -lh src/screenshots/                             # Check files

Common Issues:

Issue	Check	Fix
No screenshots	Session type in logs	Install tools for X11/Wayland
Large payloads	File sizes	Reduce SCREENSHOT_MAX_WIDTH or SCREENSHOT_JPEG_QUALITY
Stale data	latest.jpg timestamp	Restart display_manager
MQTT errors	Publish return codes	Check broker connectivity

---

Development Notes

Event Types (Scheduler Integration)

• Supported: presentation, webuntis, webpage, website
• Auto-scroll: Only for event_type: "website" (CDP injection + Chrome extension fallback)
• Display manager: Uses event_type to select renderer when available

HDMI-CEC Behavior

• Development mode: CEC auto-disabled (prevents TV cycling during testing)
• Test script: test-hdmi-cec.sh option 5 skips integration test in dev mode
• Manual testing: Options 1-4 work regardless of mode

Code Modification Guidelines

• Presentations: Always use Impressive (--auto, --wrap, --autoquit)
• Web autoscroll: Only for event_type: "website", keep CDP optional
• Screenshot changes: Remember two-process architecture (capture ≠ transmission)
• URL resolution: Handle server host placeholder in file/video URLs