infoscreen-dev/.github/copilot-instructions.md

Copilot Instructions - Infoscreen Client

Project Overview

This is an Infoscreen Client system for Raspberry Pi that creates digital signage displays. The client communicates with a server via MQTT to display presentations, videos, and web content in kiosk mode. It's designed for educational/research environments where multiple displays need to be centrally managed.

Server-Side PPTX Rendering (Update Nov 2025)

The server now performs all PPTX → PDF conversion. The client only ever downloads and displays PDF files with Impressive. Any references below to local LibreOffice conversion are legacy and can be ignored for new deployments. Keep LibreOffice only if you must support older workflows that still send raw PPTX files.

Architecture & Technology Stack

Core Technologies

• Python 3.x - Main application language
• MQTT (paho-mqtt) - Real-time messaging with server
• Impressive - PDF presenter with native auto-advance and loop support
• LibreOffice - PPTX to PDF conversion (headless mode)
• Environment Variables - Configuration management via .env files
• JSON - Data exchange format for events and configuration
• Base64 - Screenshot transmission encoding
• Threading - Background services (screenshot monitoring)

System Components

• Main Client (simclient.py) - Core MQTT client and event processor
• Display Manager (display_manager.py) - Controls display applications (presentations, videos, web)
• Discovery System - Automatic client registration with server
• Heartbeat Monitoring - Regular status updates and keepalive
• Event Processing - Handles presentation/content switching commands
• Screenshot Service - Dashboard monitoring via image capture
• File Management - Downloads and manages presentation files
• Group Management - Supports organizing clients into groups

Key Features & Functionality

MQTT Communication Patterns

• Discovery: infoscreen/discovery → infoscreen/{client_id}/discovery_ack
• Heartbeat: Regular infoscreen/{client_id}/heartbeat messages

MQTT Reconnection & Heartbeat (Nov 2025)

• The client uses Paho MQTT v2 callback API with client.loop_start() and client.reconnect_delay_set() to handle automatic reconnection.
• on_connect re-subscribes to all topics (discovery_ack, config, group_id, current group events) and re-sends discovery on reconnect to re-register with the server.
• Heartbeats are gated by client.is_connected() and retry once on NO_CONN (rc=4). Occasional rc=4 warnings are normal right after broker restarts or brief network stalls and typically followed by a successful heartbeat.
• Do not treat single rc=4 heartbeat warnings as failures. Investigate only if multiple consecutive heartbeats fail without recovery.
• Dashboard: Screenshot transmission via infoscreen/{client_id}/dashboard
• Group Assignment: Server sends group via infoscreen/{client_id}/group_id
• Events: Content commands via infoscreen/events/{group_id}

Event Types Supported

{
  "presentation": {
    "files": [{"url": "https://server/file.pptx", "filename": "file.pptx"}],
    "auto_advance": true,
    "slide_interval": 10,
    "loop": true
  },
  "web": {
    "url": "https://example.com"
  },
  "video": {
    "url": "https://server/video.mp4",
    "loop": false,
    "autoplay": true,
    "volume": 0.8
  }
}

Presentation System (Impressive-Based)

• PDF files are displayed natively with Impressive PDF presenter (no conversion needed)
• PPTX files are automatically converted to PDF using LibreOffice headless
• Auto-advance: Native Impressive --auto parameter (no xdotool needed)
• Loop mode: Impressive --wrap parameter for infinite looping
• Auto-quit: Impressive --autoquit parameter to exit after last slide
• Virtual Environment: Uses venv with pygame + pillow for reliable operation
• Reliable: Works consistently on Raspberry Pi without window focus issues

Client Identification

• Hardware Token: SHA256 hash of serial number + MAC addresses
• Persistent UUID: Stored in config/client_uuid.txt
• Group Membership: Persistent group assignment in config/last_group_id.txt

Directory Structure

~/infoscreen-dev/
├── .env                    # Environment configuration
├── README.md               # Complete project documentation
├── IMPRESSIVE_INTEGRATION.md  # Presentation system details
├── QUICK_REFERENCE.md      # Quick command reference
├── .github/                # GitHub configuration
│   └── copilot-instructions.md
├── src/                    # Source code
│   ├── simclient.py       # MQTT client (event management)
│   ├── display_manager.py # Display controller (Impressive integration)
│   ├── current_event.json # Current active event (runtime)
│   ├── config/            # Persistent client data
│   │   ├── client_uuid.txt
│   │   └── last_group_id.txt
│   ├── presentation/      # Downloaded presentation files & PDFs
│   └── screenshots/       # Screenshot captures for monitoring
├── scripts/               # Production & testing utilities
│   ├── start-dev.sh       # Start development client
│   ├── start-display-manager.sh  # Start Display Manager
│   ├── test-display-manager.sh   # Interactive testing menu
│   ├── test-impressive.sh        # Test Impressive (auto-quit)
│   ├── test-impressive-loop.sh   # Test Impressive (loop mode)
│   ├── test-mqtt.sh       # MQTT connectivity test
│   ├── test-screenshot.sh # Screenshot capture test
│   └── present-pdf-auto-advance.sh  # PDF presentation wrapper
├── logs/                  # Application logs
│   ├── simclient.log
│   └── display_manager.log
└── venv/                  # Python virtual environment

Configuration & Environment Variables

Development vs Production

• Development: ENV=development, verbose logging, frequent heartbeats
• Production: ENV=production, minimal logging, longer intervals

HDMI-CEC behavior:

• In development mode (ENV=development) the Display Manager automatically disables HDMI-CEC to avoid constantly switching the TV during local testing. The test helper scripts/test-hdmi-cec.sh also respects this: option 5 (Display Manager CEC integration) detects dev mode and skips running CEC commands. Manual options (1–4) still work for direct cec-client checks.

Key Environment Variables

# Environment
ENV=development|production
DEBUG_MODE=1|0
LOG_LEVEL=DEBUG|INFO|WARNING|ERROR

# MQTT Configuration
MQTT_BROKER=192.168.1.100        # Primary MQTT broker
MQTT_PORT=1883                   # MQTT port
MQTT_BROKER_FALLBACKS=host1,host2  # Fallback brokers

# Timing (seconds)
HEARTBEAT_INTERVAL=10            # Status update frequency
SCREENSHOT_INTERVAL=30           # Dashboard screenshot frequency
 
# File/API Server (used to download presentation files)
# Defaults to the same host as MQTT_BROKER, port 8000, scheme http.
# If incoming event URLs use host 'server' (or are host-less), simclient rewrites them to this server.
FILE_SERVER_HOST=                # optional; if empty, defaults to MQTT_BROKER
FILE_SERVER_PORT=8000            # default API port
FILE_SERVER_SCHEME=http          # http or https
# FILE_SERVER_BASE_URL=           # optional full override, e.g., http://192.168.1.100:8000

File Server URL Resolution

• The MQTT client (simclient.py) downloads presentation files listed in events.
• To avoid DNS issues when event URLs use http://server:8000/..., the client normalizes such URLs to the configured file server.
• By default, the file server host is the same as MQTT_BROKER, with port 8000 and scheme http.
• You can override behavior using .env variables above; FILE_SERVER_BASE_URL takes precedence over individual host/port/scheme.
• Inline comments in .env are supported; keep comments after a space and # so values stay clean.

Development Patterns & Best Practices

Error Handling

• Robust MQTT connection with fallbacks and retries
• Graceful degradation when services unavailable
• Comprehensive logging with rotating file handlers
• Exception handling for all external operations

State Management

• Event state persisted in current_event.json
• Client configuration persisted across restarts
• Group membership maintained with server synchronization
• Clean state transitions (delete old events on group changes)

Threading Architecture

• Main thread: MQTT communication and heartbeat
• Background thread: Screenshot monitoring service
• Thread-safe operations for shared resources

File Operations

• Automatic directory creation for all output paths
• Safe file operations with proper exception handling
• Atomic writes for configuration files
• Automatic cleanup of temporary/outdated files

Development Workflow

Local Development Setup

1. Clone repository to ~/infoscreen-dev
2. Create virtual environment: python3 -m venv venv
3. Install dependencies: pip install -r src/requirements.txt (includes pygame + pillow for PDF slideshows)
4. Configure .env file with MQTT broker settings
5. Use ./scripts/start-dev.sh for MQTT client or ./scripts/start-display-manager.sh for display manager
6. Important: Virtual environment must include pygame and pillow for PDF auto-advance to work

Testing Components

• ./scripts/test-mqtt.sh - MQTT connectivity
• ./scripts/test-screenshot.sh - Screenshot capture
• ./scripts/test-display-manager.sh - Interactive testing menu
• ./scripts/test-impressive.sh - Test auto-quit presentation mode
• ./scripts/test-impressive-loop.sh - Test loop presentation mode
• ./scripts/test-utc-timestamps.sh - Event timing validation
• Manual event testing via mosquitto_pub or test-display-manager.sh

Production Deployment

• Docker containerization available (docker-compose.production.yml)
• Systemd service integration for auto-start
• Resource limits and health checks configured
• Persistent volume mounts for data

Code Style & Conventions

Python Code Standards

• Environment variable parsing with fallback defaults
• Comprehensive docstrings for complex functions
• Logging at appropriate levels (DEBUG/INFO/WARNING/ERROR)
• Type hints where beneficial for clarity
• Exception handling with specific error types

Configuration Management

• Environment-first configuration (12-factor app principles)
• Support for inline comments in environment values
• Graceful handling of missing/invalid configuration
• Multiple environment file locations for flexibility

MQTT Message Handling

• JSON message format validation
• Robust payload parsing with fallbacks
• Topic-specific message handlers
• Retained message support where appropriate

Hardware Considerations

Target Platform

• Primary: Raspberry Pi 4/5 with desktop environment
• Storage: SSD recommended for performance
• Display: HDMI output for presentation display
• Network: WiFi or Ethernet connectivity required

System Dependencies

• Python 3.x runtime
• Network connectivity for MQTT
• Display server (X11) for screenshot capture
• Impressive - PDF presenter with auto-advance (primary presentation tool)
• pygame - Required for Impressive (installed in venv)
• Pillow/PIL - Required for Impressive PDF rendering (installed in venv)
• LibreOffice - PPTX to PDF conversion (headless mode)
• Chromium/Chrome - Web content display (kiosk mode)
• VLC or MPV - Video playbook

Video playback details (python-vlc)

• The Display Manager now prefers using python-vlc (libvlc) when available for video playback. This enables programmatic control (autoplay, loop, volume) and cleaner termination/cleanup. If python-vlc is not available, the external vlc binary is used as a fallback.
• Supported video event fields: url, autoplay (boolean), loop (boolean), volume (float 0.0-1.0). The manager converts volume to VLC's 0-100 scale.
• URLs using the placeholder host server (for example http://server:8000/...) are rewritten to the configured file server before playback. The resolution priority is: FILE_SERVER_BASE_URL > FILE_SERVER_HOST (or MQTT_BROKER) + FILE_SERVER_PORT + FILE_SERVER_SCHEME.
• Hardware-accelerated decoding errors (e.g., h264_v4l2m2m) may appear when the platform does not expose a V4L2 M2M device. To avoid these errors the Display Manager can be configured to disable hw-decoding (see README env var VLC_HW_ACCEL). By default the manager will attempt hw-acceleration when libvlc supports it.
• Fullscreen / kiosk: the manager will attempt to make libVLC windows fullscreen (remove decorations) when using python-vlc, and the README contains recommended system-level kiosk/session setup for a truly panel-free fullscreen experience.

Security & Privacy

Data Protection

• Hardware identification via cryptographic hash
• No sensitive data in plain text logs
• Local storage of minimal required data only
• Secure MQTT communication (configurable)

Network Security

• Configurable MQTT authentication (if broker requires)
• Firewall-friendly design (outbound connections only)
• Multiple broker fallback for reliability

Presentation System Architecture

How It Works

1. PDF File Received → Event contains PDF file reference → Use directly (no conversion)
2. PPTX File Received → Event contains PPTX file reference
3. Convert to PDF → LibreOffice headless: libreoffice --headless --convert-to pdf
4. Cache PDF → Converted PDF stored in presentation/ directory
5. Display with Impressive → Launch with venv environment and parameters:
   • --fullscreen - Full screen mode
   • --nooverview - No slide overview
   • --auto N - Auto-advance every N seconds
   • --wrap - Loop infinitely (if loop: true)
   • --autoquit - Exit after last slide (if loop: false)

Key Parameters

Parameter	Type	Default	Description
auto_advance	boolean	false	Enable automatic slide advancement
slide_interval	integer	10	Seconds between slides
loop	boolean	false	Loop presentation vs. quit after last slide

Why Impressive?

• ✅ Native auto-advance - No xdotool or window management hacks
• ✅ Built-in loop support - Reliable --wrap parameter
• ✅ Works on Raspberry Pi - No focus/window issues
• ✅ Simple integration - Clean command-line interface
• ✅ Maintainable - ~50 lines of code vs. 200+ with xdotool approaches

Implementation Location

• File: src/display_manager.py
• Method: start_presentation()
• Key Logic:
  1. Check if PDF file (use directly) or PPTX (needs conversion)
  2. Convert PPTX to PDF if needed (cached for reuse)
  3. Set up virtual environment for Impressive (pygame + pillow)
  4. Build Impressive command with appropriate parameters
  5. Launch process and monitor

Common Development Tasks

When working on this codebase:

1. Adding new event types: Extend the event processing logic in display_manager.py → start_display_for_event()
2. Modifying presentation behavior: Update display_manager.py → start_presentation()
3. Configuration changes: Update environment variable parsing and validation
4. MQTT topics: Follow the established infoscreen/ namespace pattern
5. Error handling: Always include comprehensive logging and graceful fallbacks
6. State persistence: Use the established config/ directory pattern
7. Testing: Use ./scripts/test-display-manager.sh for interactive testing
8. Presentation testing: Use ./scripts/test-impressive*.sh scripts
9. File download host resolution: If the API server differs from the MQTT broker or uses HTTPS, set FILE_SERVER_* in .env or adjust resolve_file_url() in src/simclient.py.

Troubleshooting Guidelines

Common Issues

• MQTT Connection: Check broker reachability, try fallback brokers
• Screenshots: Verify display environment and permissions
• File Downloads: Check network connectivity and disk space
  • If event URLs use host server and DNS fails, the client rewrites to MQTT_BROKER by default.
  • Ensure MQTT_BROKER points to the correct server IP; if the API differs, set FILE_SERVER_HOST or FILE_SERVER_BASE_URL.
  • Match scheme/port via FILE_SERVER_SCHEME/FILE_SERVER_PORT for HTTPS or non-default ports.
• Group Changes: Monitor log for group assignment messages
• Service Startup: Check systemd logs and environment configuration

Debugging Tools

• Log files in logs/simclient.log and logs/display_manager.log with rotation
• MQTT message monitoring with mosquitto_sub
• Interactive testing menu: ./scripts/test-display-manager.sh
• Component test scripts: test-impressive*.sh, test-mqtt.sh, etc.
• Process monitoring: Check for impressive, libreoffice, chromium, vlc processes

File download URL troubleshooting

• Symptoms:
  • Failed to resolve 'server' or NameResolutionError when downloading files
  • Invalid URL 'http # http or https://...' in simclient.log
• What to check:
  • Look for lines like Lade Datei herunter von: in logs/simclient.log to see the effective URL used
  • Ensure the URL host is the MQTT broker IP (or your configured file server), not server
  • Verify .env values don’t include inline comments as part of the value (e.g., keep FILE_SERVER_SCHEME=http on its own line)
• Fixes:
  • If your API is on the same host as the broker: leave FILE_SERVER_HOST empty (defaults to MQTT_BROKER), keep FILE_SERVER_PORT=8000, and set FILE_SERVER_SCHEME=http or https
  • To override fully, set FILE_SERVER_BASE_URL (e.g., http://192.168.1.100:8000); this takes precedence
  • After changing .env, restart the simclient process
• Expected healthy log sequence:
  • Lade Datei herunter von: http://<broker-ip>:8000/...
  • Followed by "GET /... HTTP/1.1" 200 and Datei erfolgreich heruntergeladen:

Important Notes for AI Assistants

Virtual Environment Requirements (Critical)

• pygame and pillow MUST be installed in venv - Required for Impressive to work
• Display manager uses venv context - Ensures Impressive has access to dependencies
• Installation command: pip install pygame pillow (already in requirements.txt)
• If pygame missing: Impressive will fail with "No module named 'pygame'" error

Presentation System

• ALWAYS use Impressive for PDF presentations (primary solution)
• DO NOT suggest xdotool approaches - they failed on Raspberry Pi due to window focus issues
• DO NOT suggest video conversion - adds complexity, had black screen issues
• PDF files are supported natively - no conversion needed
• PPTX conversion is automatic - LibreOffice headless handles PPTX→PDF
• Virtual environment is required - pygame + pillow must be available for Impressive
• Loop mode uses --wrap - not custom scripts or workarounds
• Auto-quit uses --autoquit - native Impressive parameter

Testing Approach

• Use ./scripts/test-display-manager.sh for interactive testing
• Use ./scripts/test-impressive-loop.sh to verify loop functionality
• Test individual components with specific test scripts
• Always check logs in logs/ directory for debugging

CEC testing notes:

• In development mode, the CEC integration path is skipped on purpose. To test end-to-end, either set ENV=production temporarily or use the manual options (1–4) in scripts/test-hdmi-cec.sh.

Code Changes

• Display logic is in src/display_manager.py, not simclient.py
• MQTT client (simclient.py) writes events to current_event.json
• Display Manager reads current_event.json and launches appropriate applications
• Two separate processes: simclient.py (MQTT) + display_manager.py (display control)

Documentation

• README.md - Start here for comprehensive overview
• IMPRESSIVE_INTEGRATION.md - Deep dive into presentation system
• QUICK_REFERENCE.md - Quick commands and examples
• Source code has extensive comments and logging

This system is designed for reliability and ease of maintenance in educational environments with multiple deployed clients. The Impressive-based presentation solution provides native auto-advance and loop support without complex window management hacks.

Recent changes (Oct 2025)

Summary of additions since the last Copilot instructions update:

• Added event_type handling for scheduler events: presentation, webuntis, webpage, website. The display manager now uses event_type when available to pick the correct display path.
• Implemented auto-scroll for long websites. Two strategies are available:
  • CDP injection via DevTools websocket using websocket-client and requests. The injector attempts to evaluate an auto-scroll script inside the page context. Default duration: 60s.
  • Local Chrome extension fallback (src/chrome_autoscroll) loaded with --load-extension when CDP handshake is blocked (403), ensuring reliable autoscroll operation.
• New helper scripts for debugging DevTools handshake: scripts/test_cdp.py and scripts/test_cdp_origins.py.
• Updated requirements.txt to include websocket-client.

Runtime behavior adjustments:

• HDMI-CEC is automatically disabled in development mode; both the Display Manager and the CEC integration test (option 5) honor this to protect developer setups from constant TV power cycling.

Notes for Copilot:

• When adding or modifying display logic, prefer Impressive for PDF presentations and avoid xdotool approaches for slideshow control.
• For web events, ensure autoscroll is only activated for event_type: "website" and keep the CDP injection optional/fallback-only when feasible.