infoscreen-dev/.github/copilot-instructions.md

# Copilot Instructions - Infoscreen Client

## Project Overview
This is an **Infoscreen Client** system for Raspberry Pi that creates digital signage displays. The client communicates with a server via MQTT to display presentations, videos, and web content in kiosk mode. It's designed for educational/research environments where multiple displays need to be centrally managed.

## Architecture & Technology Stack

### Core Technologies
- **Python 3.x** - Main application language
- **MQTT (paho-mqtt)** - Real-time messaging with server
- **Impressive** - PDF presenter with native auto-advance and loop support
- **LibreOffice** - PPTX to PDF conversion (headless mode)
- **Environment Variables** - Configuration management via `.env` files
- **JSON** - Data exchange format for events and configuration
- **Base64** - Screenshot transmission encoding
- **Threading** - Background services (screenshot monitoring)

### System Components
- **Main Client** (`simclient.py`) - Core MQTT client and event processor
- **Display Manager** (`display_manager.py`) - Controls display applications (presentations, videos, web)
- **Discovery System** - Automatic client registration with server
- **Heartbeat Monitoring** - Regular status updates and keepalive
- **Event Processing** - Handles presentation/content switching commands
- **Screenshot Service** - Dashboard monitoring via image capture
- **File Management** - Downloads and manages presentation files
- **Group Management** - Supports organizing clients into groups

## Key Features & Functionality

### MQTT Communication Patterns
- **Discovery**: `infoscreen/discovery` → `infoscreen/{client_id}/discovery_ack`
- **Heartbeat**: Regular `infoscreen/{client_id}/heartbeat` messages
- **Dashboard**: Screenshot transmission via `infoscreen/{client_id}/dashboard`
- **Group Assignment**: Server sends group via `infoscreen/{client_id}/group_id`
- **Events**: Content commands via `infoscreen/events/{group_id}`

### Event Types Supported
```json
{
  "presentation": {
    "files": [{"url": "https://server/file.pptx", "filename": "file.pptx"}],
    "auto_advance": true,
    "slide_interval": 10,
    "loop": true
  },
  "web": {
    "url": "https://example.com"
  },
  "video": {
    "url": "https://server/video.mp4",
    "loop": false,
    "autoplay": true,
    "volume": 0.8
  }
}
```

### Presentation System (Impressive-Based)
- **PDF files** are displayed natively with Impressive PDF presenter (no conversion needed)
- **PPTX files** are automatically converted to PDF using LibreOffice headless
- **Auto-advance**: Native Impressive `--auto` parameter (no xdotool needed)
- **Loop mode**: Impressive `--wrap` parameter for infinite looping
- **Auto-quit**: Impressive `--autoquit` parameter to exit after last slide
- **Virtual Environment**: Uses venv with pygame + pillow for reliable operation
- **Reliable**: Works consistently on Raspberry Pi without window focus issues

### Client Identification
- **Hardware Token**: SHA256 hash of serial number + MAC addresses
- **Persistent UUID**: Stored in `config/client_uuid.txt`
- **Group Membership**: Persistent group assignment in `config/last_group_id.txt`

## Directory Structure
```
~/infoscreen-dev/
├── .env                    # Environment configuration
├── README.md               # Complete project documentation
├── IMPRESSIVE_INTEGRATION.md  # Presentation system details
├── QUICK_REFERENCE.md      # Quick command reference
├── .github/                # GitHub configuration
│   └── copilot-instructions.md
├── src/                    # Source code
│   ├── simclient.py       # MQTT client (event management)
│   ├── display_manager.py # Display controller (Impressive integration)
│   ├── current_event.json # Current active event (runtime)
│   ├── config/            # Persistent client data
│   │   ├── client_uuid.txt
│   │   └── last_group_id.txt
│   ├── presentation/      # Downloaded presentation files & PDFs
│   └── screenshots/       # Screenshot captures for monitoring
├── scripts/               # Production & testing utilities
│   ├── start-dev.sh       # Start development client
│   ├── start-display-manager.sh  # Start Display Manager
│   ├── test-display-manager.sh   # Interactive testing menu
│   ├── test-impressive.sh        # Test Impressive (auto-quit)
│   ├── test-impressive-loop.sh   # Test Impressive (loop mode)
│   ├── test-mqtt.sh       # MQTT connectivity test
│   ├── test-screenshot.sh # Screenshot capture test
│   └── present-pdf-auto-advance.sh  # PDF presentation wrapper
├── logs/                  # Application logs
│   ├── simclient.log
│   └── display_manager.log
└── venv/                  # Python virtual environment
```

## Configuration & Environment Variables

### Development vs Production
- **Development**: `ENV=development`, verbose logging, frequent heartbeats
- **Production**: `ENV=production`, minimal logging, longer intervals

### Key Environment Variables
```bash
# Environment
ENV=development|production
DEBUG_MODE=1|0
LOG_LEVEL=DEBUG|INFO|WARNING|ERROR

# MQTT Configuration
MQTT_BROKER=192.168.1.100        # Primary MQTT broker
MQTT_PORT=1883                   # MQTT port
MQTT_BROKER_FALLBACKS=host1,host2  # Fallback brokers

# Timing (seconds)
HEARTBEAT_INTERVAL=10            # Status update frequency
SCREENSHOT_INTERVAL=30           # Dashboard screenshot frequency
 
# File/API Server (used to download presentation files)
# Defaults to the same host as MQTT_BROKER, port 8000, scheme http.
# If incoming event URLs use host 'server' (or are host-less), simclient rewrites them to this server.
FILE_SERVER_HOST=                # optional; if empty, defaults to MQTT_BROKER
FILE_SERVER_PORT=8000            # default API port
FILE_SERVER_SCHEME=http          # http or https
# FILE_SERVER_BASE_URL=           # optional full override, e.g., http://192.168.1.100:8000
```

### File Server URL Resolution
- The MQTT client (`simclient.py`) downloads presentation files listed in events.
- To avoid DNS issues when event URLs use `http://server:8000/...`, the client normalizes such URLs to the configured file server.
- By default, the file server host is the same as `MQTT_BROKER`, with port `8000` and scheme `http`.
- You can override behavior using `.env` variables above; `FILE_SERVER_BASE_URL` takes precedence over individual host/port/scheme.
- Inline comments in `.env` are supported; keep comments after a space and `#` so values stay clean.

## Development Patterns & Best Practices

### Error Handling
- Robust MQTT connection with fallbacks and retries
- Graceful degradation when services unavailable
- Comprehensive logging with rotating file handlers
- Exception handling for all external operations

### State Management
- Event state persisted in `current_event.json`
- Client configuration persisted across restarts
- Group membership maintained with server synchronization
- Clean state transitions (delete old events on group changes)

### Threading Architecture
- Main thread: MQTT communication and heartbeat
- Background thread: Screenshot monitoring service
- Thread-safe operations for shared resources

### File Operations
- Automatic directory creation for all output paths
- Safe file operations with proper exception handling
- Atomic writes for configuration files
- Automatic cleanup of temporary/outdated files

## Development Workflow

### Local Development Setup
1. Clone repository to `~/infoscreen-dev`
2. Create virtual environment: `python3 -m venv venv`
3. Install dependencies: `pip install -r src/requirements.txt` (includes pygame + pillow for PDF slideshows)
4. Configure `.env` file with MQTT broker settings
5. Use `./scripts/start-dev.sh` for MQTT client or `./scripts/start-display-manager.sh` for display manager
6. **Important**: Virtual environment must include pygame and pillow for PDF auto-advance to work

### Testing Components
- `./scripts/test-mqtt.sh` - MQTT connectivity
- `./scripts/test-screenshot.sh` - Screenshot capture
- `./scripts/test-display-manager.sh` - Interactive testing menu
- `./scripts/test-impressive.sh` - Test auto-quit presentation mode
- `./scripts/test-impressive-loop.sh` - Test loop presentation mode
- `./scripts/test-utc-timestamps.sh` - Event timing validation
- Manual event testing via mosquitto_pub or test-display-manager.sh

### Production Deployment
- Docker containerization available (`docker-compose.production.yml`)
- Systemd service integration for auto-start
- Resource limits and health checks configured
- Persistent volume mounts for data

## Code Style & Conventions

### Python Code Standards
- Environment variable parsing with fallback defaults
- Comprehensive docstrings for complex functions
- Logging at appropriate levels (DEBUG/INFO/WARNING/ERROR)
- Type hints where beneficial for clarity
- Exception handling with specific error types

### Configuration Management
- Environment-first configuration (12-factor app principles)
- Support for inline comments in environment values
- Graceful handling of missing/invalid configuration
- Multiple environment file locations for flexibility

### MQTT Message Handling
- JSON message format validation
- Robust payload parsing with fallbacks
- Topic-specific message handlers
- Retained message support where appropriate

## Hardware Considerations

### Target Platform
- **Primary**: Raspberry Pi 4/5 with desktop environment
- **Storage**: SSD recommended for performance
- **Display**: HDMI output for presentation display
- **Network**: WiFi or Ethernet connectivity required

### System Dependencies
- Python 3.x runtime
- Network connectivity for MQTT
- Display server (X11) for screenshot capture
- **Impressive** - PDF presenter with auto-advance (primary presentation tool)
- **pygame** - Required for Impressive (installed in venv)
- **Pillow/PIL** - Required for Impressive PDF rendering (installed in venv)
- **LibreOffice** - PPTX to PDF conversion (headless mode)
- Chromium/Chrome - Web content display (kiosk mode)
- VLC or MPV - Video playbook

### Video playback details (python-vlc)

- The Display Manager now prefers using python-vlc (libvlc) when available for video playback. This enables programmatic control (autoplay, loop, volume) and cleaner termination/cleanup. If python-vlc is not available, the external `vlc` binary is used as a fallback.
- Supported video event fields: `url`, `autoplay` (boolean), `loop` (boolean), `volume` (float 0.0-1.0). The manager converts `volume` to VLC's 0-100 scale.
- URLs using the placeholder host `server` (for example `http://server:8000/...`) are rewritten to the configured file server before playback. The resolution priority is: `FILE_SERVER_BASE_URL` > `FILE_SERVER_HOST` (or `MQTT_BROKER`) + `FILE_SERVER_PORT` + `FILE_SERVER_SCHEME`.
- Hardware-accelerated decoding errors (e.g., `h264_v4l2m2m`) may appear when the platform does not expose a V4L2 M2M device. To avoid these errors the Display Manager can be configured to disable hw-decoding (see README env var `VLC_HW_ACCEL`). By default the manager will attempt hw-acceleration when libvlc supports it.
- Fullscreen / kiosk: the manager will attempt to make libVLC windows fullscreen (remove decorations) when using python-vlc, and the README contains recommended system-level kiosk/session setup for a truly panel-free fullscreen experience.


## Security & Privacy

### Data Protection
- Hardware identification via cryptographic hash
- No sensitive data in plain text logs
- Local storage of minimal required data only
- Secure MQTT communication (configurable)

### Network Security
- Configurable MQTT authentication (if broker requires)
- Firewall-friendly design (outbound connections only)
- Multiple broker fallback for reliability

## Presentation System Architecture

### How It Works
1. **PDF File Received** → Event contains PDF file reference → Use directly (no conversion)
2. **PPTX File Received** → Event contains PPTX file reference
3. **Convert to PDF** → LibreOffice headless: `libreoffice --headless --convert-to pdf`
4. **Cache PDF** → Converted PDF stored in `presentation/` directory
5. **Display with Impressive** → Launch with venv environment and parameters:
   - `--fullscreen` - Full screen mode
   - `--nooverview` - No slide overview
   - `--auto N` - Auto-advance every N seconds
   - `--wrap` - Loop infinitely (if `loop: true`)
   - `--autoquit` - Exit after last slide (if `loop: false`)

### Key Parameters
| Parameter | Type | Default | Description |
|-----------|------|---------|-------------|
| `auto_advance` | boolean | `false` | Enable automatic slide advancement |
| `slide_interval` | integer | `10` | Seconds between slides |
| `loop` | boolean | `false` | Loop presentation vs. quit after last slide |

### Why Impressive?
- ✅ **Native auto-advance** - No xdotool or window management hacks
- ✅ **Built-in loop support** - Reliable `--wrap` parameter
- ✅ **Works on Raspberry Pi** - No focus/window issues
- ✅ **Simple integration** - Clean command-line interface
- ✅ **Maintainable** - ~50 lines of code vs. 200+ with xdotool approaches

### Implementation Location
- **File**: `src/display_manager.py`
- **Method**: `start_presentation()`
- **Key Logic**: 
  1. Check if PDF file (use directly) or PPTX (needs conversion)
  2. Convert PPTX to PDF if needed (cached for reuse)
  3. Set up virtual environment for Impressive (pygame + pillow)
  4. Build Impressive command with appropriate parameters
  5. Launch process and monitor

## Common Development Tasks

When working on this codebase:

1. **Adding new event types**: Extend the event processing logic in `display_manager.py` → `start_display_for_event()`
2. **Modifying presentation behavior**: Update `display_manager.py` → `start_presentation()`
3. **Configuration changes**: Update environment variable parsing and validation
4. **MQTT topics**: Follow the established `infoscreen/` namespace pattern
5. **Error handling**: Always include comprehensive logging and graceful fallbacks
6. **State persistence**: Use the established `config/` directory pattern
7. **Testing**: Use `./scripts/test-display-manager.sh` for interactive testing
8. **Presentation testing**: Use `./scripts/test-impressive*.sh` scripts
9. **File download host resolution**: If the API server differs from the MQTT broker or uses HTTPS, set `FILE_SERVER_*` in `.env` or adjust `resolve_file_url()` in `src/simclient.py`.

## Troubleshooting Guidelines

### Common Issues
- **MQTT Connection**: Check broker reachability, try fallback brokers
- **Screenshots**: Verify display environment and permissions
- **File Downloads**: Check network connectivity and disk space
  - If event URLs use host `server` and DNS fails, the client rewrites to `MQTT_BROKER` by default.
  - Ensure `MQTT_BROKER` points to the correct server IP; if the API differs, set `FILE_SERVER_HOST` or `FILE_SERVER_BASE_URL`.
  - Match scheme/port via `FILE_SERVER_SCHEME`/`FILE_SERVER_PORT` for HTTPS or non-default ports.
- **Group Changes**: Monitor log for group assignment messages
- **Service Startup**: Check systemd logs and environment configuration

### Debugging Tools
- Log files in `logs/simclient.log` and `logs/display_manager.log` with rotation
- MQTT message monitoring with mosquitto_sub
- Interactive testing menu: `./scripts/test-display-manager.sh`
- Component test scripts: `test-impressive*.sh`, `test-mqtt.sh`, etc.
- Process monitoring: Check for `impressive`, `libreoffice`, `chromium`, `vlc` processes

### File download URL troubleshooting
- Symptoms:
  - `Failed to resolve 'server'` or `NameResolutionError` when downloading files
  - `Invalid URL 'http      # http or https://...'` in `simclient.log`
- What to check:
  - Look for lines like `Lade Datei herunter von:` in `logs/simclient.log` to see the effective URL used
  - Ensure the URL host is the MQTT broker IP (or your configured file server), not `server`
  - Verify `.env` values don’t include inline comments as part of the value (e.g., keep `FILE_SERVER_SCHEME=http` on its own line)
- Fixes:
  - If your API is on the same host as the broker: leave `FILE_SERVER_HOST` empty (defaults to `MQTT_BROKER`), keep `FILE_SERVER_PORT=8000`, and set `FILE_SERVER_SCHEME=http` or `https`
  - To override fully, set `FILE_SERVER_BASE_URL` (e.g., `http://192.168.1.100:8000`); this takes precedence
  - After changing `.env`, restart the simclient process
- Expected healthy log sequence:
  - `Lade Datei herunter von: http://<broker-ip>:8000/...`
  - Followed by `"GET /... HTTP/1.1" 200` and `Datei erfolgreich heruntergeladen:`

## Important Notes for AI Assistants

### Virtual Environment Requirements (Critical)
- **pygame and pillow MUST be installed in venv** - Required for Impressive to work
- **Display manager uses venv context** - Ensures Impressive has access to dependencies
- **Installation command**: `pip install pygame pillow` (already in requirements.txt)
- **If pygame missing**: Impressive will fail with "No module named 'pygame'" error

### Presentation System
- **ALWAYS use Impressive** for PDF presentations (primary solution)
- **DO NOT suggest xdotool approaches** - they failed on Raspberry Pi due to window focus issues
- **DO NOT suggest video conversion** - adds complexity, had black screen issues
- **PDF files are supported natively** - no conversion needed
- **PPTX conversion is automatic** - LibreOffice headless handles PPTX→PDF
- **Virtual environment is required** - pygame + pillow must be available for Impressive
- **Loop mode uses `--wrap`** - not custom scripts or workarounds
- **Auto-quit uses `--autoquit`** - native Impressive parameter

### Testing Approach
- Use `./scripts/test-display-manager.sh` for interactive testing
- Use `./scripts/test-impressive-loop.sh` to verify loop functionality
- Test individual components with specific test scripts
- Always check logs in `logs/` directory for debugging

### Code Changes
- Display logic is in `src/display_manager.py`, not `simclient.py`
- MQTT client (`simclient.py`) writes events to `current_event.json`
- Display Manager reads `current_event.json` and launches appropriate applications
- Two separate processes: simclient.py (MQTT) + display_manager.py (display control)

### Documentation
- **README.md** - Start here for comprehensive overview
- **IMPRESSIVE_INTEGRATION.md** - Deep dive into presentation system
- **QUICK_REFERENCE.md** - Quick commands and examples
- Source code has extensive comments and logging

This system is designed for reliability and ease of maintenance in educational environments with multiple deployed clients. The Impressive-based presentation solution provides native auto-advance and loop support without complex window management hacks.

## Recent changes (Oct 2025)

Summary of additions since the last Copilot instructions update:

- Added `event_type` handling for scheduler events: `presentation`, `webuntis`, `webpage`, `website`. The display manager now uses `event_type` when available to pick the correct display path.
- Implemented auto-scroll for long websites. Two strategies are available:
  - CDP injection via DevTools websocket using `websocket-client` and `requests`. The injector attempts to evaluate an auto-scroll script inside the page context. Default duration: 60s.
  - Local Chrome extension fallback (`src/chrome_autoscroll`) loaded with `--load-extension` when CDP handshake is blocked (403), ensuring reliable autoscroll operation.
- New helper scripts for debugging DevTools handshake: `scripts/test_cdp.py` and `scripts/test_cdp_origins.py`.
- Updated `requirements.txt` to include `websocket-client`.

Notes for Copilot:
- When adding or modifying display logic, prefer Impressive for PDF presentations and avoid xdotool approaches for slideshow control.
- For web events, ensure autoscroll is only activated for `event_type: "website"` and keep the CDP injection optional/fallback-only when feasible.