olaf
fcc0dfbb0f
DB/model Add Conversion model + ConversionStatus enum (pending, processing, ready, failed) Alembic migrations: create conversions table, indexes, unique (source_event_media_id, target_format, file_hash), and NOT NULL on file_hash API Enqueue on upload (ppt|pptx|odp) in routes/eventmedia.py: compute sha256, upsert Conversion, enqueue job New routes: POST /api/conversions/<media_id>/pdf — ensure/enqueue conversion GET /api/conversions/<media_id>/status — latest status/details GET /api/files/converted/<path> — serve converted PDFs Register conversions blueprint in wsgi Worker server/worker.py: convert_event_media_to_pdf Calls Gotenberg /forms/libreoffice/convert, writes to server/media/converted/ Updates Conversion status, timestamps, error messages Fix media root resolution to /server/media Prefer function enqueue over string path; expose server.worker in package init for RQ string compatibility Queue/infra server/task_queue.py: RQ queue helper (REDIS_URL, default redis://redis:6379/0) docker-compose: Add redis and gotenberg services Add worker service (rq worker conversions) Pass REDIS_URL and GOTENBERG_URL to server/worker Mount shared media volume in prod for API/worker parity docker-compose.override: Add dev redis/gotenberg/worker services Ensure PYTHONPATH + working_dir allow importing server.worker Use rq CLI instead of python -m rq for worker Dashboard dev: run as appropriate user/root and pre-create/chown caches to avoid EACCES Dashboard dev UX Vite: set cacheDir .vite to avoid EACCES in node_modules Disable Node inspector by default to avoid port conflicts Docs Update copilot-instructions.md with conversion system: flow, services, env vars, endpoints, storage paths, and data model
117 lines
9.5 KiB
Markdown
117 lines
9.5 KiB
Markdown
# Copilot instructions for infoscreen_2025
|
||
|
||
Use this as your shared context when proposing changes. Keep edits minimal and match existing patterns referenced below.
|
||
|
||
## Big picture
|
||
- Multi-service app orchestrated by Docker Compose.
|
||
- API: Flask + SQLAlchemy (MariaDB), in `server/` exposed on :8000 (health: `/health`).
|
||
- Dashboard: React + Vite in `dashboard/`, dev on :5173, served via Nginx in prod.
|
||
- MQTT broker: Eclipse Mosquitto, config in `mosquitto/config/mosquitto.conf`.
|
||
- Listener: MQTT consumer handling discovery + heartbeats in `listener/listener.py`.
|
||
- Scheduler: Publishes active events (per group) to MQTT retained topics in `scheduler/scheduler.py`.
|
||
- Nginx: Reverse proxy routes `/api/*` and `/screenshots/*` to API; everything else to dashboard (`nginx.conf`).
|
||
|
||
## Service boundaries & data flow
|
||
- Database connection string is passed as `DB_CONN` (mysql+pymysql) to Python services.
|
||
- API builds its engine in `server/database.py` (loads `.env` only in development).
|
||
- Scheduler loads `DB_CONN` in `scheduler/db_utils.py`.
|
||
- Listener also creates its own engine for writes to `clients`.
|
||
- MQTT topics (paho-mqtt v2, use Callback API v2):
|
||
- Discovery: `infoscreen/discovery` (JSON includes `uuid`, hw/ip data). ACK to `infoscreen/{uuid}/discovery_ack`. See `listener/listener.py`.
|
||
- Heartbeat: `infoscreen/{uuid}/heartbeat` updates `Client.last_alive` (UTC).
|
||
- Event lists (retained): `infoscreen/events/{group_id}` from `scheduler/scheduler.py`.
|
||
- Per-client group assignment (retained): `infoscreen/{uuid}/group_id` via `server/mqtt_helper.py`.
|
||
- Screenshots: server-side folders `server/received_screenshots/` and `server/screenshots/`; Nginx exposes `/screenshots/{uuid}.jpg` via `server/wsgi.py` route.
|
||
|
||
- Presentation conversion (PPT/PPTX/ODP → PDF):
|
||
- Trigger: on upload in `server/routes/eventmedia.py` for media types `ppt|pptx|odp` (compute sha256, upsert `Conversion`, enqueue job).
|
||
- Worker: RQ worker runs `server.worker.convert_event_media_to_pdf`, calls Gotenberg LibreOffice endpoint, writes to `server/media/converted/`.
|
||
- Services: Redis (queue) and Gotenberg added in compose; worker service consumes the `conversions` queue.
|
||
- Env: `REDIS_URL` (default `redis://redis:6379/0`), `GOTENBERG_URL` (default `http://gotenberg:3000`).
|
||
- Endpoints: `POST /api/conversions/<media_id>/pdf` (ensure/enqueue), `GET /api/conversions/<media_id>/status`, `GET /api/files/converted/<path>` (serve PDFs).
|
||
- Storage: originals under `server/media/…`, outputs under `server/media/converted/` (prod compose mounts a shared volume for this path).
|
||
|
||
## Data model highlights (see `models/models.py`)
|
||
- Enums: `EventType` (presentation, website, video, message, webuntis), `MediaType` (file/website types), and `AcademicPeriodType` (schuljahr, semester, trimester).
|
||
- Tables: `clients`, `client_groups`, `events`, `event_media`, `users`, `academic_periods`, `school_holidays`.
|
||
- Academic periods: `academic_periods` table supports educational institution cycles (school years, semesters). Events and media can be optionally linked via `academic_period_id` (nullable for backward compatibility).
|
||
- Times are stored as timezone-aware; treat comparisons in UTC (see scheduler and routes/events).
|
||
|
||
- Conversions:
|
||
- Enum `ConversionStatus`: `pending`, `processing`, `ready`, `failed`.
|
||
- Table `conversions`: `id`, `source_event_media_id` (FK→`event_media.id` ondelete CASCADE), `target_format`, `target_path`, `status`, `file_hash` (sha256), `started_at`, `completed_at`, `error_message`.
|
||
- Indexes: `(source_event_media_id, target_format)`, `(status, target_format)`; Unique: `(source_event_media_id, target_format, file_hash)`.
|
||
|
||
## API patterns
|
||
- Blueprints live in `server/routes/*` and are registered in `server/wsgi.py` with `/api/...` prefixes.
|
||
- Session usage: instantiate `Session()` per request, commit when mutating, and always `session.close()` before returning.
|
||
- Examples:
|
||
- Clients: `server/routes/clients.py` includes bulk group updates and MQTT sync (`publish_multiple_client_groups`).
|
||
- Groups: `server/routes/groups.py` computes “alive” using a grace period that varies by `ENV`.
|
||
- Events: `server/routes/events.py` serializes enum values to strings and normalizes times to UTC.
|
||
- Media: `server/routes/eventmedia.py` implements a simple file manager API rooted at `server/media/`.
|
||
- Academic periods: `server/routes/academic_periods.py` exposes:
|
||
- `GET /api/academic_periods` — list all periods
|
||
- `GET /api/academic_periods/active` — currently active period
|
||
- `POST /api/academic_periods/active` — set active period (deactivates others)
|
||
- `GET /api/academic_periods/for_date?date=YYYY-MM-DD` — period covering given date
|
||
|
||
## Frontend patterns (dashboard)
|
||
- Vite React app; proxies `/api` and `/screenshots` to API in dev (`vite.config.ts`).
|
||
- Uses Syncfusion components; Vite config pre-bundles specific packages to avoid alias issues.
|
||
- Environment: `VITE_API_URL` provided at build/run; in dev compose, proxy handles `/api` so local fetches can use relative `/api/...` paths.
|
||
- Scheduler (appointments page): top bar includes Group and Academic Period selectors (Syncfusion DropDownList). Selecting a period calls `POST /api/academic_periods/active`, moves the calendar to today’s month/day within the period year, and refreshes a right-aligned indicator row showing:
|
||
- Holidays present in the current view (count)
|
||
- Period label (display_name or name) with a badge indicating whether any holidays exist in that period (overlap check)
|
||
|
||
Note: Syncfusion usage in the dashboard is already documented above; if a UI for conversion status/downloads is added later, link its routes and components here.
|
||
|
||
## Local development
|
||
- Compose: development is `docker-compose.yml` + `docker-compose.override.yml`.
|
||
- API (dev): `server/Dockerfile.dev` with debugpy on 5678, Flask app `wsgi:app` on :8000.
|
||
- Dashboard (dev): `dashboard/Dockerfile.dev` exposes :5173 and waits for API via `dashboard/wait-for-backend.sh`.
|
||
- Mosquitto: allows anonymous in dev; WebSocket on :9001.
|
||
- Common env vars: `DB_CONN`, `DB_USER`, `DB_PASSWORD`, `DB_HOST=db`, `DB_NAME`, `ENV`, `MQTT_USER`, `MQTT_PASSWORD`.
|
||
- Alembic: prod compose runs `alembic ... upgrade head` and `server/init_defaults.py` before gunicorn.
|
||
- Use `server/init_academic_periods.py` to populate default Austrian school years after migration.
|
||
|
||
## Production
|
||
- `docker-compose.prod.yml` uses prebuilt images (`ghcr.io/robbstarkaustria/*`).
|
||
- Nginx serves dashboard and proxies API; TLS certs expected in `certs/` and mounted to `/etc/nginx/certs`.
|
||
|
||
## Environment variables (reference)
|
||
- DB_CONN — Preferred DB URL for services. Example: `mysql+pymysql://${DB_USER}:${DB_PASSWORD}@db/${DB_NAME}`
|
||
- DB_USER, DB_PASSWORD, DB_NAME, DB_HOST — Used to assemble DB_CONN in dev if missing; inside containers `DB_HOST=db`.
|
||
- ENV — `development` or `production`; in development, `server/database.py` loads `.env`.
|
||
- MQTT_BROKER_HOST, MQTT_BROKER_PORT — Defaults `mqtt` and `1883`; MQTT_USER/MQTT_PASSWORD optional (dev often anonymous per Mosquitto config).
|
||
- VITE_API_URL — Dashboard build-time base URL (prod); in dev the Vite proxy serves `/api` to `server:8000`.
|
||
- HEARTBEAT_GRACE_PERIOD_DEV / HEARTBEAT_GRACE_PERIOD_PROD — Groups “alive” window (defaults ~15s dev / 180s prod).
|
||
- REFRESH_SECONDS — Optional scheduler republish interval; `0` disables periodic refresh.
|
||
|
||
## Conventions & gotchas
|
||
- Always compare datetimes in UTC; some DB values may be naive—normalize before comparing (see `routes/events.py`).
|
||
- Use retained MQTT messages for state that clients must recover after reconnect (events per group, client group_id).
|
||
- In-container DB host is `db`; do not use `localhost` inside services.
|
||
- No separate dev vs prod secret conventions: use the same env var keys across environments (e.g., `DB_CONN`, `MQTT_USER`, `MQTT_PASSWORD`).
|
||
- When adding a new route:
|
||
1) Create a Blueprint in `server/routes/...`,
|
||
2) Register it in `server/wsgi.py`,
|
||
3) Manage `Session()` lifecycle, and
|
||
4) Return JSON-safe values (serialize enums and datetimes).
|
||
- When extending media types, update `MediaType` and any logic in `eventmedia` and dashboard that depends on it.
|
||
- Academic periods: Events/media can be optionally associated with periods for educational organization. Only one period should be active at a time (`is_active=True`).
|
||
|
||
## Quick examples
|
||
- Add client description persists to DB and publishes group via MQTT: see `PUT /api/clients/<uuid>/description` in `routes/clients.py`.
|
||
- Bulk group assignment emits retained messages for each client: `PUT /api/clients/group`.
|
||
- Listener heartbeat path: `infoscreen/<uuid>/heartbeat` → sets `clients.last_alive`.
|
||
|
||
Questions or unclear areas? Tell us if you need: exact devcontainer debugging steps, stricter Alembic workflow, or a seed dataset beyond `init_defaults.py`.
|
||
|
||
## Academic Periods System
|
||
- **Purpose**: Organize events and media by educational cycles (school years, semesters, trimesters).
|
||
- **Design**: Fully backward compatible - existing events/media continue to work without period assignment.
|
||
- **Usage**: New events/media can optionally reference `academic_period_id` for better organization and filtering.
|
||
- **Constraints**: Only one period can be active at a time; use `init_academic_periods.py` for Austrian school year setup.
|
||
- **UI Integration**: The dashboard highlights the currently selected period and whether a holiday plan exists within that date range. Holiday linkage currently uses date overlap with `school_holidays`; an explicit `academic_period_id` on `school_holidays` can be added later if tighter association is required.
|