home-automation/README.md

# Home Automation Monorepo

A Python-based home automation system built with Poetry in a monorepo structure. Features a microservices architecture with MQTT/Redis messaging, dynamic UI with realtime updates, and flexible device configuration.

## Features

- **Dynamic Dashboard**: Responsive web UI with realtime device status via Server-Sent Events
- **MQTT Integration**: Device communication through MQTT broker with vendor abstraction
- **Realtime Updates**: Live device state updates via Redis Pub/Sub and SSE
- **Flexible Layout**: Configure rooms and device tiles via YAML
- **Multiple Device Support**: Lights with power and brightness control
- **Clean Architecture**: Separation of concerns with API-first design

## Project Structure

```
home-automation/
├── apps/                    # Applications
│   ├── api/                # API Gateway (FastAPI, port 8001)
│   │   └── main.py        # REST API, SSE endpoint, device management
│   ├── abstraction/        # MQTT ↔ Redis Bridge
│   │   └── main.py        # Protocol translation layer
│   ├── rules/              # Rules Engine (planned)
│   └── ui/                 # Web Interface (FastAPI, port 8002)
│       ├── main.py        # Jinja2 templates, API client
│       ├── api_client.py  # HTTP client for API Gateway
│       ├── templates/     # HTML templates
│       │   ├── dashboard.html  # Dynamic dashboard
│       │   └── index.html      # Legacy static UI
│       └── static/        # CSS and assets
├── packages/               # Shared packages
│   └── home_capabilities/  # Core libraries
│       ├── light.py       # Light device models
│       └── layout.py      # UI layout models
├── config/                 # Configuration files
│   ├── devices.yaml       # Device definitions with features
│   └── layout.yaml        # UI room/device layout
├── tools/                  # Development tools
│   └── sim_test_lampe.py  # Multi-device MQTT simulator
├── infra/                  # Infrastructure
│   ├── docker-compose.yml
│   └── README.md
├── pyproject.toml          # Poetry configuration
├── PORTS.md               # Port allocation
└── README.md
```

## Requirements

- Python 3.11+ (tested with 3.14.0)
- Poetry 2.2.1+
- MQTT Broker (e.g., Mosquitto)
- Redis Server

### External Services

This system requires the following external services:

- **MQTT Broker**: `172.16.2.16:1883` (configured in `config/devices.yaml`)
- **Redis Server**: `172.23.1.116:6379/8` (configured in `config/devices.yaml`)

## Setup

1. Install Poetry if you haven't already:
   ```bash
   curl -sSL https://install.python-poetry.org | python3 -
   ```

2. Install dependencies:
   ```bash
   poetry install
   ```

3. Activate the virtual environment:
   ```bash
   poetry shell
   ```

4. Configure devices and layout:
   - Edit `config/devices.yaml` for device definitions and MQTT/Redis settings
   - Edit `config/layout.yaml` for UI room organization

## Configuration

### devices.yaml

Defines available devices with their features and MQTT topics:

```yaml
devices:
  - device_id: test_lampe_1
    type: light
    features:
      power: true
      brightness: true
    topics:
      set: "vendor/test_lampe_1/set"
      state: "vendor/test_lampe_1/state"
```

### layout.yaml

Organizes devices into rooms for the UI:

```yaml
rooms:
  - name: Wohnzimmer
    devices:
      - device_id: test_lampe_1
        title: Stehlampe
        icon: "LIGHT"
        rank: 10  # Lower rank = higher priority
```

## Development

### Dependencies

Key packages installed:

- **Web Framework**: FastAPI 0.120.3, Uvicorn 0.38.0
- **Data Validation**: Pydantic 2.12.3
- **MQTT**: aiomqtt 2.4.0 (async), paho-mqtt 2.1.0 (sync)
- **Redis**: redis 7.0.1
- **HTTP Client**: httpx 0.28.1
- **Templates**: Jinja2 3.1.6
- **Config**: PyYAML 6.0.3
- **Testing**: beautifulsoup4 4.14.2

### Code Quality Tools

This project uses the following tools configured in `pyproject.toml`:

- **Ruff**: Fast Python linter
- **Black**: Code formatter
- **Mypy**: Static type checker

Run code quality checks:

```bash
# Format code
poetry run black .

# Lint code
poetry run ruff check .

# Type check
poetry run mypy .
```

### Adding New Devices

1. Add device to `config/devices.yaml`:
   ```yaml
   - device_id: new_device
     type: light
     features:
       power: true
       brightness: false
     topics:
       set: "vendor/new_device/set"
       state: "vendor/new_device/state"
   ```

2. Add device to rooms in `config/layout.yaml`:
   ```yaml
   rooms:
     - name: Kitchen
       devices:
         - device_id: new_device
           title: Kitchen Light
           icon: "LIGHT"
           rank: 5
   ```

3. Restart API and UI services (they will auto-reload if using `--reload`)

4. Device will appear in dashboard automatically!

### Extending Features

To add new device capabilities:

1. Update Pydantic models in `packages/home_capabilities/`
2. Add feature to `devices.yaml`
3. Extend dashboard template for UI controls
4. Update simulator or create new simulator for testing

## Troubleshooting

### Connection Issues

- **SSE not connecting**: Check API server is running on port 8001
- **Device not responding**: Check MQTT broker connectivity
- **No updates in UI**: Check abstraction layer and Redis connection

### Check Logs

```bash
# API logs
tail -f /tmp/api.log

# Abstraction layer logs
tail -f /tmp/abstraction.log

# UI logs
tail -f /tmp/ui.log
```

### Common Commands

```bash
# Check if services are running
ps aux | grep -E "uvicorn|abstraction"

# Check port usage
lsof -i :8001
lsof -i :8002

# Test MQTT connection
mosquitto_pub -h 172.16.2.16 -t test -m "hello"
```

### Running Applications

#### Quick Start - All Services

Start all services in the background:

```bash
# 1. Start MQTT Abstraction Layer
poetry run python -m apps.abstraction.main > /tmp/abstraction.log 2>&1 &

# 2. Start API Gateway
poetry run uvicorn apps.api.main:app --host 0.0.0.0 --port 8001 > /tmp/api.log 2>&1 &

# 3. Start UI
poetry run uvicorn apps.ui.main:app --host 0.0.0.0 --port 8002 > /tmp/ui.log 2>&1 &

# 4. Start Device Simulator (optional)
poetry run python tools/sim_test_lampe.py &
```

Stop all services:

```bash
pkill -f "uvicorn apps" && pkill -f "apps.abstraction.main" && pkill -f "sim_test_lampe"
```

#### Port Configuration

See `PORTS.md` for detailed port allocation.

- **API Server**: http://localhost:8001
- **UI Server**: http://localhost:8002

#### API Server

Start the FastAPI server with auto-reload:

```bash
# Using uvicorn directly (port 8001)
poetry run uvicorn apps.api.main:app --reload --port 8001

# Or using the main function
poetry run python -m apps.api.main
```

The API will be available at:
- API Base: http://localhost:8001
- Interactive Docs: http://localhost:8001/docs
- OpenAPI Schema: http://localhost:8001/openapi.json

Available endpoints:
- `GET /health` - Health check endpoint
- `GET /devices` - List all devices with features
- `GET /layout` - Get UI layout configuration
- `POST /devices/{device_id}/set` - Control a device
- `GET /realtime` - Server-Sent Events stream for live updates

#### UI Server

Start the web interface:

```bash
# Using uvicorn directly (port 8002)
poetry run uvicorn apps.ui.main:app --reload --port 8002

# Or using the main function
poetry run python -m apps.ui.main
```

The UI will be available at:
- **Dynamic Dashboard**: http://localhost:8002/dashboard (or http://localhost:8002)
  - Realtime device status via SSE
  - Toggle buttons with state reflection
  - Brightness sliders for dimmable lights
  - Event log for all updates
  - Responsive layout from `config/layout.yaml`
- **Legacy Static UI**: http://localhost:8002/index.html
  - Fixed layout with test_lampe_1 and test_lampe_2

#### Abstraction Layer

The MQTT-Redis bridge translates between protocols:

```bash
poetry run python -m apps.abstraction.main
```

Functions:
- Subscribes to vendor-specific MQTT topics (`vendor/*/state`)
- Publishes state changes to Redis Pub/Sub (`ui:updates`)
- Enables decoupling of UI from MQTT

#### Device Simulator

Test your setup with the multi-device simulator:

```bash
poetry run python tools/sim_test_lampe.py
```

Simulates:
- `test_lampe_1`: Light with power and brightness control
- `test_lampe_2`: Simple light with power only
- `test_lampe_3`: Simple light with power only

The simulator:
- Subscribes to `vendor/test_lampe_*/set` topics
- Maintains device state (power, brightness)
- Publishes state to `vendor/test_lampe_*/state` (retained)
- Handles graceful shutdown (sets all lights to off)

#### Other Applications

```bash
# Rules Engine (planned)
poetry run python -m apps.rules.main
```

## Architecture

### Message Flow

```
User Action (UI) 
  → HTTP POST to API Gateway (/devices/{id}/set)
    → MQTT Publish (home/light/{id}/set)
      → Abstraction Layer receives
        → MQTT Publish (vendor/{id}/set)
          → Device/Simulator receives
            → Device State Update
              → MQTT Publish (vendor/{id}/state, retained)
                → Abstraction Layer receives
                  → Redis Pub/Sub (ui:updates)
                    → API Gateway /realtime SSE
                      → UI Updates (EventSource)
```

### Key Components

1. **API Gateway** (`apps/api/main.py`)
   - Single source of truth for configuration
   - REST endpoints for device control
   - SSE endpoint for realtime updates
   - Reads `devices.yaml` and `layout.yaml`

2. **UI** (`apps/ui/main.py`)
   - Pure API consumer (no direct file access)
   - Fetches devices and layout via HTTP
   - Renders dynamic dashboard with Jinja2
   - Connects to SSE for live updates

3. **Abstraction Layer** (`apps/abstraction/main.py`)
   - Protocol translation (MQTT ↔ Redis)
   - Subscribes to vendor topics
   - Publishes to Redis for UI updates

4. **Device Simulator** (`tools/sim_test_lampe.py`)
   - Emulates physical devices
   - Responds to SET commands
   - Publishes STATE updates

## Testing

### Manual Testing

1. Start all services (see Quick Start above)

2. Open the dashboard: http://localhost:8002

3. Toggle a light and watch:
   - Button changes state (Einschalten ↔ Ausschalten)
   - Status updates in realtime
   - Event log shows all messages

4. Check MQTT traffic:
   ```bash
   # Subscribe to all topics
   mosquitto_sub -h 172.16.2.16 -t '#' -v
   
   # Publish test command
   mosquitto_pub -h 172.16.2.16 -t 'vendor/test_lampe_1/set' \
     -m '{"power":"on","brightness":75}'
   ```

5. Check Redis Pub/Sub:
   ```bash
   redis-cli -h 172.23.1.116 -p 6379 -n 8
   > SUBSCRIBE ui:updates
   ```

### API Testing

```bash
# Get all devices
curl http://localhost:8001/devices | python3 -m json.tool

# Get layout
curl http://localhost:8001/layout | python3 -m json.tool

# Control a device
curl -X POST http://localhost:8001/devices/test_lampe_1/set \
  -H "Content-Type: application/json" \
  -d '{"type":"light","payload":{"power":"on"}}'

# Test SSE stream
curl -N http://localhost:8001/realtime
```

## License

TBD