LOGAR/README.md

# LOGAR: Edge-Thin Log Analysis & Temporal Verification System

**LOGAR** is an enterprise log aggregation, verification, and anomaly detection architecture designed for heterogeneous server fleets (Windows & Linux). It combines lightweight zero-state edge forwarders with a centralized cloud hub that applies mutual TLS 1.3 (**mTLS**) authentication, dynamic PKI licensing and quota management, temporal persistence tracking across 12-hour evaluation windows, an automated 4-run rule to filter transient warnings, and immediate pass-through for critical errors before reporting verified anomalies to **Hermes**.

---

## Table of Contents
1. [Core Philosophy](#core-philosophy)
2. [Architecture & Data Flow](#architecture--data-flow)
3. [mTLS Security, Dynamic PKI & Licensing](#mtls-security-dynamic-pki--licensing)
4. [Cloud-Side Temporal Persistence & 4-Run Rule](#cloud-side-temporal-persistence--4-run-rule)
5. [Agentic Hermes & Client Management API](#agentic-hermes--client-management-api)
6. [Dynamic Machine & Domain Identification](#dynamic-machine--domain-identification)
7. [Automated Service Installers (Linux & Windows)](#automated-service-installers-linux--windows)
8. [Repository & Shippables Structure](#repository--shippables-structure)
9. [Getting Started & Deployment](#getting-started--deployment)
10. [Running Tests](#running-tests)
11. [Automated Releases via Gitea Actions](#automated-releases-via-gitea-actions)

---

## Core Philosophy

### 1. Edge Thinness & Zero State
Site agents running on Windows and Linux act strictly as lightweight forwarders:
- **No Local Database**: Clients maintain zero local SQLite or heavy cache files.
- **Source-Level Filtering**: Agents stream candidate entries from informational events up to errors (`INFO`, `WARNING`, `ERROR`, `CRITICAL`), stripping verbose debugging noise (`DEBUG`, trace entries) and skipping events older than 24 hours.
- **Transport Security (mTLS 1.3)**: Logs are streamed directly over mutual TLS 1.3 sockets with hardware-bound / machine-unique client certificates.
- **Zero Configuration Overhead**: Clients auto-bootstrap certificate enrollment on first run if configured with an enrollment secret.

### 2. Cloud-Side Temporal Persistence & Severity Routing
The central Python hub handles state and verification:
- State tracking is managed centrally in SQLite (`logar_state.db`).
- Candidate issues are evaluated over a **12-hour temporal evaluation window**.
- **Warning Persistence (4-Run Rule)**: `WARNING` issues must persist across **at least 4 consecutive runs / cycles** within the 12-hour window to be confirmed as genuine anomalies, automatically filtering transient infrastructure blips.
- **Immediate Error Pass**: Critical errors (`ERROR`, `CRITICAL`, `FATAL`) bypass the 4-run threshold and are promoted immediately to `VERIFIED` on their first occurrence.

### 3. Agentic Integration with Hermes
Instead of engineers manually sifting through logs, **Hermes** ingests pre-filtered anomalies directly from the cloud hub (`GET /api/hermes/report`), treating verified errors and 4-run validated warnings as actionable system artifacts to trigger precise notifications and remediations.

---

## Architecture & Data Flow

```mermaid
graph TB
    subgraph Edge Nodes [Lightweight Edge Forwarders]
        W[Win_Client.exe / Win_Client.py<br/>Windows Event Log Ingestion]
        L[Linux_Client.bin / Linux_Client.py<br/>systemd journalctl -p warning]
    end

    subgraph Enrollment [Dynamic PKI & Licensing]
        ENR[POST /api/client/enroll<br/>License Quota & Secret Validation]
        CA[Internal Root CA<br/>Signs RSA-2048 Client Cert]
    end

    subgraph Transport [mTLS 1.3 Security Layer]
        MTLS[Mutual TLS 1.3 Handshake<br/>Port 9443 - Client Cert Required]
        AUTH[Extract Client CN & Fingerprint<br/>Validate Active License in SQLite]
    end

    subgraph Hub [LOGAR Server Hub]
        INGEST[Length-Prefixed Frame Ingestion]
        DB[(SQLite Persistence<br/>active_issues, clients, license_config)]
        RULE{12h Window &<br/>4-Run Rule}
    end

    subgraph Downstream [Hermes Agent & Monitoring]
        API[FastAPI Reporting & Management<br/>Port 8443]
        HERMES[Hermes Agent<br/>GET /api/hermes/report]
    end

    W -->|Auto-Enrollment| ENR
    L -->|Auto-Enrollment| ENR
    ENR --> CA
    CA -->|ca.crt, client.crt, client.key| W
    CA -->|ca.crt, client.crt, client.key| L

    W -->|mTLS Stream| MTLS
    L -->|mTLS Stream| MTLS
    MTLS --> AUTH
    AUTH --> INGEST
    INGEST --> RULE
    RULE --> DB
    DB --> API
    API --> HERMES
```

---

## mTLS Security, Dynamic PKI & Licensing

### 1. TLS 1.3 Mutual Authentication (mTLS)
- **Port 9443**: Ingestion occurs exclusively over TLS 1.3 sockets with `ssl.CERT_REQUIRED`.
- Both the hub and edge clients verify each other's certificates:
  - Client verifies server certificate against `ca.crt`.
  - Server verifies client certificate against the Root CA.
- **Client CN Identification**: In the TLS handshake, the server extracts the `commonName` attribute (`client_id`), validates that the client is marked `active` in the `clients` table, updates the `last_seen` timestamp, and drops unregistered or revoked certificates immediately.

### 2. Dynamic PKI Hub Engine (`src/server_enrollment.py`)
- **Root CA**: On first run, `Server.py` creates a self-signed Root CA (`ca.crt` / `ca.key`) valid for 10 years.
- **Server TLS Certificate**: Generated automatically with Subject Alternative Names (SANs) for `localhost`, `127.0.0.1`, server IP, and hostnames.
- **Authority Key Identifiers**: Full compliance with OpenSSL 3.x and Python 3.12–3.14 via `SubjectKeyIdentifier` and `AuthorityKeyIdentifier` extensions.
- **Dynamic Client Certificates**: RSA-2048 keys and X.509 client certificates are issued on the fly via the enrollment API.

### 3. Seat Accounting & Licensing
- Stored in SQLite table `license_config`:
  - `max_seats`: Maximum concurrent active client licenses (default: 10).
  - `enrollment_secret`: Cryptographic secret required for initial client enrollment.
- Stored in SQLite table `clients`:
  - `client_id`: Unique client identifier (machine GUID or hardware hash).
  - `hostname`, `os_type`, `cert_fingerprint`, `status` (`active` / `revoked`), `first_seen`, `last_seen`.
- When a new client enrolls:
  - If `active_seats >= max_seats`, the hub rejects registration with `HTTP 403 (License seat limit reached)`.
  - Existing registered clients can re-enroll / renew seamlessly without consuming additional seats.

### 4. In-Flight Certificate Watchdog & Automated Self-Healing Renewal
- **Continuous Hub PKI Watchdog**:
  - The server hub runs a continuous background watchdog coroutine (`cert_validity_watchdog`, running every 12 hours) alongside startup checks.
  - The hub automatically inspects expiration dates of both the Root CA (`ca.crt`) and the Server TLS certificate (`server.crt`).
  - If either certificate is within 30 days of expiration, the server regenerates certificates (backing up previous keys as `ca.crt.<timestamp>.bak`) and dynamically reloads its active `ssl.SSLContext` in memory without dropping socket connections or restarting the service.
- **Client Proactive Check & Reactive Self-Healing**:
  - **Proactive Renewal**: Edge clients inspect `client.crt` before every run cycle. If the certificate expires in less than 30 days, it automatically contacts `/api/client/enroll` to renew its certificate.
  - **Reactive Self-Healing**: If the server hub Root CA rotates or a handshake fails with `ssl.SSLError` / `SSLCertVerificationError`, edge clients catch the verification exception, re-bootstrap certificate enrollment against the hub, and re-establish the connection cleanly without human intervention.

---

## Cloud-Side Temporal Persistence & 4-Run Rule

Incoming candidate logs are tracked in SQLite table `active_issues`:
- **Issue Fingerprint**: Formatted as `{site_name}:{server}:{signature}`.
- **12-Hour Evaluation Window**:
  - The hub compares `(now - last_seen)`.
  - If more than 12 hours have elapsed since the issue was last recorded, the previous window expires and the cycle resets to `run_count = 1`.
- **4-Run Rule for Warnings**:
  - The 4-run persistence threshold applies to `WARNING` (and `INFO`) events to eliminate transient operational noise.
  - Each distinct run batch increments `run_count`.
  - Warnings with `run_count < 4` are marked as `TRANSIENT` and excluded from Hermes reports.
  - When `run_count >= 4` within the active 12-hour window, the warning transitions to `VERIFIED`.
- **Immediate Verification for Errors**:
  - High-severity events (`ERROR`, `CRITICAL`, `FATAL`) **always pass immediately**.
  - On their very first ingestion (`run_count = 1`), errors are promoted directly to `VERIFIED` and surfaced to Hermes without waiting for 4 runs.

---

## Agentic Hermes & Client Management API

The server hub exposes a management and reporting REST API (default port `8443`):

### `POST /api/client/enroll`
Client enrollment endpoint:
- **Request**:
  ```json
  {
    "client_id": "web-worker-01.corp.internal",
    "hostname": "web-worker-01",
    "os": "linux",
    "enrollment_secret": "<SECRET>"
  }
  ```
- **Response**:
  ```json
  {
    "ca_cert": "-----BEGIN CERTIFICATE-----\n...",
    "client_cert": "-----BEGIN CERTIFICATE-----\n...",
    "client_key": "-----BEGIN RSA PRIVATE KEY-----\n..."
  }
  ```

### `GET /api/clients`
Returns seat quota status and registered client telemetry:
```json
{
  "active_seats": 2,
  "max_seats": 10,
  "clients": [
    {
      "client_id": "web-worker-01.corp.internal",
      "hostname": "web-worker-01",
      "os_type": "linux",
      "cert_fingerprint": "7D5B660B...",
      "status": "active",
      "first_seen": "2026-09-04 18:00:00",
      "last_seen": "2026-09-04 19:15:00"
    }
  ]
}
```

### `GET /api/hermes/report`
Returns all **verified anomalies** (immediate critical errors and warnings verified after 4 consecutive runs within the 12-hour window):
```json
[
  {
    "fingerprint": "corp.internal:web-worker-01.corp.internal:PostgresPoolExhausted",
    "site": "corp.internal",
    "server": "web-worker-01.corp.internal",
    "signature": "PostgresPoolExhausted",
    "severity": "WARNING",
    "message": "Connection pool saturated (>95%) across 4 runs",
    "os_type": "linux",
    "first_seen": "2026-09-04T07:00:00+00:00",
    "last_seen": "2026-09-04T19:00:00+00:00",
    "consecutive_runs": 4,
    "evaluation_window": "12h",
    "verified": true,
    "status": "VERIFIED"
  }
]
```

### `GET /api/hermes/all`
Diagnostic endpoint listing all candidate issues (`TRANSIENT` and `VERIFIED`).

### `GET /health`
Returns hub health, encryption fingerprint, listener ports, and mTLS status.

---

## Dynamic Machine & Domain Identification

Client configurations intentionally contain **no hardcoded machine name or site name**. Both forwarders dynamically identify their host and domain at runtime via `get_machine_identifier()`:
1. **Fully Qualified Domain Name (FQDN)**: Checked via `socket.getfqdn()`.
2. **OS-Specific Domain Discovery**:
   - **Windows**: Checks Active Directory environment variables (`USERDNSDOMAIN`, `USERDOMAIN`).
   - **Linux**: Parses `/etc/resolv.conf` `domain` and `search` directives.
3. **Reverse DNS Lookup**: Resolves canonical hostname via `socket.gethostbyaddr`.
4. **Fallback**: Local hostname `socket.gethostname()`.

The server automatically infers site attribution from domain qualifiers (e.g. `node01.corp.internal` $\rightarrow$ site `corp.internal`).

---

## Automated Service Installers (Linux & Windows)

LOGAR provides production-grade installation scripts and installer builders for automated service deployment:

### 1. Linux Service Installers
- **Client Installer (`compilation/install_linux_client.sh`)**:
  - Non-interactive script deploying to `/opt/logar-client`.
  - Automatically queries `/etc/machine-id` and enrolls with the hub via `curl`.
  - Installs and enables `logar-client.service` systemd unit.
  ```bash
  sudo ./compilation/install_linux_client.sh "http://hub.example.com:8443" "<ENROLLMENT_SECRET>"
  ```
- **Server Installer (`compilation/install_linux_server.sh`)**:
  - Deploys server to `/opt/logar-server`.
  - Configures logging and installs `logar-server.service` with `LimitNOFILE=65536`.
  ```bash
  sudo ./compilation/install_linux_server.sh
  ```

### 2. Windows Inno Setup Installers
- Built using **Inno Setup 6** and bundled with **NSSM** (`compilation/nssm.exe`):
  - **Client Setup (`compilation/installer_client.iss`)**: Compiles `LOGAR-Client-Setup.exe`. Installs `Win_Client.exe` into `{autopf}\LOGAR`, sets up `LOGAR_Client` service via NSSM with stdout/stderr redirection to `{commonappdata}\LOGAR\client.log`, and starts the service. Clean uninstallation stops and removes the service.
  - **Server Setup (`compilation/installer_server.iss`)**: Compiles `LOGAR-Server-Setup.exe`. Installs `Server.exe` and sets up `LOGAR_Server` Windows service via NSSM.

---

## Repository & Shippables Structure

```
LOGAR/
├── .gitea/
│   └── workflows/
│       ├── ci.yml                     # CI pipeline: syntax, 22 unit tests & mTLS pipeline test
│       ├── release-linux.yml          # Linux release workflow (compiles binaries & checksums)
│       └── release-windows.yml        # Windows release workflow (compiles .exe & Inno Setup installers)
├── compilation/                       # Packaging, installers, and release automation
│   ├── install_linux_client.sh        # Automated Linux client systemd installation script
│   ├── install_linux_server.sh        # Automated Linux server systemd installation script
│   ├── installer_client.iss           # Inno Setup Windows Client installer script
│   ├── installer_server.iss           # Inno Setup Windows Server installer script
│   ├── nssm.exe                       # Official 64-bit NSSM service manager binary
│   ├── package_dist.py                # Standalone binary compiler & packager
│   ├── requirements.txt               # Unified project dependencies
│   └── upload_release.py              # Gitea REST API release asset publisher
├── src/                               # Core application source modules
│   ├── __init__.py
│   ├── Server.py                      # Central mTLS server, temporal engine, and Hermes REST API
│   ├── server_enrollment.py           # Dynamic PKI, Root CA, and client certificate generator
│   ├── Win_Client.py                  # Windows edge forwarder with auto-enrollment
│   └── Linux_Client.py                # Linux edge forwarder with auto-enrollment
├── tests/                             # Automated test suites
│   ├── test_linux_client.py           # Linux client unit tests & mTLS certificate validation
│   ├── test_pipeline.py               # End-to-end mTLS integration & 4-run verification test
│   ├── test_server.py                 # Server unit tests, PKI generation, and seat quota tests
│   └── test_win_client.py             # Windows client unit tests & mTLS certificate validation
├── .gitignore                         # Ignores venv, caches, DBs, and private keys
├── README.md                          # Architecture and usage documentation
├── RELEASE_NOTES.md                   # Release history and changelog
├── server_config.sample.json          # Reference server configuration
└── out/                               # Component guides and sample configs
    ├── linux_server/
    │   ├── README.md
    │   └── server_config.sample.json
    ├── win_server/
    │   ├── README.md
    │   └── server_config.sample.json
    ├── linux_client/
    │   ├── README.md
    │   └── client_config.sample.json
    └── win_client/
        ├── README.md
        └── client_config.sample.json
```

---

## Getting Started & Deployment

### 1. Central Server Hub

1. **Install dependencies**:
   ```bash
   pip install -r compilation/requirements.txt
   ```
2. **Start the server** (generates `server_config.json`, Root CA, and server certs on first run):
   ```bash
   python src/Server.py
   ```
3. **Export a client configuration**:
   ```bash
   python src/Server.py --create-client-config --server-host <SERVER_IP> --server-port 9443 --client-out client_config.json
   ```

### 2. Windows Client Deployment
1. Download `LOGAR-Client-Setup.exe` from releases and run it, or place `Win_Client.exe` and `client_config.json` in `C:\Program Files\LOGAR`.
2. On first run with `client_config.json`, `Win_Client.exe` automatically enrolls with the hub, receives its mTLS certificates, and establishes secure streaming.

### 3. Linux Client Deployment
1. Run the automated installer:
   ```bash
   sudo ./compilation/install_linux_client.sh "http://<HUB_HOST>:8443" "<ENROLLMENT_SECRET>"
   ```
2. The installer enrolls the client, configures `/etc/logar/certs`, and activates `logar-client.service`.

---

## Running Tests

### 1. Component Unit Tests
```bash
python -m unittest discover -s tests -v
```
Runs all **22 unit tests**, covering:
- Dynamic Root CA generation and server TLS certificate issuance.
- Dynamic client certificate issuance with CN and authority key extensions.
- Enrollment secret authentication, seat limits, and certificate revocation.
- Windows & Linux event log collection, deduplication, and mTLS certificate verification.
- 12-hour evaluation window and 4-run rule progression.

### 2. End-to-End Pipeline Integration Test
```bash
# 1. Initialize test configuration
python src/Server.py --create-client-config --server-host 127.0.0.1 --server-port 9443 --client-out client_config.json

# 2. Launch server in background
python src/Server.py &

# 3. Run integration test
python tests/test_pipeline.py
```
Tests client enrollment, secret rejection, mTLS TLS 1.3 socket handshake, warning 4-run rule promotion, immediate error promotion, and Hermes report output.

---

## Automated Releases via Gitea Actions

Releases are triggered automatically on tag push (`v*`):
```bash
git tag v2.0.0
git push origin v2.0.0
```
Two dedicated workflows run in parallel:
- **`release-linux.yml`** (`ubuntu-latest`): Compiles `Linux_Client.bin` and `Server.bin`, generating checksums.
- **`release-windows.yml`** (`windows-latest`): Compiles `Win_Client.exe` and `Server.exe`, builds Inno Setup installers (`LOGAR-Client-Setup.exe`, `LOGAR-Server-Setup.exe`), and uploads all artifacts.