Files

T

me0nline 9584b2c229 Update container pipeline description in README

2026-09-25 15:21:17 +02:00

21 KiB

Raw Blame History

LOGAR: Edge-Thin Log Analysis & Temporal Verification System

LOGAR is an enterprise log aggregation, verification, and anomaly detection architecture designed for heterogeneous server fleets (Windows & Linux). It combines lightweight zero-state edge forwarders with a centralized cloud hub that applies mutual TLS 1.3 (mTLS) authentication, dynamic PKI licensing and quota management, temporal persistence tracking across 12-hour evaluation windows, an automated 4-run rule to filter transient warnings, and immediate pass-through for critical errors before reporting verified anomalies to Hermes.

Core Philosophy
Architecture & Data Flow
mTLS Security, Dynamic PKI & Licensing
Cloud-Side Temporal Persistence & 4-Run Rule
Agentic Hermes & Client Management API
Dynamic Machine & Domain Identification
Automated Service Installers (Linux & Windows)
Repository & Shippables Structure
Getting Started & Deployment
Running Tests
Automated Releases via Gitea Actions

Core Philosophy

1. Edge Thinness & Zero State

Site agents running on Windows and Linux act strictly as lightweight forwarders:

No Local Database: Clients maintain zero local SQLite or heavy cache files.
Source-Level Filtering: Agents stream candidate entries from informational events up to errors (INFO, WARNING, ERROR, CRITICAL), stripping verbose debugging noise (DEBUG, trace entries) and skipping events older than 24 hours.
Transport Security (mTLS 1.3): Logs are streamed directly over mutual TLS 1.3 sockets with hardware-bound / machine-unique client certificates.
Zero Configuration Overhead: Clients auto-bootstrap certificate enrollment on first run if configured with an enrollment secret.

2. Cloud-Side Temporal Persistence & Severity Routing

The central Python hub handles state and verification:

State tracking is managed centrally in SQLite (logar_state.db).
Candidate issues are evaluated over a 12-hour temporal evaluation window.
Warning Persistence (4-Run Rule): WARNING issues must persist across at least 4 consecutive runs / cycles within the 12-hour window to be confirmed as genuine anomalies, automatically filtering transient infrastructure blips.
Immediate Error Pass: Critical errors (ERROR, CRITICAL, FATAL) bypass the 4-run threshold and are promoted immediately to VERIFIED on their first occurrence.

3. Agentic Integration with Hermes

Instead of engineers manually sifting through logs, Hermes ingests pre-filtered anomalies directly from the cloud hub (GET /api/hermes/report), treating verified errors and 4-run validated warnings as actionable system artifacts to trigger precise notifications and remediations.

Architecture & Data Flow

graph TB
    subgraph Edge Nodes [Lightweight Edge Forwarders]
        W[Win_Client.exe / Win_Client.py<br/>Windows Event Log Ingestion]
        L[Linux_Client.bin / Linux_Client.py<br/>systemd journalctl -p warning]
    end

    subgraph Enrollment [Dynamic PKI & Licensing]
        ENR[POST /api/client/enroll<br/>License Quota & Secret Validation]
        CA[Internal Root CA<br/>Signs RSA-2048 Client Cert]
    end

    subgraph Transport [mTLS 1.3 Security Layer]
        MTLS[Mutual TLS 1.3 Handshake<br/>Port 9443 - Client Cert Required]
        AUTH[Extract Client CN & Fingerprint<br/>Validate Active License in SQLite]
    end

    subgraph Hub [LOGAR Server Hub]
        INGEST[Length-Prefixed Frame Ingestion]
        DB[(SQLite Persistence<br/>active_issues, clients, license_config)]
        RULE{12h Window &<br/>4-Run Rule}
    end

    subgraph Downstream [Hermes Agent & Monitoring]
        API[FastAPI Reporting & Management<br/>Port 8443]
        HERMES[Hermes Agent<br/>GET /api/hermes/report]
    end

    W -->|Auto-Enrollment| ENR
    L -->|Auto-Enrollment| ENR
    ENR --> CA
    CA -->|ca.crt, client.crt, client.key| W
    CA -->|ca.crt, client.crt, client.key| L

    W -->|mTLS Stream| MTLS
    L -->|mTLS Stream| MTLS
    MTLS --> AUTH
    AUTH --> INGEST
    INGEST --> RULE
    RULE --> DB
    DB --> API
    API --> HERMES

mTLS Security, Dynamic PKI & Fleet Licensing

1. TLS 1.3 Mutual Authentication (mTLS)

Port 9443: Ingestion occurs exclusively over TLS 1.3 sockets with ssl.CERT_REQUIRED.
Both the hub and edge clients verify each other's certificates:
- Client verifies server certificate against ca.crt.
- Server verifies client certificate against the Subordinate/Root CA.
Client CN Identification & Revocation: In the TLS handshake, the server extracts the commonName attribute (client_id), validates that the client is marked active in the clients table, updates the last_seen timestamp, and drops unregistered or revoked certificates immediately.

2. Master License Authority (`logar.eibl.tech`) & Seat-Bearing Subordinate CA

Master Root Authority: Standalone PKI repository & submodule (LOGAR-LICENSING/, upstream at https://gitea.eibl.tech/me0nline/LOGAR-LICENSING.git) deployed at logar.eibl.tech.
Custom Enterprise X.509 OIDs:
- OID_LOGAR_SEATS (1.3.6.1.4.1.59999.1.1): Embeds the exact licensed seat quota directly inside the signed Subordinate CA certificate.
- OID_LOGAR_LICENSE_ID (1.3.6.1.4.1.59999.1.2): Uniquely tracks enterprise license entitlement.
- OID_LOGAR_ISSUER_URL (1.3.6.1.4.1.59999.1.3): Embedded endpoint coordinate (https://logar.eibl.tech).
Exportable .lic Request & Activated .lic Packages:
- Request: The LOGAR server generates its Subordinate CA private key (certs/ca.key), Certificate Signing Request (certs/ca.csr), and packages them into an exportable logar_server.lic file.
- Activation Package: The master license authority issues an activated .lic package containing the signed Subordinate CA, granted seat quota, server metadata, and the Master Root CA (upstream_ca.crt).
- Importable via WebUI or CLI (python src/Server.py --import-lic activated.lic or --import-ca).
Self-Updating Client Servers & Dynamic Seat Expansion:
- Client servers update themselves against the master authority (logar.eibl.tech) over mTLS (python src/Server.py --sync-license or via WebUI / automated watchdog).
- Seat quotas can be increased with each renewal or updated on-demand on the master authority.
- The client server automatically detects the higher seat count in the signed certificate's OID, updates its database and runtime configuration, and reloads its in-flight ssl.SSLContext without dropping client connections or restarting services.
Automated Container Pipeline & Portainer GitOps Deployment:
- The license authority repository includes a dedicated CI/CD workflow (.gitea/workflows/build-container.yml) compiling and publishing the container image to gitea.eibl.tech/me0nline/logar-licensing:latest (authenticated via giteatoken secret) and 10.30.0.3:5000/logar-licensing:latest.
- It deploys seamlessly in Portainer using a GitOps Stack connected directly to LOGAR-LICENSING via docker-compose.yml with automatic polling or webhook redeployment and persistent volume storage (logar_licensing_data) preserving Root CA keys and database.
- Reverse proxy integration (e.g., Nginx Proxy Manager, Caddy) terminates HTTPS on port 443, verifies client mTLS against the downloadable Master Root CA (/api/ca.crt), and forwards client certificate headers (X-SSL-Client-Cert) to port 8090.

3. Integrated Fleet Licensing & WebUI Dashboard

WebUI URL: Accessible at http://<server_host>:8443/webui (or /).
Visual Seat Quota Meter: Real-time progress bar displaying active forwarders vs. allocated seat limit.
Licensing Actions:
- 1-click Export .lic Request file download.
- Interactive View & Copy CA CSR modal.
- Drag-and-drop or paste Import License (.lic / ca.crt) modal.
- Sync & Renew via mTLS button for immediate upstream synchronization and seat expansion.
Seat Management & Revocation Table:
- Lists all connected Windows and Linux edge forwarders.
- Revoke Seat: Instantly drops and blocks the client at the TLS socket handshake layer.
- Reactivate Seat: Restores the client to active status if within seat quota limits.

4. Cloud-Side In-Flight Certificate Watchdog

Continuous Hub PKI Watchdog:
- The server hub runs a continuous background watchdog coroutine (cert_validity_watchdog, running every 12 hours).
- Automatically renews Subordinate CA certificates over mTLS with logar.eibl.tech and Server TLS certificates in-flight, dynamically reloading the active ssl.SSLContext.
Client Proactive Check & Reactive Self-Healing:
- Proactive Renewal: Edge clients inspect client.crt before every run cycle. If expiring within 30 days, they contact /api/client/enroll to renew.
- Reactive Self-Healing: If the server CA rotates or a handshake error occurs, edge clients catch the exception, re-bootstrap enrollment against the hub, and re-establish streaming cleanly.

Cloud-Side Temporal Persistence & 4-Run Rule

Incoming candidate logs are tracked in SQLite table active_issues:

Issue Fingerprint: Formatted as {site_name}:{server}:{signature}.
12-Hour Evaluation Window:
- The hub compares (now - last_seen).
- If more than 12 hours have elapsed since the issue was last recorded, the previous window expires and the cycle resets to run_count = 1.
4-Run Rule for Warnings:
- The 4-run persistence threshold applies to WARNING (and INFO) events to eliminate transient operational noise.
- Each distinct run batch increments run_count.
- Warnings with run_count < 4 are marked as TRANSIENT and excluded from Hermes reports.
- When run_count >= 4 within the active 12-hour window, the warning transitions to VERIFIED.
Immediate Verification for Errors:
- High-severity events (ERROR, CRITICAL, FATAL) always pass immediately.
- On their very first ingestion (run_count = 1), errors are promoted directly to VERIFIED and surfaced to Hermes without waiting for 4 runs.

Agentic Hermes & Client Management API

The server hub exposes a management and reporting REST API (default port 8443):

`POST /api/client/enroll`

Client enrollment endpoint:

Request:

{
  "client_id": "web-worker-01.corp.internal",
  "hostname": "web-worker-01",
  "os": "linux",
  "enrollment_secret": "<SECRET>"
}

Response:

{
  "ca_cert": "-----BEGIN CERTIFICATE-----\n...",
  "client_cert": "-----BEGIN CERTIFICATE-----\n...",
  "client_key": "-----BEGIN RSA PRIVATE KEY-----\n..."
}

`GET /api/clients`

Returns seat quota status and registered client telemetry:

{
  "active_seats": 2,
  "max_seats": 10,
  "clients": [
    {
      "client_id": "web-worker-01.corp.internal",
      "hostname": "web-worker-01",
      "os_type": "linux",
      "cert_fingerprint": "7D5B660B...",
      "status": "active",
      "first_seen": "2026-09-04 18:00:00",
      "last_seen": "2026-09-04 19:15:00"
    }
  ]
}

`GET /api/hermes/report`

Returns all verified anomalies (immediate critical errors and warnings verified after 4 consecutive runs within the 12-hour window):

[
  {
    "fingerprint": "corp.internal:web-worker-01.corp.internal:PostgresPoolExhausted",
    "site": "corp.internal",
    "server": "web-worker-01.corp.internal",
    "signature": "PostgresPoolExhausted",
    "severity": "WARNING",
    "message": "Connection pool saturated (>95%) across 4 runs",
    "os_type": "linux",
    "first_seen": "2026-09-04T07:00:00+00:00",
    "last_seen": "2026-09-04T19:00:00+00:00",
    "consecutive_runs": 4,
    "evaluation_window": "12h",
    "verified": true,
    "status": "VERIFIED"
  }
]

`GET /api/hermes/all`

Diagnostic endpoint listing all candidate issues (TRANSIENT and VERIFIED).

`GET /health`

Returns hub health, encryption fingerprint, listener ports, and mTLS status.

Dynamic Machine & Domain Identification

Client configurations intentionally contain no hardcoded machine name or site name. Both forwarders dynamically identify their host and domain at runtime via get_machine_identifier():

Fully Qualified Domain Name (FQDN): Checked via socket.getfqdn().
OS-Specific Domain Discovery:
- Windows: Checks Active Directory environment variables (USERDNSDOMAIN, USERDOMAIN).
- Linux: Parses /etc/resolv.conf domain and search directives.
Reverse DNS Lookup: Resolves canonical hostname via socket.gethostbyaddr.
Fallback: Local hostname socket.gethostname().

The server automatically infers site attribution from domain qualifiers (e.g. node01.corp.internal \rightarrow site corp.internal).

Automated Service Installers (Linux & Windows)

LOGAR provides production-grade installation scripts and installer builders for automated service deployment:

1. Linux Service Installers

Client Installer (compilation/install_linux_client.sh):
- Non-interactive script deploying to /opt/logar-client.
- Automatically queries /etc/machine-id and enrolls with the hub via curl.
- Installs and enables logar-client.service systemd unit.
```
sudo ./compilation/install_linux_client.sh "http://hub.example.com:8443" "<ENROLLMENT_SECRET>"
```
Server Installer (compilation/install_linux_server.sh):
- Deploys server to /opt/logar-server.
- Configures logging and installs logar-server.service with LimitNOFILE=65536.
```
sudo ./compilation/install_linux_server.sh
```

2. Windows Inno Setup Installers

Built using Inno Setup 6 and bundled with NSSM (compilation/nssm.exe):
- Client Setup (compilation/installer_client.iss): Compiles LOGAR-Client-Setup.exe. Installs Win_Client.exe into {autopf}\LOGAR, sets up LOGAR_Client service via NSSM with stdout/stderr redirection to {commonappdata}\LOGAR\client.log, and starts the service. Clean uninstallation stops and removes the service.
- Server Setup (compilation/installer_server.iss): Compiles LOGAR-Server-Setup.exe. Installs Server.exe and sets up LOGAR_Server Windows service via NSSM.

Repository & Shippables Structure

LOGAR/
├── .gitea/
│   └── workflows/
│       ├── ci.yml                     # CI pipeline: syntax, 25 unit tests & mTLS pipeline test
│       └── release.yml                # Consolidated release workflow (Linux binaries & Windows installers)
├── compilation/                       # Packaging, installers, and release automation
│   ├── install_linux_client.sh        # Automated Linux client systemd installation script
│   ├── install_linux_server.sh        # Automated Linux server systemd installation script
│   ├── installer_client.iss           # Inno Setup Windows Client installer script
│   ├── installer_server.iss           # Inno Setup Windows Server installer script
│   ├── nssm.exe                       # Official 64-bit NSSM service manager binary
│   ├── package_dist.py                # Standalone binary compiler & packager
│   ├── requirements.txt               # Unified project dependencies
│   └── upload_release.py              # Gitea REST API release asset publisher
├── src/                               # Core application source modules
│   ├── __init__.py
│   ├── Server.py                      # Central mTLS server, temporal engine, and Hermes REST API
│   ├── server_enrollment.py           # Dynamic PKI, Root CA, and client certificate generator
│   ├── Win_Client.py                  # Windows edge forwarder with auto-enrollment
│   └── Linux_Client.py                # Linux edge forwarder with auto-enrollment
├── tests/                             # Automated test suites
│   ├── test_linux_client.py           # Linux client unit tests & mTLS certificate validation
│   ├── test_pipeline.py               # End-to-end mTLS integration & 4-run verification test
│   ├── test_server.py                 # Server unit tests, PKI generation, and seat quota tests
│   └── test_win_client.py             # Windows client unit tests & mTLS certificate validation
├── .gitignore                         # Ignores venv, caches, DBs, and private keys
├── README.md                          # Architecture and usage documentation
├── RELEASE_NOTES.md                   # Release history and changelog
├── server_config.sample.json          # Reference server configuration
└── out/                               # Component guides and sample configs
    ├── linux_server/
    │   ├── README.md
    │   └── server_config.sample.json
    ├── win_server/
    │   ├── README.md
    │   └── server_config.sample.json
    ├── linux_client/
    │   ├── README.md
    │   └── client_config.sample.json
    └── win_client/
        ├── README.md
        └── client_config.sample.json

Getting Started & Deployment

1. Central Server Hub

Install dependencies:

pip install -r compilation/requirements.txt

Start the server (generates server_config.json, Root CA, and server certs on first run):
```
python src/Server.py
```

Export a client configuration:

python src/Server.py --create-client-config --server-host <SERVER_IP> --server-port 9443 --client-out client_config.json

2. Windows Client Deployment

Download LOGAR-Client-Setup.exe from releases and run it, or place Win_Client.exe and client_config.json in C:\Program Files\LOGAR.
On first run with client_config.json, Win_Client.exe automatically enrolls with the hub, receives its mTLS certificates, and establishes secure streaming.

3. Linux Client Deployment

Run the automated installer:

sudo ./compilation/install_linux_client.sh "http://<HUB_HOST>:8443" "<ENROLLMENT_SECRET>"

The installer enrolls the client, configures /etc/logar/certs, and activates logar-client.service.

Running Tests

1. Component Unit Tests

python -m unittest discover -s tests -v

Runs all 25 unit tests, covering:

Dynamic Root CA generation and server TLS certificate issuance.
Dynamic client certificate issuance with CN and authority key extensions.
Enrollment secret authentication, seat limits, and certificate revocation.
Proactive certificate validity checks, Root CA auto-renewal, and in-flight server SSLContext reload.
Windows & Linux event log collection, deduplication, and mTLS certificate verification.
12-hour evaluation window and 4-run rule progression.

2. End-to-End Pipeline Integration Test

# 1. Initialize test configuration
python src/Server.py --create-client-config --server-host 127.0.0.1 --server-port 9443 --client-out client_config.json

# 2. Launch server in background
python src/Server.py &

# 3. Run integration test
python tests/test_pipeline.py

Tests client enrollment, secret rejection, mTLS TLS 1.3 socket handshake, warning 4-run rule promotion, immediate error promotion, and Hermes report output.

Releases & Binary Distribution

Releases can be built and published through three complementary channels:

1. Tag Push Automation (Gitea Actions)

Pushing a release tag automatically triggers the build workflows:

git tag v2.0.1
git push origin v2.0.1

The consolidated workflow release.yml defines two parallel jobs:

release-linux (ubuntu-latest): Compiles standalone native ELF binaries (Linux_Client.bin, Server.bin) and checksums.
release-windows (windows-latest): Compiles Windows executables (Win_Client.exe, Server.exe), builds Inno Setup installers, and publishes checksums (requires self-hosted Windows Act Runner).

2. Manual Workflow Dispatch (Gitea UI)

Workflows can be manually triggered on demand from the Gitea web interface:

Navigate to Actions \rightarrow Release Binaries & Installers (release.yml).
Click Run workflow, set the release tag (defaults to v2.0.1), and run.

3. Native Local Windows Build & Direct Release Upload

For environments without a registered Windows CI runner, Windows executables can be built and published directly to Gitea releases:

# 1. Build Windows binaries locally
python compilation/package_dist.py --target windows

# 2. Upload assets and release notes directly to the Gitea release
python compilation/upload_release.py --tag v2.0.1 --token <GITEA_TOKEN> --skip-build

21 KiB Raw Blame History