LOGAR/README.md

LOGAR: Edge-Thin Log Analysis & Temporal Verification System

LOGAR is an enterprise log aggregation, verification, and anomaly detection architecture designed for heterogeneous server fleets (Windows & Linux). It combines lightweight zero-state edge forwarders with a centralized cloud hub that applies mutual TLS 1.3 (mTLS) authentication, dynamic PKI licensing and quota management, temporal persistence tracking across 12-hour evaluation windows, an automated 4-run rule to filter transient warnings, and immediate pass-through for critical errors before reporting verified anomalies to Hermes.

---

Table of Contents

1. Core Philosophy
2. Architecture & Data Flow
3. mTLS Security, Dynamic PKI & Licensing
4. Cloud-Side Temporal Persistence & 4-Run Rule
5. Agentic Hermes & Client Management API
6. Dynamic Machine & Domain Identification
7. Automated Service Installers (Linux & Windows)
8. Repository & Shippables Structure
9. Getting Started & Deployment
10. Running Tests
11. Automated Releases via Gitea Actions

---

Core Philosophy

1. Edge Thinness & Zero State

Site agents running on Windows and Linux act strictly as lightweight forwarders:

• No Local Database: Clients maintain zero local SQLite or heavy cache files.
• Source-Level Filtering: Agents stream candidate entries from informational events up to errors (INFO, WARNING, ERROR, CRITICAL), stripping verbose debugging noise (DEBUG, trace entries) and skipping events older than 24 hours.
• Transport Security (mTLS 1.3): Logs are streamed directly over mutual TLS 1.3 sockets with hardware-bound / machine-unique client certificates.
• Zero Configuration Overhead: Clients auto-bootstrap certificate enrollment on first run if configured with an enrollment secret.

2. Cloud-Side Temporal Persistence & Severity Routing

The central Python hub handles state and verification:

• State tracking is managed centrally in SQLite (logar_state.db).
• Candidate issues are evaluated over a 12-hour temporal evaluation window.
• Warning Persistence (4-Run Rule): WARNING issues must persist across at least 4 consecutive runs / cycles within the 12-hour window to be confirmed as genuine anomalies, automatically filtering transient infrastructure blips.
• Immediate Error Pass: Critical errors (ERROR, CRITICAL, FATAL) bypass the 4-run threshold and are promoted immediately to VERIFIED on their first occurrence.

3. Agentic Integration with Hermes

Instead of engineers manually sifting through logs, Hermes ingests pre-filtered anomalies directly from the cloud hub (GET /api/hermes/report), treating verified errors and 4-run validated warnings as actionable system artifacts to trigger precise notifications and remediations.

---

Architecture & Data Flow

graph TB
    subgraph Edge Nodes [Lightweight Edge Forwarders]
        W[Win_Client.exe / Win_Client.py<br/>Windows Event Log Ingestion]
        L[Linux_Client.bin / Linux_Client.py<br/>systemd journalctl -p warning]
    end

    subgraph Enrollment [Dynamic PKI & Licensing]
        ENR[POST /api/client/enroll<br/>License Quota & Secret Validation]
        CA[Internal Root CA<br/>Signs RSA-2048 Client Cert]
    end

    subgraph Transport [mTLS 1.3 Security Layer]
        MTLS[Mutual TLS 1.3 Handshake<br/>Port 9443 - Client Cert Required]
        AUTH[Extract Client CN & Fingerprint<br/>Validate Active License in SQLite]
    end

    subgraph Hub [LOGAR Server Hub]
        INGEST[Length-Prefixed Frame Ingestion]
        DB[(SQLite Persistence<br/>active_issues, clients, license_config)]
        RULE{12h Window &<br/>4-Run Rule}
    end

    subgraph Downstream [Hermes Agent & Monitoring]
        API[FastAPI Reporting & Management<br/>Port 8443]
        HERMES[Hermes Agent<br/>GET /api/hermes/report]
    end

    W -->|Auto-Enrollment| ENR
    L -->|Auto-Enrollment| ENR
    ENR --> CA
    CA -->|ca.crt, client.crt, client.key| W
    CA -->|ca.crt, client.crt, client.key| L

    W -->|mTLS Stream| MTLS
    L -->|mTLS Stream| MTLS
    MTLS --> AUTH
    AUTH --> INGEST
    INGEST --> RULE
    RULE --> DB
    DB --> API
    API --> HERMES

---

mTLS Security, Dynamic PKI & Licensing

1. TLS 1.3 Mutual Authentication (mTLS)

• Port 9443: Ingestion occurs exclusively over TLS 1.3 sockets with ssl.CERT_REQUIRED.
• Both the hub and edge clients verify each other's certificates:
  • Client verifies server certificate against ca.crt.
  • Server verifies client certificate against the Root CA.
• Client CN Identification: In the TLS handshake, the server extracts the commonName attribute (client_id), validates that the client is marked active in the clients table, updates the last_seen timestamp, and drops unregistered or revoked certificates immediately.

2. Dynamic PKI Hub Engine (src/server_enrollment.py)

• Root CA: On first run, Server.py creates a self-signed Root CA (ca.crt / ca.key) valid for 10 years.
• Server TLS Certificate: Generated automatically with Subject Alternative Names (SANs) for localhost, 127.0.0.1, server IP, and hostnames.
• Authority Key Identifiers: Full compliance with OpenSSL 3.x and Python 3.12–3.14 via SubjectKeyIdentifier and AuthorityKeyIdentifier extensions.
• Dynamic Client Certificates: RSA-2048 keys and X.509 client certificates are issued on the fly via the enrollment API.

3. Seat Accounting & Licensing

• Stored in SQLite table license_config:
  • max_seats: Maximum concurrent active client licenses (default: 10).
  • enrollment_secret: Cryptographic secret required for initial client enrollment.
• Stored in SQLite table clients:
  • client_id: Unique client identifier (machine GUID or hardware hash).
  • hostname, os_type, cert_fingerprint, status (active / revoked), first_seen, last_seen.
• When a new client enrolls:
  • If active_seats >= max_seats, the hub rejects registration with HTTP 403 (License seat limit reached).
  • Existing registered clients can re-enroll / renew seamlessly without consuming additional seats.

---

Cloud-Side Temporal Persistence & 4-Run Rule

Incoming candidate logs are tracked in SQLite table active_issues:

• Issue Fingerprint: Formatted as {site_name}:{server}:{signature}.
• 12-Hour Evaluation Window:
  • The hub compares (now - last_seen).
  • If more than 12 hours have elapsed since the issue was last recorded, the previous window expires and the cycle resets to run_count = 1.
• 4-Run Rule for Warnings:
  • The 4-run persistence threshold applies to WARNING (and INFO) events to eliminate transient operational noise.
  • Each distinct run batch increments run_count.
  • Warnings with run_count < 4 are marked as TRANSIENT and excluded from Hermes reports.
  • When run_count >= 4 within the active 12-hour window, the warning transitions to VERIFIED.
• Immediate Verification for Errors:
  • High-severity events (ERROR, CRITICAL, FATAL) always pass immediately.
  • On their very first ingestion (run_count = 1), errors are promoted directly to VERIFIED and surfaced to Hermes without waiting for 4 runs.

---

Agentic Hermes & Client Management API

The server hub exposes a management and reporting REST API (default port 8443):

POST /api/client/enroll

Client enrollment endpoint:

• Request:

  {
    "client_id": "web-worker-01.corp.internal",
    "hostname": "web-worker-01",
    "os": "linux",
    "enrollment_secret": "<SECRET>"
  }
  

• Response:

  {
    "ca_cert": "-----BEGIN CERTIFICATE-----\n...",
    "client_cert": "-----BEGIN CERTIFICATE-----\n...",
    "client_key": "-----BEGIN RSA PRIVATE KEY-----\n..."
  }
  

GET /api/clients

Returns seat quota status and registered client telemetry:

{
  "active_seats": 2,
  "max_seats": 10,
  "clients": [
    {
      "client_id": "web-worker-01.corp.internal",
      "hostname": "web-worker-01",
      "os_type": "linux",
      "cert_fingerprint": "7D5B660B...",
      "status": "active",
      "first_seen": "2026-09-04 18:00:00",
      "last_seen": "2026-09-04 19:15:00"
    }
  ]
}

GET /api/hermes/report

Returns all verified anomalies (immediate critical errors and warnings verified after 4 consecutive runs within the 12-hour window):

[
  {
    "fingerprint": "corp.internal:web-worker-01.corp.internal:PostgresPoolExhausted",
    "site": "corp.internal",
    "server": "web-worker-01.corp.internal",
    "signature": "PostgresPoolExhausted",
    "severity": "WARNING",
    "message": "Connection pool saturated (>95%) across 4 runs",
    "os_type": "linux",
    "first_seen": "2026-09-04T07:00:00+00:00",
    "last_seen": "2026-09-04T19:00:00+00:00",
    "consecutive_runs": 4,
    "evaluation_window": "12h",
    "verified": true,
    "status": "VERIFIED"
  }
]

GET /api/hermes/all

Diagnostic endpoint listing all candidate issues (TRANSIENT and VERIFIED).

GET /health

Returns hub health, encryption fingerprint, listener ports, and mTLS status.

---

Dynamic Machine & Domain Identification

Client configurations intentionally contain no hardcoded machine name or site name. Both forwarders dynamically identify their host and domain at runtime via get_machine_identifier():

1. Fully Qualified Domain Name (FQDN): Checked via socket.getfqdn().
2. OS-Specific Domain Discovery:
   • Windows: Checks Active Directory environment variables (USERDNSDOMAIN, USERDOMAIN).
   • Linux: Parses /etc/resolv.conf domain and search directives.
3. Reverse DNS Lookup: Resolves canonical hostname via socket.gethostbyaddr.
4. Fallback: Local hostname socket.gethostname().

The server automatically infers site attribution from domain qualifiers (e.g. node01.corp.internal \rightarrow site corp.internal).

---

Automated Service Installers (Linux & Windows)

LOGAR provides production-grade installation scripts and installer builders for automated service deployment:

1. Linux Service Installers

• Client Installer (compilation/install_linux_client.sh):
  • Non-interactive script deploying to /opt/logar-client.
  • Automatically queries /etc/machine-id and enrolls with the hub via curl.
  • Installs and enables logar-client.service systemd unit.

  sudo ./compilation/install_linux_client.sh "http://hub.example.com:8443" "<ENROLLMENT_SECRET>"
  

• Server Installer (compilation/install_linux_server.sh):
  • Deploys server to /opt/logar-server.
  • Configures logging and installs logar-server.service with LimitNOFILE=65536.

  sudo ./compilation/install_linux_server.sh
  

2. Windows Inno Setup Installers

• Built using Inno Setup 6 and bundled with NSSM (compilation/nssm.exe):
  • Client Setup (compilation/installer_client.iss): Compiles LOGAR-Client-Setup.exe. Installs Win_Client.exe into {autopf}\LOGAR, sets up LOGAR_Client service via NSSM with stdout/stderr redirection to {commonappdata}\LOGAR\client.log, and starts the service. Clean uninstallation stops and removes the service.
  • Server Setup (compilation/installer_server.iss): Compiles LOGAR-Server-Setup.exe. Installs Server.exe and sets up LOGAR_Server Windows service via NSSM.

---

Repository & Shippables Structure

LOGAR/
├── .gitea/
│   └── workflows/
│       ├── ci.yml                     # CI pipeline: syntax, 22 unit tests & mTLS pipeline test
│       ├── release-linux.yml          # Linux release workflow (compiles binaries & checksums)
│       └── release-windows.yml        # Windows release workflow (compiles .exe & Inno Setup installers)
├── compilation/                       # Packaging, installers, and release automation
│   ├── install_linux_client.sh        # Automated Linux client systemd installation script
│   ├── install_linux_server.sh        # Automated Linux server systemd installation script
│   ├── installer_client.iss           # Inno Setup Windows Client installer script
│   ├── installer_server.iss           # Inno Setup Windows Server installer script
│   ├── nssm.exe                       # Official 64-bit NSSM service manager binary
│   ├── package_dist.py                # Standalone binary compiler & packager
│   ├── requirements.txt               # Unified project dependencies
│   └── upload_release.py              # Gitea REST API release asset publisher
├── src/                               # Core application source modules
│   ├── __init__.py
│   ├── Server.py                      # Central mTLS server, temporal engine, and Hermes REST API
│   ├── server_enrollment.py           # Dynamic PKI, Root CA, and client certificate generator
│   ├── Win_Client.py                  # Windows edge forwarder with auto-enrollment
│   └── Linux_Client.py                # Linux edge forwarder with auto-enrollment
├── tests/                             # Automated test suites
│   ├── test_linux_client.py           # Linux client unit tests & mTLS certificate validation
│   ├── test_pipeline.py               # End-to-end mTLS integration & 4-run verification test
│   ├── test_server.py                 # Server unit tests, PKI generation, and seat quota tests
│   └── test_win_client.py             # Windows client unit tests & mTLS certificate validation
├── .gitignore                         # Ignores venv, caches, DBs, and private keys
├── README.md                          # Architecture and usage documentation
├── RELEASE_NOTES.md                   # Release history and changelog
├── server_config.sample.json          # Reference server configuration
└── out/                               # Component guides and sample configs
    ├── linux_server/
    │   ├── README.md
    │   └── server_config.sample.json
    ├── win_server/
    │   ├── README.md
    │   └── server_config.sample.json
    ├── linux_client/
    │   ├── README.md
    │   └── client_config.sample.json
    └── win_client/
        ├── README.md
        └── client_config.sample.json

---

Getting Started & Deployment

1. Central Server Hub

1. Install dependencies:

   pip install -r compilation/requirements.txt
   

2. Start the server (generates server_config.json, Root CA, and server certs on first run):

   python src/Server.py
   

3. Export a client configuration:

   python src/Server.py --create-client-config --server-host <SERVER_IP> --server-port 9443 --client-out client_config.json
   

2. Windows Client Deployment

1. Download LOGAR-Client-Setup.exe from releases and run it, or place Win_Client.exe and client_config.json in C:\Program Files\LOGAR.
2. On first run with client_config.json, Win_Client.exe automatically enrolls with the hub, receives its mTLS certificates, and establishes secure streaming.

3. Linux Client Deployment

1. Run the automated installer:

   sudo ./compilation/install_linux_client.sh "http://<HUB_HOST>:8443" "<ENROLLMENT_SECRET>"
   

2. The installer enrolls the client, configures /etc/logar/certs, and activates logar-client.service.

---

Running Tests

1. Component Unit Tests

python -m unittest discover -s tests -v

Runs all 22 unit tests, covering:

• Dynamic Root CA generation and server TLS certificate issuance.
• Dynamic client certificate issuance with CN and authority key extensions.
• Enrollment secret authentication, seat limits, and certificate revocation.
• Windows & Linux event log collection, deduplication, and mTLS certificate verification.
• 12-hour evaluation window and 4-run rule progression.

2. End-to-End Pipeline Integration Test

# 1. Initialize test configuration
python src/Server.py --create-client-config --server-host 127.0.0.1 --server-port 9443 --client-out client_config.json

# 2. Launch server in background
python src/Server.py &

# 3. Run integration test
python tests/test_pipeline.py

Tests client enrollment, secret rejection, mTLS TLS 1.3 socket handshake, warning 4-run rule promotion, immediate error promotion, and Hermes report output.

---

Automated Releases via Gitea Actions

Releases are triggered automatically on tag push (v*):

git tag v1.0.4
git push origin v1.0.4

Two dedicated workflows run in parallel:

• release-linux.yml (ubuntu-latest): Compiles Linux_Client.bin and Server.bin, generating checksums.
• release-windows.yml (windows-latest): Compiles Win_Client.exe and Server.exe, builds Inno Setup installers (LOGAR-Client-Setup.exe, LOGAR-Server-Setup.exe), and uploads all artifacts.