me0nline/LOGAR
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 10 Wiki Activity
Files
3b2c3e0fd707364ef94067bdc6c219ac23c2c07b
LOGAR/README.md
T

20 KiB
Raw Blame History

LOGAR: Edge-Thin Log Analysis & Temporal Verification System

LOGAR is an enterprise log aggregation, verification, and anomaly detection architecture designed for heterogeneous server fleets (Windows & Linux). It combines lightweight zero-state edge forwarders with a centralized cloud hub that applies mutual TLS 1.3 (mTLS) authentication, dynamic PKI licensing and quota management, temporal persistence tracking across 12-hour evaluation windows, and direct anomaly pass-through enabling Hermes AI to perform all deep reasoning, anomaly evaluation, and autonomous decision-making.

Table of Contents

  1. Core Philosophy
  2. Architecture & Data Flow
  3. mTLS Security, Dynamic PKI & Licensing
  4. Cloud-Side Temporal Persistence & Direct Hermes Reasoning
  5. Agentic Hermes & Client Management API
  6. Dynamic Machine & Domain Identification
  7. Automated Service Installers (Linux & Windows)
  8. Repository & Shippables Structure
  9. Getting Started & Deployment
  10. Running Tests
  11. Automated Releases via Gitea Actions

Core Philosophy

1. Edge Thinness & Zero State

Site agents running on Windows and Linux act strictly as lightweight forwarders:

  • No Local Database: Clients maintain zero local SQLite or heavy cache files.
  • Source-Level Filtering: Agents stream candidate entries from informational events up to errors (INFO, WARNING, ERROR, CRITICAL), stripping verbose debugging noise (DEBUG, trace entries) and skipping events older than 24 hours.
  • Transport Security (mTLS 1.3): Logs are streamed directly over mutual TLS 1.3 sockets with hardware-bound / machine-unique client certificates.
  • Zero Configuration Overhead: Clients auto-bootstrap certificate enrollment on first run if configured with an enrollment secret.

2. Cloud-Side Temporal Persistence & Direct Hermes Reasoning

The central Python hub handles state and temporal tracking:

  • State tracking is managed centrally in SQLite (logar_state.db).
  • Candidate issues are tracked over a 12-hour temporal evaluation window with consecutive run counts and first/last-seen timestamps.
  • Direct Hermes Intelligence: All ingested actionable anomalies (WARNING, ERROR, INFO) are immediately promoted to VERIFIED and surfaced directly to Hermes. Hermes performs all multi-factor reasoning, pattern evaluation, and contextual thinking rather than relying on arbitrary hardcoded threshold rules.

3. Agentic Integration with Hermes

Instead of engineers manually sifting through logs, Hermes ingests verified anomalies directly from the cloud hub (GET /api/hermes/report), treating errors and warnings with full temporal metadata (run counts, timestamps) as actionable system artifacts to trigger precise notifications and remediations.

Architecture & Data Flow

graph TB
    subgraph Edge Nodes [Lightweight Edge Forwarders]
        W[Win_Client.exe / Win_Client.py<br/>Windows Event Log Ingestion]
        L[Linux_Client.bin / Linux_Client.py<br/>systemd journalctl -p warning]
    end

    subgraph Enrollment [Dynamic PKI & Licensing]
        ENR[POST /api/client/enroll<br/>License Quota & Secret Validation]
        CA[Internal Root CA<br/>Signs RSA-2048 Client Cert]
    end

    subgraph Transport [mTLS 1.3 Security Layer]
        MTLS[Mutual TLS 1.3 Handshake<br/>Port 9443 - Client Cert Required]
        AUTH[Extract Client CN & Fingerprint<br/>Validate Active License in SQLite]
    end

    subgraph Hub [LOGAR Server Hub]
        INGEST[Length-Prefixed Frame Ingestion]
        DB[(SQLite Persistence<br/>active_issues, clients, license_config)]
        RULE{12h Window &<br/>Temporal Ingestion}
    end

    subgraph Downstream [Hermes Agent & Monitoring]
        API[FastAPI Reporting & Management<br/>Port 8443]
        HERMES[Hermes Agent<br/>GET /api/hermes/report]
    end

    W -->|Auto-Enrollment| ENR
    L -->|Auto-Enrollment| ENR
    ENR --> CA
    CA -->|ca.crt, client.crt, client.key| W
    CA -->|ca.crt, client.crt, client.key| L

    W -->|mTLS Stream| MTLS
    L -->|mTLS Stream| MTLS
    MTLS --> AUTH
    AUTH --> INGEST
    INGEST --> RULE
    RULE --> DB
    DB --> API
    API --> HERMES

mTLS Security, Dynamic PKI & Fleet Licensing

1. TLS 1.3 Mutual Authentication (mTLS)

  • Port 9443: Ingestion occurs exclusively over TLS 1.3 sockets with ssl.CERT_REQUIRED.
  • Both the hub and edge clients verify each other's certificates:
    • Client verifies server certificate against ca.crt.
    • Server verifies client certificate against the Subordinate/Root CA.
  • Client CN Identification & Revocation: In the TLS handshake, the server extracts the commonName attribute (client_id), validates that the client is marked active in the clients table, updates the last_seen timestamp, and drops unregistered or revoked certificates immediately.

2. Master License Authority (logar.eibl.tech) & Seat-Bearing Subordinate CA

  • Master Root Authority: Standalone PKI repository & submodule (LOGAR-LICENSING/, upstream at https://gitea.eibl.tech/me0nline/LOGAR-LICENSING.git) deployed at logar.eibl.tech.

  • Custom Enterprise X.509 OIDs:

    • OID_LOGAR_SEATS (1.3.6.1.4.1.59999.1.1): Embeds the exact licensed seat quota directly inside the signed Subordinate CA certificate.
    • OID_LOGAR_LICENSE_ID (1.3.6.1.4.1.59999.1.2): Uniquely tracks enterprise license entitlement.
    • OID_LOGAR_ISSUER_URL (1.3.6.1.4.1.59999.1.3): Embedded endpoint coordinate (https://logar.eibl.tech).

  • Bootstrap .lic Packages & Instant WebSocket Auto-Enrollment:

    • The License Authority (logar.eibl.tech) generates initial Bootstrap .lic Packages containing Master Root CA, dedicated communication credentials (communication_cert + communication_key), CSR templates, and WebSocket coordinates (wss://logar.eibl.tech/ws/license).
    • 1-Click Import & Live Signing: Importing the .lic file on the client server (python src/Server.py --import-lic <file.lic> or via WebUI) connects directly via WebSocket, submits the server Sub-CA CSR, and immediately installs the officially signed Sub-CA license certificate.

  • Self-Updating Client Servers & Live WebSocket Sync:

    • Client servers update themselves against the master authority (wss://logar.eibl.tech/ws/license) over WebSocket (python src/Server.py --sync-license or via WebUI / automated watchdog).
    • Seat quotas can be increased with each renewal or updated on-demand in the License Authority WebUI.
    • The client server automatically receives the updated seat quota in its signed certificate, updates its SQLite database and runtime configuration, and reloads its in-flight ssl.SSLContext without dropping client connections or restarting services.

  • Automated Container Pipeline & Ephemeral NPM Pass-Through:

    • The license authority repository includes a dedicated CI/CD workflow (.gitea/workflows/build-container.yml) publishing to gitea.eibl.tech/me0nline/logar-licensing:latest (authenticated via giteatoken secret).
    • Works seamlessly with ephemeral Nginx Proxy Manager (NPM) by enabling the standard Websockets Support toggle for logar.eibl.tech.
    • Admin WebUI is protected by session login (/login, default: admin / admin_license_2026!).

3. Integrated Fleet Licensing & WebUI Dashboard

  • WebUI URL: Accessible at http://<server_host>:8443/webui (or /).
  • Visual Seat Quota Meter: Real-time progress bar displaying active forwarders vs. allocated seat limit.
  • Licensing Actions:
    • 1-click Export .lic Request file download.
    • Interactive View & Copy CA CSR modal.
    • Drag-and-drop or paste Import License (.lic / ca.crt) modal.
    • Sync & Renew via mTLS button for immediate upstream synchronization and seat expansion.
  • Seat Management & Revocation Table:
    • Lists all connected Windows and Linux edge forwarders.
    • Revoke Seat: Instantly drops and blocks the client at the TLS socket handshake layer.
    • Reactivate Seat: Restores the client to active status if within seat quota limits.

4. Cloud-Side In-Flight Certificate Watchdog

  • Continuous Hub PKI Watchdog:
    • The server hub runs a continuous background watchdog coroutine (cert_validity_watchdog, running every 12 hours).
    • Automatically renews Subordinate CA certificates over mTLS with logar.eibl.tech and Server TLS certificates in-flight, dynamically reloading the active ssl.SSLContext.
  • Client Proactive Check & Reactive Self-Healing:
    • Proactive Renewal: Edge clients inspect client.crt before every run cycle. If expiring within 30 days, they contact /api/client/enroll to renew.
    • Reactive Self-Healing: If the server CA rotates or a handshake error occurs, edge clients catch the exception, re-bootstrap enrollment against the hub, and re-establish streaming cleanly.

Cloud-Side Temporal Persistence & Direct Hermes Reasoning

Incoming candidate logs are tracked in SQLite table active_issues:

  • Issue Fingerprint: Formatted as {site_name}:{server}:{signature}.
  • 12-Hour Evaluation Window:
    • The hub compares (now - last_seen).
    • If more than 12 hours have elapsed since the issue was last recorded, the previous window expires and the cycle resets to run_count = 1.
  • Direct Verification & Temporal Telemetry:
    • Ingested anomalies are immediately promoted to VERIFIED and made available to Hermes with complete temporal metadata (first seen, last seen, consecutive run counts).
    • Hermes performs all contextual analysis, pattern recognition, and alerting logic.

Agentic Hermes & Client Management API

The server hub exposes a management and reporting REST API (default port 8443):

POST /api/client/enroll

Client enrollment endpoint:

  • Request: 
    {
  "client_id": "web-worker-01.corp.internal",
  "hostname": "web-worker-01",
  "os": "linux",
  "enrollment_secret": "<SECRET>"
}
  • Response: 
    {
  "ca_cert": "-----BEGIN CERTIFICATE-----\n...",
  "client_cert": "-----BEGIN CERTIFICATE-----\n...",
  "client_key": "-----BEGIN RSA PRIVATE KEY-----\n..."
}

GET /api/clients

Returns seat quota status and registered client telemetry:

{
  "active_seats": 2,
  "max_seats": 10,
  "clients": [
    {
      "client_id": "web-worker-01.corp.internal",
      "hostname": "web-worker-01",
      "os_type": "linux",
      "cert_fingerprint": "7D5B660B...",
      "status": "active",
      "first_seen": "2026-09-04 18:00:00",
      "last_seen": "2026-09-04 19:15:00"
    }
  ]
}

GET /api/hermes/report

Returns all verified anomalies within the active 12-hour evaluation window with full temporal metadata for direct Hermes reasoning:

[
  {
    "fingerprint": "corp.internal:web-worker-01.corp.internal:PostgresPoolExhausted",
    "site": "corp.internal",
    "server": "web-worker-01.corp.internal",
    "signature": "PostgresPoolExhausted",
    "severity": "WARNING",
    "message": "Connection pool saturated (>95%)",
    "os_type": "linux",
    "first_seen": "2026-09-04T07:00:00+00:00",
    "last_seen": "2026-09-04T19:00:00+00:00",
    "consecutive_runs": 4,
    "evaluation_window": "12h",
    "verified": true,
    "status": "VERIFIED"
  }
]

GET /api/hermes/all

Diagnostic endpoint listing all tracked anomalies in SQLite.

GET /health

Returns hub health, encryption fingerprint, listener ports, and mTLS status.

Dynamic Machine & Domain Identification

Client configurations intentionally contain no hardcoded machine name or site name. Both forwarders dynamically identify their host and domain at runtime via get_machine_identifier():

  1. Fully Qualified Domain Name (FQDN): Checked via socket.getfqdn().
  2. OS-Specific Domain Discovery:
    • Windows: Checks Active Directory environment variables (USERDNSDOMAIN, USERDOMAIN).
    • Linux: Parses /etc/resolv.conf domain and search directives.
  3. Reverse DNS Lookup: Resolves canonical hostname via socket.gethostbyaddr.
  4. Fallback: Local hostname socket.gethostname().

The server automatically infers site attribution from domain qualifiers (e.g. node01.corp.internal \rightarrow site corp.internal).

Automated Service Installers (Linux & Windows)

LOGAR provides production-grade installation scripts and installer builders for automated service deployment:

1. Linux Service Installers

  • Client Installer (compilation/install_linux_client.sh):
    • Non-interactive script deploying to /opt/logar-client.
    • Automatically queries /etc/machine-id and enrolls with the hub via curl.
    • Installs and enables logar-client.service systemd unit.
    sudo ./compilation/install_linux_client.sh "http://hub.example.com:8443" "<ENROLLMENT_SECRET>"
  • Server Installer (compilation/install_linux_server.sh):
    • Deploys server to /opt/logar-server.
    • Configures logging and installs logar-server.service with LimitNOFILE=65536.
    sudo ./compilation/install_linux_server.sh

2. Windows Inno Setup Installers

  • Built using Inno Setup 6 and bundled with NSSM (compilation/nssm.exe):
    • Client Setup (compilation/installer_client.iss): Compiles LOGAR-Client-Setup.exe. Installs Win_Client.exe into {autopf}\LOGAR, sets up LOGAR_Client service via NSSM with stdout/stderr redirection to {commonappdata}\LOGAR\client.log, and starts the service. Clean uninstallation stops and removes the service.
    • Server Setup (compilation/installer_server.iss): Compiles LOGAR-Server-Setup.exe. Installs Server.exe and sets up LOGAR_Server Windows service via NSSM.

Repository & Shippables Structure

LOGAR/
├── .gitea/
│   └── workflows/
│       ├── ci.yml                     # CI pipeline: syntax, 25 unit tests & mTLS pipeline test
│       └── release.yml                # Consolidated release workflow (Linux binaries & Windows installers)
├── compilation/                       # Packaging, installers, and release automation
│   ├── install_linux_client.sh        # Automated Linux client systemd installation script
│   ├── install_linux_server.sh        # Automated Linux server systemd installation script
│   ├── installer_client.iss           # Inno Setup Windows Client installer script
│   ├── installer_server.iss           # Inno Setup Windows Server installer script
│   ├── nssm.exe                       # Official 64-bit NSSM service manager binary
│   ├── package_dist.py                # Standalone binary compiler & packager
│   ├── requirements.txt               # Unified project dependencies
│   └── upload_release.py              # Gitea REST API release asset publisher
├── src/                               # Core application source modules
│   ├── __init__.py
│   ├── Server.py                      # Central mTLS server, temporal engine, and Hermes REST API
│   ├── server_enrollment.py           # Dynamic PKI, Root CA, and client certificate generator
│   ├── Win_Client.py                  # Windows edge forwarder with auto-enrollment
│   └── Linux_Client.py                # Linux edge forwarder with auto-enrollment
├── tests/                             # Automated test suites
│   ├── test_linux_client.py           # Linux client unit tests & mTLS certificate validation
│   ├── test_pipeline.py               # End-to-end mTLS integration & direct anomaly verification test
│   ├── test_server.py                 # Server unit tests, PKI generation, and seat quota tests
│   └── test_win_client.py             # Windows client unit tests & mTLS certificate validation
├── .gitignore                         # Ignores venv, caches, DBs, and private keys
├── README.md                          # Architecture and usage documentation
├── RELEASE_NOTES.md                   # Release history and changelog
├── server_config.sample.json          # Reference server configuration
└── out/                               # Component guides and sample configs
    ├── linux_server/
    │   ├── README.md
    │   └── server_config.sample.json
    ├── win_server/
    │   ├── README.md
    │   └── server_config.sample.json
    ├── linux_client/
    │   ├── README.md
    │   └── client_config.sample.json
    └── win_client/
        ├── README.md
        └── client_config.sample.json

Getting Started & Deployment

1. Central Server Hub

  1. Install dependencies: 
    pip install -r compilation/requirements.txt
  2. Start the server (generates server_config.json, Root CA, and server certs on first run): 
    python src/Server.py
  3. Export a client configuration: 
    python src/Server.py --create-client-config --server-host <SERVER_IP> --server-port 9443 --client-out client_config.json

2. Windows Client Deployment

  1. Download LOGAR-Client-Setup.exe from releases and run it, or place Win_Client.exe and client_config.json in C:\Program Files\LOGAR.
  2. On first run with client_config.json, Win_Client.exe automatically enrolls with the hub, receives its mTLS certificates, and establishes secure streaming.

3. Linux Client Deployment

  1. Run the automated installer: 
    sudo ./compilation/install_linux_client.sh "http://<HUB_HOST>:8443" "<ENROLLMENT_SECRET>"
  2. The installer enrolls the client, configures /etc/logar/certs, and activates logar-client.service.

Running Tests

1. Component Unit Tests

python -m unittest discover -s tests -v

Runs all 25 unit tests, covering:

  • Dynamic Root CA generation and server TLS certificate issuance.
  • Dynamic client certificate issuance with CN and authority key extensions.
  • Enrollment secret authentication, seat limits, and certificate revocation.
  • Proactive certificate validity checks, Root CA auto-renewal, and in-flight server SSLContext reload.
  • Windows & Linux event log collection, deduplication, and mTLS certificate verification.
  • 12-hour evaluation window and direct Hermes anomaly verification.

2. End-to-End Pipeline Integration Test

# 1. Initialize test configuration
python src/Server.py --create-client-config --server-host 127.0.0.1 --server-port 9443 --client-out client_config.json

# 2. Launch server in background
python src/Server.py &

# 3. Run integration test
python tests/test_pipeline.py

Tests client enrollment, secret rejection, mTLS TLS 1.3 socket handshake, direct warning promotion, immediate error promotion, and Hermes report output.

Releases & Binary Distribution

Releases can be built and published through three complementary channels:

1. Tag Push Automation (Gitea Actions)

Pushing a release tag automatically triggers the build workflows:

git tag v2.0.1
git push origin v2.0.1

The consolidated workflow release.yml defines two parallel jobs:

  • release-linux (ubuntu-latest): Compiles standalone native ELF binaries (Linux_Client.bin, Server.bin) and checksums.
  • release-windows (windows-latest): Compiles Windows executables (Win_Client.exe, Server.exe), builds Inno Setup installers, and publishes checksums (requires self-hosted Windows Act Runner).

2. Manual Workflow Dispatch (Gitea UI)

Workflows can be manually triggered on demand from the Gitea web interface:

  1. Navigate to Actions \rightarrow Release Binaries & Installers (release.yml).
  2. Click Run workflow, set the release tag (defaults to v2.0.1), and run.

3. Native Local Windows Build & Direct Release Upload

For environments without a registered Windows CI runner, Windows executables can be built and published directly to Gitea releases:

# 1. Build Windows binaries locally
python compilation/package_dist.py --target windows

# 2. Upload assets and release notes directly to the Gitea release
python compilation/upload_release.py --tag v2.0.1 --token <GITEA_TOKEN> --skip-build
Reference in New Issue View Git Blame Copy Permalink