me0nline/LOGAR
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 4 Wiki Activity
Files
e634b060df11674d9ef00af9204f4a84337a10da
LOGAR/README.md
T
me0nline 78344e7677
Release Binaries / release (push) Failing after 7m17s
Details
Configure Gitea Actions release workflow to compile and publish standalone binaries via runner
2026-09-03 21:43:39 +02:00

12 KiB
Raw Blame History

LOGAR: Edge-Thin Log Analysis & Temporal Verification System

LOGAR is an enterprise log aggregation, verification, and anomaly detection architecture designed for heterogeneous server fleets (Windows & Linux). It combines lightweight zero-state edge forwarders with a centralized cloud hub that applies OpenPGP encryption, authenticated TCP streaming, temporal persistence tracking across 12-hour evaluation windows, and an automated 4-run rule to filter out transient infrastructure blips before reporting verified anomalies to Hermes.

Table of Contents

  1. Core Philosophy
  2. Architecture & Data Flow
  3. Security & Cryptographic Model
  4. Cloud-Side Temporal Persistence & 4-Run Rule
  5. Agentic Hermes Integration
  6. Dynamic Machine & Domain Identification
  7. Repository & Shippables Structure
  8. Getting Started & Installation
  9. Running Tests

Core Philosophy

1. Edge Thinness & Zero State

Site agents running on Windows and Linux act strictly as lightweight forwarders:

  • No Local Database: Clients maintain zero state and no local SQLite or cache files.
  • Source-Level Noise Stripping: Conversational, informational, and debugging log noise (INFO, DEBUG, audit entries) is dropped directly at the source.
  • End-to-End Encryption: Logs are encrypted using the server's OpenPGP public key before leaving the edge node.
  • Secure TCP Sockets: Ingestion occurs over low-overhead authenticated TCP sockets rather than bulky HTTP/HTTPS endpoints.

2. Cloud-Side Temporal Persistence

The central Python/TCP hub handles the heavy lifting:

  • State tracking is managed centrally in SQLite (logar_state.db).
  • Candidate issues are evaluated over a 12-hour temporal evaluation window.
  • An issue must persist across at least 4 consecutive runs / cycles to be confirmed as a genuine system anomaly. Transient blips and sporadic spikes are filtered out automatically.

3. Agentic Integration with Hermes

Instead of human engineers manually diving through noisy logs, Hermes ingests pre-filtered, 4-run validated anomalies directly from the cloud hub (GET /api/hermes/report), treating them as verified system artifacts to trigger precise team notifications.

Architecture & Data Flow

graph TB
    subgraph Edge Nodes [Zero-State Edge Forwarders]
        W[Win_Client.py / Win_Client.exe<br/>Windows Event Log Application]
        L[Linux_Client.py / Linux_Client.bin<br/>systemd journalctl -p warning]
    end

    subgraph Security Layer [Security & Framing]
        E[OpenPGP Payload Encryption<br/>Server Public Key & Fingerprint]
        S[Length-Prefixed Framing<br/>4-byte Big-Endian + Auth Envelope]
    end

    subgraph Cloud Hub [LOGAR Central Server Hub]
        TCP[Authenticated TCP Listener<br/>Port 9443]
        DEC[OpenPGP Decryption<br/>Server Private Key]
        DB[(SQLite Persistence<br/>active_issues & ingest_runs)]
        RULE{12h Window &<br/>4-Run Rule}
    end

    subgraph Agentic Reporting [Downstream Integration]
        API[FastAPI / Uvicorn Reporting<br/>Port 8443]
        HERMES[Hermes Agent<br/>GET /api/hermes/report]
    end

    W --> E
    L --> E
    E --> S
    S -->|TCP Stream| TCP
    TCP --> DEC
    DEC --> RULE
    RULE --> DB
    DB --> API
    API --> HERMES

Security & Cryptographic Model

Pure-Python OpenPGP (RFC 4880)

  • Zero OS Binary Dependency: Utilizes pgpy and cryptography in pure Python. No native GnuPG or gpg binary installation is required on the server, Windows nodes, or Linux nodes.
  • First-Run Automatic Key Generation: On the first launch, if server_config.json is missing, Server.py automatically generates:
    • An OpenPGP RSA 2048 keypair with encryption-only usage flags.
    • An armored private key (private_key) and public key (public_key).
    • A SHA-256 public encryption fingerprint (server_fingerprint).
    • A cryptographically random authentication secret token (auth_token).
  • Client Configuration Exporter: 
    python Server.py --create-client-config --server-host 127.0.0.1 --server-port 9443 --client-out client_config.json
    Produces an anonymous client config containing only the server socket coordinates, authentication token, and the encryption-only public key & fingerprint.
  • Socket Protocol Framing:
    • [4 bytes big-endian unsigned int] : Total envelope length.
    • [JSON Envelope] : 
      {
  "auth_token": "<SECRET_TOKEN>",
  "timestamp": "2026-09-03T...",
  "encrypted_payload": "-----BEGIN PGP MESSAGE-----\n..."
}
    • Unauthorized clients or invalid authentication tokens are rejected immediately.

Cloud-Side Temporal Persistence & 4-Run Rule

Incoming candidate logs are tracked in SQLite table active_issues:

  • Issue Fingerprint: Formatted as {site_name}:{server}:{signature}.
  • 12-Hour Evaluation Window:
    • When an issue is observed, the hub compares (now - last_seen).
    • If more than 12 hours have passed since the issue was last recorded, the previous window is expired and the cycle resets to run_count = 1 with status TRANSIENT.
  • 4-Run Rule:
    • For each distinct run batch, run_count increments.
    • Issues with run_count < 4 are marked as TRANSIENT and ignored by downstream reporting.
    • When run_count >= 4 within the active 12-hour window, the status transitions to VERIFIED.

Agentic Hermes Integration

The server hub serves a REST reporting API (default port 8443):

GET /api/hermes/report

Returns exclusively verified anomalies that have satisfied the 4-run rule within the active 12-hour evaluation window:

[
  {
    "fingerprint": "corp.internal:web-app-01.corp.internal:NginxWorkerCrash",
    "site": "corp.internal",
    "server": "web-app-01.corp.internal",
    "signature": "NginxWorkerCrash",
    "severity": "ERROR",
    "message": "Worker process 4120 terminated with signal 11",
    "os_type": "linux",
    "first_seen": "2026-09-03T09:00:00+00:00",
    "last_seen": "2026-09-03T21:00:00+00:00",
    "consecutive_runs": 4,
    "evaluation_window": "12h",
    "verified": true,
    "status": "VERIFIED"
  }
]

GET /api/hermes/all

Diagnostic endpoint listing all active issues (both TRANSIENT candidate blips and VERIFIED anomalies).

GET /health

Returns hub health, encryption fingerprint, and listener ports.

Dynamic Machine & Domain Identification

Client configurations intentionally contain no machine name or site name. Both forwarders dynamically identify their host and domain at runtime via get_machine_identifier():

  1. Fully Qualified Domain Name (FQDN): Checked via socket.getfqdn().
  2. OS-Specific Domain Discovery:
    • Windows: Checks Active Directory environment variable USERDNSDOMAIN / USERDOMAIN.
    • Linux: Parses /etc/resolv.conf domain and search directives.
  3. Reverse DNS Lookup: Resolves canonical hostname via socket.gethostbyaddr.
  4. Fallback: Local hostname socket.gethostname().

The server automatically infers site attribution from domain qualifiers (e.g. node01.corp.internal \rightarrow site corp.internal).

Repository & Shippables Structure

LOGAR/
├── .gitignore                      # Ignore venv, caches, DBs, and private keys
├── requirements.txt                # Unified dependencies
├── README.md                       # Comprehensive documentation
├── Server.py                       # Central TCP server and Hermes API
├── Win_Client.py                   # Windows edge forwarder
├── Linux_Client.py                 # Linux edge forwarder
├── test_pipeline.py                # End-to-end integration test
└── out/                            # Standalone shippable distributions
    ├── server/
    │   ├── Server.py               # Python source
    │   ├── server_config.sample.json
    │   ├── requirements.txt
    │   ├── README.md
    │   └── test/
    │       └── test_server.py      # Server unit tests
    ├── win_client/
    │   ├── Win_Client.py           # Python source
    │   ├── client_config.sample.json
    │   ├── requirements.txt
    │   ├── README.md
    │   └── test/
    │       └── test_win_client.py  # Windows client unit tests
    └── linux_client/
        ├── build_bin.sh            # PyInstaller native ELF compiler script
        ├── Linux_Client.py         # Python source
        ├── client_config.sample.json
        ├── requirements.txt
        ├── README.md
        └── test/
            └── test_linux_client.py# Linux client unit tests

Getting Started & Installation

1. Central Server Hub

  1. Install dependencies: 
    pip install -r requirements.txt
  2. Start the server (generates server_config.json and keypair on first run): 
    python Server.py
  3. Export a client configuration: 
    python Server.py --create-client-config --server-host <SERVER_IP> --server-port 9443 --client-out client_config.json

2. Windows Client Deployment

  1. Copy Win_Client.py (and requirements.txt) plus client_config.json to the target machine.
  2. Run manually or schedule via Task Scheduler (every 3 hours): 
    python Win_Client.py --hours 6

3. Linux Client Deployment

  1. Copy Linux_Client.py (and requirements.txt) plus client_config.json to /opt/logar/.
  2. (Optional) Run build_bin.sh to compile a standalone ELF binary if desired.
  3. Run via cron or systemd timer: 
    0 */3 * * * python3 /opt/logar/Linux_Client.py --hours 6

Running Tests

1. Component-Specific Unit Tests

Each component in out/ includes its own isolated test suite:

# Server tests (config generation, SQLite persistence, 4-run rule)
python out/server/test/test_server.py

# Windows client tests (config anonymity, machine ID, OpenPGP encryption)
python out/win_client/test/test_win_client.py

# Linux client tests (config anonymity, journalctl priority filter, OpenPGP)
python out/linux_client/test/test_linux_client.py

2. End-to-End Pipeline Integration Test

Start the server in one shell and run the pipeline test:

python test_pipeline.py

This tests invalid token rejection, encrypted socket streaming, database persistence, status promotion upon the 4th run, and the Hermes API output.

Automated Releases via Gitea Actions

Releases are automated via .gitea/workflows/release.yml using your Gitea action runner:

Publishing a Release

Whenever you want to release a new version with compiled standalone binaries:

git tag v1.0.0
git push origin v1.0.0

What Gitea Actions Does Automatically:

  1. Gitea runner executes the workflow on tag push.
  2. Runs package_dist.py to compile native standalone binaries:
    • Linux_Client.bin (standalone binary)
    • Server.bin (standalone server binary)
    • Win_Client.pyz (standalone executable zipapp)
    • SHA256SUMS.txt (checksums)
  3. Publishes the Gitea release using gitea-release-action and attaches the compiled binary assets.

(Note: You can also use upload_release.py from your Windows machine to upload Windows .exe binaries directly if desired).

Reference in New Issue View Git Blame Copy Permalink