Skip to content

Application for handling and Inspecting logs from various microservices and endpoints.

Notifications You must be signed in to change notification settings

anmolchhabra21/log-capture-analysis

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

15 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Log Capture & Inspection

Scalabe and Multi-threaded Go Backend for ingesting bulk logs, logs from different services, and also via a direct HTTP log publisher endpoint, all with advanced analytics filtering and a clean UI Interface.

Table of Contents
  1. About The Project
  2. Getting Started

About The Project

This solution provides a Scalable and Efficient implementation for log ingestion and analysis, leveraging the Light-weight multithreading power of Golang and search-indexing capability of ElasticSearch for effective log ingestion, storage, and analysis. It is capable of accepting logs from two distinct sources: a Kafka queue(that moves down to taking logs from various different Services) or direct HTTP API requests. It also offers a web-based user interface for log analysis, supporting various filters and search queries including regex. image Architecture Diagram
Project Video

Features

  • Kafka queue for streamlined Log processing
  • Regular expression search on all fields
  • Text-search across all fields
  • Efficient search queries leveraging Elastic DB
  • Scalable & Efficient Storage using sharding provided by Elastic DB
  • Search in given time range
  • HTTP endpoint for posting logs
  • Ingestion Buffer & Batch processing
  • Filters on specific fields
  • Combination of all filters & search

Built With

  • Golang
  • Python
  • ElasticSearch
  • Kafka
  • Gin
  • React

Getting Started

Prerequisites

  • Python
  • Node JS & npm
  • Golang
  • Docker (Other services can be installed directly or with Docker)

Installation & usage

  • Clone the repo

  • Start the dependencies, you can use the docker-compose to setup:

    docker compose up -d
  • You can check if all the services (ElasticSearch, Zookeeper, Kafka) are up and running by

    docker ps
  • Setup Log ingestion server

    1. Go to log-server directory

        cd log-server/
    2. Install golang dependencies

      go mod download
    3. Install Python dependencies

      pip install -r requirements.txt
    4. Start the ingestion server

      cd server
      GIN_MODE=release go run .

      The server should now be up and running on http://localhost:3000.

    5. This script is like a stress test, for a simultaneous bulk log insertion, the current value is 500, but can be modified as per requirements. Configure LOGS_LENGTH in log-server/tests/performance_test.py.

      python tests/performance_test.py
  • Setup UI

    1. Go to frontend directory
        cd frontend/
    2. Install NPM dependencies
      npm install
    3. Start the React app
      npm start
    4. Frontend should be up at - http://localhost:3006
  • Simulate Log Publishers (logs via http endpoint)

    1. Assuming all the services are up and running.

    2. Start publisher script. Go to log-producers

      cd log-server/log-producers

      Start a producer to simulate service using --topic option in different shells. Configured topics: auth,database,email,payment,server,services, for example

      python producer.py --topic payment
      python producer.py --topic auth