NetFlow Analysis Tool

A web-based network flow analysis tool for visualizing University of Oregon network traffic patterns. Built with SvelteKit frontend and SQLite database backend for efficient querying of netflow data.

Features

• Interactive Data Visualization: Chart.js-powered line graphs with logarithmic scaling
• Flexible Time Grouping: Aggregate data by month, date, or hour
• Multiple Metrics: Track 15 different network statistics including flows, packets, and bytes by protocol
• Router Filtering: Support for multiple routers
• Date Range Selection: Query specific time periods with intuitive controls
• Dynamic X-Axis: Automatically formats labels based on selected grouping (YYYY-MM, MM/DD/YY, DD:HH)

Architecture

Data Processing Pipeline

STATS

1. Raw netflow files from a configurable location
2. Python script (netflow-db/db.py) processes files using nfdump and populates SQLite database
3. SvelteKit API queries database with better-sqlite3 for fast aggregation
4. Frontend renders interactive charts with real-time filtering

MAAD

1. Raw netflow files are processed on demand for source/destination IP addresses
2. IPv4 addresses are piped into MAAD CLI
3. Charts are rendered on the frontend

Database Schema

SQLite database (flowStats.db) with netflow_stats table containing:

• Router identification and timestamps
• Flow counts by protocol (TCP, UDP, ICMP, Other)
• Packet counts by protocol
• Byte counts by protocol
• Sequence failure tracking
• Indexed for efficient time-range queries

Getting Started

Prerequisites

• Node.js 20+
• Python 3.x with nfdump available
• SSH access to pinot, to use ONRG database

Installation

1. Clone the repository

   bash

   1	git clone https://gitlab.com/onrg/netflow-analysis
   2	cd netflow-analysis

2. Install frontend dependencies

   bash

   1	cd netflow-webapp
   2	npm install

3. Populate database (if needed)

   bash

   1	cd ../netflow-db
   2	python db.py

4. Start development server

   bash

   1	cd ../netflow-webapp
   2	npm run dev

5. Access via SSH tunnel

   bash

   1	ssh -L 5173:localhost:5173 user@pinot

   Then open http://localhost:5173 in your browser

Usage

Web Interface

• Date Range: Select start and end dates for analysis
• Time Options: Choose specific time or analyze full day
• Router Selection: Filter by different routers
• Metrics: Select from 15 available network statistics
• Grouping: Aggregate by month, date, or hour for different perspectives

Available Metrics

• Flows (Total, TCP, UDP, ICMP, Other)
• Packets (Total, TCP, UDP, ICMP, Other)
• Bytes (Total, TCP, UDP, ICMP, Other)

Development

Frontend Commands

Database Management

Technical Stack

• Frontend: SvelteKit 2.x, TypeScript, TailwindCSS 4.x
• Charts: Chart.js with logarithmic scaling
• Database: SQLite with better-sqlite3 Node.js integration
• UI Components: Shadcn-Svelte with custom calendar components
• Data Processing: Python with nfdump command-line tool

Access Requirements

This application requires SSH tunnel access to the UO research infrastructure where netflow data is stored. It's designed specifically for research environment use with proper authentication and network access.

Known Issues/Bugs