README.md

NetFlow Analysis Tool

A web-based network flow analysis tool for visualizing University of Oregon network traffic patterns. Built with SvelteKit frontend and SQLite database backend for efficient querying of netflow data.

Features

Interactive Data Visualization: Chart.js-powered line graphs with logarithmic scaling
Flexible Time Grouping: Aggregate data by month, date, or hour
Multiple Metrics: Track 15 different network statistics including flows, packets, and bytes by protocol
Router Filtering: Support for multiple routers
Date Range Selection: Query specific time periods with intuitive controls
Dynamic X-Axis: Automatically formats labels based on selected grouping (YYYY-MM, MM/DD/YY, DD:HH)

Architecture

Database

Data Access

zsh

1	> cd netflow-db
2	> sqlite3 flowStats.db

Now you're connected to the database. Try some of the following to explore the shape of the database

zsh

1	sqlite> .tables
2	sqlite> .schema ip_stats
3	sqlite> .schema netflow_stats

Some example queries:

sql

1	-- Netflow dashboard: daily flow/packet summary for selected routers and time window
2	SELECT
3	DATE(time_start, 'unixepoch') AS day,
4	router,
5	SUM(flows_total) AS flows,
6	SUM(packets_total) AS packets,
7	SUM(bytes_total) AS bytes
8	FROM netflow_stats
9	WHERE router IN ('router1', 'router2')
10	AND time_start BETWEEN strftime('%s', '2025-01-01') AND strftime('%s', '2025-01-08')
11	GROUP BY day, router
12	ORDER BY day ASC, router ASC;
13
14	-- Drilldown: 30-minute granularity for flows vs protocols (mirrors chart drill interaction)
15	SELECT
16	strftime('%Y-%m-%d %H:%M', time_start, 'unixepoch') AS bucket,
17	SUM(flows_tcp) AS flows_tcp,
18	SUM(flows_udp) AS flows_udp,
19	SUM(flows_icmp) AS flows_icmp
20	FROM netflow_stats
21	WHERE router = 'router1'
22	AND time_start BETWEEN strftime('%s', '2025-01-03') AND strftime('%s', '2025-01-04')
23	GROUP BY bucket
24	ORDER BY bucket;
25
26	-- IP statistics chart: per-router unique source/destination IP counts
27	SELECT
28	router,
29	bucket_start,
30	sa_ipv4_count,
31	da_ipv4_count,
32	sa_ipv6_count,
33	da_ipv6_count
34	FROM ip_stats
35	WHERE granularity = '1h'
36	AND router IN ('router1', 'router2')
37	AND bucket_start BETWEEN strftime('%s', '2025-01-01') AND strftime('%s', '2025-01-02')
38	ORDER BY router, bucket_start;

Data Processing Pipeline

STATS

Raw netflow files from a configurable location
Python script (netflow-db/db.py) processes files using nfdump and populates SQLite database
SvelteKit API queries database with better-sqlite3 for fast aggregation
Frontend renders interactive charts with real-time filtering

MAAD

Raw netflow files are processed on demand for source/destination IP addresses
IPv4 addresses are piped into MAAD CLI
Charts are rendered on the frontend

Database Schema

SQLite database (flowStats.db) with netflow_stats table containing:

Router identification and timestamps
Flow counts by protocol (TCP, UDP, ICMP, Other)
Packet counts by protocol
Byte counts by protocol
Sequence failure tracking
Indexed for efficient time-range queries

Getting Started

Prerequisites

Node.js 20+
Python 3.x with nfdump available
SSH access to pinot, to use ONRG database

Installation

Clone the repository
bash
1 git clone https://gitlab.com/onrg/netflow-analysis
2 cd netflow-analysis
Install frontend dependencies
bash
1 cd netflow-webapp
2 npm install
Populate database (if needed)
bash
1 cd ../netflow-db
2 python db.py
Start development server
bash
1 cd ../netflow-webapp
2 npm run dev
Access via SSH tunnel
bash
1 ssh -L 5173:localhost:5173 user@pinot
Then open http://localhost:5173 in your browser

Usage

Web Interface

Date Range: Select start and end dates for analysis
Time Options: Choose specific time or analyze full day
Router Selection: Filter by different routers
Metrics: Select from 15 available network statistics
Grouping: Aggregate by month, date, or hour for different perspectives

Available Metrics

Flows (Total, TCP, UDP, ICMP, Other)
Packets (Total, TCP, UDP, ICMP, Other)
Bytes (Total, TCP, UDP, ICMP, Other)

Development

Frontend Commands

bash

1	cd netflow-webapp/
2
3	npm run dev # Start development server
4	npm run build # Build for production
5	npm run preview # Preview production build
6	npm run check # TypeScript type checking
7	npm run format # Format with Prettier
8	npm run lint # Run ESLint and Prettier checks

Database Management