NeuroVision: A Self-Adapting Event-Driven Visual Cortex

NeuroVision is a research-grade proof of concept for neuromorphic-inspired visual intelligence. It uses a conventional webcam, but behaves like a simplified visual cortex: it reacts to visual change, converts events into spikes, learns important regions, habituates to repetitive stimuli, detects novelty, switches cognitive states, and runs YOLOv8 only when expensive AI inference is justified.

This is designed as a hackathon-ready AI/edge-computing demo, not a simple motion detector.

System Architecture

Webcam frames
  -> event_encoder.py       sparse visual change events
  -> spike_generator.py     spike events and spike raster map
  -> stdp_learning.py       learned attention memory
  -> habituation.py         familiar-stimulus suppression
  -> novelty_detector.py    unusual pattern scoring
  -> attention_engine.py    salience fusion
  -> brain_state.py         SLEEP / LEARNING / ATTENTION / ALERT
  -> ai_trigger.py          gated YOLOv8 inference
  -> dashboard.py           real-time research dashboard
  -> metrics.py             workload and energy analysis

Neuromorphic Inspiration

Traditional AI camera systems process every frame continuously. NeuroVision takes inspiration from biological vision and event cameras:

• The retina prioritizes change.
• Neurons communicate with sparse spikes.
• Repeated sensory patterns can become less attention-worthy.
• Novel events receive priority.
• Expensive cognition is recruited only when needed.

The result is an event-driven pipeline that keeps monitoring cheap and responsive while dramatically reducing expensive inference calls.

Event-Based Sensing

event_encoder.py compares consecutive grayscale frames. Pixels whose intensity difference exceeds a configurable threshold become events. Inactive frames are ignored by downstream event processing.

Important parameter:

--threshold 32

Raise the threshold for fewer events. Lower it for more sensitivity.

Spike-Based Computation

spike_generator.py converts event pixels into spike objects with:

• position
• polarity
• strength
• timestamp

Spike density corresponds to the amount of motion or visual change. The dashboard shows a live spike raster where positive and negative changes are colored separately.

STDP-Inspired Learning

stdp_learning.py implements a simplified spatial form of spike-timing-dependent plasticity. Regions with repeated spikes strengthen an attention memory map. Regions without activity decay.

Example:

• A doorway with repeated movement becomes important.
• An unused wall fades toward low importance.

The learned map is visualized as a live heatmap.

Habituation

habituation.py models biological habituation. Repetitive non-informative stimuli become familiar and are gradually suppressed by the attention engine.

Example:

• A fan initially causes spike activity.
• After repeated exposure, its habituation score rises.
• Its effective salience decreases.

This helps NeuroVision avoid wasting AI inference on stale background activity.

Novelty Detection

novelty_detector.py maintains a historical activity pattern. Current activity that differs from history receives a novelty score. Novel events immediately increase attention and can push the brain state into ALERT.

Brain-State Logic

brain_state.py exposes four high-level states:

• SLEEP: low activity or habituated stimulus
• LEARNING: new pattern being incorporated
• ATTENTION: learned important region is active
• ALERT: unexpected high-priority activity

Transitions depend on spike density, novelty score, attention score, and habituation score.

YOLOv8 AI Trigger

ai_trigger.py integrates YOLOv8 through ultralytics, but does not run it continuously.

YOLOv8 runs only when at least one gated condition is met:

• spike density exceeds threshold
• novelty score exceeds threshold
• important learned region activates
• salience score exceeds threshold

This models an edge AI system where cheap neuromorphic sensing gates expensive object detection.

If ultralytics or the YOLO weights are unavailable, the dashboard still runs and shows the gated status. Install ultralytics for real YOLOv8 detections.

Energy-Saving Analysis

metrics.py estimates:

• traditional processing cost
• neuromorphic processing cost
• frames received
• frames ignored
• events generated
• spikes generated
• YOLO activations
• skipped YOLO executions
• computation reduction
• estimated energy savings

The goal is to demonstrate roughly 80-95% reduction in expensive AI processing during mostly static or repetitive scenes.

Installation

Python 3.10 or newer is recommended. A virtual environment is strongly recommended.

python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt

Equivalent manual install:

pip install opencv-python numpy matplotlib ultralytics

Running

python main.py

Press q in the OpenCV dashboard to quit.

For the futuristic PyQt6 AI operating-system dashboard:

python main_qt.py

The PyQt6 screen is designed as a single-window AI Brain OS for 1920x1080 displays: camera, neural cortex, learned attention, event retina, novelty, familiarity, AI reasoning, and energy savings are all visible at once.

To run the PyQt6 dashboard without loading YOLOv8:

python main_qt.py --disable-yolo

Useful options:

python main.py --threshold 28 --downsample 2 --cell-size 24
python main.py --disable-yolo
python main.py --model yolov8n.pt

On exit, NeuroVision saves:

neurovision_performance_summary.png

Dashboard Panels

The real-time dashboard shows:

• Original visual stream
• Sparse event map
• Spike raster activity
• STDP attention memory
• Novelty heatmap
• Habituation heatmap
• Current brain state
• YOLOv8 gated detections
• Metrics and energy savings

Only the first panel is the raw camera feed. The other panels are internal representations computed from the same feed, like different layers of a simplified visual cortex.

For a plain-English explanation of what each panel means, see:

REPORT.md

Future Improvements

• Replace webcam frame differencing with a real event camera.
• Add spiking neural network inference for event classification.
• Persist learned attention maps across sessions.
• Add region-level semantic memory.
• Use optical-flow-aware novelty instead of grid activity only.
• Deploy the gate on edge hardware such as Jetson, Raspberry Pi AI Kit, or Intel NPU devices.
• Add adaptive thresholds based on ambient lighting and sensor noise.