DST-FallNet
Real-Time Multimodal Fall Detection via
Audio-Visual Fusion and DST on Edge Devices

Main features

• Visual fall detection — MoveNet pose estimation and CNN action classification
• Audio anomaly detection — MFCC features with LSTM
• Dempster-Shafer Theory (DST) decision-level fusion
• GPIO output for LED / buzzer alerts
• CLI dashboard for real-time system status over SSH

Why multimodal, and why on the edge?

With an aging population, falls are a leading cause of injury and death among the elderly. Most detection systems rely on a single modality — a camera or a microphone — and each has blind spots.

Limits of single-modality systems

• Accuracy ceiling: vision suffers under poor lighting and occlusion; audio is fooled by background noise — both cause false alarms and misses.
• Heavy compute: most deep models assume a powerful PC or cloud, unsuitable for constrained hardware.
• Privacy & latency: uploading frames to the cloud raises privacy concerns and cannot give real-time alerts.

Hardware used

• Jetson Nano 2GB
• USB camera
• USB microphone
• LED and buzzer (GPIO control)

Dempster-Shafer Theory over naive averaging

Simple score averaging breaks down under uncertainty and conflict: the camera may be unsure in dim light, the microphone may misread noise. Averaging those scores can actively hurt accuracy.

Four cooperating threads

The system runs as four concurrent threads so sensing, fusion, and hardware control never block each other — essential for real-time behavior on a 2GB board.

1. Visual pose inference: capture frames, preprocess, run MoveNet ONNX → 17 keypoints + confidence; compute body tilt to flag suspicious postures.
2. Audio recognition: record 3-second clips, convert to MFCC, run the LSTM ONNX → a fall-sound score (0–1).
3. Fusion & CNN verification: fuse vision + audio scores with DST; if above threshold, a 5-class CNN confirms the action and decides whether to fire GPIO.
4. GPIO & button monitoring: drive LED / buzzer outputs, watch the reset button, clear fall events, and reset system state.

MoveNet pose + custom CNN verification

MoveNet pose estimation

MoveNet gives fast, accurate 17-keypoint pose estimation, ideal for embedded devices. Keypoints and confidence feed a body-tilt and center-of-gravity estimate as the first fall cue. The model is quantized to INT8 and converted from TFLite to ONNX, shrinking size and compute with minimal accuracy loss for stable real-time use on Jetson Nano.

Custom 5-class CNN verification

When the DST fusion score crosses the threshold, a custom CNN classifies the action into standing, sitting, lying, bending, or crawling with a confidence score. This second stage sharply cuts false positives.

• Dataset: public Fall Detection Dataset — 15,402 labeled images.
• Split: 80% train / 20% validation, with live accuracy & loss monitoring.
• Input: 192×192×4; inference ~20–40 ms; accuracy 87.4%.

MFCC features + LSTM sound classifier

The microphone continuously records, segmenting into 3-second clips. After noise reduction and volume normalization, each clip becomes a 40-dimension MFCC feature map (3 s, 22050 Hz) fed to the LSTM.

Late fusion at the decision level

Each modality produces its own decision; their scores become belief assignments and DST fuses them into a unified confidence plus a conflict coefficient K. This late (decision-level) fusion lets each model play to its strengths and stays robust when one modality is noisy.

• Threshold & trigger: once fused confidence passes the threshold, the CNN confirms; a high-risk action (lying / crawling) above the alert threshold fires the buzzer / LED and logs the event.
• Conflict handling: when modalities disagree, K reallocates trust to the more reliable source, preventing single-sensor failure.
• Configurable: choose vision-only, audio-only, or both, and tune thresholds per environment (quiet home vs. noisy public space).

Lightweight monitoring for embedded use

A text-based dashboard shows live status in organized tables with color coding (red = alert, green = normal). It runs over SSH, ideal for headless edge devices.

• Runtime status: memory, CPU, latency
• MoveNet keypoints & confidence
• Audio MFCC features & LSTM scores
• DST fusion score & conflict coefficient
• CNN multi-class results & alert state
• Event log with time, score, and action type

Model performance summary

The DST fusion step adds only microseconds, so multimodal robustness comes essentially for free relative to the model inference cost.

Source code & how to run

The full embedded system lives in the DST-FallNet repository on GitHub. Separate repos hold the CNN and LSTM training code; exported ONNX models are loaded at runtime.

Project directory structure

DST-FallNet/
├── src/                # Main source code
├── models/             # ONNX model files
├── assets/             # Documentation images
├── abnormal_images/    # Runtime abnormal captures
├── requirements.txt    # Python dependencies
└── README.md

Where it goes next

• Convert ONNX models to TensorRT to further speed up edge inference.
• Move from public datasets to custom-collected data for higher real-world relevance.
• Add a notification system for instant caregiver alerts on a detected fall.
• Benchmark single-model vs. multimodal accuracy in real-world validation.