Ngoma Platform
User Manual
1. What is Ngoma?
Ngoma is the intelligent control and analytics interface for the Ngoma home intelligence robot platform. It runs on Android tablets, iPhones, iPads, web browsers, Windows and macOS desktops, and as the embedded touch interface on the robot's own display (RPi CM4 + DSI screen).
From a single application, you can:
- Monitor home occupancy, activity, and ambient conditions via the AI Perception System
- Stream home entertainment, ambient audio, and video content from SoundCentral
- Control every Zigbee and Z-Wave smart environment device via Home Assistant
- Issue voice commands via tap-to-talk or always-on ambient microphones
- Run autonomous security patrols and manage facial recognition of enrolled household members
- Track live health and activity data of household members wearing a KnightGuard wearable
2. Getting Started
First Launch
On first launch Ngoma checks for Home Assistant credentials. If none are present you will see the Configuration Required screen.
Option A — Enter credentials now:
Open Settings
Tap the Open Settings button on the configuration screen.
Enter your HA connection details
Under Home Assistant Connection, enter your HA base URL (e.g.,
http://192.168.1.100:8123) and your long-lived access token.Save and connect
Tap Save. The app reconnects and loads the main dashboard.
Option B — Continue in limited mode:
Tap Continue without configuration. Smart environment features are unavailable, but entertainment and wellness features work if their credentials were supplied at deployment.
Signing in to SoundCentral
Open Settings
Tap the ⚙ icon in the top-right corner of the app bar.
Sign in
Tap SoundCentral Account → Sign In.
Authorise your device
Ngoma displays a device code and QR code. On any authorised device, open your SoundCentral app and enter the code — or scan the QR code.
Complete sign-in
Complete sign-in in your SoundCentral app. Ngoma detects success automatically. Your account name and subscription status appear in Settings. The session persists across restarts.
3. Navigation Overview
Ngoma has four main sections accessed from the bottom navigation bar, plus a Settings screen in the top-right corner:
| Icon | Section | Purpose |
|---|---|---|
| ♪ | Entertainment & Comms | Media streaming, announcements, voice control, display management |
| 🏠 | Smart Environment | Device control, scenes, thermostat, blinds, alert feed |
| 🛡 | Security | Sentry mode, facial recognition, autonomous navigation |
| ♡ | Wellness | Live health data from KnightGuard wearables |
The Settings screen is accessed via the ⚙ icon in the top-right of the app bar — it is not a bottom-nav tab.
4. Entertainment & Communications
The Entertainment screen is Ngoma's media and communications hub. Live perception status cards (people count, detected activity, ambient sound) appear at the top. Each media type follows as a scrollable section below.
4.1 Audio Playback
The Now Playing card appears when a track is loaded, showing cover art, track title and artist, a seek bar, transport controls (Previous ⏮ / Play⏸Pause▶ / Next ⏭), and a volume slider.
To play music:
Browse the music library
Scroll to the Music Library section.
Select a track
Tap any track row — it loads into the player and begins playing immediately.
Control playback
Use the Now Playing card to pause, skip, seek, or adjust volume.
4.2 Video
The Featured Video section shows the first video from SoundCentral's content feed. Tap the thumbnail to start playback. The player provides full transport controls (play/pause, seek, volume, full-screen). The video widget is lazy-mounted — it does not consume resources until you tap play.
4.3 Live TV
Live channels are displayed in a three-column grid, each tile showing the channel name and a LIVE badge.
Select a channel
Tap any channel tile — the inline HLS player loads below the grid.
Dismiss
Tap the tile again to dismiss the player.
If a stream fails to load, the player retries automatically up to three times. After three failures, an error banner appears — try a different channel and retry the original later.
4.4 Live Radio
All available stations are listed with station name and a pulsing LIVE indicator.
Select a station
Tap any station row — it loads into the embedded radio player.
Monitor playback
The station name and a LIVE badge appear in the player. There is no seek bar for live streams.
Stop or switch
Tap Stop to end playback, or tap a different station to switch instantly.
4.5 Mood Soundscapes
Three AI-driven audio presets automatically select the appropriate live radio station based on detected workspace activity.
| Preset | Best for |
|---|---|
| 🎵 Relaxing | Quiet environments, low-occupancy periods, end-of-day wind-down |
| 💼 Working | Focus sessions, individual work, high-productivity periods |
| 🕺 Active | Team gatherings, training sessions, high-energy environments |
Tap a preset chip to activate it. The matching station loads automatically into the player. The active chip is highlighted.
4.6 Audio Output Routing
The Cast To section controls where audio plays. Tap a chip to select the output.
| Option | Behaviour |
|---|---|
| Speaker | Default — plays through the device's own speakers. |
| Bluetooth | Routes to a connected Bluetooth speaker via A2DP. Connect the speaker in system Bluetooth settings first. |
| AirPlay | Enables AirPlay 2. Select the output device from iOS/macOS Control Centre. |
| Chromecast | Requires Google Cast SDK — shows an informational message if not bundled. |
| DLNA | Requires DLNA discovery library — shows an informational message if not bundled. |
4.7 Display & Projector Control
| Control | Description |
|---|---|
| Auto-Focus | Toggle — enables automatic focus adjustment. |
| Auto-Keystone | Toggle — enables automatic keystone correction. |
| H. Keystone | Slider — horizontal keystone adjustment (±40). |
| V. Keystone | Slider — vertical keystone adjustment (±40). |
| Projection Angle | Slider — projection angle (0°–45°). |
Tap Apply & Project to send the current settings to the backend. Changes take effect immediately.
4.8 Voice Control
The Voice Control card supports two modes:
Tap-to-talk:
Activate listening
Tap the microphone button — it turns red and a pulsing animation confirms the app is listening.
Speak your command
Speak clearly at a normal pace. No need to slow down.
Command executes
The recognised phrase appears in the card and the command executes automatically. Tap the microphone again to stop listening early.
Ambient always-on (Jetson Whisper):
Start ambient mode
Tap the ear icon to start ambient listening mode. The card shows "Ambient active — Jetson Whisper" and a waveform indicator.
Hands-free operation
Ngoma continuously processes audio using the Jetson on-device Whisper model. Recognised phrases appear in real time as commands execute.
Stop ambient mode
Tap the ear icon again to stop. Tap-to-talk continues to work independently regardless of ambient mode status.
5. Smart Environment
The Smart Environment screen provides a real-time dashboard of all connected Home Assistant devices, environmental controls, and workspace scenes.
5.1 Device Dashboard
The Connected Devices card shows the total count of Zigbee and Z-Wave devices. Tap View All to open the full Devices screen.
- Zigbee and Z-Wave tabs separate devices by protocol.
- Each row shows the device icon, friendly name, entity ID, and current state.
- Context-aware inline controls: toggle switch for lights, chip for scenes, text for sensors.
- Tap the Refresh button (bottom-right) to reload all device states.
5.2 Thermostat & Blinds
Thermostat: Shows the current temperature from the configured thermostat entity. Updates automatically when Home Assistant reports a change.
Blinds: Shows the current position (0% = fully closed, 100% = fully open). Drag the slider to set a new position — the command sends to HA immediately.
5.3 Workspace Scenes
| Scene | Description |
|---|---|
| 🌙 Evening Mode | Dim, warm lighting — end of business day or wind-down periods |
| ☀️ Morning Routine | Bright, neutral lighting — opening hours and morning productivity |
| 🎬 Presentation Mode | Low ambient lighting optimised for screen-based presentations |
| 💼 Focus Mode | Bright, focused lighting — intensive work sessions |
Tap any scene chip to activate it. A confirmation message appears. To use custom scenes, update the entity IDs in Settings → Entity IDs.
5.4 Alert Feed
The bottom of the Smart Environment screen shows recent notifications from the HA logbook, colour-coded by severity:
| Colour | Severity |
|---|---|
| ■ Red | Critical — requires immediate attention |
| ■ Amber | Warning — review when convenient |
| ■ Blue | Info — informational only |
The feed refreshes at the interval configured in Settings → Alert Poll Interval (default: 30 seconds).
6. Security
The Security screen provides full visibility and control over patrol, facial recognition, and autonomous navigation operations.
6.1 Sensor Status
The Sensor Status card shows Depth Camera, LiDAR, and Microphone in green (active) or grey (inactive). A Processing Load bar shows the current AI pipeline load percentage.
6.2 Facial Recognition
The Facial Recognition card shows known personnel visible in the camera frame (name, role, confidence score) and an unknown-faces banner when unrecognised individuals are detected.
Enrolling a new person or visitor (admin only):
Open Manage Personnel
Go to Settings → Manage Personnel → Add Person.
Enter details
Enter the person's name and relation (e.g., Employee, Contractor, Guest).
Enrol
Tap Enrol. The Jetson runs a liveness check and captures the face using ArcFace deep learning.
Confirm
On success the person appears in the list immediately, stored in both the local database and on the Jetson edge device.
Removing a person:
Open Manage Personnel
Go to Settings → Manage Personnel.
Delete
Tap the person's row and select Delete. They are removed from both the local database and the Jetson profile store.
6.3 Autonomous Navigation
| Control | Description |
|---|---|
| Obstacle Avoidance toggle | Enable/disable the Nav2 obstacle avoidance layer. Default: on. Recommended to leave enabled at all times. |
| Waypoint chips | Up to five named destinations configured at deployment: e.g., Entrance, Floor 1, Floor 2, Server Room, Reception. |
Enable obstacle avoidance
Confirm Obstacle Avoidance is toggled on (recommended).
Select a waypoint
Tap a waypoint chip — it highlights and an en-route indicator appears.
Arrival confirmation
Arrival is confirmed by a Jetson status update (within 120 seconds). If navigation fails, an error message describes the reason (path blocked, motor fault, timeout).
6.4 Sentry Mode
Sentry Mode activates the patrol behaviour in Home Assistant, putting the robot into autonomous patrol mode along your configured route.
Enable Sentry Mode
Toggle the Sentry Mode switch to ON. The patrol scene is sent to Home Assistant.
Review patrol schedule
The patrol schedule row shows the configured patrol times for your deployment.
Start a manual patrol
To begin immediately, tap Start Manual Patrol. Waypoint progress updates in real time (e.g., "Waypoint 2 of 5").
Deactivate
Toggle the Sentry Mode switch to OFF to cancel the patrol.
7. Wellness
The Wellness screen shows live health and activity data from a KnightGuard Wear OS wearable, polled from the Knights Health API every 30 seconds.
KNIGHTS_BASE_URL and KNIGHTS_API_KEY credentials, and (3) ensure the wearer's KnightGuard Wear OS app is installed and authenticated.7.1 Live Vitals
Heart Rate (BPM)
| Zone | Range | Colour |
|---|---|---|
| Low | < 50 BPM | Blue |
| Normal | 50–100 BPM | Green |
| Elevated | 100–120 BPM | Amber |
| High | 120–150 BPM | Orange |
| Critical | > 150 BPM | Red |
Blood Oxygen — SpO₂ (%)
| Zone | Range | Colour |
|---|---|---|
| Normal | ≥ 95% | Green |
| Low | 90–94% | Amber |
| Critical | < 90% | Red |
Body Temperature (°C)
| Zone | Range | Colour |
|---|---|---|
| Hypothermia | < 35.0°C | Blue |
| Normal | 35.0–37.5°C | Green |
| Fever | 37.5–39.0°C | Amber |
| High Fever | > 39.0°C | Red |
7.2 Activity Today
| Metric | Description | Daily Goal |
|---|---|---|
| Steps | Total steps taken today | 10,000 steps |
| Calories | Kilocalories burned | — |
| Distance | Metres travelled | — |
| Speed | Current speed in m/s | — |
7.3 Movement Quality
- Motion Intensity — computed from accelerometer magnitude √(x²+y²+z²). Higher values indicate more vigorous movement.
- Balance / Stability — computed from gyroscope magnitude. Lower values indicate steadier, more controlled movement.
7.4 24-Hour Activity Chart
Plots hourly activity for the past 24 hours. Colour-coded bars:
| Colour | Activity Level |
|---|---|
| No bar | None — no movement detected |
| Blue | Low activity |
| Amber | Medium activity |
| Red | High activity |
8. Settings
Access Settings via the ⚙ icon in the top-right of the app bar.
8.1 SoundCentral Account
| Action | How |
|---|---|
| Sign in | Tap Sign In → scan QR code or enter device code in your SoundCentral app |
| View account | Name, email, and subscription status are shown when signed in |
| Sign out | Tap Sign Out → confirm → session cleared |
8.2 Manage Personnel
| Action | How |
|---|---|
| View enrolled people | Tile shows total enrolled count |
| Add a person | Tap Manage Personnel → Add Person → enter name and relation → tap Enrol |
| Delete a person | Open person list → tap entry → tap Delete |
8.3 Home Assistant Connection
| Field | Description |
|---|---|
| Base URL | HA server address with no trailing slash. Example: http://192.168.1.100:8123 |
| Access Token | Long-lived HA token. Generate in HA → Profile → Long-Lived Access Tokens. Stored in hardware-backed secure storage on the device. |
8.4 Entity IDs
Six Home Assistant entity IDs are configurable in Settings. Tap Validate entities to check all six against your live HA instance — each field shows ✅ (found) or ❌ (not found).
| Field | Default Entity ID |
|---|---|
| Thermostat | climate.living_room_thermostat |
| Blinds | cover.living_room_blinds |
| Evening scene | scene.living_room_evening |
| Morning scene | scene.living_room_morning |
| Presentation scene | scene.living_room_movie_night |
| Focus Mode scene | scene.living_room_work_mode |
8.5 Device Filters
Comma-separated substrings that control which HA entities appear on the Devices screen. Default: zigbee,zwave. Examples:
zigbee,zwave— show only Zigbee and Z-Wave deviceslight,switch,climate— show all lights, switches, and thermostats- (empty) — show all HA entities
8.6 Alert Poll Interval
How frequently Ngoma polls the HA logbook for new alerts. Default: 30 seconds. Minimum: 5 seconds. Recommended: 15–60 seconds (lower values increase Home Assistant API load). Enter in seconds and tap Save.
9. Perception System
The perception system provides Ngoma with real-time awareness of its environment, running on NVIDIA Jetson AGX Thor edge hardware and publishing results via MQTT. The status cards at the top of the Entertainment screen display:
| Card | What it shows |
|---|---|
| 👤 People | Number of people currently detected in the monitored zone |
| 🏃 Activity | Detected workspace activity (Working, Relaxing, Meeting, Moving) |
| 🔊 Sound | Ambient sound classification (Music, Conversation, Silence, Alert) |
Simulation mode: Before the Jetson is deployed, the perception system cycles through realistic simulated values. All features — including mood soundscapes and auto-ducking — are fully functional in simulation.
Production mode: When the Mosquitto broker and Jetson AI pipeline are running, the cards show live data. No application changes are required to switch from simulation to production.
Auto-ducking activates automatically when sound class is Conversation — audio drops to 20% and restores when the classification changes.
10. Voice Commands Reference
Playback Commands
| Command | Action |
|---|---|
| "Play" | Resume or start playback |
| "Pause" | Pause current playback |
| "Stop" | Stop playback entirely |
| "Next" | Skip to the next track |
| "Previous" | Return to the previous track |
Volume Commands
| Command | Action |
|---|---|
| "Volume up" | Increase volume by 10% |
| "Volume down" | Decrease volume by 10% |
| "Mute" | Mute all audio |
| "Unmute" | Restore audio to previous level |
Smart Environment Commands
| Command | Action |
|---|---|
| "Turn on [device name]" | Turn on the named Home Assistant device |
| "Turn off [device name]" | Turn off the named Home Assistant device |
Device names should match the friendly name as configured in Home Assistant. Example: "Turn on reception lights" or "Turn off floor two heating".
11. FAQ & Troubleshooting
"Configuration Required" screen on launch
Cause: Home Assistant credentials were not provided at deployment time.
Fix: Tap Open Settings and enter your HA base URL and access token. Or tap Continue without configuration to use the platform in limited mode.
Smart environment shows stale data or a "stale data" banner
Cause: Ngoma cannot reach Home Assistant.
Fix: Confirm HA is running and accessible on your network. Verify the base URL has no trailing slash. The app continues to serve from cached device states until connectivity is restored.
Entertainment shows a loading spinner indefinitely
Cause: SoundCentral credentials are missing, or the server is temporarily unreachable.
Fix: Ensure you are signed in via Settings → SoundCentral Account. The client retries with exponential backoff — wait 30–60 seconds and pull to refresh.
Live TV stream shows an error
The player retries automatically up to three times with increasing delays. Try a different channel and retry the original after a few minutes.
Voice control microphone button does nothing
Fix: Go to System Settings → Apps → Ngoma → Permissions and enable Microphone access. The app cannot request this permission again once denied — it must be granted manually in system settings.
Ambient voice mode shows "Whisper unavailable"
Fix: Confirm the Jetson wyoming-whisper service is running and reachable at the configured host and port (default 192.168.1.200:10300). Tap-to-talk continues to work independently at all times.
Wellness screen shows "Not Configured"
All three conditions must be met: (1) signed in to SoundCentral, (2) deployment includes KNIGHTS_BASE_URL and KNIGHTS_API_KEY, (3) the wearer's KnightGuard Wear OS app is installed and authenticated. Contact your account manager if credentials are missing.
Sentry Mode shows a red error
Create a scene in Home Assistant named sentry_mode_patrol. Refer to your deployment guide for the required scene structure. There is no silent fallback — the error is visible immediately to prevent undetected misconfiguration.
Entity validation shows ❌ for all fields
Confirm the HA base URL and access token are correct. Browse HA → Developer Tools → States to find the correct entity IDs, then update them in Settings and re-validate.
Facial recognition does not detect enrolled household members
The Jetson backend (Mosquitto + ArcFace pipeline) must be deployed and connected. Enrolment data stored locally in the app will be used as soon as the Jetson is connected — no re-enrolment required.
12. Glossary
- AirPlay 2
- Apple's wireless audio/video streaming protocol for routing audio to Apple devices and compatible speakers.
- ArcFace
- A deep learning face recognition model. Runs on the Jetson to identify enrolled household members in real time.
- Ambient mode
- Voice recognition mode where the microphone is always listening and streaming audio to the Jetson wyoming-whisper service for on-device processing.
- Bluetooth A2DP
- Advanced Audio Distribution Profile — the Bluetooth standard for streaming high-quality audio to wireless speakers and headphones.
- Entity ID
- A unique identifier for a device or automation in Home Assistant. Format:
<domain>.<location>_<device>. - HLS
- HTTP Live Streaming — the protocol used for live TV and radio streams.
- Home Assistant (HA)
- An open-source home/building automation platform. Ngoma uses it as the smart environment control layer for Zigbee and Z-Wave devices.
- Jetson AGX Thor
- NVIDIA's edge AI compute module. Powers the Ngoma platform's AI pipelines (vision, audio, navigation).
- KnightGuard
- A Wear OS health monitoring wearable from Knights Health. Streams biometric sensor data to the Knights Health API for display in the Wellness screen.
- MQTT
- Message Queuing Telemetry Transport — a lightweight publish/subscribe protocol used for Jetson ↔ platform real-time communication.
- Nav2
- The ROS 2 navigation stack. Runs on the Jetson and handles autonomous path planning and obstacle avoidance.
- Perception system
- The AI pipeline on the Jetson that detects people, classifies activity, and recognises ambient sounds in real time.
- Sentry Mode
- An autonomous patrol mode where the robot moves through configured waypoints, monitoring for security events.
- SoundCentral
- Ngoma's media content provider, offering audio tracks, video content, live TV channels, and live radio stations.
- Wyoming protocol
- A lightweight JSON-over-TCP protocol for streaming audio to ASR services. Used by wyoming-whisper for ambient voice recognition.
- YAMNet
- A deep learning model for audio event classification. Detects ambient sounds (conversation, music, alerts) on the Jetson.
- YOLOv8
- A real-time object detection model used on the Jetson for people counting and pose estimation.
- Zigbee
- A low-power wireless mesh protocol widely used for smart environment sensors, lighting, and switches.
- Z-Wave
- A wireless building automation protocol operating at 900 MHz for improved range and reduced interference compared to 2.4 GHz alternatives.