24 KiB
name, description
| name | description |
|---|---|
| protocols-mqtt-messaging | MQTT pub-sub messaging for IoT and real-time applications |
MQTT Messaging
Scope: MQTT protocol, pub-sub patterns, QoS levels, broker configuration, IoT integration Lines: ~400 Last Updated: 2025-10-27
When to Use This Skill
Activate this skill when:
- Implementing MQTT pub-sub messaging systems
- Building IoT device communication
- Designing lightweight messaging protocols
- Implementing real-time sensor data collection
- Setting up MQTT brokers (Mosquitto, EMQX, HiveMQ)
- Configuring QoS levels and message delivery guarantees
- Implementing retained messages and last will
- Designing topic hierarchies and wildcard subscriptions
- Securing MQTT with TLS and authentication
- Integrating with cloud IoT platforms (AWS IoT Core, Azure IoT Hub)
Core Concepts
MQTT Architecture
MQTT (Message Queuing Telemetry Transport): Lightweight pub-sub protocol designed for constrained devices and unreliable networks.
Key characteristics:
- Publish-Subscribe: Decoupled message pattern (publishers don't know subscribers)
- Lightweight: Minimal 2-byte header overhead
- QoS levels: Three levels of delivery guarantees (0, 1, 2)
- TCP/IP transport: Reliable, ordered delivery (or WebSocket for browsers)
- Broker-based: Central broker routes messages between clients
- Topic-based routing: Hierarchical topics with wildcard subscriptions
Architecture components:
Publisher → Broker → Subscriber(s)
↓
Topics + QoS
↓
Retained Messages
Last Will Testament
MQTT Basics
Connection Flow
Client Broker
| |
|-- CONNECT (clientId, auth) ----->|
| |
|<-- CONNACK (session present) -----|
| |
|-- SUBSCRIBE (topic, QoS) -------->|
| |
|<-- SUBACK (granted QoS) ----------|
| |
|-- PUBLISH (topic, payload, QoS) ->|
| |
|<-- PUBACK (if QoS 1+) ------------|
| |
|-- DISCONNECT ---------------------->|
QoS Levels
QoS 0 - At Most Once (Fire and forget):
- No acknowledgment or retransmission
- Message may be lost
- Fastest, lowest overhead
- Use case: Non-critical data (sensor readings where loss is acceptable)
QoS 1 - At Least Once (Acknowledged delivery):
- Message acknowledged with PUBACK
- Message may be delivered multiple times
- Moderate overhead
- Use case: Important messages where duplicates are acceptable
QoS 2 - Exactly Once (Assured delivery):
- Four-way handshake (PUBLISH → PUBREC → PUBREL → PUBCOMP)
- Message delivered exactly once
- Highest overhead
- Use case: Critical messages (billing, alarms)
# QoS comparison
qos_0 = client.publish("sensor/temp", "22.5", qos=0) # No ACK
qos_1 = client.publish("sensor/temp", "22.5", qos=1) # PUBACK required
qos_2 = client.publish("alarm/fire", "triggered", qos=2) # 4-way handshake
Topic Design
Topic Hierarchy
Topics are UTF-8 strings using / as separator:
# Good topic hierarchy
home/bedroom/temperature
home/bedroom/humidity
home/livingroom/temperature
factory/line1/machine5/status
factory/line1/machine5/telemetry
# Topic components
{domain}/{location}/{device}/{measurement}
Wildcards
Single-level wildcard (+):
- Matches one topic level
home/+/temperaturematcheshome/bedroom/temperatureandhome/kitchen/temperature
Multi-level wildcard (#):
- Matches zero or more topic levels (must be last character)
home/#matcheshome/bedroom/temperatureandhome/bedroom/humidity
# Wildcard subscriptions
client.subscribe("home/+/temperature") # All rooms' temperature
client.subscribe("home/bedroom/#") # All bedroom sensors
client.subscribe("#") # All topics (expensive!)
Topic Best Practices
DO:
- Use hierarchical structure:
domain/location/device/metric - Keep topics descriptive and consistent
- Use lowercase with underscores:
home/living_room/temp - Limit topic depth (3-7 levels)
- Avoid leading/trailing slashes
DON'T:
- Don't use spaces or special characters
- Don't use topics starting with
$(reserved for broker) - Don't subscribe to
#wildcard (high load) - Don't embed data in topics:
sensor/temp/22.5(use payload)
Python Client Implementation
Basic Publisher
import paho.mqtt.client as mqtt
import json
import time
class MQTTPublisher:
def __init__(self, broker, port=1883, client_id=None):
self.broker = broker
self.port = port
self.client = mqtt.Client(client_id=client_id)
# Callbacks
self.client.on_connect = self.on_connect
self.client.on_publish = self.on_publish
self.client.on_disconnect = self.on_disconnect
def on_connect(self, client, userdata, flags, rc):
if rc == 0:
print(f"Connected to {self.broker}:{self.port}")
else:
print(f"Connection failed: {rc}")
def on_publish(self, client, userdata, mid):
print(f"Message {mid} published")
def on_disconnect(self, client, userdata, rc):
if rc != 0:
print(f"Unexpected disconnect: {rc}")
def connect(self):
self.client.connect(self.broker, self.port, keepalive=60)
self.client.loop_start()
def publish(self, topic, payload, qos=0, retain=False):
"""Publish message to topic"""
if isinstance(payload, dict):
payload = json.dumps(payload)
result = self.client.publish(topic, payload, qos=qos, retain=retain)
return result.mid
def disconnect(self):
self.client.loop_stop()
self.client.disconnect()
# Usage
publisher = MQTTPublisher("mqtt.example.com")
publisher.connect()
# Publish sensor data
publisher.publish("home/bedroom/temperature", "22.5", qos=1)
publisher.publish("home/bedroom/humidity", "65", qos=1)
# Publish JSON
publisher.publish("sensor/data", {
"device_id": "sensor_01",
"temperature": 22.5,
"humidity": 65,
"timestamp": time.time()
}, qos=1)
time.sleep(2)
publisher.disconnect()
Basic Subscriber
import paho.mqtt.client as mqtt
import json
class MQTTSubscriber:
def __init__(self, broker, port=1883, client_id=None):
self.broker = broker
self.port = port
self.client = mqtt.Client(client_id=client_id)
# Callbacks
self.client.on_connect = self.on_connect
self.client.on_message = self.on_message
self.client.on_subscribe = self.on_subscribe
self.client.on_disconnect = self.on_disconnect
# Message handlers
self.handlers = {}
def on_connect(self, client, userdata, flags, rc):
if rc == 0:
print(f"Connected to {self.broker}:{self.port}")
# Resubscribe on reconnect
for topic, qos in self.handlers.keys():
client.subscribe(topic, qos)
else:
print(f"Connection failed: {rc}")
def on_message(self, client, userdata, msg):
topic = msg.topic
payload = msg.payload.decode()
qos = msg.qos
print(f"Received: {topic} (QoS {qos}): {payload}")
# Call registered handler
for (handler_topic, handler_qos), handler_func in self.handlers.items():
if self.topic_matches(topic, handler_topic):
handler_func(topic, payload, qos)
def on_subscribe(self, client, userdata, mid, granted_qos):
print(f"Subscribed (QoS {granted_qos})")
def on_disconnect(self, client, userdata, rc):
if rc != 0:
print(f"Unexpected disconnect: {rc}")
def topic_matches(self, topic, pattern):
"""Check if topic matches pattern with wildcards"""
# Simplified matching (paho does this internally)
return mqtt.topic_matches_sub(pattern, topic)
def connect(self):
self.client.connect(self.broker, self.port, keepalive=60)
self.client.loop_start()
def subscribe(self, topic, qos=0, handler=None):
"""Subscribe to topic with optional handler"""
self.client.subscribe(topic, qos)
if handler:
self.handlers[(topic, qos)] = handler
def disconnect(self):
self.client.loop_stop()
self.client.disconnect()
# Usage
subscriber = MQTTSubscriber("mqtt.example.com")
# Register handler
def handle_temperature(topic, payload, qos):
temp = float(payload)
if temp > 25:
print(f"High temperature alert: {temp}°C")
subscriber.subscribe("home/+/temperature", qos=1, handler=handle_temperature)
subscriber.connect()
# Keep running
try:
while True:
time.sleep(1)
except KeyboardInterrupt:
subscriber.disconnect()
Advanced Features
Retained Messages
Retained messages are stored by the broker and delivered to new subscribers immediately.
# Publish retained message (e.g., device status)
client.publish("device/status", "online", qos=1, retain=True)
# New subscribers immediately receive last retained message
# Use case: Device status, configuration, last known value
Use cases:
- Device online/offline status
- Last known sensor value
- System configuration
Clear retained message:
client.publish("device/status", "", qos=1, retain=True) # Empty payload clears
Last Will and Testament (LWT)
LWT is a message automatically sent by broker if client disconnects unexpectedly.
client = mqtt.Client()
# Set last will (sent if client disconnects without DISCONNECT)
client.will_set(
topic="device/status",
payload="offline",
qos=1,
retain=True
)
client.connect("mqtt.example.com")
# If client crashes or loses connection, broker publishes "offline"
Use cases:
- Device presence detection
- Connection monitoring
- Automatic cleanup
Clean Session vs Persistent Session
Clean Session = True (default):
- Broker discards all session state on disconnect
- Subscriptions not preserved
- Queued messages discarded
Clean Session = False (persistent):
- Broker stores subscriptions and QoS 1+ messages
- Messages delivered when client reconnects
- Use same client ID to resume session
# Persistent session
client = mqtt.Client(client_id="sensor_01", clean_session=False)
client.connect("mqtt.example.com")
# Subscribe with QoS 1
client.subscribe("commands/sensor_01", qos=1)
# Even if disconnected, messages are queued and delivered on reconnect
Security
TLS/SSL Encryption
import ssl
client = mqtt.Client()
# TLS with server certificate verification
client.tls_set(
ca_certs="/path/to/ca.crt",
certfile="/path/to/client.crt", # Optional client cert
keyfile="/path/to/client.key", # Optional client key
tls_version=ssl.PROTOCOL_TLSv1_2
)
client.connect("mqtt.example.com", port=8883) # TLS port
Username/Password Authentication
client = mqtt.Client()
client.username_pw_set(username="sensor_01", password="secret")
client.connect("mqtt.example.com")
Token-Based Authentication (AWS IoT Core)
# AWS IoT Core uses mutual TLS (certificate-based)
client = mqtt.Client()
client.tls_set(
ca_certs="AmazonRootCA1.pem",
certfile="device.pem.crt",
keyfile="device.pem.key",
tls_version=ssl.PROTOCOL_TLSv1_2
)
client.connect("xxxxxx-ats.iot.us-east-1.amazonaws.com", port=8883)
MQTT Brokers
Mosquitto (Open Source)
Lightweight broker for development and small deployments.
# Install
sudo apt-get install mosquitto mosquitto-clients
# Start broker
mosquitto -c /etc/mosquitto/mosquitto.conf
# Test with CLI
mosquitto_sub -h localhost -t "test/topic"
mosquitto_pub -h localhost -t "test/topic" -m "Hello"
Configuration (/etc/mosquitto/mosquitto.conf):
# Basic config
listener 1883
allow_anonymous true
# TLS config
listener 8883
cafile /path/to/ca.crt
certfile /path/to/server.crt
keyfile /path/to/server.key
require_certificate false
# Authentication
password_file /etc/mosquitto/passwd
EMQX (Scalable, Enterprise)
High-performance broker for large-scale IoT deployments.
# Docker deployment
docker run -d --name emqx \
-p 1883:1883 \
-p 8083:8083 \
-p 8084:8084 \
-p 8883:8883 \
-p 18083:18083 \
emqx/emqx:latest
# Dashboard: http://localhost:18083 (admin/public)
Features:
- Clustering and scalability (millions of connections)
- Rule engine for data processing
- Built-in authentication (JWT, database, HTTP)
- Metrics and monitoring
HiveMQ (Enterprise)
Production-grade broker with enterprise features.
Features:
- High availability and clustering
- Kafka integration
- Security (TLS, OAuth, LDAP)
- Management dashboard
Common Patterns
Request-Response Pattern
MQTT is pub-sub, but request-response can be implemented:
# Client sends request with reply topic
request_payload = json.dumps({
"command": "get_status",
"reply_to": f"responses/{client_id}"
})
client.subscribe(f"responses/{client_id}", qos=1)
client.publish("commands/device_01", request_payload, qos=1)
# Device responds to reply_to topic
def on_message(client, userdata, msg):
request = json.loads(msg.payload)
reply_topic = request.get("reply_to")
response = {"status": "online", "uptime": 3600}
client.publish(reply_topic, json.dumps(response), qos=1)
IoT Device Telemetry
# Device sends periodic telemetry
def send_telemetry():
while True:
telemetry = {
"device_id": "sensor_01",
"temperature": read_temperature(),
"humidity": read_humidity(),
"battery": get_battery_level(),
"timestamp": time.time()
}
client.publish(
f"devices/{device_id}/telemetry",
json.dumps(telemetry),
qos=1
)
time.sleep(60) # Every minute
Command and Control
# Backend sends commands to devices
client.publish(
"commands/sensor_01/update_config",
json.dumps({"interval": 30}),
qos=1
)
# Device subscribes to commands
client.subscribe("commands/sensor_01/#", qos=1)
def on_message(client, userdata, msg):
command = msg.topic.split("/")[-1]
payload = json.loads(msg.payload)
if command == "update_config":
update_config(payload)
elif command == "reboot":
reboot_device()
Anti-Patterns
❌ Subscribing to # wildcard: Receives all messages (broker load)
✅ Use specific wildcards: home/+/temperature
❌ Using QoS 2 everywhere: Highest overhead (4-way handshake) ✅ Use QoS 0 for non-critical data, QoS 1 for most cases, QoS 2 only when required
❌ Large payloads (>1 MB): MQTT is for small messages ✅ Keep payloads small (<10 KB), use external storage for large data
❌ Embedding data in topics: sensor/temp/22.5
✅ Use topics for routing, payloads for data: sensor/temp with payload 22.5
❌ No authentication: Open broker accessible to anyone ✅ Use TLS + username/password or client certificates
❌ Ignoring connection failures: No reconnection logic ✅ Implement exponential backoff reconnection
❌ Not using retained messages for status: Subscribers miss status ✅ Use retained messages for device status, config
Level 3: Resources
Overview
This skill includes comprehensive Level 3 resources for deep MQTT implementation knowledge and practical tools.
Resources include:
- REFERENCE.md (3,200+ lines): Complete technical reference covering MQTT protocol, brokers, patterns
- 3 executable scripts: Config validation, QoS testing, broker benchmarking
- 7 production examples: Complete implementations across brokers, platforms, languages
REFERENCE.md
Location: skills/protocols/mqtt-messaging/resources/REFERENCE.md
Comprehensive technical reference (3,200+ lines) covering:
Core Topics:
- MQTT protocol fundamentals (v3.1.1, v5.0)
- Pub-sub architecture and message flow
- QoS levels (0, 1, 2) and delivery guarantees
- Topic design and wildcard patterns
- Connection management and keep-alive
- Retained messages and last will
- Clean vs persistent sessions
- Security (TLS, authentication, authorization)
- MQTT brokers (Mosquitto, EMQX, HiveMQ)
- MQTT over WebSocket
- AWS IoT Core integration
- Azure IoT Hub integration
- MQTT v5.0 features
- Performance optimization
- Testing strategies
- Anti-patterns
Key Sections:
- Protocol Fundamentals: MQTT v3.1.1/v5.0, packet types, control packets
- QoS Levels: Detailed flow diagrams for QoS 0/1/2
- Topic Design: Hierarchies, wildcards, best practices
- Broker Setup: Mosquitto, EMQX, HiveMQ configuration
- Security: TLS/mTLS, authentication methods, authorization
- Client Implementation: Python, Node.js, Go examples
- Cloud Integration: AWS IoT Core, Azure IoT Hub
- MQTT v5.0: New features (user properties, reason codes, shared subscriptions)
- Performance: Connection pooling, batching, compression
- Anti-Patterns: Common mistakes and solutions
Format: Markdown with extensive code examples in Python, Node.js, and shell scripts
Scripts
Three production-ready executable scripts in resources/scripts/:
1. validate_mqtt_config.py (550 lines)
Purpose: Validate MQTT broker and topic configurations
Features:
- Parse Mosquitto and EMQX config files
- Validate topic naming conventions
- Check security settings (TLS, auth)
- Detect misconfigurations
- Validate ACL rules
- Output as JSON or human-readable text
Usage:
# Validate Mosquitto config
./validate_mqtt_config.py --config /etc/mosquitto/mosquitto.conf
# JSON output
./validate_mqtt_config.py --config /etc/mosquitto/mosquitto.conf --json
# Check topics
./validate_mqtt_config.py --check-topics --topics-file topics.txt
# Validate ACL
./validate_mqtt_config.py --config mosquitto.conf --check-acl
Checks:
- Config: Listener ports, TLS settings, auth configuration
- Topics: Naming conventions, hierarchy depth, wildcard usage
- Security: TLS version, certificate paths, password files
- ACL: Permission rules, topic patterns, user access
2. test_mqtt_qos.py (600 lines)
Purpose: Test MQTT QoS levels and message delivery
Features:
- Test QoS 0, 1, 2 delivery
- Measure message loss and duplicates
- Test retained messages
- Test last will and testament
- Connection reliability testing
- Latency measurements
- JSON output for CI/CD
Usage:
# Test all QoS levels
./test_mqtt_qos.py --broker mqtt.example.com --test-all
# Test QoS 1
./test_mqtt_qos.py --broker mqtt.example.com --qos 1 --count 100
# Test retained messages
./test_mqtt_qos.py --broker mqtt.example.com --test-retained
# Test LWT
./test_mqtt_qos.py --broker mqtt.example.com --test-lwt
# JSON output
./test_mqtt_qos.py --broker mqtt.example.com --test-all --json
Metrics: Delivery rate, message loss, duplicates, latency (min, avg, p95, max)
3. benchmark_mqtt_broker.py (650 lines)
Purpose: Benchmark MQTT broker performance and scalability
Features:
- Test concurrent connections (100s to 1000s)
- Measure throughput (messages/sec)
- Pub-sub latency testing
- Connection establishment time
- Memory and CPU monitoring
- Ramp-up testing
- JSON output for reporting
Usage:
# Benchmark with 1000 connections
./benchmark_mqtt_broker.py --broker mqtt.example.com --connections 1000 --duration 60
# Test throughput
./benchmark_mqtt_broker.py --broker mqtt.example.com --test throughput --messages 10000
# Test latency
./benchmark_mqtt_broker.py --broker mqtt.example.com --test latency --count 100
# Ramp-up test
./benchmark_mqtt_broker.py --broker mqtt.example.com --connections 5000 --ramp-up 120 --duration 300
# JSON output
./benchmark_mqtt_broker.py --broker mqtt.example.com --connections 1000 --json
Metrics: Connections/sec, messages/sec, latency (p50, p95, p99), memory usage
Examples
Seven production-ready examples in resources/examples/:
1. python/publisher.py
Complete Python publisher implementation:
- Connection management with reconnection
- QoS 0/1/2 support
- Retained messages
- JSON payload serialization
- Error handling and logging
- CLI interface
2. python/subscriber.py
Complete Python subscriber implementation:
- Topic subscription with wildcards
- Message handlers
- Persistent sessions
- Automatic reconnection
- Message queueing
- CLI interface
3. mosquitto/mosquitto.conf
Production Mosquitto configuration:
- Multiple listeners (1883, 8883, 9001 WebSocket)
- TLS/SSL configuration
- Password authentication
- ACL file for authorization
- Logging and persistence
- Connection limits
4. emqx/docker-compose.yml
EMQX deployment with Docker:
- EMQX broker with clustering support
- Prometheus metrics
- Grafana dashboards
- MQTT over WebSocket
- Health checks
- Volume mounts for persistence
5. aws_iot/iot_device.py
AWS IoT Core integration:
- Certificate-based authentication (mutual TLS)
- Device shadows
- Thing registry
- Job execution
- Fleet provisioning
- Telemetry publishing
6. node/mqtt_client.js
Node.js client implementation:
- MQTT.js library
- Async/await patterns
- Reconnection logic
- Event handlers
- TypeScript types
7. tls/generate_certs.sh
TLS certificate generation script:
- CA certificate creation
- Server certificate
- Client certificates
- Mosquitto configuration for TLS
- Testing with mosquitto_pub/sub
Quick Start
1. Validate broker config:
cd skills/protocols/mqtt-messaging/resources/scripts
./validate_mqtt_config.py --config ../examples/mosquitto/mosquitto.conf --json
2. Test QoS levels:
./test_mqtt_qos.py --broker localhost --test-all
3. Run examples:
cd ../examples
# Start Mosquitto
mosquitto -c mosquitto/mosquitto.conf
# Run publisher (in another terminal)
python python/publisher.py --broker localhost --topic test/topic --message "Hello MQTT"
# Run subscriber (in another terminal)
python python/subscriber.py --broker localhost --topic test/#
4. Deploy with Docker:
cd emqx
docker-compose up -d
# Dashboard: http://localhost:18083 (admin/public)
5. Benchmark broker:
cd ../scripts
./benchmark_mqtt_broker.py --broker localhost --connections 100 --duration 30
File Structure
skills/protocols/mqtt-messaging/
├── mqtt-messaging.md (this file)
└── resources/
├── REFERENCE.md (3,200+ lines)
├── scripts/
│ ├── validate_mqtt_config.py (550 lines) - Config validation
│ ├── test_mqtt_qos.py (600 lines) - QoS testing
│ └── benchmark_mqtt_broker.py (650 lines) - Broker benchmarking
└── examples/
├── python/
│ ├── publisher.py - Python publisher
│ └── subscriber.py - Python subscriber
├── mosquitto/
│ ├── mosquitto.conf - Mosquitto config
│ └── acl.conf - ACL rules
├── emqx/
│ └── docker-compose.yml - EMQX deployment
├── aws_iot/
│ └── iot_device.py - AWS IoT Core client
├── node/
│ └── mqtt_client.js - Node.js client
└── tls/
└── generate_certs.sh - TLS certificate generation
Resources Summary
| Category | Item | Lines | Description |
|---|---|---|---|
| Reference | REFERENCE.md | 3,200+ | Complete technical reference |
| Scripts | validate_mqtt_config.py | 550 | Config validation tool |
| test_mqtt_qos.py | 600 | QoS testing tool | |
| benchmark_mqtt_broker.py | 650 | Broker benchmarking tool | |
| Examples | python/publisher.py | 180 | Python publisher |
| python/subscriber.py | 200 | Python subscriber | |
| mosquitto.conf | 120 | Mosquitto configuration | |
| docker-compose.yml | 100 | EMQX deployment | |
| iot_device.py | 250 | AWS IoT Core client | |
| mqtt_client.js | 150 | Node.js client | |
| generate_certs.sh | 100 | TLS cert generation |
Total: 6,100+ lines of production-ready resources
Related Skills
protocols-grpc-implementation.md- gRPC for microservice communicationrealtime-websocket-implementation.md- WebSocket for bidirectional messagingpubsub-patterns.md- Pub-sub architecture patternsiot-device-management.md- IoT device lifecycle managementmessage-queue-patterns.md- Message queuing architectures
Last Updated: 2025-10-27 Format Version: 1.0 (Atomic)