hanzozt/zt
1
0
Fork 0
mirror of https://github.com/hanzozt/zt.git synced 2026-10-30 15:04:55 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
release-v1.7.x
zt/CHANGELOG.md
T
Paul Lorenz 9ede8e94f4 Update changelog
2025-11-21 12:21:45 -05:00

135 KiB
Raw Permalink Blame History

Release 1.7.2

What's New

The 1.7.0 release changed legacy (non-JWT) sessions to be in the JWT format, even when not using an OIDC flow. The 1.7.+ controller and router require the information in the JWT, and are thus incompatible with legacy service sessions issued from older controllers. This release ensures that all service sessions are cleared to avoid running clients from failing with cached older sessions.

Component Updates and Bug Fixes

  • github.com/openziti/ziti: v1.7.1 -> v1.7.2
    • Issue #3382 - Legacy service sessions generated pre-1.7.x are incompatible with v1.7.+ and need to be cleared

Release 1.7.1

What's New

This release updates the build to use Go 1.25.+. This is the only change in the release.

Release 1.7.0

What's New

  • proxy.v1 config type
  • Alert Events (Beta)
  • Azure Service Bus Event Sink (contributed by @ffaraone)

New proxy.v1 Config Type

Added support for dynamic service proxies with configurable binding and protocol options. This allows Edge Routers and Tunnelers to create proxy endpoints that can forward traffic for Ziti services.

This differs from intercept.v1 in that intercept.v1 will intercept traffic on specified IP ip addresses or DNS entries to forward to a service using tproxy or tun interface, depending on implementation.

A proxy on the other hand will just start a regular TCP/UDP listener on the configured port, so traffic will have to be configured for that destination.

Example proxy.v1 Configuration:

  {
    "port": 8080,
    "protocols": ["tcp"],
    "binding": "0.0.0.0"
  }

Configuration Properties:

  • port (required): Port number to listen on (1-65535)
  • protocols (required): Array of supported protocols (tcp, udp)
  • binding (optional): Interface to bind to. For the ER/T defaults to the configured lanIF config property.

This config type is currently supported by the ER/T when running in either proxy or tproxy mode.

Alert Events

A new alert event type has been added to allow Ziti components to emit alerts for issues that network operators can address. Alert events are generated when components encounter problems such as service configuration errors or resource availability issues.

Alert events include:

  • Alert source type and ID (currently supports routers, with controller and SDK support planned for future releases)
  • Severity level (currently supports error, with info and warning planned for future releases)
  • Alert message and supporting details
  • Related entities (router, identity, service, etc.) associated with the alert

Example alert event when a router cannot bind a configured network interface:

  {
    "namespace": "alert",
    "event_src_id": "ctrl1",
    "timestamp": "2021-11-08T14:45:45.785561479-05:00",
    "alert_source_type": "router",
    "alert_source_id": "DJFljCCoLs",
    "severity": "error",
    "message": "error starting proxy listener for service 'test'",
    "details": [
      "unable to bind eth0, no address"
    ],
    "related_entities": {
      "router": "DJFljCCoLs",
      "identity": "DJFljCCoLs",
      "service": "3DPjxybDvXlo878CB0X2Zs"
    }
  }

Alert events can be consumed through the standard event system and logged to configured event handlers for monitoring and alerting purposes.

These events are currently in Beta, as the format is still subject to change. Once they've been in use in production for a while and proven useful, they will marked as stable.

Azure Service Bus Event Sink

GitHub user @ffaraone contributed this feature, which adds support for streaming controller events to Azure Service Bus. The new logger enables real-time event streaming from the OpenZiti controller to Azure Service Bus queues or topics, providing integration with Azure-based monitoring and analytics systems.

To enable the Azure Service Bus event logger, add configuration to the controller config file under the events section:

  events:
    serviceBusLogger:
      subscriptions:
        - type: circuit
        - type: session
        - type: metrics
          sourceFilter: .*
          metricFilter: .*
        # Add other event types as needed
      handler:
        type: servicebus
        format: json
        connectionString: "Endpoint=sb://your-namespace.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=your-key"
        topic: "ziti-events"          # Use 'topic' for Service Bus topic
        # queue: "ziti-events-queue"  # Or use 'queue' for Service Bus queue
        bufferSize: 100                # Optional, defaults to 50

  • Required configuration:

    • format: Event format, currently supports only json
    • connectionString: Azure Service Bus connection string
    • Either topic or queue: Destination name (mutually exclusive)

  • Optional configuration:

    • bufferSize: Internal message buffer size (default: 50)

Component Updates and Bug Fixes

  • github.com/openziti/agent: v1.0.31 -> v1.0.33

  • github.com/openziti/channel/v4: v4.2.28 -> v4.2.41

  • github.com/openziti/edge-api: v0.26.47 -> v0.26.50

  • github.com/openziti/foundation/v2: v2.0.72 -> v2.0.79

    • Issue #455 - Correctly close goroutine pool when external close is signaled
    • Issue #452 - Goroutine pool with a min worker count of 1 can drop to 0 workers due to race condition

  • github.com/openziti/identity: v1.0.111 -> v1.0.118

    • Issue #68 - Shutdown file watcher when stopping identity watcher

  • github.com/openziti/runzmd: v1.0.80 -> v1.0.84

  • github.com/openziti/sdk-golang: v1.2.3 -> v1.2.10

    • Issue #818 - Full re-auth should not clear services list, as that breaks the on-change logic
    • Issue #817 - goroutines can get stuck when iterating over randomized HA controller list
    • Issue #736 - Migrate from github.com/mailru/easyjson
    • Issue #813 - SDK doesn't stop close listener when it detects that a service being hosted gets deleted
    • Issue #811 - Credentials are lost when explicitly set
    • Issue #807 - Don't send close from rxer to avoid blocking
    • Issue #800 - Tidy create service session logging

  • github.com/openziti/secretstream: v0.1.39 -> v0.1.41

  • github.com/openziti/storage: v0.4.26 -> v0.4.31

  • github.com/openziti/transport/v2: v2.0.188 -> v2.0.198

  • github.com/openziti/go-term-markdown: v1.0.1 (new)

  • github.com/openziti/ziti: v1.6.8 -> v1.7.0

    • Issue #3264 - Add support for streaming events to Azure Service Bus
    • Issue #3321 - Health Check API missing base path on discovery endpoint
    • Issue #3323 - router/tunnel static services fail to bind unless new param protocol is defined
    • Issue #3309 - Detect link connections meant for another router
    • Issue #3286 - edge-api binding doesn't have the correct path on discovery endpoints
    • Issue #3297 - stop promoting hotfixes downstream
    • Issue #3295 - make ziti tunnel service:port pairs optional
    • Issue #3291 - replace decommissioned bitnami/kubectl
    • Issue #3277 - Router can deadlock on closing a connection if the incoming data channel is full
    • Issue #3269 - Add host-interfaces config type
    • Issue #3258 - Add config type proxy.v1 so proxies can be defined dynamically for the ER/T
    • Issue #3259 - Interfaces config type not added due to wrong name
    • Issue #3265 - Forwarding errors should log at debug, since they are usual part of circuit teardown
    • Issue #3261 - ER/T dialed xgress connections may only half-close when peer is fully closed
    • Issue #3207 - Allow router embedders to customize config before start

Release 1.6.8

What's New

  • Bug fixes and library updates
  • Session Events for JWT Sessions
  • OIDC Fix when using a separate certificate for the API

Session Events for JWT sessions

When using JWT sessions, instead of legacy sessions, session events will now be created. There is a new provider field in session events, whose value will either be legacy or jwt.

OIDC Fix

There was an issue where OIDC authentication would fail if the API was configured with a different certificate than the controller's root identity certificate.

The v1.2.3 release of the Go SDK made OIDC the default, if the controller supported it. Since the quickstart uses separate certs certs, this was quickly noticed. If using the v1.2.3 release of the Go SDK, and affected by this issue, updating to OpenZiti controller v1.6.8 should resolve the problem.

Component Updates and Bug Fixes

Release 1.6.7

What's New

  • Bug fixes and library updates

Component Updates and Bug Fixes

Release 1.6.6

What's New

  • SDK Flow Control Updates
  • Multi-underlay links
  • Nested Identity App Data

SDK Flow Control Updates

The circuit testing for sdk flow control is complete. Many fixes were made. The SDK flow control code is still considered experimental, in that the features or API may change. However, it should now be feature complete and relatively stable.. Once it has been in production use for a reasonable period and no further changes are anticipated, it will be marked stable.

Multi-underlay Link

In previous releases, routers would attempt to set up two connections per link, one for payloads and one for acks. If either one failed, the whole link would be torn down. With this release, links can be made up of a user-configurable number of connections.

Link Connection Types

Link connections are of two types:

  • default - These may carry payloads and acks. As long as there is at least one default connection, the link will stay up.
  • ack - These may carry only acks. They act as a prioritization mechanism for acks. There may be zero ack connections.

The desired number of default and ack channels can be configured in the router configuration.

link:
  dialers:
    - binding: transport

      # Target number of default connections. Allowed range 1-100. Defaults to 3.
      maxDefaultConnections: 3
  
      # Target number of ack connections. Allowed range 1-100. Defaults to 1.
      maxAckConnections: 1

      # Time to delay making additional connections after the initial connection. Defaults to 3s
      # Reduces connection churn when routers are dialing each other at the same time.
      startupDelay: 3s

It's recommended to configure at least two connections per link.

Why Multiple Connections?

  1. They allow for link continuity even if one of the connections goes down.
  2. They can keep traffic moving if one of the connections stalls for some reason.
  3. Using multiple links also multiples the number of OS buffers in use, although the amount of per-connection buffers can also be bumped up at the OS level.

Why a ACK Priority Connection?

If a payload gets dropped, it will need to be retransmitted. If an ack gets dropped, a payload that's already been received will be retransmitted. Acks are also generally much smaller than payloads. The faster we can deliver them, the faster the flow control logic can react.

How Many Connections?

At least two. However, having more connections doesn't increase the physical bandwidth available between routers. Some additional connections provide additional resilience and perhaps more performance due to increased OS resources. However, the benefits diminish quickly. More than the default of three is unlikely to provide much benefit.

How is traffic load-balanced?

There is a queue for payloads and other for acks. Default connections pull from both queues, ack connections only pull from the ack queue. Because connections pull from the queues, if one connection is slower it will naturally pull fewer messages than other connections.

Backwards Compatibility

When creating links to a router older than 1.6.6, routers will fallback to the old logic and dial one payload and one ack channel.

Link Events

Links will now report their connections to the controller. They are now reported when listing links using ziti fabric list links.

Here is an example from a test setup.

$ ziti fabric list links 'skip 3 limit 2'
╭────────────────────────┬───────────────────────┬────────────────────────┬─────────────┬─────────────┬─────────────┬───────────┬────────┬───────────┬──────────────────────────────────────────────────────────────╮
│ ID                     │ DIALER                │ ACCEPTOR               │ STATIC COST │ SRC LATENCY │ DST LATENCY │ STATE     │ STATUS │ FULL COST │ CONNECTIONS                                                  │
├────────────────────────┼───────────────────────┼────────────────────────┼─────────────┼─────────────┼─────────────┼───────────┼────────┼───────────┼──────────────────────────────────────────────────────────────┤
│ 101OzJLiMrrFSpwT0LnYOY │ router-eu-central-3.7 │ router-eu-central-2.11 │           1 │       2.7ms │       2.7ms │ Connected │     up │         5 │ link.default: tcp:10.0.0.230:40028 -> tcp:54.93.210.111:6011 │
│                        │                       │                        │             │             │             │           │        │           │ link.default: tcp:10.0.0.230:40032 -> tcp:54.93.210.111:6011 │
│                        │                       │                        │             │             │             │           │        │           │ link.ack: tcp:10.0.0.230:46092 -> tcp:54.93.210.111:6011     │
│                        │                       │                        │             │             │             │           │        │           │ link.default: tcp:10.0.0.230:46096 -> tcp:54.93.210.111:6011 │
│ 101YAe327nSngeRIXeKR0T │ router-eu-central-3.5 │ router-us-east-4.17    │           1 │      91.5ms │      91.4ms │ Connected │     up │       183 │ ack: tcp:10.0.0.230:57574 -> tcp:13.220.214.103:6017         │
│                        │                       │                        │             │             │             │           │        │           │ payload: tcp:10.0.0.230:57568 -> tcp:13.220.214.103:6017     │
╰────────────────────────┴───────────────────────┴────────────────────────┴─────────────┴─────────────┴─────────────┴───────────┴────────┴───────────┴──────────────────────────────────────────────────────────────╯
results: 4-5 of 79803

A link is considered created once it has an initial default connection. The link will then attempt to reach the desired count of default and ack connections. Whenever a new underlay connection is established or closes, the controller will be notified and an event will be generated.

Link event example:

{
  "namespace": "link",
  "event_src_id": "ctrl_client",
  "timestamp": "2025-07-11T10:35:01.614896435-04:00",
  "event_type": "connectionsChanged",
  "link_id": "7mCYLrQAiO93du7SLGDeXf",
  "connections": [
    {
      "id": "link.default",
      "local_addr": "tcp:127.0.0.1:33682",
      "remote_addr": "tcp:127.0.0.1:4024"
    },
    {
      "id": "link.default",
      "local_addr": "tcp:127.0.0.1:33686",
      "remote_addr": "tcp:127.0.0.1:4024"
    },
    {
      "id": "link.ack",
      "local_addr": "tcp:127.0.0.1:33696",
      "remote_addr": "tcp:127.0.0.1:4024"
    },
    {
      "id": "link.default",
      "local_addr": "tcp:127.0.0.1:33702",
      "remote_addr": "tcp:127.0.0.1:4024"
    }
  ]
}

NOTES

  1. Link events show the full set of connections for the current state instead of the change.
  2. New routers dialing older routers will still report link connections. See the second link in the list above.
  3. Old routers will not report connections.

Nested Identity App Data

Identity app data may now be a full JSON document, rather than just a single layer map. There are also some additional CLI methods to work with the data:

$ ziti edge create identity test --app-data foo=bar
$ ziti edge create identity test --app-data-json '{ "foo" : "bar", "test" : { "nested" : true, "number" : 234 } }'
$ ziti edge create identity test --app-data-json-file test-app-data.json 

$ ziti edge update identity test --app-data foo=bar
$ ziti edge update identity test --app-data-json '{ "foo" : "bar", "test" : { "nested" : true, "number" : 234 } }'
$ ziti edge update identity test --app-data-json-file test-app-data.json

Component Updates and Bug Fixes

Release 1.6.5

What's New

Bugfixes and dependency updates.

Component Updates and Bug Fixes

Release 1.6.4

This version was intentionally skipped and not released because the 1.6.3 FIPS binary had an erroneous internal version of 1.6.4.

Release 1.6.3

What's New

  • Router Network Interface Discovery

Router Interface Discovery

Routers can now discover their network interfaces and publish this information to the controller.

This feature will be used in the future to allow controller side configuration of router link listeners and edge listeners.

Update Router Configuration

There is new router configuration to manage this:

interfaceDiscovery:
  # This feature is enabled by default, but can be disabled by setting this to true
  disabled: false

  # How often to poll for interface changes. Defaults to 1 minute
  checkInterval: 1m

  # How often to report the current set of interfaces, when nothing has changed.
  # This is a failsafe reporting mechanism, in the very unlikely event that an 
  # earlier change report was lost or disregarded due to distributed controller
  # eventual consistency
  minReportInterval: 24h

Update REST APIs

Network interfaces, where reported, can now be viewed on the following endpoints.

  • routers
  • edge-routers
  • identities (if the router is an ER/T, with an associated identities)

At some point in the future, we expect to allow SDKs to also optionally report their network interfaces as well. Those will be available via the identities REST API.

Example:

$ ziti fabric list routers 'name="edge-router-1"' -j | jq
{
  "data": [
    {
      "_links": {
        "self": {
          "href": "./routers/oLvcT6VepI"
        },
        "terminators": {
          "href": "./routers/oLvcT6VepI/terminators"
        }
      },
      "createdAt": "2025-03-24T04:35:59.077Z",
      "id": "oLvcT6VepI",
      "tags": {},
      "updatedAt": "2025-06-11T14:29:22.083Z",
      "connected": false,
      "cost": 0,
      "disabled": false,
      "fingerprint": "b7b03c55be77df0ec57e49d8fe2610e0b99fa61c",
      "interfaces": [
        {
          "addresses": [
            "192.168.3.29/24",
            "aaaa::aaaa:aaaa:aaaa:aaaa/64"
          ],
          "hardwareAddress": "aa:aa:aa:aa:aa:aa",
          "index": 4,
          "isBroadcast": true,
          "isLoopback": false,
          "isMulticast": true,
          "isRunning": true,
          "isUp": true,
          "mtu": 1500,
          "name": "wifi0"
        },
        {
          "addresses": null,
          "hardwareAddress": "aa:aa:aa:aa:aa:aa",
          "index": 2,
          "isBroadcast": true,
          "isLoopback": false,
          "isMulticast": true,
          "isRunning": false,
          "isUp": true,
          "mtu": 1500,
          "name": "eth1"
        },
        {
          "addresses": [
            "192.168.1.2/24",
            "aaaa::aaaa:aaa:aaaa:aaaa/64"
          ],
          "hardwareAddress": "aa:aa:aa:aa:aa:aa",
          "index": 16,
          "isBroadcast": true,
          "isLoopback": false,
          "isMulticast": true,
          "isRunning": true,
          "isUp": true,
          "mtu": 1500,
          "name": "eth0"
        },
        {
          "addresses": [
            "127.0.0.1/8",
            "::1/128"
          ],
          "hardwareAddress": "",
          "index": 1,
          "isBroadcast": false,
          "isLoopback": true,
          "isMulticast": false,
          "isRunning": true,
          "isUp": true,
          "mtu": 65536,
          "name": "lo"
        }
      ],
      "listenerAddresses": null,
      "name": "edge-router-1",
      "noTraversal": false
    }
  ],
  "meta": {
    "filterableFields": [
      "id",
      "isSystem",
      "name",
      "fingerprint",
      "cost",
      "createdAt",
      "updatedAt",
      "tags",
      "noTraversal",
      "disabled",
      "connected"
    ],
    "pagination": {
      "limit": 10,
      "offset": 0,
      "totalCount": 1
    }
  }
}

Note that addresses have been sanitized.

Component Updates and Bug Fixes

Release 1.6.2

What's New

  • System Certificate Authentication Improper Chain Detection
  • Authentication Events
  • Multi-underlay channel group secret
  • Flag to disable posture check functionality

System Certificate Authentication Improper Chain Detection

Previous versions of SDKs and controllers issued and stored client certificate chains with differing levels of fidelity. Depending on when an identity was enrolled, it may or may not have a proper client certificate chain. In single controller environments, the controller will automatically include its known intermediates to help create valid x509 certificate chains back to the network's Root CA.

In HA environments, each controller does not know of all the intermediates in the system by default. In order to support dynamically scaling controllers, clients must store and provide a proper client certificate chain (a leaf certificate followed by all necessary intermediate CA certs). Identities enrolled with SDKs that did not store the chain or controllers that did not provide the chain will encounter authentication issues in multi-controller environments.

To help determine which identities have issues, the system has been augmented to allow detection of problematic certificate authenticators. Firstly, authenticators have flags on them that are set during authentication to detect the current behavior of a specific client that owns and is invoking the certificate authenticator.

Detecting Improper Client Cert Chains As A Client SDK

The current API Sessions endpoint (https://<host>/edge/client/v1/current-api-session) now returns a value named improperClientCertChain as a boolean value. If it is not present or false no action should be taken. If true it means that the current API Session was authenticated with an internal PKI issued certificate where the client did not provide a full chain to the root CA during authentication; indicating a problem with the certificate storage mechanism in the application or due to the controller version used during enrollment/extension not providing a chain. The SDK should proactively opt to begin certificate extension on its own to obtain a proper chain. Authentication succeeded in this case because the controller relied upon a deprecated certificate pool that happen to include the necessary intermediate CAs.

Detecting Clients Without Proper Chains

After a client has authenticated with the system via a network-issued certificate at least one time, a number of fields are set depending on what the client provided. These values can be reviewed via the Edge Management API for Authenticators (edge/management/v1/authenticators). Where an authenticator has isIssuedByNetwork set to true and lastAuthResolvedToRoot set to false, indicates that the related identity/client is not providing a chain when it should.

Additionally, if authenticator events are enabled and being processed, events will have a field improper_client_cert_chain set to true (see Authentication Events below)

Fixing Clients Chains

Once an authenticator has been identified as problematic, an administrator should verify the client is using the newest possible versions of its SDK and either re-enroll it or request the identity to extend the next time it authenticates. Both scenarios result in a new certificate and chain being provided to the client.

  • Re-Enrollment removes the current authenticator, making authentication impossible until enrollment is completed. Once completed, a new authenticator is created, and the new certificate generated from the process can be used to authenticate. Humans or other external automation processes have to deliver and consume the new enrollment JWT.
  • Extension leaves the authenticator in place and authentication can still occur. When the extension process is completed, the authenticator is updated and will use the new certificate generated from the process. SDKs will handle the process by raising SDK specific events to drive the process.

While related and similar, using one over the other depends on the situation. Re-Enrollment is best for when an client or its host are unrecoverable; either through damage, decommissioning, or suspected to be compromised. Extension is useful when the client and host are in a known good status and one simply wants to issue new certificates to said client. Extension is also "hands off" from the client as long as the client is using SDKs that support enrollment request/key rolling.

Useful Authenticator Values

Authenticators are an entity within OpenZiti and can be queried via the CLI (ziti edge list authenticators) or Edge Management API (GET https://<host>/edge/management/v1/authenticators). Below are properties that are useful for determining clients with improper chains.

  • isIssuedByNetwork (boolean) indicates the authenticator is a certificate that was issued by the network
  • isExtendRequested (boolean) indicated the authenticator will report to the client to extend its certificate on next authentication.
  • isKeyRollRequested (boolean) indicates if key rolling is requested with an outstanding isExtendRequested
  • extendRequestedAt (string, date time) indicates when an outstanding extension was requested, otherwise null or not provided
  • lastAuthResolvedToRoot (boolean) indicates if the last time the client authenticated, it provided a certificate chain that resolved to the root CA. Only valid if isIssuedByNetwork is true.

Re-Enrollment

Re-enrollment can be accomplished via the CLI or the Edge Management API. Re-enrollment removes the current authenticator, stopping the client from authenticating. All of the underlying configuration for the related identity is preserved. The newly created enrollment will have a new enrollment JWT that will be consumed during the enrollment process and will result in a new client certificate and chain.

CLI:

ziti edge update authenticator cert <id> --re-enroll [--duration <duration>]

Edge Management API:

POST /edge/management/v1/authenticators/:id/re-enroll

{
    expiresAt: "2025-05-21T13:47:26Z"
}

Both of the above provide an enrollment id and or enrollment JWT token as output. These can be consumed in various OpenZiti applications or CLI commands.

CLI Enrollment

 ziti edge enroll -h
enroll an identity

Usage:
  ziti edge enroll path/to/jwt [flags]

Flags:
      --ca string         Additional trusted certificates
  -c, --cert string       The certificate to present when establishing a connection.
  -h, --help              help for enroll
  -n, --idname string     Names the identity. Ignored if not 3rd party auto enrollment
  -j, --jwt string        Enrollment token (JWT file). Required
  -k, --key string        The key to use with the certificate. Optionally specify the engine to use. supported engines: [parsec]
  -a, --keyAlg RSA|EC     Crypto algorithm to use when generating private key (default RSA)
  -o, --out string        Output configuration file.
  -p, --password string   Password for updb enrollment, prompted if not provided and necessary
      --rm                Remove the JWT on success
  -u, --username string   Username for updb enrollment, prompted if not provided and necessary
  -v, --verbose           Enable verbose logging

OpenZiti Tunneler Enrollment

ziti-edge-tunnel add --jwt "$(< ./in-file.jwt)" --identity myIdentityName

Request Extension

Extension leaves the authenticator in place, but notifies SDKs that they should generate a CSR to request a new certificate. Once the process is complete, the client will have a new certificate and chain that must be used for later authentication requests. This can be initiated through the CLI or Edge Management API.

CLI:

ziti edge update authenticator cert <id> --request-extend [--request-key-roll]

Edge Management API:

POST /edge/management/v1/authenticators/:id/request-extend

{
    rollKeys: true,
}

These commands will cause the authenticator to be updated to have isExtendRequested, extendRequestedAt, and optionally isKeyRollRequested` updated on the authenticator. These values will be used to signal the client to extend their certificate on the next successful authentication attempt.

Authentication Events

A new event names space authentication has been added for events with two types success and fail. These can be enabled via the controller configuration files.

Event Documentation:

// An AuthenticationEvent is emitted when an authentication attempt is made
//
// Types of authentication events
//   - fail - authentication failed
//   - success - authentication succeeded
//
// Types of authentication methods
//   - updb - username password from the internal database
//   - cert - a certificate, either first party or 3rd party
//   - ext-jwt - an external JWT from an IDP

Example Controller Configuration:

events:
  jsonLogger:
    subscriptions:
      - type: authentication
#          - success
#          - fail
    handler:
      type: file
      format: json
      path: ${TMPDIR}/ziti-events.log

Example Output:

{
  "namespace": "authentication",
  "event_src_id": "ctrl_client",
  "timestamp": "2025-05-21T10:24:32.6532054-04:00",
  "event_type": "fail",
  "type": "password",
  "authenticator_id": "B0JrMDPGxd",
  "external_jwt_signer_id": "",
  "identity_id": "BGJabPP0K",
  "auth_policy_id": "default",
  "remote_address": "127.0.0.1:51587",
  "success": false,
  "reason": "could not authenticate, password does not match",
  "improper_client_cert_chain": false
}

Multi-underlay channel group secret

For additional security the experimental multi-underlay channel code now requires that clients provide a shared secret. This ensures that channels are get the expected underlays without requiring much larger group ids. On the client side this will require the go sdk version to be v1.1.1 or greater.

Disabling posture check functionality

As posture check functionality can have performance impacts, it can now be disabled, for users who don't need it.

This is controlled in the controller config file.

edge:
  # Set to true to disable posture check functionality
  disablePostureChecks: false

Component Updates and Bug Fixes

Release 1.6.1

What's New

  • Bug fixes and library updates
  • Ability to request that SDKs extend and optionally roll their key
  • Address translations can now be specified in host.v1 service configuration

Ability to request that SDKs extend and optionally roll their key

It is now possible for administrators to flag specific certificate authenticators as needed to extend their current certificate early and/or optionally roll the keypair that underpins the certificate. This capability only works for certificates issued by the OpenZiti network. If '3rd party CAs' are in use, those certificate authenticators will not work with this system.

SDKs must support this capability for it to have any effect, and the application utilizing the SDK must respond to the certificate extension events to store certificate credentials.

This capability is located in the Management API at /edge/management/v1/authenticators/{id>/request-extend. Its payload is currently and optional boolean value for rollKeys that can be set to true/false and defaults to false if not provided.

This can also be issued via the CLI:

> ziti edge update authenticator cert -h
Request a specific certificate authenticator to --requestExtend or --requestKeyRoll, --requestKeyRoll implies --requestExtend

Usage:
  ziti edge update authenticator cert <authenticatorId> [--requestExtend] [--requestKeyRoll] [flags]

Flags:
  -h, --help             help for cert
  -e, --requestExtend    Specify the certificate authenticator should be flagged for extension
  -r, --requestKeyRoll   Specify the certificate authenticator should be flagged for key rolling, implies --requestExtend

Requesting an extension flags new fields on a certificate authenticator in the values isExtendRequest and isKeyRollRequested. These values are set to false after the client performs a certificate extension. The CLI has been updated to report these values on certificate authenticators via ziti edge list authenticators.

These values are also present on the /edge/client/v1/current-api-session endpoint when a client has use certificate authentication to initiate an API Session using a certificate authenticator.

Additionally, a log of key rolling activity per authenticator will be available in a future release.

host.v1 Address Translation

The host.v1 service configuration type now includes a forwardAddressTranslations field that specifies how a hosting tunneler should translate destination IPs from the client when connecting to the underlay application.

Component Updates and Bug Fixes

Release 1.6.0

What's New

  • Bug fixes and library updates
  • Change to cluster add peer
  • Experimental multi-underlay SDK support
  • Experimental SDK flow-control support

Cluster Add Peer Change

The ziti agent cluster add command no longer supports the --id argument for providing the peer id. The add operation will now always connect to the peer, verify the certs and get the peer advertise address and id from the peer directly. This will ensure that the peer is reachable and valid before it is added to the cluster.

Multi-underlay SDK support

For SDKs which support it, the edge router now supports a separate control channel connection along side the data connection. If the SDK doesn't request separate channels, the edge router will continue to work with a single connection. This feature is still experimental and may have bugs, may be changed or be removed.

SDK Flow-control

For SDKs which support it, the edge router now supports delegating flow control to the SDK. This feature is still experimental and may have bugs, may be changed or be removed.

Component Updates and Bug Fixes

  • github.com/openziti/channel/v4: v3.0.39 -> v4.0.6

    • Issue #182 - MultiListener can deadlock
    • Issue #180 - Add GetUserData to Channel interface
    • Issue #176 - Multi-channel need a mechanism to notify the txer that the underlay has closed
    • Issue #172 - Support multi-underlay channels

  • github.com/openziti/identity: v1.0.100 -> v1.0.101

    • Issue #64 - Support a way to check if a cert/serverCert can be saved

  • github.com/openziti/metrics: v1.3.0 -> v1.4.1

    • Issue #53 - Add reporter useful for emitting metrics to stdout

  • github.com/openziti/sdk-golang: v0.25.1 -> v1.1.0

    • Issue #702 - [Go SDK] Support xgress flow control from the SDK
    • Issue #722 - Move xgress impl to SDK
    • Issue #717 - ER connection race condition can leak connections
    • Issue #689 - Concurrent map iteration and modification in getEdgeRouterConn causes panic
    • Issue #701 - Support multi-underlay channels for edge router connections

  • github.com/openziti/transport/v2: v2.0.167 -> v2.0.168

  • github.com/openziti/xweb/v2: v2.3.0 -> v2.3.1

  • github.com/openziti/ziti: v1.5.4 -> v1.6.0

    • Issue #3005 - Always check that a controller is reachable and valid before adding it to an HA controller cluster
    • Issue #2986 - [Router] Support xgress flow control from the SDK
    • Issue #2999 - OIDC JWT backed sessions cannot verify extended certs
    • Issue #2997 - Add Authenticator Id to OIDC JWTs/return for current-api-session
    • Issue #2904 - Support client certificate authorities in TLS handshake
    • Issue #2973 - CLI: add a subcommand to retrieve network JWT
    • Issue #2984 - Extend enrollments does not return a full chain
    • Issue #2930 - Support multi-underlay channels for the edge SDK
    • Issue #2978 - Create loop4 sim for testing circuit contention and scale
    • Issue #2981 - Remove PayloadBufferForwarder API from xgress retransmitter
    • Issue #2906 - Controller not removed from DB controller store when removed from controller
    • Issue #2922 - Validate node address before adding to cluster
    • Issue #2932 - Fix router data model 'create public key' related errors
    • Issue #2919 - Make xgress pluggable, so it can be used from the SDK
    • Issue #2955 - Extract xgress inspection types
    • Issue #2954 - Encapsulate xgress metrics
    • Issue #2952 - Remove global payload ingester
    • Issue #2951 - Remove global xgress retransmitter
    • Issue #2950 - Move router specific xgress code to a new xgress_router package
    • Issue #2920 - Make xgress acker configurable

Release 1.5.4

What's new

  • Bug fixes

Component Updates and Bug Fixes

Release 1.5.3

What's New

  • This release updates the Go version from 1.23 to 1.24

Release 1.5.2

What's New

  • This release reverts a change refactoring some flow-control apis, as the change caused a panic

Release 1.5.1

What's New

  • Bug fixes and features

Component Updates and Bug Fixes

Release 1.5.0

What's New

  • Bug fixes and features
  • Change to router endpoints file default name
  • Updated Cluster Defaults
  • Updates to terminator costing
  • Router metrics changes

Router Endpoints File

The router endpoints file used to have a default name of endpoints. As it is a YAML file, the file now has a default name of endpoints.yml. This may affect existing setups running the beta HA code. If this is a concern, the endpoints file path can be configured:

ctrl:
  endpointsFile: /path/to/endpoints.file

Updated Cluster Defaults

The following defaults have changed.

cluster:
  # How many updates before creating a new snapshot. 
  #
  # New default: 500
  # Old default: 8192
  snapshotThreshold: 500 

  # How many old entries to keep around, so that a stream of
  # entries can be sent to sync peers, instead of sending an
  # entire snapshot
  #
  # New default: 500
  # Old default: 10240
  trailingLogs: 500

Terminator Costing Changes

When a terminator is selected for a service dial, and that dial results in a failure, a failure cost is added to that terminator. This will bias future dials towards other terminators, if they are available.

The failure cost can be reduced by successful dials. Failure cost is also reduced over time. In previous releases this was a fixed credit of 5, every minute. This is now changing to an exponential amount, based on time since the last failure.

If X is minutes since last failure, the credit is now: min(10, 2 ^ (X/5)).

Router Metrics Changes

There are four new router metrics, focused on visibility into flow control.

  • xgress.blocked_by_local_window_rate - meter which ticks whenever an xgress becomes blocked by the local window being full
  • xgress.blocked_by_remote_window_rate - meter which ticks whenever an xgress becomes blocked by the remote receive buffer being full
  • xgress.blocked_time - timer which tracks how long xgresses are in a blocked state.
  • xgress_edge.long_data_queue_time - timer that tracks times to process incoming data payloads to xgress_edge.

The xgress_edge.long_data_queue_time will be controller by a router config file setting. It will default to disabled. The other metrics will always be enabled.

Router metrics has two new configuration setting:

metrics:
  # Number of usage events to be able to queue. Defaults to 256. If this queue backs up, it can
  # slow down dispatch of data from an SDK onto the fabric.
  eventQueueSize: 256

  # If set to true, enables the xgress_edge.long_data_queue_time metric. Defaults to false.
  enableDataDelayMetric: false

Component Updates and Bug Fixes

Release 1.4.3

What's New

  • Bug fixes

Component Updates and Bug Fixes

Release 1.4.2

  • Bug fixes

Component Updates and Bug Fixes

Release 1.4.1

What's New

  • Bug fixes

Component Updates and Bug Fixes

Release 1.4.0

What's New

  • Changes to backup/restore and standalone to HA migrations
  • Use cluster consistently for cluster operations
  • Event Doc and Consistency
  • ziti ops verify changes
    • Moved ziti ops verify-network to ziti ops verify network
    • Moved ziti ops verify traffic to ziti ops verify traffic
    • Added ziti ops verify ext-jwt-signer oidc to help users with configuring OIDC external auth
    • Added ziti ops verify ext-jwt-signer oidc to help users with configuring OIDC external auth
  • Router Controller Endpoint Changes
  • Bug fixes

Config Changes

NOTE: For HA configuration, the raft: config stanza is now named cluster:.

Example:

cluster:
  dataDir: ./data

Event Doc and Consistency

The event types are now exhaustively documented as part of the OpenZiti Reference Documentation.

During documentation, some inconsistencies were found the following changes were made.

Namespace Cleanup

Namespaces have been cleaned up, with the following changes:

  • edge.apiSessions -> apiSession
  • fabric.circuits -> circuit
  • edge.entityCount -> entityCount
  • fabric.links -> link
  • fabric.routers -> router
  • services -> service
  • edge.sessions -> session
  • fabric.terminators -> terminator
  • fabric.usage -> usage

Note that service events used services in the config file, but service.events in the event itself. The old namespaces still work. If the old event type is used in the config file, the old namespace will be in the events as well

Timestamp field

The following event types now have a timestamp field:

  • service
  • usage

This timestamp is the time the event was generated.

Event Source ID

All event types now have a new field: event_src_id. This field is the id of the controller which emitted the event. This may be useful in HA environments, during event processing.

Cluster Operations Naming

The CLI tools under ziti fabric raft are now found at ziti ops cluster.

The Raft APIs available in the fabric management API are now namespaced under Cluster instead.

Backup/Restore/HA Migrations

What restoring from a DB snapshot has in common with migrating from a standalone setup to a RAFT enabled one, is that the controller is changing in a way that the router might not notice.

Now that routers have a simplified data model, they need know if the controller database has gone backwards. In the case of a migration to an HA setup, they need to know that the data model index has changed, likely resetting back to close to zero.

To facilitate this, the database now has a timeline identifier. This is shared among controllers and is sent to routers along with the data state. When the controller restores to a snapshot of previous state, or when the the controller moves to a raft/HA setup, the timeline identifier will change.

When the router requests data model changes, it will send along the current timeline identifier. If the controller sees that the timeline identifier is different, it knows to send down the full data state.

Implementation Notes

In general this is all handled behind the scenes. The current data model index and timeline identifier can be inspected on controllers and routers using:

ziti fabric inspect router-data-model-index

Example

$ ziti fabric inspect router-data-model-index
Results: (3)
ctrl1.router-data-model-index
index: 25
timeline: MMt19ldHR

vEcsw2kJ7Q.router-data-model-index
index: 25
timeline: MMt19ldHR

ctrl2.router-data-model-index
index: 25
timeline: MMt19ldHR

Whenever we create a database snapshot now, the snapshot will contain a flag indicating that the timeline identifier needs to be changed. When a standalone controller starts up, if that flag is set, the controller changes the timeline identifier and resets the flag.

When an HA cluster is initialized using an existing controller database it also changes the timeline id.

HA DB Restore

There's a new command to restore an HA cluster to an older DB snapshot.

ziti agent controller restore-from-db </path/to/database.file>

Note that when a controller is already up and running and receives a snapshot to apply, it will move the database into place and then shutdown, expecting to be restarted. This is because there is caching in various places and restartingi makes sure that everything is coherent with the changes database.

Router Controller Endpoint Updates

Endpoints File Config

The config setting for controller the endpoints file location has changed.

It was:

ctrl:
  dataDir: /path/to/dir

The endpoints file would live in that directory but always be called endpoints.

This is replaced by a more flexible endpointsFile.

ctrl:
  endpointsFile: /path/to/endpoints.file

The default location is unchanged, which is a file named endpoints in the same directory as the router config file.

Enrollment

The router enrollment will now contain the set of known controllers at the time the router as enrollled. This also works for standalone controllers, as long as the advertiseAddress settings is set.

Example

ctrl:
  listener: tls:0.0.0.0:6262
  options:
    advertiseAddress: tls:ctrl1.ziti.example.com

This means that the controller no longer needs to be set manually in the config file, enrollment should handle initializing the value appropriately.

Component Updates and Bug Fixes

Release 1.3.3

What's New

  • Bug Fixes

Component Updates and Bug Fixes

  • github.com/openziti/ziti: v1.3.2 -> v1.3.3
    • Issue #2694 - Router should use router data model if it has more than one controller configured, regardless of controller configuration

Release 1.3.2

What's New

  • Bug Fixes

Component Updates and Bug Fixes

  • github.com/openziti/ziti: v1.3.1 -> v1.3.2
    • Issue #2692 - Routers get empty ctrl list on upgrade
    • Issue #2689 - OIDC authentication with form data requires "id" in form data, authReqeustID in query string is ignored

Release 1.3.1

What's New

  • Bug Fixes

Component Updates and Bug Fixes

  • github.com/openziti/ziti: v1.3.0 -> v1.3.1
    • Issue #2682 - HA Controller panics when bootstrapping by setting the db variable in the configuration
    • Issue #2683 - Controller fails to save peer configuration on a fresh install
    • Issue #2684 - Controller emits duplicate cluster events on startup

Release 1.3.0

What's New

  • Router Data Model enabled by default
  • Bug fixes
  • Controller Health Check HA Update (from @nenkoru)

Router Data Model

As part of the controller HA work, a stripped down version of the data model can now be distributed to the routers, allowing routers to make some authorization/authentication decisions. This code has existed for some time, but after testing and validation, is now enabled by default.

It can still be disabled at the controller level using new configuration. Note that the router data model is required for HA functionality, so if the controller is running in HA mode, it cannot be disabled.

routerDataModel:
  # Controls whether routers are told to enable functionality dependent on the router data model
  # Defaults to true
  enabled: true 

  # How many model changes to buffer so that routers can be updated iteratively. If a router requests
  # data that's no longer available, it will receive the full data model
  logSize: 10000

HA Changes

Routers no longer require the ha: enabled flag be set in the configuration. Routers should work correctly whether connecting to HA or non-HA controllers.

NOTE: If the controller a router is connected changes modes, specifically if the controller goes from supporting the router data model to not, or vice-versa, the router will shutdown so that it can restart with the correct mode.

Controller Health Check HA Update

This feature was contributed by @nenkoru.

The controller health check can now optionally return information about raft and leadership when the /controller/raft path is provided.

$ curl -k https://localhost:1280/health-checks/controller/raft
{
    "data": {
        "checks": [
            {
                "healthy": true,
                "id": "bolt.read",
                "lastCheckDuration": "0s",
                "lastCheckTime": "2025-01-14T19:42:13Z"
            }
        ],
        "healthy": true
    },
    "meta": {},
    "raft": {
        "isLeader": true,
        "isRaftEnabled": true
    }
}

Note the raft section, which indicates if raft is enabled and if the queried controller is currently the leader. If the controller/raft path isn't present in the request, the result should be unchanged from previous releases.

When querying the controller/raft health, if raft is enabled but the controller is not the leader, the check will return an HTTP status of 429.

Component Updates and Bug Fixes

Release 1.2.2

What's New

  • Bug fixes and continuing progress on controller HA

Component Updates and Bug Fixes

Release 1.2.1

What's New

  • Bug fixes and continuing progress on controller HA

Component Updates and Bug Fixes

Release 1.2.0

What's New

  • New Router Metrics
  • Changes to identity connect status
  • HA Bootstrap Changes
  • Connect Events
  • SDK Events
  • Bug fixes and other HA work

New Router Metrics

The following new metrics are available for edge routers:

  1. edge.connect.failures - meter tracking failed connect attempts from sdks This tracks failures to not having a valid token. Other failures which happen earlier in the connection process may not be tracked here.
  2. edge.connect.successes - meter tracking successful connect attempts from sdks
  3. edge.disconnects - meter tracking disconnects of previously successfully connected sdks
  4. edge.connections - gauge tracking count of currently connected sdks

Identity Connect Status

Ziti tracks whether an identity is currently connected to an edge router. This is the hasEdgeRouterConnection field on Identity.

Identity connection status used to be driven off of heartbeats from the edge router. This feature doesn't work correctly when running with controller HA.

To address this, while also providing more operation insight, connect events were added (see below for more details on the events themselves).

The controller can be configured to use status from heartbeats, connect events or both. If both are used as source, then if either reports the identity as connected, then it will show as connected. This is intended for when you have a mix of routers and they don't all yet supported connect events.

The controller now also aims to be more precise about identity state. There is a new field on Identity: edgeRouterConnectionStatus. This field can have one of three values:

  • offline
  • online
  • unknown

If the identity is reported as connected to any ER, it will be marked as online. If the identity has been reported as connected, but the reporting ER is now offline, the identity may still be connected to the ER. While in this state it will be marked as 'unknown'. After a configurable interval, it will be marked as offline.

New controller config options:

identityStatusConfig:
  # valid values ['heartbeats', 'connect-events', 'hybrid']
  # defaults to 'hybrid' for now
  source: connect-events 

  # determines how often we scan for disconnected routers
  # defaults to 1 minute
  scanInterval: 1m

  # determines how long an identity will stay in unknown status before it's marked as offline
  # defaults to 5m
  unknownTimeout: 5m

HA Bootstrapping Changes

Previously bootstrapping the RAFT cluster and initializing the controller with a default administrator were separate operations. Now, the raft cluster will be bootstrapped whenever the controller is initialized.

The controller can be initialized as follows:

  1. Using ziti agent controller init
  2. Using ziti agent controller init-from-db
  3. Specifying a db: entry in the config file. This is equivalent to using ziti agent controller init-from-db.

Additionally:

  1. minClusterSize has been removed. The cluster will always be initialized with a size of 1.
  2. bootstrapMembers has been renamed to initialMembers. If initialMembers are specified, the bootstrapping controller will attempt to add them after bootstrap has been complete. If they are invalid they will be ignored. If they can't be reached (because they're not running yet), the controller will continue to retry until they are reached, or it is restarted.

Connect Events

These are events generated when a successful connection is made to a controller, from any of:

  1. Identity, using the REST API
  2. Router
  3. Controller (peer in an HA cluster)

They are also generated when an SDK connects to a router.

Controller Configuration

events:
  jsonLogger:
    subscriptions:
      - type: connect
    handler:
      type: file
      format: json
      path: /tmp/ziti-events.log

Router Configuration

connectEvents:
  # defaults to true. 
  # If set to false, minimal information about which identities are connected will still be 
  # sent to the controller, so the `edgeRouterConnectionStatus` field can be populated, 
  # but connect events will not be generated.
  enabled: true

  # The interval at which connect information will be batched up and sent to the controller. 
  # Shorter intervals will improve data resolution on the controller. Longer intervals could
  # more efficient.
  batchInterval: 3s

  # The router will also periodically sent the full state to the controller, to ensure that 
  # it's in sync. It will do this automatically if the router gets disconnected from the 
  # controller, or if the router is unable to send a connect events messages to the controller.
  # This controls how often the full state will be sent under ordinairy conditions
  fullSyncInterval: 5m

  # If enabled is set to true, the router will collect connect events and send them out
  # at the configured batch interval. If there are a huge number of connecting identities
  # or if the router is disconnected from the controller for a time, it may be unable to
  # send events. In order to prevent queued events from exhausting memory, a maximum 
  # queue size is configured. 
  # Default value 100,000
  maxQueuedEvents: 100000

Example Events

{
  "namespace": "connect",
  "src_type": "identity",
  "src_id": "ji2Rt8KJ4",
  "src_addr": "127.0.0.1:59336",
  "dst_id": "ctrl_client",
  "dst_addr": "localhost:1280/edge/management/v1/edge-routers/2L7NeVuGBU",
  "timestamp": "2024-10-02T12:17:39.501821249-04:00"
}

{
  "namespace": "connect",
  "src_type": "router",
  "src_id": "2L7NeVuGBU",
  "src_addr": "127.0.0.1:42702",
  "dst_id": "ctrl_client",
  "dst_addr": "127.0.0.1:6262",
  "timestamp": "2024-10-02T12:17:40.529865849-04:00"
}

{
  "namespace": "connect",
  "src_type": "peer",
  "src_id": "ctrl2",
  "src_addr": "127.0.0.1:40056",
  "dst_id": "ctrl1",
  "dst_addr": "127.0.0.1:6262",
  "timestamp": "2024-10-02T12:37:04.490859197-04:00"
}

SDK Events

Building off of the connect events, there are events generated when an identity/sdk comes online or goes offline.

events:
  jsonLogger:
    subscriptions:
      - type: sdk
    handler:
      type: file
      format: json
      path: /tmp/ziti-events.log

{
  "namespace": "sdk",
  "event_type" : "sdk-online",
  "identity_id": "ji2Rt8KJ4",
  "timestamp": "2024-10-02T12:17:39.501821249-04:00"
}

{
  "namespace": "sdk",
  "event_type" : "sdk-status-unknown",
  "identity_id": "ji2Rt8KJ4",
  "timestamp": "2024-10-02T12:17:40.501821249-04:00"
}

{
  "namespace": "sdk",
  "event_type" : "sdk-offline",
  "identity_id": "ji2Rt8KJ4",
  "timestamp": "2024-10-02T12:17:41.501821249-04:00"
}

Component Updates and Bug Fixes

Release 1.1.15

What's New

  • Bug fixes, enhancements and continuing progress on controller HA

Component Updates and Bug Fixes

Release 1.1.14

What's New

  • Bug fixes, enhancements and continuing progress on controller HA

Component Updates and Bug Fixes

Release 1.1.13

This release will not be promoted, as a test binary was unintentionally released in the release archives.

Release 1.1.12

What's New

  • Bug fixes, enhancements and continuing progress on controller HA
  • Data corruption Fix

Data Corruption Fix

Previous to version 1.1.12, the controller would not handle changes to the policy type of a service policy. Specifically if the type was changed from Bind -> Dial, or Dial -> Bind, a set of denormalized data would be left behind, leaving the permissions with the old policy type.

Example:

  1. Identity A has Bind access to service B via Bind service policy C.
  2. The policy type of service policy C is changed from Bind to Dial.
  3. The service list would now likely show that Identity A has Dial and Bind access to service B, instead of just Dial access.

Mitigation/Fixing Bad Data

If you encounter this problem, the easiest and safest way to solve the problem is to to delete and recreate the affected service policy.

If changing policy types is something you do on a regular basis, and can't upgrade to a version with the fix, you can work around the issue by deleting and recreating policies, instead of updating them.

If you're not sure if you have ever changed a policy type, there is a database integrity check tool which can be run which looks for data integrity errors. It is run against a running system.

Start the check using:

ziti fabric db start-check-integrity

This kicks off the operation in the background. The status of the check can be seen using:

ziti fabric db check-integrity-status

By default this is a read-only operation. If the read-only run reports errors, it can be run with the -f flag, which will have it try to fix errors. The data integrity errors caused by this bug should all be fixable by the integrity checker.

ziti fabric db start-check-integrity -f

WARNINGS:

  • Always make a database snapshot before running the integrity checker: ziti db fabric snapshot <optional path
  • The integrity checker can be very resource intensive, depending on the size of your data model. It is recommended that you run the integrity checker when the system is otherwise not busy.

Component Updates and Bug Fixes

Release 1.1.11

What's New

  • This release updates to Go v1.23
  • Updates to the latest version of golangci-lint, to allow it to work with the new version of Go
  • Linter fixes to address issues caught by updated linter

Release 1.1.10

What's New

  • Bug fixes, enhancements and continuing progress on controller HA

Component Updates and Bug Fixes

Release 1.1.9

What's New

  • Bug fixes, enhancements and continuing progress on controller HA
  • Includes a performance update (Issue #2340) which should improve connection ramp times. Previously circuits would start with a relatively low window size and ramp slowly. Circuits will now start with a large initial window size and scale back if they can't keep up
  • Added ziti ops verify-network. A command to aid when configuring the overlay network, this command will check config files for obvious mistakes
  • Added ziti ops verify-traffic. Another command to aid when configuring the overlay network, this command will ensure the overlay network is able to pass traffic

Component Updates and Bug Fixes

Release 1.1.8

What's New

  • Bug fixes, enhancements and continuing progress on controller HA

Component Updates and Bug Fixes

  • github.com/openziti/edge-api: v0.26.20 -> v0.26.23

    • Issue #120 - Add API for retrieving services referencing a config
    • Issue #121 - Add API for retrieving the set of attribute roles used by posture checks

  • github.com/openziti/sdk-golang: v0.23.38 -> v0.23.39

    • Issue #596 - SDK should submit selected config types to auth and service list APIs
    • Issue #593 - SDK Golang OIDC Get API Session Returns Wrong Value

  • github.com/openziti/storage: v0.2.47 -> v0.3.0

    • Issue #80 - Set indexes aren't cleaned up when referenced entities are deleted, only when they change
    • Issue #78 - Allow searching for things without case sensitivity

  • github.com/openziti/ziti: v1.1.7 -> v1.1.8

    • Issue #2121 - Use router data model for edge router tunnel
    • Issue #2245 - Add ability to retrieve a list of services that reference a config
    • Issue #2089 - Enhance Management API to list Posture Check Roles
    • Issue #2209 - /edge/v1/external-jwt-signers needs to be open
    • Issue #2010 - Add config information to router data model
    • Issue #1990 - Implement subscriber model for identity/service events in router
    • Issue #2240 - Secondary ext-jwt Auth Policy check incorrectly requires primary ext-jwt auth to be enabled

Release 1.1.7

What's New

  • Release actions fixes
  • Fix for a flaky acceptance test

Release 1.1.6

What's New

  • Trust Domain Configuration
  • Controller HA Beta 2

Trust Domain Configuration

OpenZiti controllers from this release forward will now require a trust domain to be configured. High Availability (HA) controllers already have this requirement. HA Controllers configure their trust domain via SPIFFE ids that are embedded in x509 certificates.

For feature parity, non-HA controllers will now have this same requirement. However, as re-issuing certificates is not always easily done. To help with the transition, non-HA controllers will have the ability to have their trust domain sourced from the controller configuration file through the root configuration value trustDomain. The configuration field which takes a string that must be URI hostname compatible (see: https://github.com/spiffe/spiffe/blob/main/standards/SPIFFE-ID.md). If this value is not defined, a trust domain will be generated from the root CA certificate of the controller.

For networks that will be deployed after this change, it is highly suggested that a SPIFFE id is added to certificates. The ziti pki create ... tooling supports the --spiffe-id option to help handle this scenario.

Generated Trust Domain Log Messages

The following log messages are examples of warnings produced when a controller is using a generated trust domain:

WARNING this environment is using a default generated trust domain [spiffe://d561decf63d229d66b07de627dbbde9e93228925], 
  it is recommended that a trust domain is specified in configuration via URI SANs or the 'trustDomain' field

WARNING this environment is using a default generated trust domain [spiffe://d561decf63d229d66b07de627dbbde9e93228925], 
  it is recommended that if network components have enrolled that the generated trust domain be added to the 
  configuration field 'additionalTrustDomains'

Trust domain resolution:

  • Non-HA controllers

    • Prefers SPIFFE ids in x509 certificate URI SANs, looking at the leaf up the signing chain
    • Regresses to trustDomain in the controller configuration file if not found
    • Regress to generating a trust domain from the server certificates root CA, if the above do not resolve

  • HA Controllers

    • Requires x509 SPIFFE ids in x509 certificate URI SANs

Additional Trust Domains

When moving between trust domains (i.e. from the default generated to a new named one), the controller supports having other trust domains. The trust domains do not replace certificate chain validation, which is still checked and enforced.

Additional trust domains are configured in the controller configuration file under the root field additionalTrustDomains. This field is an array of hostname safe strings.

The most common use case for this is field is if a network has issued certificates using the generated trust domain and now wants to transition to a explicitly defined one.

Controller HA Beta 2

This release can be run in HA mode. The code is still beta, as we're still finding and fixing bugs. Several bugs have been fixed since Beta 1 and c-based SDKs and tunnelers now work in HA mode. The smoketest can now be run with HA controllers and clients.

For more information:

Component Updates and Bug Fixes

  • github.com/openziti/storage: v0.2.45 -> v0.2.46

    • Issue #76 - Add support for non-boltz symbols to the the boltz stores

  • github.com/openziti/ziti: v1.1.5 -> v1.1.6

    • Issue #2171 - Routers should consider control channels unresponsive if they are not connected
    • Issue #2219 - Add inspection for router connections
    • Issue #2195 - cached data model file set to
    • Issue #2222 - Add way to get read-only status from cluster nodes
    • Issue #2191 - Change raft list cluster members element name from values to data to match rest of REST api
    • Issue #785 - ziti edge update service-policy to empty/no posture checks fails
    • Issue #2205 - Merge fabric and edge model code
    • Issue #2165 - Add network id

Release 1.1.5

What's New

  • Bug fixes

Component Updates and Bug Fixes

Release 1.1.4

What's New

  • Controller HA Beta 1
  • Bug fixes

Controller HA Beta 1

This release can be run in HA mode. The code is still beta, as we're still finding and fixing bugs. Several bugs have been fixed since Alpha 3 and c-based SDKs and tunnelers now work in HA mode. The smoketest can now be run with HA controllers and clients.

For more information:

Component Updates and Bug Fixes

Release 1.1.3

What's New

  • Sticky Terminator Selection
  • Linux and Docker deployments log formats no longer default to the simplified format option and now use logging library defaults: json for non-interactive, text for interactive.

NOTE: This release is the first since 1.0.0 to be marked promoted from pre-release. Be sure to check the release notes for the rest of the post-1.0.0 releases to get the full set of changes.

Stick Terminator Strategy

This release introduces a new terminator selection strategy sticky. On every dial it will return a token to the dialer, which represents the terminator used in the dial. This token maybe passed in on subsequent dials. If no token is passed in, the strategy will work the same as the smartrouting strategy. If a token is passed in, and the terminator is still valid, the same terminator will be used for the dial. A terminator will be consideder valid if it still exists and there are no terminators with a higher precedence.

This is currently only supported in the Go SDK.

Go SDK Example

ziti edge create service test --terminator-strategy sticky

	conn := clientContext.Dial("test")
	token := conn.Conn.GetStickinessToken()
	_ = conn.Close()

	dialOptions := &ziti.DialOptions{
		ConnectTimeout:  time.Second,
		StickinessToken: token,
	}
	conn = clientContext.DialWithOptions("test", dialOptions))
	nextToken := conn.Conn.GetStickinessToken()
	_ = conn.Close()

Component Updates and Bug Fixes

Release 1.1.2

What's New

  • Bug fixes and minor enhancements

Component Updates and Bug Fixes

  • github.com/openziti/sdk-golang: v0.23.32 -> v0.23.35
  • github.com/openziti/ziti: v1.1.1 -> v1.1.2
    • Issue #2032 - Auto CA Enrollment Fails w/ 400 Bad Request
    • Issue #2026 - Root Version Endpoint Handling 404s
    • Issue #2002 - JWKS endpoints may not refresh on new KID
    • Issue #2007 - Identities for edge routers with tunneling enabled sometimes show hasEdgeRouterConnection=false even though everything is OK
    • Issue #1983 - delete of non-existent entity causes panic when run on follower controller

Release 1.1.1

What's New

HA Alpha 3

This release can be run in HA mode. The code is still alpha, as we're still finding and fixing bugs.

For more information:

New Contributors

Thanks to new contributors

  • @Vrashabh-Sontakke

Component Updates and Bug Fixes

Release 1.1.0

What's New

  • HA Alpha2
  • Deployments Alpha
    • Linux packages provide systemd services for controller and router. Both depend on existing package openziti which provides the ziti command line tool.
      • openziti-controller provides ziti-controller.service
      • openziti-router provides ziti-router.service
    • Container images for controller and router now share the bootstrapping logic with the packages, so they support the same configuration options.

HA Alpha2

This release can be run in HA mode. The code is still alpha, so there are still some bugs and missing features, however basic functionality work with the exceptions noted. See the HA Documementation for instructions on setting up an HA cluster.

Known Issues

  • JWT Session exchange isn't working with Go SDK clients
    • This means Go clients will need to be restarted once their sessions expire
  • Service/service policy changes might not be reflected in routers
    • Changes to policy may not yet properly sync to the routers, causing unexpected behavior with ER/Ts running in HA mode

More information can be found on the HA Project Board

Component Updates and Bug Fixes

Release 1.0.0

About 1.0

What does marking OpenZiti as 1.0 mean?

Backwards Compatibility

We've guaranteed API stability for SDK clients for years and worked hard to ensure that routers and controllers would be backwards and forward compatible. However, we have had a variety of management API changes and CLI changes. For post 1.0 releases we expect to make additions to the APIs and CLI, but won't remove anything until it's been first marked as deprecated and then only with a major version bump.

Stability and Scale

Recent releases have seen additional testing using chaos testing techniques. These tests involve setting up relatively large scale environments, knocking out various components and then verifying that the network is able to return to a stable state. These test are run for hours to try and eliminate race conditions and distributed state machine problems.

OpenZiti is also being used as underlying infrastrcture for the zrok public service. Use of this network has grown quickly and proven that it's possible to build ziti native apps that can scale up.

Backward Incompatible Changes to pre-1.0 releases

Administrators no longer have access to dial/bind all services by default. See below for details.

What's New

  • Administrators no longer have access to dial/bind all services by default.
  • TLS Handshakes can now be rate limited in the controller
  • TLS Handshake timeouts can now be set on the controller when using ALPN
  • Bugfixes

DEFAULT Bind/Dial SERVICE PERMISSIONS FOR Admin IDENTITIES HAVE CHANGED

Admin identities were able to Dial and Bind all services regardless of the effective service policies prior to this release. This could lead to a confusing situation where a tunneler that was assuming an Admin identity would put itself into an infinite connect-loop when a service's host.v1 address overlapped with any addresses in its intercept configuration.

Please create service policies to grant Bind or Dial permissions to Admin identities as needed.

TLS Handshake

A TLS handhshake rate limiter can be enabled. This is useful in cases where there's a flood of TLS requests and the controller can't handle them all. It can get into a state where it can't respond to TLS handshakes quickly enough, so the clients time out. They then retry, adding to the the load. The controller ends up wasting time doing work that isn't use.

This uses the same rate limiting as the auth rate limiter.

Additionally the server side handshake timeout can now be configured.

Configuration:

tls: 
  handshakeTimeout: 15s

  rateLimiter:
    # if disabled, no tls handshake rate limiting with be enforced
    enabled: true
    # the smallest window size for tls handshakes
    minSize: 5
    # the largest allowed window size for tls handshakes
    maxSize: 5000
    # after how long to consider a handshake abandoned if neither success nor failure was reported
    timeout: 30s

New metrics:

  • tls_handshake_limiter.in_process - number of TLS handshakes in progress
  • tls_handshake_limiter.window_size - number of TLS handhshakes allowed concurrently
  • tls_handshake_limiter.work_timer - timer tracking how long TLS handshakes are taking

Component Updates and Bug Fixes

Reference in New Issue View Git Blame Copy Permalink