Fortio Dynamic Flags

Dynamic, thread-safe flag variables that can be modified at runtime through files, URL endpoint, or Kubernetes configmap changes.

See History section below.

This sounds crazy. Why?

File-based or command-line configuration can only be changed when a service restarts. Dynamic flags provide flexibility in normal operations and emergencies. Two examples:

• A new feature launches that you want to A/B test. You want to gradually enable it for a certain fraction of user requests (1%, 5%, 20%, 50%, 100%) without the need to restart servers.
• Your service is getting overloaded and you want to disable certain costly features. You can't afford restarting because you'd lose important capacity.

All of this can be done simultaneously across a whole shard of your services.

Features

• compatible with standard go flag package
• dynamic flag that are thread-safe and efficient
  • Dyn[T] generic, or
  • DynBool
  • DynInt64
  • DynFloat64
  • DynString
  • DynDuration
  • DynStringSlice
  • DynStringSet
  • DynJSON - a flag that takes an arbitrary JSON struct
• validator functions for each flag, allows the user to provide checks for newly set values
• notifier functions allow user code to be subscribed to flag changes
• Kubernetes ConfigMap watcher, see configmap/README.md.
• a HandlerFunc endpoint.ListFlags that allows for easy inspection of the service's runtime configuration
• a HandlerFunc endpoint.SetFlag that let's you update the flag values

Here's a teaser of the debug endpoint:

Examples

Declare a single flag.FlagSet in some public package (e.g. common.SharedFlagSet) that you'll use throughout your server or stick to flag.CommandLine default flagset for your binary.

Dynamic JSON flag with a validator and notifier

var (
  limitsConfigFlag = dflag.DynJSON(
    common.SharedFlagSet,
    "rate_limiting_config",
    &rateLimitConfig{ DefaultRate: 10, Policy: "allow"},
    "Config for service's rate limit",
  ).WithValidator(rateLimitConfigValidator).WithNotifier(onRateLimitChange)
)

This declares a JSON flag of type rateLimitConfig with a default value. Whenever the config changes (statically or dynamically) the rateLimitConfigValidator will be called. If it returns no errors, the flag will be updated and onRateLimitChange will be called with both old and new, allowing the rate-limit mechanism to re-tune.

Dynamic feature flags

var (
  featuresFlag = dflag.DynStringSlice(common.SharedFlagSet, "enabled_features", []string{"fast_index"}, "list of enabled feature markers")
)
...
func MyHandler(resp http.ResponseWriter, req *http.Request) {
   ...
   if existsInStringSlice("fast_index", featuresFlag.Get()) {
     doFastIndex(req)
   }
   ...
}

All access to featuresFlag, which is a []string flag, is synchronized across go-routines using atomic pointer swaps.

Library versus caller style

NEW:

// In the library "libfoo" package
var MyConfig = dflag.New("default value", "explanation of what that is for").WithValidator(myValidator)
// In the caller/users, bind to an actual flag:
dflag.Flag("foocfg", libfoo.MyConfig) // defines -foocfg flag

Complete example

See a http server complete example or the fortio.org/scli package for easy reuse/configuration.

History

This came from https://github.com/ldemailly/go-flagz, a fork of the code originally on https://github.com/mwitkow/go-flagz and https://github.com/improbable-eng/go-flagz with initial changes to get the go modules to work, reduce boiler plate needed for configmap watcher, avoid panic when there is extra whitespace, make the watcher work with regular files and relative paths and switched to standard golang flags.

And further changes, simplification, etc... as part of fortio.

Including rewrite and simplifications taking advantage of go 1.18 and newer generics support (use versions in fortio prior to 1.33 if you want to use the older per type implementation)

And now moved to a toplevel package in the fortio org.

For a similar project for JVM languages (Java, scala) see java-flagz

Thanks to @mwitkow for having created this originally.

Status

This code is production quality. It's been running happily in production in its earlier incarnation at Improbable for years and now everywhere fortio runs.

License

dflag (was go-flagz) is released under the Apache 2.0 license. See the LICENSE file for details.