go-taskflow: A General-purpose Task-parallel Programming Framework with an integrated visualizer and profiler for Go

Go-Taskflow#

codecov Go Reference Go Report Card Mentioned in Awesome Go

A taskflow-like General-purpose Task-parallel Programming Framework with an integrated visualizer and profiler for Go, inspired by taskflow-cpp, with Go’s native capabilities and simplicity, suitable for complex dependency management in concurrent tasks.

Feature#

  • High extensibility: Easily extend the framework to adapt to various specific use cases.

  • Native Go’s concurrency model: Leverages Go’s goroutines to manage concurrent task execution effectively.

  • User-friendly programming interface: Simplify complex task dependency management using Go.

  • Static\Subflow\Conditional\Cyclic tasking: Define static tasks, condition nodes, nested subflows and cyclic flow to enhance modularity and programmability.

    Static Subflow Condition Cyclic
  • Priority Task Schedule: Define tasks’ priority, higher priority tasks will be scheduled first.

  • Built-in visualization & profiling tools: Generate visual representations of tasks and profile task execution performance using integrated tools, making debugging and optimization easier.

Use Cases#

  • Data Pipeline: Orchestrate data processing stages that have complex dependencies.

  • Workflow Automation: Define and run automation workflows where tasks have a clear sequence and dependency structure.

  • Parallel Tasking: Execute independent tasks concurrently to fully utilize CPU resources.

Example#

import latest version: go get -u github.com/noneback/go-taskflow

go-taskflow/examples/conditional/condition.go

Understand Condition Task Correctly#

Condition Node is special in taskflow-cpp. It not only enrolls in Condition Control but also in Looping.

Our repo keeps almost the same behavior. You should read ConditionTasking to avoid common pitfalls.

How to use visualize taskflow#

if err := gotaskflow.Visualize(tf, os.Stdout); err != nil {
        log.Fatal(err)
}

Visualize generates raw strings in dot format, use dot to draw a DAG svg.

dot

How to use profile taskflow#

if err :=exector.Profile(os.Stdout);err != nil {
        log.Fatal(err)
}

Profile generates raw strings in flamegraph format, use flamegraph to draw a flamegraph svg.

flg

What’s more#

Any Features Request or Discussions are all welcomed.