Generators receive data (strings) from clients (via API) and automatically spawn workflows when number of calls exceed a threshold. Data is then available as an argument to the Colonies process.
Generators are a server module controlled by the COLONIES_MODULE_GENERATORS_ENABLED environment variable (default true). If the module is disabled, the generator API endpoints return a typed "module disabled" error.
The quickest way to get a server is the all-in-one container described in Deployment.md:
docker run -d --name colonies -p 50080:50080 \
-v colonies-data:/var/lib/postgresql/data \
colonyos/colonies-allinone:latestAlternatively, run a local development server against an existing Postgres with colonies server start --initdb --insecure.
colonies generator add --spec ./examples/generators/generator_workflow.json --name testgenerator --trigger 5Output:
INFO[0000] Generator added GeneratorID=f3a433d0a428ddd21fba2b82659db40dfc4e70771a29e2a19743ad80033749d7 GeneratorName=testgenerator Timeout=-1 Trigger=5
The generator can also take an optional --timeout flag (seconds). If set, a workflow is spawned after the timeout expires even if the number of packed calls has not reached the trigger threshold, as long as at least one call has been packed.
The workflow just echoes the args. It looks like this (see examples/generators/generator_workflow.json):
[
{
"nodename": "generator_example",
"funcname": "echo",
"args": [],
"conditions": {
"executortype": "cli",
"dependencies": []
}
}
]The workflows spawned by the generator target executors of type cli, for example one of the executors in colonyos/executors or one written against the Go SDK (see examples/fibonacci/solver/solver.go). For a quick demo, the Colonies CLI itself can act as an executor. Register and approve one (this uses the colony private key from the COLONIES_COLONY_PRVKEY environment variable):
colonies executor create --name generator_executor --type cli --approve --keypath /tmp/generator_prvkey --idpath /tmp/generator_executoridcolonies generator pack --generatorid f3a433d0a428ddd21fba2b82659db40dfc4e70771a29e2a19743ad80033749d7 --arg hello1
colonies generator pack --generatorid f3a433d0a428ddd21fba2b82659db40dfc4e70771a29e2a19743ad80033749d7 --arg hello2
colonies generator pack --generatorid f3a433d0a428ddd21fba2b82659db40dfc4e70771a29e2a19743ad80033749d7 --arg hello3
colonies generator pack --generatorid f3a433d0a428ddd21fba2b82659db40dfc4e70771a29e2a19743ad80033749d7 --arg hello4
colonies generator pack --generatorid f3a433d0a428ddd21fba2b82659db40dfc4e70771a29e2a19743ad80033749d7 --arg hello5Notice that a workflow is spawned after the last pack call, as number of pack calls > trigger. The spawned process is now waiting in the queue (colonies process psw) with the packed data as args. Pull and complete it with the CLI executor:
colonies process assign --prvkey $(cat /tmp/generator_prvkey) --timeout 10
colonies process close -p <processid> --prvkey $(cat /tmp/generator_prvkey)If we look up the process we get:
colonies process get --processid 3806424831e78001fd7157a387ca9ab414ef908f0649eeed7e9fee691438db01Output:
Process:
+--------------------+------------------------------------------------------------------+
| ID | 3806424831e78001fd7157a387ca9ab414ef908f0649eeed7e9fee691438db01 |
| IsAssigned | True |
| AssignedExecutorID | eeefa45d65b75c6ec3e11fedd2b120909607da830bade4f1953e55ccbad417c1 |
| State | Successful |
| Priority | 0 |
| SubmissionTime | 2022-08-23 22:19:14 |
| StartTime | 2022-08-23 22:19:14 |
| EndTime | 2022-08-23 22:19:14 |
| WaitDeadline | 0001-01-01 01:12:12 |
| ExecDeadline | 0001-01-01 01:12:12 |
| WaitingTime | 9.712ms |
| ProcessingTime | 12.305ms |
| Retries | 0 |
| ErrorMsg | |
+--------------------+------------------------------------------------------------------+
Function Specification:
+-------------+-----------------------------+
| Func | echo |
| Args | hello1 hello2 hello3 hello4 |
| | hello5 |
| KwArgs | None |
| MaxWaitTime | 0 |
| MaxExecTime | -1 |
| MaxRetries | 0 |
| Label | |
+-------------+-----------------------------+
Conditions:
+------------------+------+
| Colony | dev |
| ExecutorNames | None |
| ExecutorType | cli |
| Location | None |
| Dependencies | |
| Nodes | 0 |
| CPU | |
| Memory | |
| Processes | 0 |
| ProcessesPerNode | 0 |
| Storage | |
| Walltime | 0 |
| GPUName | |
| GPUs | 0 |
| GPUPerNode | 0 |
| GPUMemory | |
+------------------+------+
Attributes:
+------------------------------------------------------------------+--------+--------------------------------+------+
| ID | KEY | VALUE | TYPE |
+------------------------------------------------------------------+--------+--------------------------------+------+
| b4159b8813a657d20b88ad7231bc388cbc2b1e296e1c5bf02e28c44486ac95b1 | output | hello1 hello2 hello3 hello4 | Out |
| | | hel... | |
+------------------------------------------------------------------+--------+--------------------------------+------+