Design Principles¶

This section contains a number of principles that should be followed when extending Krake. The principles are very similar to the Kubernetes design principles.

API¶

Control Logic¶

Functionality must be level-based, meaning the system must operate correctly given the desired state and the current/observed state, regardless of how many intermediate state updates may have been missed. Event/Edge-triggered behavior must be just an optimization.

Note

There should be a CAP-like theorem regarding the trade-offs between driving control loops via polling or events about simultaneously achieving high performance, reliability, and simplicity – pick any two.
Assume an open world: continually verify assumptions and gracefully adapt to external events and/or actors.

Tip

For example, Krake allows users to kill Kubernetes resources under control of a Kubernetes application controller; the controller just replaces the killed resource.
Do not assume any state transition or state that cannot be determined by observation.
Do not assume a component’s decisions will not be overridden or rejected, nor for the component to always understand why.

Tip

For example, etcd may reject writes. The scheduler may not be able to schedule applications. A Kubernetes cluster may reject requests.
Retry, but back off and/or make alternative decisions.
Components should be self-healing.

Tip

For example, if some state must be kept, e.g. cached, the content needs to be periodically refreshed, so that if an item does get incorrectly stored or a deletion event is missed, the kept state will be soon synchronized, ideally on timescales that are shorter than what will attract attention from humans.
Component behavior should degrade gracefully. Actions should be prioritized such that the most important activities can continue to function even when overloaded and/or in states of partial failure.

Architecture¶

Only the API server communicate with etcd, and no other components, e.g. scheduler, garbage collector, etc.
Components should continue to do what they were last told in the absence of new instructions, e.g. due to network partition or component outage.
All components should keep all relevant state in memory all the time. The API server writes through to etcd, other components write through to the API server, and they watch for updates made by other clients.
Watch is preferred over polling.

Extensibility¶

All components should be replaceable. This means there is no strong coupling between components.

Tip

For example, the different scheduler should be usable without any changes in another component.
Krake is extended with new technologies/platforms by adding new APIs.

Availability¶

Note

High-availability (HA) is about removing single point of failure (SPOF).

HA is achieved by service replication.

Todo

It needs to be decided on which level replication is introduced.

Coarse grained: Replicate “Krake master” with all included components, e.g. API server, controllers etc.
Fine grained: Replicate single components. If a component is stateful – relevant state should be kept in memory as stated in section Architecture – the components should follow an active-passive principle where only one replica of a component is active at the same time. A etcd lease may be a good option for this but only the API should have direct access to etcd. A solution for this would be to introduce special API endpoints for electing a leader across multiple replicas.

Development¶

Self-hosting of all components is the goal.
Use standard tooling and de facto standards of the Python ecosystem.
Keep dependencies as small as possible, but do not reinvent the wheel.