Small Task Packing in the big.LITTLE MP Reference Patch Set

What is small task packing?
----
Simply that the scheduler will fit as many small tasks on a single CPU
as possible before using other CPUs. A small task is defined as one
whose tracked load is less than 90% of a NICE_0 task. This is a change
from the usual behavior since the scheduler will normally use an idle
CPU for a waking task unless that task is considered cache hot.


How is it implemented?
----
Since all small tasks must wake up relatively frequently, the main
requirement for packing small tasks is to select a partly-busy CPU when
waking rather than looking for an idle CPU. We use the tracked load of
the CPU runqueue to determine how heavily loaded each CPU is and the
tracked load of the task to determine if it will fit on the CPU. We
always start with the lowest-numbered CPU in a sched domain and stop
looking when we find a CPU with enough space for the task.

Some further tweaks are necessary to suppress load balancing when the
CPU is not fully loaded, otherwise the scheduler attempts to spread
tasks evenly across the domain.


How does it interact with the HMP patches?
----
Firstly, we only enable packing on the little domain. The intent is that
the big domain is intended to spread tasks amongst the available CPUs
one-task-per-CPU. The little domain however is attempting to use as
little power as possible while servicing its tasks.

Secondly, since we offload big tasks onto little CPUs in order to try
to devote one CPU to each task, we have a threshold above which we do
not try to pack a task and instead will select an idle CPU if possible.
This maintains maximum forward progress for busy tasks temporarily
demoted from big CPUs.


Can the behaviour be tuned?
----
Yes, the load level of a 'full' CPU can be easily modified in the source
and is exposed through sysfs as /sys/kernel/hmp/packing_limit to be
changed at runtime. The presence of the packing behaviour is controlled
by CONFIG_SCHED_HMP_LITTLE_PACKING and can be disabled at run-time
using /sys/kernel/hmp/packing_enable.
The definition of a small task is hard coded as 90% of NICE_0_LOAD
and cannot be modified at run time.


Why do I need to tune it?
----
The optimal configuration is likely to be different depending upon the
design and manufacturing of your SoC.

In the main, there are two system effects from enabling small task
packing.

1. CPU operating point may increase
2. wakeup latency of tasks may be increased

There are also likely to be secondary effects from loading one CPU
rather than spreading tasks.

Note that all of these system effects are dependent upon the workload
under consideration.


CPU Operating Point
----
The primary impact of loading one CPU with a number of light tasks is to
increase the compute requirement of that CPU since it is no longer idle
as often. Increased compute requirement causes an increase in the
frequency of the CPU through CPUfreq.

Consider this example:
We have a system with 3 CPUs which can operate at any frequency between
350MHz and 1GHz. The system has 6 tasks which would each produce 10%
load at 1GHz. The scheduler has frequency-invariant load scaling
enabled. Our DVFS governor aims for 80% utilization at the chosen
frequency.

Without task packing, these tasks will be spread out amongst all CPUs
such that each has 2. This will produce roughly 20% system load, and
the frequency of the package will remain at 350MHz.

With task packing set to the default packing_limit, all of these tasks
will sit on one CPU and require a package frequency of ~750MHz to reach
80% utilization. (0.75 = 0.6 * 0.8).

When a package operates on a single frequency domain, all CPUs in that
package share frequency and voltage.

Depending upon the SoC implementation there can be a significant amount
of energy lost to leakage from idle CPUs. The decision about how
loaded a CPU must be to be considered 'full' is therefore controllable
through sysfs (sys/kernel/hmp/packing_limit) and directly in the code.

Continuing the example, lets set packing_limit to 450 which means we
will pack tasks until the total load of all running tasks >= 450. In
practise, this is very similar to a 55% idle 1Ghz CPU.

Now we are only able to place 4 tasks on CPU0, and two will overflow
onto CPU1. CPU0 will have a load of 40% and CPU1 will have a load of
20%. In order to still hit 80% utilization, CPU0 now only needs to
operate at (0.4*0.8=0.32) 320MHz, which means that the lowest operating
point will be selected, the same as in the non-packing case, except that
now CPU2 is no longer needed and can be power-gated.

In order to use less energy, the saving from power-gating CPU2 must be
more than the energy spent running CPU0 for the extra cycles. This
depends upon the SoC implementation.

This is obviously a contrived example requiring all the tasks to
be runnable at the same time, but it illustrates the point.


Wakeup Latency
----
This is an unavoidable consequence of trying to pack tasks together
rather than giving them a CPU each. If you cannot find an acceptable
level of wakeup latency, you should turn packing off.

Cyclictest is a good test application for determining the added latency
when configuring packing.


Why is it turned off for the VersatileExpress V2P_CA15A7 CoreTile?
----
Simply, this core tile only has power gating for the whole A7 package.
When small task packing is enabled, all our low-energy use cases
normally fit onto one A7 CPU. We therefore end up with 2 mostly-idle
CPUs and one mostly-busy CPU. This decreases the amount of time
available where the whole package is idle and can be turned off.

