Re: [GSOC] Add SMT/HT awareness to DragonFlyBSD scheduler

[Mihai Carabas <mihai.carabas@xxxxxxxxx>]

--14dae93408b3deb93004bc7c15cd
Content-Type: text/plain; charset=ISO-8859-1

Hi again,

See my comments inline.

On Fri, Mar 30, 2012 at 10:27 PM, Tobias Weingartner <weingart@tepid.org>wrote:

> This is one of the better emails on this subject I've seen in a long time.
> Very nice to see the amount of work you have put into this so far.  If I
> may, I'd love to make a couple comments.  These are not meant to
> discourage you in any way, but comments from the peanut gallery.
>

Thanks for your review. There is no problem in pointing the problems. This
is the reason I had posted, to get more insight of this.

> After a brief documentation I can start working on the following goals:
> >
> > A. Detect how many threads have each core
> >  -  adding a generic way of detecting how many threads each core has
> (create
> > a map of siblings for example)
>
> Is this similar to what I was yaking about above?  :)
>

Yes. But here I included the start-up configuration of CPU info and detect
if the physical CPU has support for HT and propagate this to the scheduler
(as in the project description states: "ground work" ).




> >  -  guard all new added the information regarding HyperThreading. For
> this,
> > I will add an option (Enable HT) in the config file [5].
>
> I'm not 100% sure how the HT stuff affects things today.  It used to
> be that HT was
> definitely about shared resources, and running an "MP" style workload
> on an HT cpu
> was really not going to get you much.  I think that most cpu's today
> have much better
> HT implementations, and they can largely be treated as a full "MP"
> core.  Matt may
> have some better insights into this though.


 I think we should keep separate the code that is specific to HT. Here is a
matter of design decision.


 > B. Effective SMT aware scheduling task
> >    In order to achieve the final goal, I have to make a design of how
> > threads will get scheduled. I will have to treat different cases:
> > - "passive" scheduling - select a free physical CPU (if available) for
> the
> > next thread to schedule
> > - "active" scheduling - if a physical CPU becomes idle and on another CPU
> > are running two threads, migrate one of them on the idle physical cpu
> > - what task to get from a list - get the one that was scheduled on that
> > domain (physical CPU), no matter on what logical CPU has run before
> > - tasks should stick to the same domain (physical CPU)
>
> In general, I believe that "moving" threads to a new CPU should be
> "harder" than
> just "Oh, look, a free core, and we have more than 1 thread here,
> let's move" type
> of heuristic.  There should be some sort of hysteresis, or resistance
> to this type
> of movement, such that we don't end up flip-flopping all the time.
> Also, there are
> different reasons or goals to scheduling.  The effective/best use of
> all cpu cores
> is one of them.  IE: using all the cores as much as possible.  Another
> is to identify
> when cores/resources (such as memory, bus bandwidth, etc) are being too
> highly
> utilized, and then splitting load.  Conversely consolidating when
> things "shrink" in
> the utilization department.  Note, that the first, naiive method of
> scheduling will
> end up eating battery life on a laptop.  You're keeping all the cores
> as busy as you
> possibly can, while other methods may not be as optimal in terms of cycles
> available to compute things, they may significantly change the power
> profile of
> the machine.
>

I am aware of the power consumption when always scheduling on an idle
physical CPU, instead of a logical. But this is the goal of the project:
first use all the physical cores and then start using logical. The
flip-flap problem will be solved with one of the things I stated (each
thread should stick to its own domain/physical CPU).


I certainly don't expect anyone to solve these things all at once, but
> it would be
> really cool if these things are kept in mind, and the method used to solve
> one
> item, could be general enough that the basic infrastructure can be used to
> solve
> the same thing for another goal.  In other words, if I wish to create a
> "laptop"
> scheduler, I should be able to use the same data structures and methods as
> your
> "high performance" scheduler, but simply change the "logic" part to
> suit my goals.
> Does that make sense?
>
Yes it does. Actually as I was studying the linux kernel scheduling there
is a sched_smt_power_savings<http://lxr.linux.no/linux+v3.3/+code=sched_smt_power_savings>
options
that control the behaviour of the scheduler (it is configurable through
procfs from userspace).

Thanks for your feedback,
Mihai

--14dae93408b3deb93004bc7c15cd
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div class=3D"gmail_quote">Hi again,<div><br></div><div>See my comments inl=
ine.<br><br><div class=3D"gmail_quote"><div class=3D"im">On Fri, Mar 30, 20=
12 at 10:27 PM, Tobias Weingartner <span dir=3D"ltr">&lt;<a href=3D"mailto:=
weingart@tepid.org" target=3D"_blank">weingart@tepid.org</a>&gt;</span> wro=
te:<br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">This is one of the better emails on this sub=
ject I&#39;ve seen in a long time.<br>
Very nice to see the amount of work you have put into this so far. =A0If I<=
br>
may, I&#39;d love to make a couple comments. =A0These are not meant to<br>
discourage you in any way, but comments from the peanut gallery.<br></block=
quote><div><br></div></div><div>Thanks for your review. There is no problem=
 in pointing the problems. This is the reason I had posted, to get more ins=
ight of this.=A0</div>
<div class=3D"im">
<div><br></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex=
;border-left:1px #ccc solid;padding-left:1ex"><div>
&gt; After a brief documentation I can start working on the following goals=
:<br>
&gt;<br>
&gt; A. Detect how many threads have each core<br>
&gt; =A0- =A0adding a generic way of detecting how many threads each core h=
as (create<br>
&gt; a map of siblings for example)<br>
<br>
</div>Is this similar to what I was yaking about above? =A0:)<br></blockquo=
te><div>=A0</div></div><div>Yes. But here I included the start-up configura=
tion of CPU info and detect if the physical CPU has support for HT and prop=
agate this to the scheduler (as in the project description states: &quot;<s=
pan style=3D"line-height:16px;font-family:Tahoma,sans-serif">ground work&qu=
ot;</span>=A0).</div>
<div class=3D"im">
<div><br></div><div><br></div><div>=A0</div><blockquote class=3D"gmail_quot=
e" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">=
<div>&gt; =A0- =A0guard all new added the information regarding HyperThread=
ing. For this,<br>


&gt; I will add an option (Enable HT) in the config file [5].<br>
<br>
</div>I&#39;m not 100% sure how the HT stuff affects things today. =A0It us=
ed to<br>
be that HT was<br>
definitely about shared resources, and running an &quot;MP&quot; style work=
load<br>
on an HT cpu<br>
was really not going to get you much. =A0I think that most cpu&#39;s today<=
br>
have much better<br>
HT implementations, and they can largely be treated as a full &quot;MP&quot=
;<br>
core. =A0Matt may<br>
have some better insights into this though.</blockquote><div><br></div></di=
v><div>=A0I think we should keep separate the code that is specific to HT. =
Here is a matter of design decision.</div><div><div class=3D"h5"><div><br>
</div><div><br></div><blockquote class=3D"gmail_quote" style=3D"margin:0 0 =
0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
&gt; B. Effective SMT aware scheduling task<br>
&gt; =A0 =A0In order to achieve the final goal, I have to make a design of =
how<br>
&gt; threads will get scheduled. I will have to treat different cases:<br>
&gt; - &quot;passive&quot; scheduling - select a free physical CPU (if avai=
lable) for the<br>
&gt; next thread to schedule<br>
&gt; - &quot;active&quot; scheduling - if a physical CPU becomes idle and o=
n another CPU<br>
&gt; are running two threads, migrate one of them on the idle physical cpu<=
br>
&gt; - what task to get from a list - get the one that was scheduled on tha=
t<br>
&gt; domain (physical CPU), no matter on what logical CPU has run before<br=
>
&gt; - tasks should stick to the same domain (physical CPU)<br>
<br>
</div>In general, I believe that &quot;moving&quot; threads to a new CPU sh=
ould be &quot;harder&quot; than<br>
just &quot;Oh, look, a free core, and we have more than 1 thread here,<br>
let&#39;s move&quot; type<br>
of heuristic. =A0There should be some sort of hysteresis, or resistance<br>
to this type<br>
of movement, such that we don&#39;t end up flip-flopping all the time.<br>
Also, there are<br>
different reasons or goals to scheduling. =A0The effective/best use of<br>
all cpu cores<br>
is one of them. =A0IE: using all the cores as much as possible. =A0Another<=
br>
is to identify<br>
when cores/resources (such as memory, bus bandwidth, etc) are being too hig=
hly<br>
utilized, and then splitting load. =A0Conversely consolidating when<br>
things &quot;shrink&quot; in<br>
the utilization department. =A0Note, that the first, naiive method of<br>
scheduling will<br>
end up eating battery life on a laptop. =A0You&#39;re keeping all the cores=
<br>
as busy as you<br>
possibly can, while other methods may not be as optimal in terms of cycles<=
br>
available to compute things, they may significantly change the power profil=
e of<br>
the machine.<br></blockquote><div><br></div></div></div><div>I am aware of =
the power consumption when always scheduling on an idle physical CPU, inste=
ad of a logical. But this is the goal of the project: first use all the phy=
sical cores and then start using logical. The flip-flap problem will be sol=
ved with one of the things I stated (each thread should stick to its own do=
main/physical CPU).</div>
<div class=3D"im">
<div><br></div><div><br></div><blockquote class=3D"gmail_quote" style=3D"ma=
rgin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I certainly don&#39;t expect anyone to solve these things all at once, but<=
br>
it would be<br>
really cool if these things are kept in mind, and the method used to solve =
one<br>
item, could be general enough that the basic infrastructure can be used to =
solve<br>
the same thing for another goal. =A0In other words, if I wish to create a &=
quot;laptop&quot;<br>
scheduler, I should be able to use the same data structures and methods as =
your<br>
&quot;high performance&quot; scheduler, but simply change the &quot;logic&q=
uot; part to<br>
suit my goals.<br>
Does that make sense?<br></blockquote></div><div>Yes it does. Actually as I=
 was studying the linux kernel scheduling there is a=A0<a href=3D"http://lx=
r.linux.no/linux+v3.3/+code=3Dsched_smt_power_savings" style target=3D"_bla=
nk">sched_smt_power_savings</a>=A0options that control the behaviour of the=
 scheduler (it is configurable through procfs from userspace).=A0</div>

<div><br></div><div>Thanks for your feedback,</div><div>Mihai</div></div></=
div>
</div><br>

--14dae93408b3deb93004bc7c15cd--