Posted	Nick	Remark
#openstack-nova - 2018-03-15
22:02:42	mriedem	having a blueprint tag on it makes it more awkward to backport
22:02:48	dansmith	well,
22:02:50	mriedem	i don't have the context on the thing,
22:02:51	dansmith	it's not a bug in nova
22:02:56	mriedem	just saw 'meltdown'
22:03:05	dansmith	it's a bug in .. I dunno.. the whole arrangement of things
22:03:17	mriedem	in computers?
22:03:22	mriedem	just say computers
22:03:31	dansmith	nova doesn't let you specify a thing that only the operator can know, and is needed to mitigate performance loss after applying the meltdown fixes
22:03:35	dansmith	it's completely a nova feature,
22:03:55	dansmith	but operators will need it to fix a bug or regressed behavior in lots of other things (kernel, qemu, etc)
22:04:16	mriedem	hmm, this reminds me,
22:04:47	mriedem	stable/ocata and stable/pike started crapping recently (right before the ptg) because of qemu / libvirt package updates for CVEs related to meltdown and spectre in the UCA
22:04:49	dansmith	so I guess I'd leave it as a bp and just call it out in the backport as why we're making an exception
22:04:58	mriedem	mnaser: ^
22:05:01	mriedem	you might be interested in that
22:05:16	mriedem	https://review.openstack.org/#/c/534384/
22:06:14	mriedem	mnaser had to patch devstack on ocata and pike specifically to wait longer for nova-compute to start up b/c libvirt startup was going from like 10 seconds to a minute
22:06:18	mriedem	dansmith: ^ sound like the same issue?
22:06:36	dansmith	libvirt startup .. meaning a guest?
22:06:46	mriedem	no, libvirtd
22:06:52	mriedem	or the connection to libvirt
22:07:00	dansmith	mm, I dunno, that's a question for kashyap I think
22:07:08	dansmith	maybe if the fixes have been applied on the host,
22:07:16	dansmith	but pcid isn't passed it makes our guests just super slow for some things
22:07:16	mriedem	https://review.openstack.org/#/c/547676/
22:07:18	dansmith	so it could be
22:07:29	mriedem	https://launchpad.net/~ubuntu-cloud-archive/+archive/ubuntu/ocata-staging/+sourcepub/8774271/+listing-archive-extra
22:07:40	mriedem	the libvirt cve is i nthere
22:07:51	dansmith	I mean, that's what the assertion is about the problem we're trying to solve
22:07:56	dansmith	that _some_ things are like 30+% slower after the fixes
22:08:36	mnaser	so the reason why it affected ocata and pike was because first of all devstack didnt use pike uca in pike, it used ocata uca
22:08:48	mnaser	and the older release of uca didn't have this feature where it could ask qemu what cpu features it supported
22:09:06	mnaser	so on libvirt start up with older libvirt instances, it would boot up a very small tiny qemu instance to probe it for things like cpu features etc
22:09:14	dansmith	dear god
22:09:15	mnaser	which would make the start up super sloooow
22:09:56	dansmith	so that's unrelated I believe, unless the need for the new cpu feature discovery was because of the meltdown patch,
22:10:18	dansmith	but I expect kashyap's fix won't solve that as it's a qemu-missing-a-thing problem
22:10:24	mriedem	ok
22:21:59	mriedem	dansmith: if you guys have vgpu people internally besides bauzas that can take a look at this https://bugs.launchpad.net/nova/+bug/1752463
22:22:00	openstack	Launchpad bug 1752463 in OpenStack Compute (nova) "Attaching virtual GPU devices to guests in nova" [Medium,Incomplete]
22:26:30	melwitt	mriedem, dansmith: so we're agreed to approve https://blueprints.launchpad.net/nova/+spec/libvirt-cpu-model-extra-flags as a bp and we'll backport it, calling out in the backport why we're making an exception and backporting bp work. is that right?
22:26:54	dansmith	melwitt: I'm cool with that yeah
22:27:51	dansmith	mriedem: maybe, but that seems pretty low priority and can just wait for him to be back right?
22:28:24	mriedem	dansmith: sure
22:37:01	melwitt	mriedem: are you cool with the cpu model flags bp and the idea of backporting it?
22:37:51	mriedem	i'm not crazy about it being a bp if we know we're going to backport it
22:38:04	mriedem	i'd rather pretend it's not and say it's a bug fix
22:39:19	mriedem	https://review.openstack.org/#/c/534384/5//COMMIT_MSG@28 definitely makes it sound feature-y too though
22:39:30	mriedem	"besides the fix, you can also configure nova for other goodies!"
22:39:56	melwitt	heh. yeah. this is definitely an odd case
22:40:55	mriedem	now, if pcid as the extra flag is the only thing you'd care about if you only care about the fix,
22:41:24	mriedem	then this could be a [workarounds] option with a boolean to set pcid or not,
22:41:27	mriedem	and backport that
22:41:43	mriedem	then the free-form wild west extra flags thing that is more of a feature could only be master
22:41:51	mriedem	and we'd deprecate the workaround in master
22:42:00	mriedem	dansmith: ^ what do you think about that?
22:44:26	dansmith	I dunno, it seems like too much concern over a relatively minor thing
22:44:34	dansmith	and I kinda expect that we'll backport the main thing ourselves anywaym
22:44:46	dansmith	just so that people with queens and people with rocky don't need a different fix for this
22:44:55	dansmith	but if that's how it has to go, then okay
22:46:21	mriedem	i'm not the ptl, not my say, but i'd be more comfortable with that
22:46:34	dansmith	I guess my point is,
22:46:49	dansmith	we're going to backport the meat of the feature anyway, because we need it for the workaround
22:47:07	dansmith	and constraining the workaround to one flag doesn't really do anything to make it safer,
22:47:19	dansmith	unless the operator is going to just do crazy stuff with it (not sure what that would be even),
22:47:33	mriedem	^ is what i'm worried about, they try the 'other use cases' in stable,
22:47:39	mriedem	and are all of a sudden reporting bugs about it
22:47:51	dansmith	but if they're going to do that,
22:47:52	dansmith	they'll do it with rocky when it's out
22:48:13	dansmith	if we have a workaround, then people have to set that to make the thing work, and then when they roll to rocky (or S) and performance regresses for some reason they forgot about,
22:48:31	dansmith	there will be bugs where we tell them that that workaround was a one-time deal and now they have to set pcid in this other field
22:48:44	dansmith	just seems like it's too much monkeywork so we don't have a blueprint on a backport,
22:48:54	mriedem	well, we'd use the workaround if it were set while it's deprecated, but we'd mark it deprecated to drop it later, but you'd have rocky to transition your tooling
22:48:56	dansmith	when we can clearly point to not-our-fault reasons why this is needed and not high risk
22:49:15	dansmith	yeah, it just doesn't seem worth the pain to me
22:49:50	mriedem	precedent is another concern for me here,
22:49:57	mriedem	true it's a no-op by default,
22:49:59	cfriesen	dansmith: I agree with you, but I also suspect someone is going to try enabling pdpe1gb or vmx on queens.
22:50:08	mriedem	but then people could use that justification against us later
22:50:12	dansmith	cfriesen: so?
22:50:23	melwitt	the workaround idea seems good until we get to the point where people would have to go from workaround -> full feature to not regress performance
22:50:33	cfriesen	dansmith: so mriedem's point about it likely causing more bug reports against stable
22:50:49	mriedem	melwitt: it's just updating your configs in rocky
22:50:51	dansmith	cfriesen: but those bug reports will be against whatever release we allow the full feature in
22:50:54	mriedem	to not use the deprecated option
22:51:10	dansmith	mriedem: every time we do that it's upgrade pain and it just seems like we have too much of that already
22:51:25	cfriesen	dansmith: not necessarily, nested vmx for example is buggy in older systems but may be stable by the time rocky is released
22:51:26	melwitt	mriedem: but wouldn't you also have to add "pcid" to your rocky config elsewhere then?
22:51:27	dansmith	I get the precedent thing
22:51:42	mriedem	melwitt: yes, you'd do it on the new option
22:51:44	dansmith	cfriesen: that's a super weak reason :)
22:52:02	dansmith	cfriesen: because (a) I don't think it will be and (b) people can already do crazy stuff with lots of options
22:52:24	dansmith	cfriesen: you could work around this by just defining your own cpu models in the libvirt system xml, and you can do way crazier stuff to your systems that way
22:52:55	cfriesen	dansmith: true enough. for what it's worth, adding pdpe1gb seems to be generally fine.
22:53:06	dansmith	heh
22:53:07	cfriesen	for the subset of intel hosts I've dealt with anyway
22:54:27	melwitt	we can just slap "experimental" on it on the backport, use at your own risk
22:54:45	dansmith	it's not experimental but sure :)
22:55:32	dansmith	melwitt: I need to be done for the day anyway,

Earlier   Later