| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-04-27 | |||
| 12:08:19 | fried_rice | sean-k-mooney: Which of course means that those specs are dead in the water until nrp works in alloc cands. | |
| 12:08:21 | sean-k-mooney | fried_rice: yes i just find its eaiser to do that kind of thing when you have something concreate to test it with | |
| 12:28:46 | openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Fix the policy argument in server-group-create https://review.openstack.org/564699 | |
| 12:31:19 | leakypipes | sean-k-mooney: https://review.openstack.org/#/c/554529/ | |
| 12:32:01 | sean-k-mooney | leakypipes: oh cool i see its jsut pending a +w | |
| 12:38:40 | fried_rice | sean-k-mooney: tbc, that patch does *not* enable nrp in alloc cands. There's half a dozen patches on top of it that get closer. But the actual microversion patch to make it work through the API has not yet been proposed (to my knowledge). | |
| 12:39:41 | sean-k-mooney | fried_rice: ok im just reading through the patch now | |
| 12:40:01 | fried_rice | sean-k-mooney: Best of luck. | |
| 12:40:14 | fried_rice | sean-k-mooney: If you understand this sqla stuff, we could use your help :) | |
| 12:40:29 | kashyap | sean-k-mooney: When you can, want to ACK this: https://review.openstack.org/#/c/563926/ | |
| 12:40:38 | kashyap | (It's the Lift the restriction of choices for `cpu_model_extra_flags`) | |
| 12:40:59 | kashyap | sean-k-mooney: After discussion with QEMU / libvirt folks, I've lifted restriction for all three CPU modes | |
| 12:41:01 | sean-k-mooney | fried_rice: i understad some of it but leakypipes still runs rings around me when it comes to sql | |
| 12:41:26 | kashyap | (See the config file help text for a valid use case.) | |
| 12:41:49 | sean-k-mooney | kashyap: adding flag to host-passthrough may be unsafe | |
| 12:42:03 | kashyap | sean-k-mooney: I have a good counter-argument | |
| 12:42:25 | kashyap | sean-k-mooney: I present you ... this test case: https://kashyapc.fedorapeople.org/CPU-flags-and-host-passthrough/readme.txt | |
| 12:42:54 | sean-k-mooney | kashyap: you mean the fixed the fact kvm can crash internally when using nested virt and emulating cpu instruction not supported by the host | |
| 12:42:55 | kashyap | sean-k-mooney: In brief: Sometimes QEMU disables certain CPU features (like Intel's "invtsc" (the Invariant TSC) | |
| 12:43:17 | kashyap | In that case, you _do_ want to specify explicitly the CPU flag *even with* 'host-passthrough' | |
| 12:43:34 | kashyap | sean-k-mooney: No, a different case ^ | |
| 12:43:59 | fried_rice | stephenfin: Remember I was having trouble rebuilding my docs env under tox? | |
| 12:44:15 | kashyap | sean-k-mooney: Even if your baremetal host has 'invtsc', if you do 'host-passthrough' and check /proc/cpuinfo, you won't find it: | |
| 12:44:18 | kashyap | $ cat /proc/cpuinfo | grep nonstop_tsc | |
| 12:44:21 | kashyap | $ echo $? | |
| 12:44:23 | kashyap | 1 | |
| 12:44:44 | kashyap | sean-k-mooney: However, if you do specify the flag explicitly with 'host-passthrough', and again check /proc/cpuinfo in the guest, you _will_ find it. | |
| 12:44:48 | sean-k-mooney | kashyap: hehe yes that case makes sense. i just wanted to point out that kvm + nested virt + emulating some instruction not supported by the host cpu has been buggy in the past so we have to be careful what you add | |
| 12:44:55 | fried_rice | stephenfin: I dorked with it some more this morning and was able to get around it by adding --no-cache to the pip command (in the [testenv] section of tox.ini). | |
| 12:45:09 | kashyap | sean-k-mooney: Okido; Yeah, I see your other point | |
| 12:45:36 | kashyap | sean-k-mooney: That is strictly in the wheelhouse of the Operator; if they insist on shooting themselves in the foot, we can't help | |
| 12:46:13 | sean-k-mooney | kashyap: true but we can at least tell them the gun is loaded when we give it to them in docs :) | |
| 12:46:47 | kashyap | Haha | |
| 12:46:59 | kashyap | These gun metaphors make me twitch | |
| 12:47:24 | sean-k-mooney | ya i guess in current climate they can sometimes be in poor taste | |
| 12:47:25 | kashyap | sean-k-mooney: If you have a preferred wording, please note in the review | |
| 12:48:22 | sean-k-mooney | kashyap: well im conflited on that as personaly i consider it to be a kvm/kernel bug so nova should not have to tell you about it. | |
| 12:48:28 | kashyap | Yeah | |
| 12:48:37 | kashyap | Probably let's not confuse | |
| 12:48:51 | kashyap | The help text for that already requires undivided attention to read and apply it properly | |
| 12:49:12 | kashyap | We'll get to it when we get the first bug filed in that area | |
| 12:49:54 | sean-k-mooney | the old example of what used to trigger it was adding AVX on a cpu that did not have it caused a fallback to sse4 instruction in qemu but that fallback was buggy when used with kvm and nested virt turned on | |
| 12:50:33 | sean-k-mooney | i think that has been fix however. | |
| 12:51:56 | kashyap | Aside: What does AVX do? | |
| 12:52:15 | kashyap | Ah, found it: https://en.wikipedia.org/wiki/Advanced_Vector_Extensions | |
| 12:53:28 | sean-k-mooney | kashyap: yep its basicaly extra wide simd instructions. which is why you can emulate them with sse instrutions at a performace penalty but still better then not using avx at all | |
| 12:54:00 | kashyap | I see, thanks for the explanation | |
| 13:00:13 | sean-k-mooney | kashyap: out of interest does cpu model Haswell-noTSX-IBRS have meltdown/specter fixed | |
| 13:01:26 | sean-k-mooney | the fact that its IBRS and not noIBRS implies it doing speclitive execution | |
| 13:11:50 | mdbooth | jgriffith: https://bugs.launchpad.net/nova/+bug/1767363 | |
| 13:11:50 | openstack | Launchpad bug 1767363 in OpenStack Compute (nova) "Deleting 2 instances with a common multi-attached volume can leave the volume attached" [Undecided,New] | |
| 13:12:20 | kashyap | sean-k-mooney: Was AFK briefly | |
| 13:12:35 | kashyap | sean-k-mooney: Yes, and no | |
| 13:13:19 | sean-k-mooney | kashyap: yes to meltdown and not to specter because qemu/kvm/kernel is patch to prevent that? | |
| 13:13:35 | kashyap | Exactly | |
| 13:14:05 | sean-k-mooney | kashyap: cool then IBRS makes sense | |
| 13:14:07 | kashyap | sean-k-mooney: And more over: those -noIBRS models _doesn't_ include 'PCID' (some got confused) | |
| 13:16:01 | sean-k-mooney | kashyap: right well i guess it you dont do speculatve branch execution then you dont need pcid to create seperate user/kernel adress contextes in the guest to mitigate the security risk | |
| 13:16:43 | kashyap | sean-k-mooney: Hmm, didn't think that way; that's a good point. | |
| 13:23:26 | stephenfin | sean-k-mooney: Sure is. I tested on an AIO DevStack deployment with OVS and things get shunted to the correct NUMA node | |
| 13:23:43 | sean-k-mooney | stephenfin: cool | |
| 13:25:30 | openstackgerrit | Matt Riedemann proposed openstack/nova master: libvirt: Lift the restriction of choices for `cpu_model_extra_flags` https://review.openstack.org/563926 | |
| 13:30:34 | sean-k-mooney | finucannot: did you see my comment on https://review.openstack.org/#/c/564439/1 | |
| 13:31:16 | kashyap | mriedem: Nice of you to fix the nits yourself | |
| 13:31:21 | sean-k-mooney | finucannot: also cool i might see if i can test it with ovs-dpdk but it will be next week at the earliest before i do | |
| 13:33:07 | finucannot | sean-k-mooney: That would be great. I was planning to do that myself. I also want to do some multi-host testing (to properly validate the scheduler aspects of things) though live migration is off the cards, seeing as this requires CPU pinning | |
| 13:33:49 | finucannot | sean-k-mooney: Question though: both you and jaypipes keep using the term "network backend". What do mean? | |
| 13:33:57 | finucannot | e.g. https://review.openstack.org/#/c/564439/1/nova/objects/numa.py@164 | |
| 13:35:21 | sean-k-mooney | finucannot: when i was using "network backend" i was referning to ovs vs linux bridge vs vpp vs sriov ... | |
| 13:35:29 | leakypipes | bauzas: so I've yet to see a single operator respond with a custom scheduler filter being used in the openstack-operators survey on scheduler filters. | |
| 13:35:50 | finucannot | sean-k-mooney: So neutron agents? | |
| 13:36:08 | sean-k-mooney | finucannot: so neutron can only have one tunnel enpoint ip for the ovs agent but if you also ran linux bridge on the host it could have a different one | |
| 13:36:13 | sean-k-mooney | finucannot: ya basically | |
| 13:36:22 | bauzas | leakypipes: maybe they're shy ? :p | |
| 13:38:41 | finucannot | sean-k-mooney: Yeah, I think we need to settle on terminology here. So to start, we've got neutron agents. Theoretically you can run many of these on one host but we're going to consider that out of scope here, I guess | |
| 13:39:13 | sean-k-mooney | finucannot: normally you will have sriov + one other maxium per host | |
| 13:40:20 | sean-k-mooney | finucannot: you can have more but gernally running more then two agent does not by you anything usefull and is a pain to mainatin | |
| 13:43:20 | sean-k-mooney | leakypipes: there were a few responces on the openstack-dev thread but ya i guess either operators dont want to share or have not seen it | |
| 13:44:57 | sean-k-mooney | leakypipes: looking at the responces only the AggregateInstanceExtraSpecsFilter and AggregateImagePropertiesIsolation need complex metadata stored on the host aggreates which cant be offloaded to placement | |
| 13:45:27 | leakypipes | sean-k-mooney: and ComputeCapabilitiesFilter. | |
| 13:46:04 | leakypipes | sean-k-mooney: the AggregateInstanceExtraSpecsFilter and AggregateImagePropertiesIsolation stuff will be handled fully by superdan's placement-request-filter work. | |
| 13:46:19 | superdan | um | |
| 13:46:21 | superdan | fully? | |
| 13:46:26 | leakypipes | FULLY! | |
| 13:46:31 | leakypipes | because... superdan. | |
| 13:46:46 | sean-k-mooney | leakypipes: hum i assumed that we would be able to modle the compute capablities could be handeled by traits on the compute node RP but i admit i have not looked at what that does in a while | |
| 13:47:27 | superdan | the former is what you use for general flavor-to-aggregate mapping right? | |
| 13:47:27 | leakypipes | sean-k-mooney: it is this ludicrousness that is problematic to "port" to placement: https://github.com/openstack/nova/blob/master/nova/scheduler/filters/compute_capabilities_filter.py#L49-L56 | |
| 13:47:46 | leakypipes | superdan: yes. and the latter is tenant to aggregate mapping | |
| 13:48:05 | superdan | leakypipes: no, the latter is for image-based mapping | |
| 13:48:15 | superdan | leakypipes: there is a dedicated tenant filter for tenant-to-aggregate | |
| 13:48:28 | leakypipes | superdan: ack, apologies, yep. | |
| 13:48:48 | leakypipes | superdan: IsolatedHostsFilter? | |
| 13:49:02 | leakypipes | nope, that's the image one too | |
| 13:49:13 | sean-k-mooney | leakypipes: https://github.com/openstack/nova/blob/master/nova/scheduler/filters/aggregate_multitenancy_isolation.py | |
| 13:49:13 | leakypipes | AggregateMultiTenancyIsolation | |
| 13:49:15 | leakypipes | :0 | |
| 13:49:24 | superdan | right, that's the tenant one | |
| 13:49:25 | leakypipes | sean-k-mooney: just beat ya. :) | |
| 13:49:59 | leakypipes | sean-k-mooney: also, see this craziness here: https://github.com/openstack/nova/blob/master/nova/scheduler/filters/compute_capabilities_filter.py#L76-L107 | |