| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-03-20 | |||
| 20:11:34 | sean-k-mooney | mriedem: for https://review.openstack.org/554591 are the network-vif-deleted event generated by neutron port being deleted as part of instance deleteion or are they neutron vif unplugged events form the ports being removed from ovs? | |
| 20:11:40 | efried | mriedem: That would be lovely. Are we at that point yet? | |
| 20:11:52 | mriedem | efried: i'm ok with that if melwitt is | |
| 20:12:17 | mriedem | sean-k-mooney: the former | |
| 20:12:29 | melwitt | mriedem, efried: sounds fine to me | |
| 20:12:52 | efried | Cool beans. edmondsw esberglu ^. | |
| 20:12:52 | sean-k-mooney | mriedem: ok then ya that makes sense to me. i just was not sure where the event was comming from | |
| 20:13:26 | efried | mriedem, melwitt: But for this release, since we're already here, use the bp/spec that's proposed, yah? | |
| 20:13:39 | mriedem | shrug, i personally don't care for the wishlist | |
| 20:13:53 | mriedem | because it just seems really random | |
| 20:13:55 | melwitt | is the proposed spec all just feature parity stuff? | |
| 20:14:05 | mriedem | and i see adding support for hot plugging vifs was added after PS3 | |
| 20:14:05 | efried | melwitt: yes | |
| 20:14:05 | edmondsw | I'm happy to abandon this and start using specless blueprints this release | |
| 20:14:13 | efried | ++ | |
| 20:14:31 | mriedem | there are no actual design details in the spec, so it's not really useful imo | |
| 20:14:34 | sean-k-mooney | mriedem: so just to follow on from that, does that mean if i delete a neutron port that is bound to an instance that automatically results in a notifcation to nova to call detach interface on the virt driver? | |
| 20:14:36 | mriedem | so specless bp is fine | |
| 20:14:36 | edmondsw | yep | |
| 20:14:47 | melwitt | sounds reasonable to me | |
| 20:14:48 | mriedem | sean-k-mooney: yup | |
| 20:15:12 | efried | Sweet. Swat that paperwork down! | |
| 20:15:51 | sean-k-mooney | mriedem: hum ok is that documented anywhere. it kind of makes sense i just would not have expected neutron to allow you to delete teh port in that case | |
| 20:16:04 | edmondsw | mriedem to the evacuate question... I'm honestly not sure if there's anything more to it than a) supporting moving to another host in general and b) updating the support matrix | |
| 20:16:21 | jaypipes | dansmith: will try to answer it first thing in the morning... been a day. having a beer. | |
| 20:16:34 | dansmith | jaypipes: oh you said you were getting to work, so I figured you were around | |
| 20:16:54 | dansmith | mah bad, tomorrow is fine | |
| 20:17:57 | melwitt | getting to work cleaning dog poodoo | |
| 20:18:07 | mriedem | edmondsw: someone on your dev or qa team could actually test it first | |
| 20:18:11 | mriedem | to flush out any obvious problems | |
| 20:18:21 | edmondsw | mriedem oh definitely | |
| 20:18:29 | edmondsw | was just talking from a code perspective | |
| 20:18:58 | mriedem | sean-k-mooney: good question; i didn't trace the requests in the tempest job to see if the port was actually detached before it was deleted, or what the order was there, | |
| 20:19:15 | mriedem | sean-k-mooney: should be pretty easy to find out - create a server from a pre-existing port and try to delete the port | |
| 20:19:24 | mriedem | i don't have a devstack handy | |
| 20:19:32 | mriedem | i know you can't delete an attached volume | |
| 20:19:48 | sean-k-mooney | mriedem: ill give it a try i think i have an env running | |
| 20:23:26 | openstackgerrit | Eric Berglund proposed openstack/nova master: DNM: EXPERIMENTAL: Set proc_units_factor to 0.1 in VMBuilder init https://review.openstack.org/554688 | |
| 20:23:49 | sean-k-mooney | mriedem: so ya i can create the port, boot with it then delete it and the interface gets detatched from the running vm | |
| 20:23:59 | sean-k-mooney | mriedem: im just going to check the xml to confirm | |
| 20:24:35 | sean-k-mooney | mriedem: yep the vm no has no nics | |
| 20:25:08 | sean-k-mooney | i wonder if the revers works. can i create a neutron port and then set its owner to a vm and trigger an attache | |
| 20:27:04 | cfriesen | So in the discussion around https://review.openstack.org/#/c/552924/ something interesting has come up...currently vcpus and 4KB pages can be consumed on a compute-node bases (where it floats across the whole compute node) or on a numa-node basis (where it's constrained to a single numa node) depending on whether the instance has a numa topology or not. (https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver. | |
| 20:27:06 | cfriesen | py#L4435) This makes it hard to reason about resource tracking. | |
| 20:30:54 | mriedem | sean-k-mooney: no the reverse doesn't work, | |
| 20:31:05 | mriedem | not might get a network-vif-plugged event but it won't know what to do with it and just log a warning | |
| 20:31:10 | mriedem | *nova | |
| 20:31:33 | sean-k-mooney | mriedem: ya just tried it. | |
| 20:31:40 | mriedem | the fact you can delete a port while it's attached is pretty scary | |
| 20:31:43 | sean-k-mooney | i create another port and did openstack port set --device 55a08a3b-fd27-495f-85da-39eedd5fde9c --device-owner "compute:nova" --host "ubuntu" myport | |
| 20:32:01 | sean-k-mooney | it ended up binding the port on the host correctly | |
| 20:32:18 | sean-k-mooney | im going to do a hard reboot but i think it will then update the xml correctly | |
| 20:32:31 | mriedem | in the compute api we check instance states when detaching a port, | |
| 20:32:34 | edleafe | dansmith: sorry, was away. Yes, the idea for member_of was "member of at least one of these". The typical use case would be only 1 agg. | |
| 20:32:39 | mriedem | but the networking api wouldn't have that | |
| 20:32:47 | mriedem | so you could detach ports by deleting them while an instance is migrating | |
| 20:33:14 | mriedem | which seems, not good | |
| 20:34:02 | cfriesen | does it mess up our accounting? | |
| 20:34:59 | cfriesen | why is "while it's migrating" different than "while it's running"? | |
| 20:35:34 | sean-k-mooney | mriedem: strange the hard reboot did not cause the interface to show up either. i must be missing something that the attach/detach whould have done | |
| 20:36:24 | dansmith | edleafe: yeah, so that's kinda what I was thinking at first, but then I tried to apply it to two things at once, so: | |
| 20:36:42 | dansmith | edleafe: imagine I have a tenant-restricted aggregate and the user boots with a requested AZ | |
| 20:37:08 | dansmith | edleafe: if I just concatenate the tenant and az list, then I won't restrict them properly | |
| 20:37:18 | efried | cfriesen: Are you suggesting that somewhere in the stack there will be code responsible for translating the allocation_request for [PCPU:2 from NUMA_RP_1] to [PCPU:2 from NUMA_RP_1 + VCPU:2 from compute node RP] ?? | |
| 20:37:37 | mriedem | cfriesen: it was an example of a bad time to detach a port | |
| 20:37:49 | cfriesen | efried: I'm suggesting that our logic right now is not rigorous. :) | |
| 20:37:53 | dansmith | edleafe: I can restrict the tenant aggs to only the one that matches the az, but they might not be the same size (and thus are different) | |
| 20:38:00 | sean-k-mooney | right we could have already caluated the new xml and miss the port is being removed | |
| 20:38:02 | mriedem | we do refresh the nw info cache when we get network-vif-deleted events | |
| 20:38:18 | dansmith | edleafe: I think placement _could_ know whether one is a subset of the other, but I can't really | |
| 20:38:43 | efried | dansmith, edleafe: Do we need member_of=all:... ? | |
| 20:38:46 | edleafe | dansmith: yeah, that really isn't how placement aggs should work | |
| 20:38:49 | dansmith | edleafe: so, just trying to figure out what to do there.. maybe we can have a hangout with jaypipes (et al) tomorrow after I have stewed on it a bit and make sure we're happy with what we've got | |
| 20:39:05 | cfriesen | efried: but other than that, I think we had been talking about tracking PCPU entirely separately from VCPU (where PCPU is the number of "dedicated" pcpus we have, and VCPU is the number of VCPUs we can support) | |
| 20:39:06 | efried | I guess if pressed, I would have said we were implementing in: for parity with GET /rps, but wouldn't actually be using it. | |
| 20:39:06 | edleafe | dansmith: we could add an 'and' case by leaving off the 'in:' operator | |
| 20:39:07 | dansmith | efried: well, I thought of that, but I'm not sure that's right either | |
| 20:39:16 | cfriesen | efried: PCPUs are always consumed on a per-numa-node basis | |
| 20:39:38 | cfriesen | efried: VCPUs can be consumed either per-numa-node or per-compute node, which is a problem for resource tracking. | |
| 20:40:26 | efried | cfriesen: Just so. Which is why I was of the opinion that we should only be presenting that inventory from within the NUMA node RP. | |
| 20:40:31 | dansmith | efried: if you have two groups of computes and both are legit for a tenant, then you don't want all | |
| 20:40:58 | efried | dansmith: Yeah, I wasn't really up on the use cases for GET /a_c?member_of | |
| 20:41:00 | dansmith | efried: almost want member_of=one:$agg1,$agg2&member_of=$agg3 | |
| 20:41:02 | cfriesen | efried: but that doesn't line up with the current logic, where you can consume a VCPU (and 4KB pages) from the entire compute node | |
| 20:41:06 | sean-k-mooney | mriedem: fyi http://paste.openstack.org/show/706642/ any guess why the "attach" by recreating the port and hard rebooting did not work? the port look identical to me bar the fact the status is down on the new port | |
| 20:41:33 | efried | dansmith: But I knew we would want in:[any] for parity. | |
| 20:41:47 | dansmith | efried: edleafe: I'll try to come up with a few different hard examples to talk about and we can decide whether we care (or care right now) about them | |
| 20:42:23 | efried | Yeah, sounds good. I mean, the bp is approved and completed, but heck, let's throw another one after it. | |
| 20:42:23 | edleafe | dansmith: ok. If it's needed, it shouldn't be too big of a change | |
| 20:42:27 | cfriesen | efried: and for various reasons it's really hard to track 4KB pages, since the host can consume them from either compute node at will. (Unless you run host stuff in a separate cgroup or container to provide hard limits.) | |
| 20:42:38 | dansmith | edleafe: okay | |
| 20:42:46 | mriedem | sean-k-mooney: not really | |
| 20:42:46 | efried | cfriesen: s/either compute node/either NUMA node/ ? | |
| 20:43:00 | cfriesen | efried: whoops,yes | |
| 20:43:55 | cfriesen | efried: we try to account for memory fairly tightly on our compute nodes, and had to make 4KB pages use "preferred" rather than "strict" numa mempolicy because we were hitting the oom killer | |
| 20:44:11 | efried | cfriesen: You could have your memory expressed in terms of resource class MEMORY_4K_PAGE. | |
| 20:44:20 | efried | cfriesen: But I guess that's not the issue you're stuck on. | |
| 20:44:51 | efried | cfriesen: You're concerned about being allowed to get those pages from separate NUMA nodes in cases where a) NUMA affinity is not strictly required; and b) you can't get 'em all from one. | |
| 20:45:39 | efried | cfriesen: So what you can actually do here is express your resource requests as separate numbered groups. | |
| 20:45:57 | cfriesen | efried: sort of. currently if you have an instance numa_topology nova will strictly constrain your instance to a single host numa node. | |