| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-06-21 | |||
| 16:38:33 | openstackgerrit | Stephen Finucane proposed openstack/nova master: conf: Add '[neutron] physnets' and related options https://review.openstack.org/564440 | |
| 16:38:34 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Start populating NUMACell.network_info field https://review.openstack.org/564441 | |
| 16:38:35 | openstackgerrit | Stephen Finucane proposed openstack/nova master: network: Rename '_get_phynet_info' to '_get_network_info' https://review.openstack.org/577233 | |
| 16:45:36 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: deduplicate {placement|api}_database help text https://review.openstack.org/577229 | |
| 16:45:48 | openstackgerrit | Zack Cornelius proposed openstack/nova master: Fix nits from change I676291ec0faa1dea0bd5050ef8e3426d171de4c6 https://review.openstack.org/576985 | |
| 16:45:49 | openstackgerrit | Zack Cornelius proposed openstack/nova master: Implement discard for file backed memory https://review.openstack.org/576994 | |
| 16:46:08 | zcorneli | dansmith: stephenfin: ^ | |
| 16:47:42 | stephenfin | zcorneli: ack | |
| 17:04:11 | stephenfin | zcorneli, dansmith: Any reason we don't enable discard by default and simply require libvirt 4.4.0 instead of 4.0.0? | |
| 17:04:27 | dansmith | stephenfin: because 4.4.0 was _just_ releases | |
| 17:04:31 | dansmith | like a week ago or something | |
| 17:04:48 | dansmith | and, discard is just a shutdown-time perf optimization | |
| 17:05:00 | dansmith | so it'd limit who can use this for quite a while for a small optimization | |
| 17:05:13 | zcorneli | what he said ^ | |
| 17:05:25 | zcorneli | Beat me to it | |
| 17:06:45 | stephenfin | dansmith, zcorneli: It's just that we're going to be carrying this stuff ad infinitum. Seems like a lot for an optimization https://review.openstack.org/#/c/576994/2/nova/objects/migrate_data.py | |
| 17:07:09 | stephenfin | But I guess we can just ignore it once the minimum reaches that in like 5 years time :) | |
| 17:07:53 | zcorneli | stephenfin: We should be able to ignore it down the line, once the minimum for all supported versions is 4.4 | |
| 17:07:55 | dansmith | stephenfin: we could rev that object without too much trouble once we raise the minimum enough to make it worth it | |
| 17:08:34 | stephenfin | dansmith: Yeah, it's more an irritation than a serious concern. +2/+Wd now | |
| 17:09:07 | stephenfin | Spotted a few nits but again, nothing technically off | |
| 17:09:46 | stephenfin | zcorneli: That must be the smallest amount of revisions required for a substantial patch that I've ever seen. Give yourself a clap on the back | |
| 17:10:16 | stephenfin | /worry about the stuff we missed 🙈 | |
| 17:11:08 | zcorneli | stephenfin: Both, of course | |
| 17:11:58 | zcorneli | stephenfin, dansmith: Thanks for all the help getting that merged in | |
| 17:40:08 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Tighten up ReportClient use of generation https://review.openstack.org/556669 | |
| 17:46:32 | zcorneli | dansmith: https://review.openstack.org/#/c/574792/7 | |
| 17:51:28 | dansmith | zcorneli: cool, we need devstack cores to look at that | |
| 17:51:30 | dansmith | mtreinish: ^ | |
| 17:58:28 | openstackgerrit | Eric Fried proposed openstack/nova master: Tighten up ReportClient use of generation https://review.openstack.org/556669 | |
| 18:11:35 | mtreinish | dansmith: ok, I put it on my list. I'll look at it after I get back from lunch | |
| 18:12:01 | dansmith | mtreinish: thanks, I left some comments on things for you to respond to, but it's fairly minor | |
| 19:23:55 | SpamapS | Running into confusing stuff with cells v2 | |
| 19:24:50 | SpamapS | we had a "rogue cell" recently and since deleting it we're not seeing nova-api listen to the cell rabbitmq's | |
| 19:24:59 | SpamapS | have restarted everything | |
| 19:25:46 | SpamapS | and nova-conductor isn't either | |
| 19:28:59 | SpamapS | the symptom is that when nova-compute sends back the 'vif-plugged' neutron is never informed and never finishes activating network ports | |
| 19:30:47 | dansmith | SpamapS: you have that backwards | |
| 19:31:01 | dansmith | SpamapS: vif-plugged comes _from_ neutron to nova-api, over the bus, and to nova-compute | |
| 19:31:49 | dansmith | if you deleted a cell, you should make sure that your instance and host mapping records all refer to legit cell records | |
| 19:32:35 | SpamapS | have verified all mappings are correct | |
| 19:32:48 | SpamapS | how does neutron know to send the vif-plugged event? | |
| 19:33:07 | SpamapS | we see os-vif successfully plugged logs | |
| 19:33:21 | SpamapS | but then nova-compute times out waiting for network-vif-plugged | |
| 19:33:41 | SpamapS | begging the question (we've been tracing through the code all last night and today) .. what actually triggers that? | |
| 19:33:49 | dansmith | it sends it in response to us doing plug on the compute side and asking neutron to wire it up | |
| 19:34:13 | dansmith | not sure if that request to neutron actually starts in os-vif now or not (it didn't use to) | |
| 19:35:17 | dansmith | do you see the event come into nova-api from neutron? | |
| 19:35:33 | openstackgerrit | Lenny Verkhovsky proposed openstack/nova master: WIP workaround for RHEL7.5 qemu 2.0 https://review.openstack.org/577280 | |
| 19:36:02 | SpamapS | dansmith: not since our unsuccessful deploy (we assume it's because of our rogue cell that the deploy created, but it could of course be something else) | |
| 19:36:32 | SpamapS | so that's where we've been trying to find what tells neutron to try that. | |
| 19:36:56 | dansmith | looks like it's in os-vif now, as I don't see (in paths that take os-vif) where we do it in nova anymore | |
| 19:37:49 | dansmith | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5604-L5607 | |
| 19:37:58 | dansmith | that context won't return until the event is received | |
| 19:38:09 | SpamapS | correct | |
| 19:38:10 | dansmith | so we start the context, do something that will trigger the event, and on __exit__ we wait for the event | |
| 19:38:11 | SpamapS | we're timing out there | |
| 19:39:02 | SpamapS | dansmith: and that do something is..... | |
| 19:39:17 | dansmith | plug | |
| 19:42:03 | SpamapS | dansmith: right, I see that, but how does neutron find out that you plugged? | |
| 19:42:29 | dansmith | I'm looking through os-vif now | |
| 19:42:36 | SpamapS | I had thought it was something like, nova-compute calls os_vif.<plugin>.plug(things) and then sends some kind of message to the conductors to say "we plugged!" and the conductors call neutron. | |
| 19:42:46 | dansmith | no | |
| 19:43:40 | dansmith | I dunno, I don't really see os-vif calling neutron anywhere | |
| 19:43:52 | SpamapS | nor do I | |
| 19:44:37 | SpamapS | only nova.network.neutronv2 calls neutron AFAICT | |
| 19:45:13 | dansmith | well, that's our interface for it of course | |
| 19:46:49 | SpamapS | IIRC, network-vif-plugged is sent when a port goes from DOWN -> ACTIVE | |
| 19:47:01 | SpamapS | and the neutron agent does that transition (we're using linuxbridge) | |
| 19:47:15 | SpamapS | so then I'm looking for what tells the neutron agent to do *that* | |
| 19:50:10 | jroll | doesn't that happen when the 'host' is sent in the portbinding part of the port body? (whether during port create or update | |
| 19:52:03 | jroll | which I guess would be here: https://github.com/openstack/nova/blob/master/nova/network/neutronv2/api.py#L1072 | |
| 19:53:09 | SpamapS | right, I think that triggers binding | |
| 19:53:32 | dansmith | the key is, we can't do it until we've created the tap or whatever type of interface | |
| 19:53:49 | dansmith | which is why this interlock exists | |
| 19:54:38 | SpamapS | the tap happens | |
| 19:55:36 | SpamapS | so, IIRC, nova-compute doesn't call that, nova-conductor calls that | |
| 19:55:47 | dansmith | nova conductor calls what? | |
| 19:55:53 | dansmith | bind? | |
| 19:56:21 | SpamapS | well I mean, nova-conductor is the only one making calls to the neutron API | |
| 19:56:38 | dansmith | no, that's definitely not true | |
| 19:56:39 | jroll | _update_ports_for_instance is called by allocate_for_instance, which was in nova-compute last I checked | |
| 19:56:48 | dansmith | right | |
| 19:57:20 | dansmith | but, the thing that triggers this plug event has to happen within that context or we could miss it | |
| 19:57:21 | SpamapS | ah ok | |
| 19:57:56 | SpamapS | so, the port gets a binding_host_id, but does not get status set to ACTIVE | |
| 19:58:29 | jroll | sounds like something is failing in neutron, then | |
| 19:59:01 | dansmith | could be that, but neutron is supposed to indicate failure to us if it does | |
| 19:59:20 | jroll | or locking up in neutron, I guess :) | |
| 19:59:36 | dansmith | or neutron is sending it and it's not hitting nova-api, which is fairly common | |
| 19:59:57 | jroll | neutron sets the port to ACTIVE, no? | |
| 20:00:01 | SpamapS | network-vif-plugged will not be sent until ACTIVE | |
| 20:00:10 | SpamapS | agents make ports ACTIVE AFAICT | |
| 20:02:15 | dansmith | SpamapS: which version is this again? and specifically which version of os-vif? | |
| 20:02:32 | SpamapS | Pike | |
| 20:02:34 | dansmith | IIRC, ovs used to be the only one that supported events at all, | |
| 20:02:40 | dansmith | oh okay, that's new enough I think | |
| 20:02:48 | dansmith | are you sure these events were coming in before? | |
| 20:03:01 | dansmith | and that you haven't just flipped vif-plugging-is-fatal recently or something? | |
| 20:03:01 | SpamapS | os-vif==1.7.0 | |