Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-01-25
22:12:39 mriedem yeah, the traits are generated from their directory structure within os-traits
22:12:44 jroll oh gdi
22:12:54 mriedem https://docs.openstack.org/os-traits/latest/user/index.html
22:15:54 jroll mriedem: efried: that did it, now I feel dumb, thanks :)
22:16:08 mriedem don't feel dumb
22:16:16 efried For not understanding the bizarre architecture of os-traits? Yeah, what mriedem said.
22:16:30 mriedem ironicclient change is 85 in the gate
22:16:44 efried I always just GET /traits from the API to find my testable traits.
22:17:29 efried Because you'll also get tripped up by this one at some point in the future: Your compute's os-traits may not be the same as the placement server's os-traits.
22:18:35 efried which is why https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L1005-L1009
22:22:13 openstackgerrit Jackie Truong proposed openstack/nova master: Reduce complexity of _from_db_object https://review.openstack.org/537728
22:22:14 openstackgerrit Jackie Truong proposed openstack/nova master: Add trusted_certs to instance_extra https://review.openstack.org/537897
22:22:14 openstackgerrit Jackie Truong proposed openstack/nova master: Add trusted_certs object https://review.openstack.org/489408
22:22:15 openstackgerrit Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949
22:22:15 openstackgerrit Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204
22:25:17 dansmith woot
22:25:50 dansmith I shall gaze upon thy patches upon my return from a brief respite
22:25:51 mriedem melwitt: this makes me think about the eventlet/python thing in the ML https://review.openstack.org/#/c/537811/1
22:26:10 jackie-truong *\o/*
22:26:29 melwitt are those pom poms?
22:28:05 melwitt mriedem: interesting
22:29:55 melwitt seems like it could be related. nothing makes too much sense so far, only cells v1 job is getting constant libvirt keepalive timeouts, starting about 10 days ago, and only on stable
22:30:10 melwitt pike and ocata
22:30:36 mriedem melwitt: unrelated, but related to lyarwood's change to hard reboot and blow things away all the time https://bugs.launchpad.net/nova/+bug/1744361
22:30:37 openstack Launchpad bug 1744361 in neutron "test_server_security_groups failed to reboot with "Domain not found: no domain with matching uuid" because of missing vif event from linuxbridge agent" [Undecided,Confirmed]
22:31:14 mriedem i thought there were some assertions in the review that we didn't need to wait for vif plugged events, or something along those lines
22:31:52 melwitt ah, suck
22:31:52 mriedem https://review.openstack.org/#/c/400384/
22:32:26 melwitt well, I had commented that we do need to wait for them since we are re-plugging vifs. but I didn't realize we'd be in a vm_state that would ignore them
22:32:44 jroll efried: fun, thanks
22:33:02 melwitt that is, I didn't expect the events to be ignored when they're being waited for
22:36:06 mriedem melwitt: it's not saying it's ignored b/c of the instance state
22:36:20 mriedem it's saying the compute got the event but no callback was registered waiting for it
22:36:28 mriedem Jan 19 10:33:35.986648 ubuntu-xenial-rax-iad-0002010825 nova-compute[28907]: DEBUG nova.compute.manager [None req-4bcf94cf-2048-4e99-94e6-646519900469 service nova] [instance: dd2ec3b0-f909-4104-ae2a-9b878d936ed4] No waiting events found dispatching network-vif-plugged-567525a4-4b16-4004-91d5-545086d01227 {{(pid=28907) pop_instance_event /opt/stack/new/nova/nova/compute/manager.py:363}}
22:37:49 melwitt mriedem: okay, I was going by the last comment in the bug. okay, so maybe there's a bug in the _create_domain_and_network vifs_already_plugged logic somewhere
22:38:29 melwitt or maybe vifs_already_plugged doesn't mean what I thought it means
22:40:45 mriedem i wonder if we don't register the callback before the vif plug event has already happened
22:41:10 mriedem no that doesn't make sense
22:41:15 mriedem we register the callback before calling plug_vifs
22:41:26 melwitt "Timeout waiting for vif plugging callback for instance with vm_state active and task_state reboot_started_hard.: Timeout: 300 seconds" so that means it was waiting
22:45:02 melwitt so "Received unexpected event" happens when there weren't events in self.instance_events in compute manager. so how do they get there ...
22:45:07 dansmith mriedem: yeah the whole design of that thing is so you can't trigger before you're registered
22:56:18 mriedem i don't know what to do about this bug https://bugs.launchpad.net/nova/+bug/1740123
22:56:20 openstack Launchpad bug 1740123 in OpenStack Compute (nova) "Can we create server group with empty policies?" [Undecided,New]
22:56:25 mriedem it's not a bug per se
22:56:51 melwitt mriedem: it seems like the patch is related to the new bug. I think the only way this can happen is if _get_neutron_events(network_info) is returning no events, meaning there are no vifs in network_info. I'm looking to see where/how does network_info get reset after a reboot
22:57:34 mriedem melwitt: if it doesn't return any events, wouldn't we just not wait for anything?
22:57:39 mriedem and then no timeout?
22:57:44 melwitt I would have thought network_info would remain as it was during a reboot (network_info came from when we originally allocated ports, right?)
22:58:06 mriedem yes network_info should have whatever ports are attached to the instance, regardless of whether or not the vifs are actually plugged
22:58:13 mriedem it's like bdms
22:58:15 mriedem for ports
22:58:25 melwitt good point ... one would think. I'm not that familiar with how the events work. you pass a list of events to the "wait" context manager
22:59:48 melwitt yeah, looks like it would just pass by all the waiting if that list is empty. so the list isn't empty. yet it's not matching up the events
23:03:57 mriedem we should probably log the list of stuff we're registering when we register it
23:04:08 mriedem for debuggabilityamortiziation
23:04:38 melwitt yeah. cause so far this don't make no sense
23:05:07 dansmith you mean log at debug level I assume.. otherwise it's too chatty
23:05:21 mriedem yes debug
23:05:30 melwitt debugaroo
23:05:42 mriedem i can't be chattier than the virt topology shit we log 20 things for
23:05:52 mriedem even though we don't do any stuff with that in the gate
23:05:55 mriedem *it
23:07:42 openstackgerrit Jim Rollenhagen proposed openstack/nova master: Add functional tests for virt driver get_traits() method https://review.openstack.org/538047
23:07:49 jroll mriedem: ^ all yours, that worked out well
23:09:08 openstackgerrit Jim Rollenhagen proposed openstack/nova master: Add functional tests for virt driver get_traits() method https://review.openstack.org/538047
23:09:12 jroll and pep8'd
23:10:30 mriedem damn it
23:10:33 mriedem comment in the last patchset
23:10:43 mriedem seems like those could be one test,
23:10:46 mriedem but idk
23:11:02 melwitt hm, the 'No waiting events found dispatching' message means that the result from _pop_event was no_events_sentinel which has another log message in it 'Unexpected attempt to pop events during shutdown'
23:11:20 mriedem i think that's when the compute service is shutting down
23:11:21 mriedem not the guest
23:11:48 melwitt oh, there's another way to get no_events_sentinel, if no result for self._events.get(instance.uuid)
23:12:15 melwitt so it should be from that
23:15:02 mriedem rloo: TheJulia: jroll: is someone putting out a release? https://review.openstack.org/#/c/532622/
23:15:33 rloo mriedem: TheJulia is in charge of that!
23:15:55 TheJulia we have a reno in the gate that should only take like 10 minutes
23:16:07 mriedem yeah i see
23:16:30 mriedem so, you could start the release patch with a depends-on the reno patch
23:16:36 openstackgerrit Jim Rollenhagen proposed openstack/nova master: Add functional tests for virt driver get_traits() method https://review.openstack.org/538047
23:16:38 mriedem then you just have to mod the git hash once it merges
23:16:57 mriedem i.e. we could start stacking up the dependencies for the release
23:19:41 TheJulia docs and pep8, even less time
23:23:16 openstackgerrit OpenStack Release Bot proposed openstack/python-novaclient master: Update reno for stable/queens https://review.openstack.org/538061
23:29:20 mriedem TheJulia: ship it
23:30:05 mriedem woot
23:30:07 mriedem https://review.openstack.org/538063
23:30:32 TheJulia yup
23:30:42 TheJulia had to just fix an excess space
23:31:06 TheJulia but yup
23:33:19 mriedem jroll: since you've got the series localm
23:33:29 mriedem want to change the depends-on in https://review.openstack.org/#/c/532288/ to https://review.openstack.org/#/c/538066/
23:33:35 mriedem and add the requirements.txt change in nova
23:33:52 mriedem *test-requirements.txt
23:42:38 mriedem the gpu patches failed on the volume backup thing, so i'm going to wait to recheck those
23:44:36 openstackgerrit Matt Riedemann proposed openstack/nova master: Implement get_traits() for the ironic virt driver https://review.openstack.org/532288
23:44:37 openstackgerrit Matt Riedemann proposed openstack/nova master: Add functional tests for virt driver get_traits() method https://review.openstack.org/538047
23:44:37 mriedem jroll: nvm i did it

Earlier   Later