| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-11 | |||
| 17:39:22 | sean-k-mooney | adrianc: after migrating the vm unicast packet from the source node to the vm are droped | |
| 17:39:58 | sean-k-mooney | in my case i have an all in one node and a compute node | |
| 17:40:17 | sean-k-mooney | the vm started on teh all-in-one node and i migrated it to the compute-only node | |
| 17:40:41 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle tags in _bury_in_cell0 https://review.openstack.org/621856 | |
| 17:40:57 | adrianc | sean-k-mooney: on the source the VF used for macVtap has MAC address of zero ? | |
| 17:41:01 | sean-k-mooney | i can see the packets being transmited with tshark on teh all-in-one node but they are not recived on the destination | |
| 17:41:30 | sean-k-mooney | adrianc: ill check it not set to the vm mac any more but not sure its 0 | |
| 17:42:24 | sean-k-mooney | adrianc: its set to ba:2a:94:20:f7:ba | |
| 17:42:53 | sean-k-mooney | the vm mac is fa:16:3e:0e:87:08 | |
| 17:42:54 | adrianc | sean-k-mooney: so not an openstack VM MAC | |
| 17:43:01 | sean-k-mooney | correct | |
| 17:43:29 | adrianc | sean-k-mooney: mac appears well set on destination VF ? | |
| 17:43:50 | sean-k-mooney | yep both the macvtap and the dest vf have the mac set | |
| 17:44:20 | sean-k-mooney | if i reboot the vm the dhcp request from the vm is recived on the other node and it replys | |
| 17:44:40 | adrianc | sean-k-mooney: so the code does what its suppose to do :) now need to figure out the issue | |
| 17:44:48 | sean-k-mooney | however the unicast dhcp offer is transmitted on the all-in-one node but not recived on the compute | |
| 17:45:08 | sean-k-mooney | well you can live migrate back and forth | |
| 17:45:53 | adrianc | sean-k-mooney: mmmm, maybe mac is still in FDB on the source ? | |
| 17:45:53 | sean-k-mooney | if i migate the vm back the networking works agiag | |
| 17:46:09 | sean-k-mooney | adrianc: i checked with bridge fdb show and no | |
| 17:46:37 | sean-k-mooney | it is actully never there | |
| 17:46:57 | sean-k-mooney | the ipv6 multicast mac is set on the host that has the vm 33:33:ff:0e:87:08 | |
| 17:48:03 | sean-k-mooney | but fa:16:3e:0e:87:08 never show up in bridge fdb show either when the vm is sawned or before or after migration | |
| 17:48:32 | adrianc | sean-k-mooney: let me try to repro on my setup | |
| 17:52:50 | sean-k-mooney | adrianc: sure no worries im going to grab something to eat | |
| 17:54:18 | adrianc | sean-k-mooney: probably some minor tweaking needed (i hope) my setup seems busted, ill take a look tomorrow (20:00 here) | |
| 17:54:39 | sean-k-mooney | adrianc: fyi if i assign an ip on the same subnet to the pf i can ping the vm on the host its on | |
| 17:54:50 | sean-k-mooney | adrianc: sure no worries | |
| 17:54:57 | openstackgerrit | Chris Dent proposed openstack/os-resource-classes master: Initial automated testing for os-resource-classes https://review.openstack.org/624366 | |
| 17:54:58 | openstackgerrit | Chris Dent proposed openstack/os-resource-classes master: Rename ORDERED_CLASSES to STANDARDS https://review.openstack.org/624369 | |
| 17:54:59 | openstackgerrit | Chris Dent proposed openstack/os-resource-classes master: Sync STANDARDS with placement/nova rc_fields https://review.openstack.org/624370 | |
| 17:55:00 | openstackgerrit | Chris Dent proposed openstack/os-resource-classes master: Tune up documentation to be more useful https://review.openstack.org/624384 | |
| 17:55:01 | adrianc | bon appetit | |
| 19:12:14 | melwitt | mriedem: not sure what you'll think of this, workaround for non-shared instance dirs not being cleaned up during evacuate with ceph https://review.openstack.org/618478 | |
| 19:31:32 | mriedem | melwitt: i think it's ugly | |
| 19:32:13 | mriedem | comments inline anyway | |
| 19:32:24 | melwitt | I think it's ugly too, which is why I've struggled with it | |
| 19:32:55 | melwitt | I can't think of another way to handle it in a backportable way | |
| 19:47:24 | openstackgerrit | Merged openstack/nova stable/queens: Create BDMs/tags in cell with instance when over-quota https://review.openstack.org/623934 | |
| 19:50:43 | openstack | bug 1807723 in OpenStack Compute (nova) "swap multiattach volume intermittently fails when servers are on different hosts" [Medium,Confirmed] https://launchpad.net/bugs/1807723 | |
| 19:50:43 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: add debug logging for bug 1807723 https://review.openstack.org/624181 | |
| 20:12:55 | melwitt | mriedem, dansmith, gibi: there's a proposal to switch the os-traits, os-resource-classes, and osc-placement projects to use the placement.config gerrit acls here https://review.openstack.org/624387 to add your opinion if you have one | |
| 20:13:31 | codyhill | looks like the config-drive doesn’t write the vendor_data.json to the config-drive. Any help would be great. | |
| 20:13:31 | codyhill | Hello folks. I was working with the #openstack-ironic folks about trying to get vendor_data.json to work with baremetal deployments with Nova and Ironic. So far the issue i’m having is that cloud-init doesn’t seem to “Discover” the baremetal node as an OpenStack Datasource. So it runs the generice EC2 setup. Which doesn’t include vendor_data.json. The recomendation from the folks in #openstack-ironic was to try and use config drive. And it | |
| 20:15:29 | mriedem | melwitt: isn't the only difference the core team at this point? | |
| 20:15:48 | mriedem | which right now is a difference of cdent and tetsuro in placement-core | |
| 20:15:55 | melwitt | yeah | |
| 20:17:00 | mriedem | ok voted | |
| 20:17:09 | melwitt | thanks | |
| 20:30:18 | codyhill | To add a little more info to my situation. I am running OpenStack Newton. And I’m using static Vendor Data (not dynamic). | |
| 20:36:37 | mriedem | codyhill: and you have vendordata_jsonfile_path set properly in nova.conf for the nova-compute service? | |
| 20:36:43 | mriedem | https://docs.openstack.org/newton/config-reference/compute/config-options.html | |
| 20:37:16 | mriedem | and you have vendordata_providers = StaticJson? | |
| 20:37:25 | mriedem | *StaticJSON | |
| 20:40:18 | mriedem | i think either of those will work | |
| 20:40:27 | mriedem | but building the config driver happens on the nova-compute host | |
| 20:40:29 | mriedem | not the api node | |
| 20:40:34 | mriedem | *config drive | |
| 20:40:42 | codyhill | mriedem: I havn’t touched nove_compute. only nove API | |
| 20:40:45 | codyhill | Let me try that. | |
| 20:41:00 | mriedem | as for ironic, i'm not sure what support the ironic driver had for config drive in newton | |
| 20:41:40 | mriedem | but yeah my guess is since vendordata_jsonfile_path isn't set in nova.conf on nova-compute it's just not serving anything up | |
| 20:49:53 | codyhill | mriedem: Thanks for your help. I’ll give this a shot. (I’m in a meeting right now) But i’ll report my findings in about an hour. | |
| 20:50:18 | mriedem | good luck | |
| 20:55:02 | openstackgerrit | Merged openstack/nova stable/queens: Move the nova-next job in-tree and update it https://review.openstack.org/620777 | |
| 20:55:09 | openstackgerrit | Merged openstack/nova stable/queens: Test websocketproxy with TLS in the nova-next job https://review.openstack.org/620776 | |
| 20:55:15 | openstackgerrit | Merged openstack/nova stable/queens: Make the nova-next job voting and gating https://review.openstack.org/620888 | |
| 21:08:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: debug old compute API service version lookup https://review.openstack.org/624438 | |
| 21:25:53 | melwitt | mriedem: I noticed you have commented "recheck slow node" on some patches and was wondering, are you categorizing a failure to boot due to a timeout waiting for network-vif-plugged as that? example: http://logs.openstack.org/35/575735/4/gate/tempest-slow/7e5803b/controller/logs/screen-n-cpu.txt.gz#_Dec_11_16_33_46_458364 | |
| 21:26:20 | mriedem | network-vif-plugged takes about 5 seconds | |
| 21:26:35 | mriedem | so if we timeout because we ddn't get that after 5 minutes, we likely screwed up somewhere | |
| 21:26:47 | mriedem | and rax isn't the slow nodes | |
| 21:26:57 | melwitt | ok | |
| 21:27:36 | melwitt | I'll check if neutron sent it. if so, we probably have another race or issue where we get the event before we start waiting or something | |
| 21:28:19 | mriedem | note 2 seconds before that a vif is unplugged http://logs.openstack.org/35/575735/4/gate/tempest-slow/7e5803b/controller/logs/screen-n-cpu.txt.gz#_Dec_11_16_33_44_406723 | |
| 21:29:13 | mriedem | there are a bunch of unplugged vif messages for that same request | |
| 21:29:37 | mriedem | http://logs.openstack.org/35/575735/4/gate/tempest-slow/7e5803b/controller/logs/screen-n-cpu.txt.gz#_Dec_11_16_33_43_583022 | |
| 21:29:37 | mriedem | this is the timeout waiting for network-vif-plugged | |
| 21:29:45 | melwitt | yeah, I saw that. I thought maybe it's some cleanup it does in response to the timeout (the _cleanup_failed_start routine) | |
| 21:30:14 | melwitt | unplugs all the vifs and then raises VirtualInterfaceCreateException | |
| 21:30:26 | mriedem | http://logs.openstack.org/35/575735/4/gate/tempest-slow/7e5803b/controller/logs/screen-n-cpu.txt.gz#_Dec_11_16_28_41_912716 | |
| 21:30:26 | mriedem | looks like this is the first plug | |
| 21:30:32 | mriedem | which is 5 minutes before | |
| 21:31:02 | mriedem | likely yes, spawn() times out, unplugs vifs, meanwhile tempest is also timing out and destroying the instance concurrently which is doing its own unplugs | |
| 21:31:56 | mriedem | jesus does that test really plug 7 ports to the same vm? | |
| 21:32:05 | mriedem | 5'), ('network-vif-plugged', u'a5b3eb7e-696c-4870-abf1-b3ee43545dbe'), ('network-vif-plugged', u'ede8a324-5ed5-4737-bf6c-1b2f21b4f090'), ('network-vif-plugged', u'99e00244-9a75-4a25-9742-5c67b86ba0b2'), ('network-vif-plugged', u'11c3d630-37f4-4fd3-ba88-8355f7253ea6'), ('network-vif-plugged', u'05bb2644-68b1-4f07-a44e-f2277b444792')] for instance with vm_state building and task_state spawning.: Timeout: 300 seconds | |
| 21:32:05 | mriedem | Dec 11 16:33:43.583022 ubuntu-xenial-rax-iad-0001127841 nova-compute[368]: WARNING nova.virt.libvirt.driver [None req-8a094b29-c0e9-445c-848e-3c827f681858 tempest-TaggedBootDevicesTest-731133270 tempest-TaggedBootDevicesTest-731133270] [instance: 01e6b7fe-bfa0-4d51-bcdf-03e65971ed74] Timeout waiting for [('network-vif-plugged', u'c1777114-78f7-4533-8725-d01139d29c02'), ('network-vif-plugged', u'caade525-9701-47b4-a5d1-bc11076 | |
| 21:32:42 | mriedem | Dec 11 16:28:45.973940 ubuntu-xenial-rax-iad-0001127841 nova-compute[368]: DEBUG nova.compute.manager [req-9fb4fde1-0cb0-4896-b0c3-5f8535a0fbd1 req-17f33770-c6cf-4fef-ade4-72fab2d05def service nova] [instance: 01e6b7fe-bfa0-4d51-bcdf-03e65971ed74] Processing event network-vif-plugged-caade525-9701-47b4-a5d1-bc11076c47f5 {{(pid=368) _process_instance_event /opt/stack/nova/nova/compute/manager.py:7952}} | |
| 21:33:02 | melwitt | I wasn't sure, I'm not that familiar with the tagged devices test. I didn't get that far yet | |
| 21:33:18 | mriedem | artom wrote it | |
| 21:33:19 | mriedem | Dec 11 16:28:48.063359 ubuntu-xenial-rax-iad-0001127841 nova-compute[368]: DEBUG nova.compute.manager [req-4f071ef5-d1a5-4cfb-820b-2d562666e534 req-86897002-4096-4477-a6ae-19515f45c40f service nova] [instance: 01e6b7fe-bfa0-4d51-bcdf-03e65971ed74] Processing event network-vif-plugged-a5b3eb7e-696c-4870-abf1-b3ee43545dbe {{(pid=368) _process_instance_event /opt/stack/nova/nova/compute/manager.py:7952}} | |
| 21:33:34 | mriedem | Dec 11 16:28:48.059696 ubuntu-xenial-rax-iad-0001127841 nova-compute[368]: DEBUG nova.compute.manager [req-4f071ef5-d1a5-4cfb-820b-2d562666e534 req-86897002-4096-4477-a6ae-19515f45c40f service nova] [instance: 01e6b7fe-bfa0-4d51-bcdf-03e65971ed74] Processing event network-vif-plugged-ede8a324-5ed5-4737-bf6c-1b2f21b4f090 {{(pid=368) _process_instance_event /opt/stack/nova/nova/compute/manager.py:7952}} | |
| 21:33:47 | mriedem | Dec 11 16:28:48.061523 ubuntu-xenial-rax-iad-0001127841 nova-compute[368]: DEBUG nova.compute.manager [req-4f071ef5-d1a5-4cfb-820b-2d562666e534 req-86897002-4096-4477-a6ae-19515f45c40f service nova] [instance: 01e6b7fe-bfa0-4d51-bcdf-03e65971ed74] Processing event network-vif-plugged-99e00244-9a75-4a25-9742-5c67b86ba0b2 {{(pid=368) _process_instance_event /opt/stack/nova/nova/compute/manager.py:7952}} | |
| 21:33:59 | mriedem | Dec 11 16:28:48.055772 ubuntu-xenial-rax-iad-0001127841 nova-compute[368]: DEBUG nova.compute.manager [req-4f071ef5-d1a5-4cfb-820b-2d562666e534 req-86897002-4096-4477-a6ae-19515f45c40f service nova] [instance: 01e6b7fe-bfa0-4d51-bcdf-03e65971ed74] Processing event network-vif-plugged-11c3d630-37f4-4fd3-ba88-8355f7253ea6 {{(pid=368) _process_instance_event /opt/stack/nova/nova/compute/manager.py:7952}} | |
| 21:34:11 | mriedem | Dec 11 16:28:48.057381 ubuntu-xenial-rax-iad-0001127841 nova-compute[368]: DEBUG nova.compute.manager [req-4f071ef5-d1a5-4cfb-820b-2d562666e534 req-86897002-4096-4477-a6ae-19515f45c40f service nova] [instance: 01e6b7fe-bfa0-4d51-bcdf-03e65971ed74] Processing event network-vif-plugged-05bb2644-68b1-4f07-a44e-f2277b444792 {{(pid=368) _process_instance_event /opt/stack/nova/nova/compute/manager.py:7952}} | |
| 21:34:14 | mriedem | looks like we got 6 of 7 | |
| 21:34:45 | mriedem | c1777114-78f7-4533-8725-d01139d29c02 was the missing port event | |
| 21:36:21 | melwitt | that's a lot of ports | |
| 21:37:49 | mriedem | not for cfriesen | |
| 21:37:55 | melwitt | haha | |