| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-12-11 | |||
| 17:31:21 | stephenfin | It actually is https://docs.openstack.org/nova/latest/admin/pci-passthrough.html#configure-pci-devices-compute | |
| 17:31:26 | stephenfin | But it's hidden | |
| 17:31:30 | stephenfin | sort of | |
| 17:31:36 | stephenfin | "From the Newton release, to resize guest with PCI device, configure the PCI alias on the compute node as well." | |
| 17:31:57 | sean-k-mooney | stephenfin: the whitelist should be set on the compute and alais on the contoller that is how i have understood it was ment to work | |
| 17:32:07 | stephenfin | Me too | |
| 17:33:01 | sean-k-mooney | your updating the config docs mind patching https://docs.openstack.org/nova/latest/admin/pci-passthrough.html#configure-pci-devices-compute to make that bold | |
| 17:34:23 | sean-k-mooney | luckally i was seting it on all nodes just because i adapted my controler local.conf into my compute node one | |
| 17:34:56 | sean-k-mooney | stephenfin: speaking of which do you have 2 node capably of sriov or one? | |
| 17:35:20 | stephenfin | sean-k-mooney: Can do, and I only have one | |
| 17:35:53 | sean-k-mooney | ok i have spent most of the day debuging a netwrokg issue | |
| 17:36:29 | sean-k-mooney | im going to need to use a node inbetween my current 2 nodes to sniff packets and see which side is broken | |
| 17:37:46 | sean-k-mooney | adrianc: FYI at least on my hardware the sriov live migration patches are not working correctly | |
| 17:38:58 | adrianc | sean-k-mooney: what are you seeing ? | |
| 17:39:22 | sean-k-mooney | adrianc: after migrating the vm unicast packet from the source node to the vm are droped | |
| 17:39:58 | sean-k-mooney | in my case i have an all in one node and a compute node | |
| 17:40:17 | sean-k-mooney | the vm started on teh all-in-one node and i migrated it to the compute-only node | |
| 17:40:41 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Handle tags in _bury_in_cell0 https://review.openstack.org/621856 | |
| 17:40:57 | adrianc | sean-k-mooney: on the source the VF used for macVtap has MAC address of zero ? | |
| 17:41:01 | sean-k-mooney | i can see the packets being transmited with tshark on teh all-in-one node but they are not recived on the destination | |
| 17:41:30 | sean-k-mooney | adrianc: ill check it not set to the vm mac any more but not sure its 0 | |
| 17:42:24 | sean-k-mooney | adrianc: its set to ba:2a:94:20:f7:ba | |
| 17:42:53 | sean-k-mooney | the vm mac is fa:16:3e:0e:87:08 | |
| 17:42:54 | adrianc | sean-k-mooney: so not an openstack VM MAC | |
| 17:43:01 | sean-k-mooney | correct | |
| 17:43:29 | adrianc | sean-k-mooney: mac appears well set on destination VF ? | |
| 17:43:50 | sean-k-mooney | yep both the macvtap and the dest vf have the mac set | |
| 17:44:20 | sean-k-mooney | if i reboot the vm the dhcp request from the vm is recived on the other node and it replys | |
| 17:44:40 | adrianc | sean-k-mooney: so the code does what its suppose to do :) now need to figure out the issue | |
| 17:44:48 | sean-k-mooney | however the unicast dhcp offer is transmitted on the all-in-one node but not recived on the compute | |
| 17:45:08 | sean-k-mooney | well you can live migrate back and forth | |
| 17:45:53 | sean-k-mooney | if i migate the vm back the networking works agiag | |
| 17:45:53 | adrianc | sean-k-mooney: mmmm, maybe mac is still in FDB on the source ? | |
| 17:46:09 | sean-k-mooney | adrianc: i checked with bridge fdb show and no | |
| 17:46:37 | sean-k-mooney | it is actully never there | |
| 17:46:57 | sean-k-mooney | the ipv6 multicast mac is set on the host that has the vm 33:33:ff:0e:87:08 | |
| 17:48:03 | sean-k-mooney | but fa:16:3e:0e:87:08 never show up in bridge fdb show either when the vm is sawned or before or after migration | |
| 17:48:32 | adrianc | sean-k-mooney: let me try to repro on my setup | |
| 17:52:50 | sean-k-mooney | adrianc: sure no worries im going to grab something to eat | |
| 17:54:18 | adrianc | sean-k-mooney: probably some minor tweaking needed (i hope) my setup seems busted, ill take a look tomorrow (20:00 here) | |
| 17:54:39 | sean-k-mooney | adrianc: fyi if i assign an ip on the same subnet to the pf i can ping the vm on the host its on | |
| 17:54:50 | sean-k-mooney | adrianc: sure no worries | |
| 17:54:57 | openstackgerrit | Chris Dent proposed openstack/os-resource-classes master: Initial automated testing for os-resource-classes https://review.openstack.org/624366 | |
| 17:54:58 | openstackgerrit | Chris Dent proposed openstack/os-resource-classes master: Rename ORDERED_CLASSES to STANDARDS https://review.openstack.org/624369 | |
| 17:54:59 | openstackgerrit | Chris Dent proposed openstack/os-resource-classes master: Sync STANDARDS with placement/nova rc_fields https://review.openstack.org/624370 | |
| 17:55:00 | openstackgerrit | Chris Dent proposed openstack/os-resource-classes master: Tune up documentation to be more useful https://review.openstack.org/624384 | |
| 17:55:01 | adrianc | bon appetit | |
| 19:12:14 | melwitt | mriedem: not sure what you'll think of this, workaround for non-shared instance dirs not being cleaned up during evacuate with ceph https://review.openstack.org/618478 | |
| 19:31:32 | mriedem | melwitt: i think it's ugly | |
| 19:32:13 | mriedem | comments inline anyway | |
| 19:32:24 | melwitt | I think it's ugly too, which is why I've struggled with it | |
| 19:32:55 | melwitt | I can't think of another way to handle it in a backportable way | |
| 19:47:24 | openstackgerrit | Merged openstack/nova stable/queens: Create BDMs/tags in cell with instance when over-quota https://review.openstack.org/623934 | |
| 19:50:43 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: add debug logging for bug 1807723 https://review.openstack.org/624181 | |
| 19:50:43 | openstack | bug 1807723 in OpenStack Compute (nova) "swap multiattach volume intermittently fails when servers are on different hosts" [Medium,Confirmed] https://launchpad.net/bugs/1807723 | |
| 20:12:55 | melwitt | mriedem, dansmith, gibi: there's a proposal to switch the os-traits, os-resource-classes, and osc-placement projects to use the placement.config gerrit acls here https://review.openstack.org/624387 to add your opinion if you have one | |
| 20:13:31 | codyhill | Hello folks. I was working with the #openstack-ironic folks about trying to get vendor_data.json to work with baremetal deployments with Nova and Ironic. So far the issue i’m having is that cloud-init doesn’t seem to “Discover” the baremetal node as an OpenStack Datasource. So it runs the generice EC2 setup. Which doesn’t include vendor_data.json. The recomendation from the folks in #openstack-ironic was to try and use config drive. And it | |
| 20:13:31 | codyhill | looks like the config-drive doesn’t write the vendor_data.json to the config-drive. Any help would be great. | |
| 20:15:29 | mriedem | melwitt: isn't the only difference the core team at this point? | |
| 20:15:48 | mriedem | which right now is a difference of cdent and tetsuro in placement-core | |
| 20:15:55 | melwitt | yeah | |
| 20:17:00 | mriedem | ok voted | |
| 20:17:09 | melwitt | thanks | |
| 20:30:18 | codyhill | To add a little more info to my situation. I am running OpenStack Newton. And I’m using static Vendor Data (not dynamic). | |
| 20:36:37 | mriedem | codyhill: and you have vendordata_jsonfile_path set properly in nova.conf for the nova-compute service? | |
| 20:36:43 | mriedem | https://docs.openstack.org/newton/config-reference/compute/config-options.html | |
| 20:37:16 | mriedem | and you have vendordata_providers = StaticJson? | |
| 20:37:25 | mriedem | *StaticJSON | |
| 20:40:18 | mriedem | i think either of those will work | |
| 20:40:27 | mriedem | but building the config driver happens on the nova-compute host | |
| 20:40:29 | mriedem | not the api node | |
| 20:40:34 | mriedem | *config drive | |
| 20:40:42 | codyhill | mriedem: I havn’t touched nove_compute. only nove API | |
| 20:40:45 | codyhill | Let me try that. | |
| 20:41:00 | mriedem | as for ironic, i'm not sure what support the ironic driver had for config drive in newton | |
| 20:41:40 | mriedem | but yeah my guess is since vendordata_jsonfile_path isn't set in nova.conf on nova-compute it's just not serving anything up | |
| 20:49:53 | codyhill | mriedem: Thanks for your help. I’ll give this a shot. (I’m in a meeting right now) But i’ll report my findings in about an hour. | |
| 20:50:18 | mriedem | good luck | |
| 20:55:02 | openstackgerrit | Merged openstack/nova stable/queens: Move the nova-next job in-tree and update it https://review.openstack.org/620777 | |
| 20:55:09 | openstackgerrit | Merged openstack/nova stable/queens: Test websocketproxy with TLS in the nova-next job https://review.openstack.org/620776 | |
| 20:55:15 | openstackgerrit | Merged openstack/nova stable/queens: Make the nova-next job voting and gating https://review.openstack.org/620888 | |
| 21:08:23 | openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: debug old compute API service version lookup https://review.openstack.org/624438 | |
| 21:25:53 | melwitt | mriedem: I noticed you have commented "recheck slow node" on some patches and was wondering, are you categorizing a failure to boot due to a timeout waiting for network-vif-plugged as that? example: http://logs.openstack.org/35/575735/4/gate/tempest-slow/7e5803b/controller/logs/screen-n-cpu.txt.gz#_Dec_11_16_33_46_458364 | |
| 21:26:20 | mriedem | network-vif-plugged takes about 5 seconds | |
| 21:26:35 | mriedem | so if we timeout because we ddn't get that after 5 minutes, we likely screwed up somewhere | |
| 21:26:47 | mriedem | and rax isn't the slow nodes | |
| 21:26:57 | melwitt | ok | |
| 21:27:36 | melwitt | I'll check if neutron sent it. if so, we probably have another race or issue where we get the event before we start waiting or something | |
| 21:28:19 | mriedem | note 2 seconds before that a vif is unplugged http://logs.openstack.org/35/575735/4/gate/tempest-slow/7e5803b/controller/logs/screen-n-cpu.txt.gz#_Dec_11_16_33_44_406723 | |
| 21:29:13 | mriedem | there are a bunch of unplugged vif messages for that same request | |
| 21:29:37 | mriedem | this is the timeout waiting for network-vif-plugged | |
| 21:29:37 | mriedem | http://logs.openstack.org/35/575735/4/gate/tempest-slow/7e5803b/controller/logs/screen-n-cpu.txt.gz#_Dec_11_16_33_43_583022 | |
| 21:29:45 | melwitt | yeah, I saw that. I thought maybe it's some cleanup it does in response to the timeout (the _cleanup_failed_start routine) | |
| 21:30:14 | melwitt | unplugs all the vifs and then raises VirtualInterfaceCreateException | |
| 21:30:26 | mriedem | looks like this is the first plug | |
| 21:30:26 | mriedem | http://logs.openstack.org/35/575735/4/gate/tempest-slow/7e5803b/controller/logs/screen-n-cpu.txt.gz#_Dec_11_16_28_41_912716 | |
| 21:30:32 | mriedem | which is 5 minutes before | |
| 21:31:02 | mriedem | likely yes, spawn() times out, unplugs vifs, meanwhile tempest is also timing out and destroying the instance concurrently which is doing its own unplugs | |
| 21:31:56 | mriedem | jesus does that test really plug 7 ports to the same vm? | |
| 21:32:05 | mriedem | Dec 11 16:33:43.583022 ubuntu-xenial-rax-iad-0001127841 nova-compute[368]: WARNING nova.virt.libvirt.driver [None req-8a094b29-c0e9-445c-848e-3c827f681858 tempest-TaggedBootDevicesTest-731133270 tempest-TaggedBootDevicesTest-731133270] [instance: 01e6b7fe-bfa0-4d51-bcdf-03e65971ed74] Timeout waiting for [('network-vif-plugged', u'c1777114-78f7-4533-8725-d01139d29c02'), ('network-vif-plugged', u'caade525-9701-47b4-a5d1-bc11076 | |