| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-01-17 | |||
| 19:53:22 | mriedem | there is a known neutron bug | |
| 19:53:56 | mriedem | https://bugs.launchpad.net/neutron/+bug/1811515 | |
| 19:54:03 | openstack | Launchpad bug 1811515 in neutron "pyroute2.NetNS don't work properly with concurrency in oslo.privsep" [Critical,In progress] - Assigned to Slawek Kaplonski (slaweq) | |
| 19:54:31 | mriedem | luckily https://review.openstack.org/#/c/631275/ | |
| 20:50:16 | melwitt | nova meeting in 10 minutes | |
| 20:55:25 | efried | mriedem: thanks. Some of them are ssh failures, yes. | |
| 21:04:03 | mriedem | melwitt: os-vif was released with that change https://review.openstack.org/#/c/602384/ | |
| 21:04:34 | melwitt | mriedem: ack | |
| 21:05:16 | sean-k-mooney | yes there is a conflict with libvirt | |
| 21:05:29 | sean-k-mooney | os-vif creates teh port then libvirt deletes it and creates it again | |
| 21:05:41 | sean-k-mooney | tests that only check for port active race | |
| 21:05:53 | sean-k-mooney | tetsts taht check the interface is pingable would be fien | |
| 21:06:19 | sean-k-mooney | e.g. if they wait for the vm to be accessabel before sshing then it wont break | |
| 21:07:09 | sean-k-mooney | im looking in to this week | |
| 21:08:59 | mriedem | dansmith: https://bugs.launchpad.net/nova/+bug/1812196 | |
| 21:09:04 | openstack | Launchpad bug 1812196 in OpenStack Compute (nova) "when using templated cells urls, validation fails with create_cell when using {scheme} and {port}" [Undecided,New] | |
| 21:10:12 | dansmith | um | |
| 21:10:28 | dansmith | those shouldn't be parsed by nova-manage | |
| 21:12:18 | dansmith | hrm | |
| 21:12:35 | dansmith | well, that's kindof the problem I guess, but.. hmm | |
| 21:14:27 | dansmith | cripes can this always have been broken with nova-manage? | |
| 21:23:34 | mriedem | i haven't dug into the bug, just noticed it in triage | |
| 21:27:00 | dansmith | yeah, this is a pretty dumb ass mistake.. mah bad | |
| 21:29:47 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix using template cell urls with nova-manage https://review.openstack.org/631649 | |
| 21:29:51 | dansmith | mriedem: ^ we'll want that backported to .. at least rocky, IIRC | |
| 21:41:40 | melwitt | thanks tonyb | |
| 21:43:01 | tonyb | first nova review in $ages | |
| 21:43:26 | melwitt | :) | |
| 21:58:46 | openstackgerrit | Dakshina Ilangovan proposed openstack/nova-specs master: Nova changes to support Local Resource Management that uses Resource Management Daemon https://review.openstack.org/630817 | |
| 22:01:19 | mriedem | dansmith: might have just been a result of https://github.com/openstack/nova/commit/bc6ca87a6a01ecbda926a78d07cadbd4e3ab97e0 not taking into account templated urls | |
| 22:01:27 | mriedem | i'm not sure which came first | |
| 22:01:55 | dansmith | mriedem: nah, mine was after | |
| 22:18:24 | openstackgerrit | Dan Smith proposed openstack/nova master: Fix using template cell urls with nova-manage https://review.openstack.org/631649 | |
| 22:20:06 | mriedem | e-r uncategorized should be fixed, will take time to regen the page | |
| 22:20:09 | mriedem | https://review.openstack.org/#/c/631653/1 | |
| 22:41:23 | mriedem | dansmith: +2 | |
| 22:42:20 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Send context.global_id on neutron calls https://review.openstack.org/627603 | |
| 23:10:14 | melwitt | mriedem: looks like we need to be the T specs directory created https://review.openstack.org/630037 (based on seeing someone propose https://review.openstack.org/630817 earlier) | |
| 23:11:32 | mriedem | done | |
| 23:11:58 | melwitt | ty | |
| 23:23:42 | openstackgerrit | Merged openstack/nova-specs master: Create specs directory for Train https://review.openstack.org/630037 | |
| #openstack-nova - 2019-01-18 | |||
| 00:46:14 | openstackgerrit | Merged openstack/nova master: Fix incompatible version handling in BuildRequest https://review.openstack.org/631179 | |
| 01:00:49 | openstackgerrit | Hang Yang proposed openstack/nova master: Fix port dns_name reset https://review.openstack.org/631684 | |
| 01:19:05 | takashin | oomichi: Would you check my comment in https://review.openstack.org/#/c/627282/ ? | |
| 01:39:20 | openstackgerrit | Jack Ding proposed openstack/nova master: Flavor extra spec and image properties validation https://review.openstack.org/620706 | |
| 01:40:12 | openstackgerrit | Jack Ding proposed openstack/nova master: Correct instance port binding for rebuilds https://review.openstack.org/603844 | |
| 01:40:35 | openstackgerrit | Jack Ding proposed openstack/nova master: Preserve UEFI NVRAM variable store https://review.openstack.org/621646 | |
| 01:41:01 | openstackgerrit | Jack Ding proposed openstack/nova master: Improve libvirt image and snapshot handling https://review.openstack.org/616692 | |
| 02:32:35 | fanzhang | we can pass the connect_reties args, but in https://github.com/openstack/keystoneauth/blob/master/keystoneauth1/loading/adapter.py#L205 there are only few args. | |
| 02:32:35 | fanzhang | Hi edleafe, sorry to borther you, I notice that you have some commits to keystoneauth, I'm wondering that do you have any idea to config 'connect_retries' of Adapter in Nova or in Placement? We've met some ('Connection aborted.', BadStatusLine("''",) errors while trying to get allocation candidates, I am not sure adding retries would help. But it seems https://github.com/openstack/keystoneauth/blob/master/keystoneauth1/adapter.py#L46 here | |
| 09:00:13 | openstackgerrit | Merged openstack/nova master: Fix overcommit for NUMA-based instances https://review.openstack.org/629281 | |
| 09:00:54 | jangutter | :-O finally! | |
| 09:08:22 | openstackgerrit | Bhagyashri Shewale proposed openstack/os-resource-classes master: Add PCPU standard resource class https://review.openstack.org/631711 | |
| 09:10:51 | openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: libvirt: Add workaround to cleanup instance dir when using rbd https://review.openstack.org/628726 | |
| 09:12:10 | openstackgerrit | Elod Illes proposed openstack/nova stable/rocky: Fix incompatible version handling in BuildRequest https://review.openstack.org/631731 | |
| 11:04:15 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif master: Import IP implementation modules outside privsep context https://review.openstack.org/629025 | |
| 11:49:52 | openstackgerrit | Daniel Alvarez proposed openstack/os-vif master: Don't set iface-id when creating an OVS port https://review.openstack.org/631782 | |
| 11:52:08 | openstackgerrit | Daniel Alvarez proposed openstack/os-vif master: Don't set iface-id when creating an OVS port https://review.openstack.org/631782 | |
| 12:15:27 | openstackgerrit | Daniel Alvarez proposed openstack/os-vif master: Don't set iface-id when creating an OVS port https://review.openstack.org/631782 | |
| 14:15:55 | edleafe | fanzhang: Sorry, but I haven't worked with keystone in many years. Perhaps someone else here might have more insight into how to debug the BadStatusLine errors. | |
| 14:18:14 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Send context.global_id on neutron calls https://review.openstack.org/627604 | |
| 14:34:55 | mriedem | i've been seeing this traceback in n-cpu logs in the gate, it seems harmless but is ugly, and i haven't dug into what introduced it http://logs.openstack.org/80/630980/5/check/grenade-py3/49818d8/logs/screen-n-cpu.txt.gz#_Jan_17_08_04_31_907868 | |
| 14:35:01 | mriedem | something trying to hit the db from nova-compute | |
| 14:35:10 | mriedem | w/o going through conductor | |
| 14:36:20 | cdent | ah, interesting, I saw that in the vmware ci logs and assumed it was the vmware ci being set up incorrectly (it is, but apparently not because of that) | |
| 14:39:30 | mriedem | kashyap: https://review.openstack.org/#/q/topic:bp/support-qemu-native-tls-for-live-migration+status:open would be OK except for you have libvirt 4.0.0 as a minimum requirement in a couple of places | |
| 14:39:34 | mriedem | which i'm guessing is a typo | |
| 14:40:15 | kashyap | Meeting hell, will check back shortly here; sorry. | |
| 14:41:11 | mriedem | cdent: looks like it's actually in rocky too... | |
| 14:41:33 | mriedem | i think i know what it is | |
| 14:42:46 | kashyap | mriedem: Damn, indeed :-( it is 4.4.0 | |
| 14:42:58 | mriedem | https://review.openstack.org/#/q/Icddbe4760eaff30e4e13c1e8d3d5d3f489dac3c4 | |
| 14:43:10 | mriedem | dansmith: i think ^ is causing the tracebacks in the n-cpu logs on startup | |
| 14:43:13 | kashyap | Eagle eyes. /me curses himself for missing that detail | |
| 14:44:03 | mriedem | well, maybe not | |
| 14:44:43 | mriedem | https://bugs.launchpad.net/nova/+bug/1812398 | |
| 14:44:44 | openstack | Launchpad bug 1812398 in OpenStack Compute (nova) "DBNotAllowed tracebacks in nova-compute logs" [High,Confirmed] | |
| 14:44:45 | dansmith | hmm | |
| 14:45:06 | cdent | mriedem: that conditional seems a bit weird anyway: having a configuration for the api database doesn't mean you have a real connection does it? | |
| 14:45:14 | cdent | (in that process) | |
| 14:45:39 | mriedem | it means compute is configured to be able to reach the api db | |
| 14:45:42 | mriedem | up-calls | |
| 14:47:19 | cdent | that's all it takes? So (accidentally?) reusing conf between a controller node and compute node is bad? | |
| 14:47:41 | cdent | or even between a nova-compute process on a controller node and the rest of the nova processes | |
| 14:48:00 | mriedem | there are some cases where people have to allow nova-compute to hit the api db | |
| 14:48:04 | mriedem | for things that aren't properly isolated yet | |
| 14:48:27 | dansmith | hang on, | |
| 14:48:31 | mriedem | docshttps://docs.openstack.org/nova/latest/user/cellsv2-layout.html#operations-requiring-upcalls | |
| 14:48:49 | cdent | thanks mriedem | |
| 14:48:50 | mriedem | this traceback isn't giving me much | |
| 14:48:57 | dansmith | this traceback would be for the non-api db not api | |
| 14:49:03 | dansmith | yeah, it's confusingly ambiguous | |
| 14:49:24 | mriedem | Jan 17 08:04:31.900276 ubuntu-xenial-inap-mtl01-0001764122 nova-compute[29129]: DEBUG nova.context [None req-6462a6fa-ac8b-4802-bed8-18c335fc623f None None] Found 2 cells: 00000000-0000-0000-0000-000000000000(cell0),2d97f3c8-5ea1-41e1-b849-6cfbf74eefec(cell1) {{(pid=29129) load_cells /opt/stack/old/nova/nova/context.py:479}} | |
| 14:49:32 | mriedem | something is hitting the api db here... | |
| 14:51:00 | dansmith | yeah, so maybe the deal is, | |
| 14:51:09 | dansmith | it hit the api database to get the cell mappings (which it seems to have), | |
| 14:51:24 | dansmith | but then is hitting the db prohibition for compute while trying to get service | |
| 14:51:51 | mriedem | http://logs.openstack.org/80/630980/5/check/grenade-py3/49818d8/logs/screen-n-cpu.txt.gz#_Jan_17_08_04_32_547867 | |
| 14:51:51 | mriedem | well this taint right | |
| 14:51:56 | mriedem | Jan 17 08:04:32.547867 ubuntu-xenial-inap-mtl01-0001764122 nova-compute[29129]: DEBUG oslo_service.service [None req-6462a6fa-ac8b-4802-bed8-18c335fc623f None None] command line args: ['--config-file', '/etc/nova/nova.conf'] {{(pid=29129) log_opt_values /usr/local/lib/python3.5/dist-packages/oslo_config/cfg.py:3008}} | |