| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-16 | |||
| 21:33:53 | jroll | can we please land https://review.openstack.org/#/c/527423/ . | |
| 21:33:56 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Make sure rebalance saves the compute node https://review.openstack.org/545464 | |
| 21:34:07 | fried_rice | jroll: Here's a quickie to make sure that save() is happening ^ | |
| 21:34:40 | fried_rice | jroll: But where the error happens, we're clearly running that code. | |
| 21:34:45 | jroll | er, the one I'm looking at is queens->master | |
| 21:34:47 | jroll | yeah | |
| 21:35:01 | fried_rice | So whereas I agree we should land that backport, that's not gonna be your fix here. | |
| 21:35:15 | mriedem | jroll: it doesn't | |
| 21:35:20 | jroll | sorry, we've been having issues with this job since before queens was cut, so I've been getting confused | |
| 21:35:23 | mriedem | compute_nodes hang out until manually removed | |
| 21:35:29 | jroll | damn | |
| 21:35:56 | mriedem | there are some bugs that tssurya opened for removing compute nodes and providers | |
| 21:36:14 | jroll | I was hoping compute nodes did get deleted but not RPs | |
| 21:36:18 | jroll | that would explain things | |
| 21:36:26 | mriedem | https://bugs.launchpad.net/nova/+bug/1749734 | |
| 21:36:26 | openstack | Launchpad bug 1749734 in OpenStack Compute (nova) "Purge the compute_node records, resource provider records and host_mappings when doing force delete of the host" [Medium,Confirmed] - Assigned to Surya Seetharaman (tssurya) | |
| 21:36:30 | jroll | thanks | |
| 21:37:26 | fried_rice | jroll: I can get a little more aggressive with the hacking. | |
| 21:38:27 | jroll | fried_rice: feel free, I'm just trying to wrap my head around some of this | |
| 21:38:48 | jroll | this seems wrong. http://logs.openstack.org/50/544750/10/check/ironic-grenade-dsvm-multinode-multitenant/d7a1ee7/logs/screen-n-cpu.txt.gz#_Feb_16_17_06_33_592313 | |
| 21:39:09 | jroll | oh, there we are http://logs.openstack.org/50/544750/10/check/ironic-grenade-dsvm-multinode-multitenant/d7a1ee7/logs/screen-n-cpu.txt.gz#_Feb_16_17_04_22_814020 | |
| 21:39:35 | jroll | but we can't talk to placement, so the RP stays | |
| 21:39:57 | jroll | because Feb 16 17:04:22.591129 ubuntu-xenial-rax-ord-0002580076 nova-compute[28778]: DEBUG nova.virt.ironic.driver [None req-969cdb75-026b-4cac-ba09-3f3be962a09d service nova] Returning 0 available node(s) {{(pid=28778) get_available_nodes /opt/stack/old/nova/nova/virt/ironic/driver.py:757}} | |
| 21:40:01 | jroll | because ironic is down | |
| 21:40:05 | jroll | got dang. | |
| 21:40:30 | jroll | I remember this code landing to fix something else | |
| 21:40:47 | fried_rice | jroll: Is there any chance that this ironic node being rebalanced has allocations? | |
| 21:41:05 | jroll | fried_rice: yes, we create an instance before the upgrade AFAIK | |
| 21:41:43 | fried_rice | So we have to do more than just delete the old RP. We have to move his allocations to the new one. This ain't gonna work. | |
| 21:41:50 | jroll | this crap is burning us: https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L607-L616 | |
| 21:41:57 | jroll | right, we shouldn't be deleting the compute node or the RP | |
| 21:42:16 | jroll | so here's what's going on, in short: | |
| 21:42:26 | fried_rice | The answer is quite simply to make sure the compute node doesn't change its freakin uuid. | |
| 21:42:42 | jroll | n-cpu has a bunch of ironic nodes it's managing | |
| 21:42:48 | jroll | ironic goes down for upgrade | |
| 21:42:54 | jroll | n-cpu does a RT update | |
| 21:42:59 | jroll | n-cpu can't reach ironic | |
| 21:43:18 | jroll | n-cpu thinks all the ironic nodes are gone, for good, as if ironic returned I have no nodes | |
| 21:43:25 | jroll | n-cpu deletes the compute_node records | |
| 21:43:28 | jroll | ironic comes back | |
| 21:43:40 | jroll | n-cpu does an RT update, sees nodes, creates compute_node records | |
| 21:44:17 | jroll | meanwhile, n-cpu couldn't delete the resource providers from placement, and so it tries to create new ones and *boom* | |
| 21:45:02 | fried_rice | n-cpu deletes the compute_node records? | |
| 21:45:02 | jroll | btw, s/ironic goes down for upgrade/keystone goes down for upgrade/, which is why neither ironic nor placement can be reached | |
| 21:45:17 | fried_rice | I thought we decided that wasn't happening. | |
| 21:45:21 | jroll | yes | |
| 21:45:25 | jroll | http://logs.openstack.org/50/544750/10/check/ironic-grenade-dsvm-multinode-multitenant/d7a1ee7/logs/screen-n-cpu.txt.gz#_Feb_16_17_04_22_814020 | |
| 21:45:40 | jroll | because: http://logs.openstack.org/50/544750/10/check/ironic-grenade-dsvm-multinode-multitenant/d7a1ee7/logs/screen-n-cpu.txt.gz#_Feb_16_17_04_22_591129 | |
| 21:45:48 | jroll | because: https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L607-L616 | |
| 21:46:42 | jroll | (maybe it is apache that is down, don't know, don't care, things are down and n-cpu can't deal) | |
| 21:47:44 | jroll | for the curious, the commit message and bug here explain why we're returning an empty list of nodes there: https://review.openstack.org/#/c/487925/ | |
| 21:48:04 | jroll | fried_rice: that all make sense? | |
| 21:48:11 | jroll | TheJulia: ^ fyi, I think I nailed it down. | |
| 21:50:33 | jroll | thanks, my gas can is nearly empty | |
| 21:53:33 | jroll | turns out lying to the resource tracker is wrong, who'da thought | |
| 21:57:05 | jroll | so the only way n-cpu will blow up by ironic not being reachable is because of this: https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L524 | |
| 21:57:10 | jroll | which will happen on the first RT run | |
| 21:57:54 | jroll | or not even? wtf | |
| 21:58:29 | jroll | oh, there used to be a _refresh_cache() there | |
| 21:58:54 | jroll | but even before 487925 we would return an empty list | |
| 21:59:57 | jroll | ah jeez https://github.com/openstack/nova/commit/cce06a1e9855d9eed3f7c653200853f23466d791 | |
| 22:01:19 | jroll | hm, 5pm friday | |
| 22:01:35 | fried_rice | jroll: Need anything from me? | |
| 22:01:52 | jroll | fried_rice: whiskey may be needed | |
| 22:01:54 | jroll | :) | |
| 22:01:56 | fried_rice | Can https://review.openstack.org/545464 be abandoned? | |
| 22:02:06 | jroll | yes, believe so | |
| 22:02:27 | fried_rice | jroll: The fact that you're a time zone ahead of me indicates I have no way of getting you a bottle in time to save you. | |
| 22:02:27 | TheJulia | jroll: I will buy you whiskey in Dublin | |
| 22:02:34 | fried_rice | Yeah, that ^ | |
| 22:02:36 | jroll | heh | |
| 22:02:57 | TheJulia | And next time I'm through your part of the country, I'll make a point of bringing really good whiskey on my RV | |
| 22:03:15 | jroll | :o <3 | |
| 22:16:19 | mrjk | About nova.cfg, something is not clear. Let's say I've conductor, api and any other service. Most of settings are in the DEFAULT section, but is it possible to override some default parameters for a specific service ? Let's say I want to change the debug mode for only one service, and not the others ... | |
| 22:17:01 | mriedem | debug is global | |
| 22:17:19 | mriedem | as long as you have all of your controller services running on the same host, they are going to share config from the [DEFAULT] section | |
| 22:17:21 | mrjk | How could I find out this info by myself ? | |
| 22:17:50 | mriedem | you could split configs and create an /etc/nova/nova-api.conf which has config specific to your API service | |
| 22:17:59 | mriedem | and remove debug from the base /etc/nova/nova.conf | |
| 22:18:05 | mriedem | then run the service with both config files | |
| 22:18:19 | mrjk | Ok, this would be the way to go. I was unsure, I believed there was a kind defaulting/override values | |
| 22:18:21 | mriedem | nova-api --config-file /etc/nova/nova.conf --config-file /etc/nova/nova-api.conf | |
| 22:26:28 | openstackgerrit | Matthew Treinish proposed openstack/nova master: Remove single quotes from posargs on stestr run commands https://review.openstack.org/545476 | |
| 22:26:43 | mtreinish | melwitt: ^^^ but lets test this and make sure I'm not just seeing things... | |
| 22:27:16 | melwitt | k, I'll try it | |
| 22:42:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix error handling in compute API for multiattach errors https://review.openstack.org/545478 | |
| 22:42:54 | mriedem | well i wish i would have found this before we cut RC2 ^ because that's an annoying UX problem | |
| 22:46:28 | openstackgerrit | Jim Rollenhagen proposed openstack/nova master: ironic: stop lying to the RT when ironic is down https://review.openstack.org/545479 | |
| 22:46:54 | jroll | TheJulia: fried_rice: ^ that fixes it, but will crash at startup if ironic is down | |
| 22:47:26 | fried_rice | jroll: Maybe we *should* crash at startup if ironic is down. | |
| 22:48:17 | jroll | fried_rice: yeah, I kind of agree, kind of don't, regardless crashing when ironic is down was a huge pain in CI in the past that I don't want to live again | |
| 22:48:37 | jroll | I also feel like I want to be able to start my computes whenever and have them do stuff when ironic comes back | |
| 22:49:19 | jroll | though I don't believe in upgrading nova and ironic at the same time (or even the same maintenance window), other people do and this makes their life easier | |
| 22:49:37 | mriedem | nova-compute doesn't start if we can't connect to libvirt | |
| 22:49:52 | mriedem | i think the same for powervm? | |
| 22:49:58 | mriedem | not sure about hyperv/xen/vmware | |
| 22:50:17 | mriedem | nova-compute shouldn't come up, | |
| 22:50:27 | mriedem | because then the service will say it's up, and be around for scheduling, | |
| 22:50:38 | mriedem | and will just not work if the scheduler picks it and the hypervisor is gone | |
| 22:50:44 | jroll | mriedem: yeah, but libvirt isn't some external service, that just means you've configured your hypervisor wrong | |