| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-02-16 | |||
| 21:38:27 | jroll | fried_rice: feel free, I'm just trying to wrap my head around some of this | |
| 21:38:48 | jroll | this seems wrong. http://logs.openstack.org/50/544750/10/check/ironic-grenade-dsvm-multinode-multitenant/d7a1ee7/logs/screen-n-cpu.txt.gz#_Feb_16_17_06_33_592313 | |
| 21:39:09 | jroll | oh, there we are http://logs.openstack.org/50/544750/10/check/ironic-grenade-dsvm-multinode-multitenant/d7a1ee7/logs/screen-n-cpu.txt.gz#_Feb_16_17_04_22_814020 | |
| 21:39:35 | jroll | but we can't talk to placement, so the RP stays | |
| 21:39:57 | jroll | because Feb 16 17:04:22.591129 ubuntu-xenial-rax-ord-0002580076 nova-compute[28778]: DEBUG nova.virt.ironic.driver [None req-969cdb75-026b-4cac-ba09-3f3be962a09d service nova] Returning 0 available node(s) {{(pid=28778) get_available_nodes /opt/stack/old/nova/nova/virt/ironic/driver.py:757}} | |
| 21:40:01 | jroll | because ironic is down | |
| 21:40:05 | jroll | got dang. | |
| 21:40:30 | jroll | I remember this code landing to fix something else | |
| 21:40:47 | fried_rice | jroll: Is there any chance that this ironic node being rebalanced has allocations? | |
| 21:41:05 | jroll | fried_rice: yes, we create an instance before the upgrade AFAIK | |
| 21:41:43 | fried_rice | So we have to do more than just delete the old RP. We have to move his allocations to the new one. This ain't gonna work. | |
| 21:41:50 | jroll | this crap is burning us: https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L607-L616 | |
| 21:41:57 | jroll | right, we shouldn't be deleting the compute node or the RP | |
| 21:42:16 | jroll | so here's what's going on, in short: | |
| 21:42:26 | fried_rice | The answer is quite simply to make sure the compute node doesn't change its freakin uuid. | |
| 21:42:42 | jroll | n-cpu has a bunch of ironic nodes it's managing | |
| 21:42:48 | jroll | ironic goes down for upgrade | |
| 21:42:54 | jroll | n-cpu does a RT update | |
| 21:42:59 | jroll | n-cpu can't reach ironic | |
| 21:43:18 | jroll | n-cpu thinks all the ironic nodes are gone, for good, as if ironic returned I have no nodes | |
| 21:43:25 | jroll | n-cpu deletes the compute_node records | |
| 21:43:28 | jroll | ironic comes back | |
| 21:43:40 | jroll | n-cpu does an RT update, sees nodes, creates compute_node records | |
| 21:44:17 | jroll | meanwhile, n-cpu couldn't delete the resource providers from placement, and so it tries to create new ones and *boom* | |
| 21:45:02 | fried_rice | n-cpu deletes the compute_node records? | |
| 21:45:02 | jroll | btw, s/ironic goes down for upgrade/keystone goes down for upgrade/, which is why neither ironic nor placement can be reached | |
| 21:45:17 | fried_rice | I thought we decided that wasn't happening. | |
| 21:45:21 | jroll | yes | |
| 21:45:25 | jroll | http://logs.openstack.org/50/544750/10/check/ironic-grenade-dsvm-multinode-multitenant/d7a1ee7/logs/screen-n-cpu.txt.gz#_Feb_16_17_04_22_814020 | |
| 21:45:40 | jroll | because: http://logs.openstack.org/50/544750/10/check/ironic-grenade-dsvm-multinode-multitenant/d7a1ee7/logs/screen-n-cpu.txt.gz#_Feb_16_17_04_22_591129 | |
| 21:45:48 | jroll | because: https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L607-L616 | |
| 21:46:42 | jroll | (maybe it is apache that is down, don't know, don't care, things are down and n-cpu can't deal) | |
| 21:47:44 | jroll | for the curious, the commit message and bug here explain why we're returning an empty list of nodes there: https://review.openstack.org/#/c/487925/ | |
| 21:48:04 | jroll | fried_rice: that all make sense? | |
| 21:48:11 | jroll | TheJulia: ^ fyi, I think I nailed it down. | |
| 21:50:33 | jroll | thanks, my gas can is nearly empty | |
| 21:53:33 | jroll | turns out lying to the resource tracker is wrong, who'da thought | |
| 21:57:05 | jroll | so the only way n-cpu will blow up by ironic not being reachable is because of this: https://github.com/openstack/nova/blob/master/nova/virt/ironic/driver.py#L524 | |
| 21:57:10 | jroll | which will happen on the first RT run | |
| 21:57:54 | jroll | or not even? wtf | |
| 21:58:29 | jroll | oh, there used to be a _refresh_cache() there | |
| 21:58:54 | jroll | but even before 487925 we would return an empty list | |
| 21:59:57 | jroll | ah jeez https://github.com/openstack/nova/commit/cce06a1e9855d9eed3f7c653200853f23466d791 | |
| 22:01:19 | jroll | hm, 5pm friday | |
| 22:01:35 | fried_rice | jroll: Need anything from me? | |
| 22:01:52 | jroll | fried_rice: whiskey may be needed | |
| 22:01:54 | jroll | :) | |
| 22:01:56 | fried_rice | Can https://review.openstack.org/545464 be abandoned? | |
| 22:02:06 | jroll | yes, believe so | |
| 22:02:27 | fried_rice | jroll: The fact that you're a time zone ahead of me indicates I have no way of getting you a bottle in time to save you. | |
| 22:02:27 | TheJulia | jroll: I will buy you whiskey in Dublin | |
| 22:02:34 | fried_rice | Yeah, that ^ | |
| 22:02:36 | jroll | heh | |
| 22:02:57 | TheJulia | And next time I'm through your part of the country, I'll make a point of bringing really good whiskey on my RV | |
| 22:03:15 | jroll | :o <3 | |
| 22:16:19 | mrjk | About nova.cfg, something is not clear. Let's say I've conductor, api and any other service. Most of settings are in the DEFAULT section, but is it possible to override some default parameters for a specific service ? Let's say I want to change the debug mode for only one service, and not the others ... | |
| 22:17:01 | mriedem | debug is global | |
| 22:17:19 | mriedem | as long as you have all of your controller services running on the same host, they are going to share config from the [DEFAULT] section | |
| 22:17:21 | mrjk | How could I find out this info by myself ? | |
| 22:17:50 | mriedem | you could split configs and create an /etc/nova/nova-api.conf which has config specific to your API service | |
| 22:17:59 | mriedem | and remove debug from the base /etc/nova/nova.conf | |
| 22:18:05 | mriedem | then run the service with both config files | |
| 22:18:19 | mrjk | Ok, this would be the way to go. I was unsure, I believed there was a kind defaulting/override values | |
| 22:18:21 | mriedem | nova-api --config-file /etc/nova/nova.conf --config-file /etc/nova/nova-api.conf | |
| 22:26:28 | openstackgerrit | Matthew Treinish proposed openstack/nova master: Remove single quotes from posargs on stestr run commands https://review.openstack.org/545476 | |
| 22:26:43 | mtreinish | melwitt: ^^^ but lets test this and make sure I'm not just seeing things... | |
| 22:27:16 | melwitt | k, I'll try it | |
| 22:42:52 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix error handling in compute API for multiattach errors https://review.openstack.org/545478 | |
| 22:42:54 | mriedem | well i wish i would have found this before we cut RC2 ^ because that's an annoying UX problem | |
| 22:46:28 | openstackgerrit | Jim Rollenhagen proposed openstack/nova master: ironic: stop lying to the RT when ironic is down https://review.openstack.org/545479 | |
| 22:46:54 | jroll | TheJulia: fried_rice: ^ that fixes it, but will crash at startup if ironic is down | |
| 22:47:26 | fried_rice | jroll: Maybe we *should* crash at startup if ironic is down. | |
| 22:48:17 | jroll | fried_rice: yeah, I kind of agree, kind of don't, regardless crashing when ironic is down was a huge pain in CI in the past that I don't want to live again | |
| 22:48:37 | jroll | I also feel like I want to be able to start my computes whenever and have them do stuff when ironic comes back | |
| 22:49:19 | jroll | though I don't believe in upgrading nova and ironic at the same time (or even the same maintenance window), other people do and this makes their life easier | |
| 22:49:37 | mriedem | nova-compute doesn't start if we can't connect to libvirt | |
| 22:49:52 | mriedem | i think the same for powervm? | |
| 22:49:58 | mriedem | not sure about hyperv/xen/vmware | |
| 22:50:17 | mriedem | nova-compute shouldn't come up, | |
| 22:50:27 | mriedem | because then the service will say it's up, and be around for scheduling, | |
| 22:50:38 | mriedem | and will just not work if the scheduler picks it and the hypervisor is gone | |
| 22:50:44 | jroll | mriedem: yeah, but libvirt isn't some external service, that just means you've configured your hypervisor wrong | |
| 22:51:05 | mriedem | vcenter is an external service | |
| 22:51:07 | jroll | and at least in ironic's case, there won't be any resources to schedule to, until it can connect to ironic | |
| 22:51:19 | jroll | or I guess there will, sigh | |
| 22:51:23 | fried_rice | In powervm, it looks like we'll hold up init_host for a while if we can't talk to the hypervisor, and then we'll ultimately blow up. | |
| 22:51:50 | mriedem | i thought someone's dog was going to get walked? | |
| 22:51:54 | mriedem | i can hear him whining from here | |
| 22:51:56 | jroll | good point | |
| 22:52:00 | fried_rice | But I've got a nice TODO there to make it work like I73a34eb6e0ca32d03e54d12a5e066b2ed4f19a61 which will actually disable the compute service (but not crash it) in that case. | |
| 22:52:00 | jroll | bbiab | |
| 22:53:16 | mriedem | melwitt: did you figure this out? https://review.openstack.org/#/c/340614/18/nova/compute/api.py@2029 | |
| 22:53:21 | mriedem | you had >1 attachment right> | |
| 22:53:22 | mriedem | ? | |
| 22:53:35 | mriedem | and that's why the volume status wasn't changing to 'available'? | |
| 22:55:39 | melwitt | mriedem: I had multiple attachments because I was having trouble getting the code path to hit in devstack. so I tried the scenario multiple times with the same volume by reset-state on it | |
| 22:55:45 | melwitt | and didn't notice it was building up attachments | |
| 22:56:14 | melwitt | so once I started from a clean slate, new volume and did the scenario, it worked as expected. the volume actually had no attachments in the fresh volume case. it was 'reserved' with no attachments | |
| 22:56:42 | melwitt | then the attachment_delete change it from 'reserved' -> 'available', then the volume_api.delete deleted the volume properly | |
| 22:57:34 | melwitt | so I think all is well | |