| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-23 | |||
| 21:01:26 | mnaser | uhoh | |
| 21:01:38 | mnaser | y'all are never gonna listen to me again | |
| 21:01:38 | dansmith | I was writing off a lot of that given that he's flat, but I guess that ain't right | |
| 21:01:41 | melwitt | jroll: +W | |
| 21:01:46 | mnaser | i just noticed a failure with | |
| 21:01:47 | mnaser | "WSREP has not yet prepared node for application use" | |
| 21:01:54 | jroll | melwitt: thanks | |
| 21:02:29 | mnaser | that is quite possibly the cause why | |
| 21:02:40 | dansmith | I don't know what that menas | |
| 21:02:41 | mnaser | it might have happened exactly in between the update for cells? | |
| 21:02:54 | mnaser | that message comes up if something happens with the galera cluster and it refuses to accept mysql requests | |
| 21:03:04 | mnaser | aka "i am bootstrapping and syncing up with other masters, leave me alone till im ready" | |
| 21:03:08 | dansmith | ah | |
| 21:03:14 | mnaser | so it refuses to complete read/writes | |
| 21:03:16 | openstackgerrit | Eric Fried proposed openstack/nova master: [placement] Add /reshaper handler for POST https://review.openstack.org/576927 | |
| 21:03:17 | openstackgerrit | Eric Fried proposed openstack/nova master: Make get_allocations_for_resource_provider sane https://review.openstack.org/584598 | |
| 21:03:17 | dansmith | well, that would explain a dropped update | |
| 21:03:18 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: Real get_allocs_for_consumer https://review.openstack.org/584599 | |
| 21:03:19 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: get_allocations_for_provider_tree https://review.openstack.org/584648 | |
| 21:03:20 | openstackgerrit | Eric Fried proposed openstack/nova master: reshaper: Look up provider if not in inventories https://review.openstack.org/585033 | |
| 21:03:21 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: _reshape helper, placement min bump https://review.openstack.org/585034 | |
| 21:03:27 | mnaser | urgh, sorry for the noise :( | |
| 21:03:41 | mriedem | mnaser: so you're thinking that happened during the save to make the instance in error state in cell0? | |
| 21:03:47 | mnaser | somehow galera manages to find more people than me to bother! | |
| 21:03:55 | mnaser | mriedem: my guess.. i cant imagine any other scenario | |
| 21:04:03 | mnaser | though i'm still so confused how like | |
| 21:04:08 | mriedem | btw, we could just put those updates on the instance before calling create() here so it's a single operation https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1155 | |
| 21:04:10 | mnaser | i cant manage to find a SINGLE log entry for that instance uuid | |
| 21:04:51 | dansmith | mriedem: the state->error stuff you mean? | |
| 21:04:57 | mriedem | dansmith: right | |
| 21:05:02 | dansmith | yeah | |
| 21:05:03 | mriedem | saves us the other instance.save() later | |
| 21:05:09 | dansmith | although | |
| 21:05:14 | dansmith | I was wondering if that would mess up the notificaitons | |
| 21:05:23 | dansmith | so you'd get a create notification in error state, | |
| 21:05:34 | dansmith | instead of a create of scheduling and then another updated one with error, then deleted, etc | |
| 21:06:34 | mriedem | i don't know if that's a big problem | |
| 21:07:03 | mriedem | depends on the consumer i guess | |
| 21:07:19 | dansmith | well, if you're currently watching for instance.updated,vm_state=ERROR for your logging thing | |
| 21:07:20 | mriedem | but i'm pretty sure we are not good about notifications as a proper state machine | |
| 21:07:23 | dansmith | and now it's instance.created | |
| 21:07:24 | dansmith | yeah | |
| 21:07:28 | dansmith | obvi | |
| 21:07:35 | mriedem | still thinking this is a bad idea https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1178 | |
| 21:07:59 | dansmith | we have to pass there or we won't clean up the BRs | |
| 21:08:32 | dansmith | log it maybe if you want | |
| 21:08:47 | mriedem | mnaser: did you say the instance mapping existed for this instance but the cell_mapping was NULL? | |
| 21:08:56 | dansmith | yes | |
| 21:09:08 | mriedem | which could be the case if the instance.save() in _set_vm_state_and_notify blew up | |
| 21:10:09 | mnaser | yes ^ | |
| 21:10:20 | mriedem | dansmith: also, doesn't it seem like we should bury in cell0 here? https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1244 | |
| 21:10:25 | mriedem | seems like a glaring omission | |
| 21:11:01 | mriedem | well, | |
| 21:11:07 | dansmith | I dunno, | |
| 21:11:10 | dansmith | we've picked a host/cell | |
| 21:11:11 | mriedem | i guess the idea is that if the user deleted the build request, and we don't create it in a cell, | |
| 21:11:13 | mriedem | then you can't list/show it | |
| 21:11:45 | dansmith | ah right this is the pre-create lookup | |
| 21:12:15 | dansmith | could argue that you should still be able to show deleted I guess | |
| 21:12:22 | dansmith | but not sure it's worth changing now | |
| 21:12:46 | mriedem | heh i guess i added that https://github.com/openstack/nova/commit/8ba92778fe14b47ad4ff5b53022e0550a93f37d3 | |
| 21:12:51 | mriedem | i knew it looked familiar | |
| 21:12:57 | dansmith | so on second thought it looks perfect? :) | |
| 21:13:05 | mriedem | it is pretty f'ing nice | |
| 21:13:29 | dansmith | you lost me about two minutes ago since 'Josie' came up on the jukebox | |
| 21:13:47 | dansmith | volume -> 100%, butt -> out of seat | |
| 21:14:29 | mriedem | well it is pretty funky | |
| 21:15:18 | eandersson | What is the general strategy for dealing with hypervisor maintenance (e.g. os updates etc)? | |
| 21:15:43 | eandersson | We wanted to look into a similar implementation to what Amazon does | |
| 21:16:12 | eandersson | Like tagging a hypervisor for downtime | |
| 21:16:31 | mriedem | there used to be a maintenance guide in the ops guide | |
| 21:16:32 | mnaser | i think that's beyond the scope of nova and more like business/operational processes | |
| 21:16:42 | mriedem | but i don't know where the ops guide stuff is being worked on now | |
| 21:17:17 | mriedem | https://github.com/openstack/openstack-manuals/blob/mitaka-eol/doc/ops-guide/source/ops_maintenance.rst | |
| 21:17:50 | eandersson | Yea - I was hoping that maybe someone wrote some exteranl tools for managing this already, or maybe even a feature inbuilt into nova to allow you to set a timer on a compute | |
| 21:18:09 | mnaser | i mean | |
| 21:18:10 | mriedem | didn't we talk about this at some point you and i? | |
| 21:18:14 | eandersson | We did | |
| 21:18:16 | mriedem | you can mark a compute as disabled from scheduling | |
| 21:18:26 | mnaser | and the user will see the status as 'maintenance' or something | |
| 21:18:27 | mriedem | https://developer.openstack.org/api-ref/compute/#compute-services-os-services | |
| 21:18:27 | mnaser | in the api | |
| 21:18:37 | mnaser | i think there is a hoststatus field or something | |
| 21:18:41 | mnaser | unless thats' admin only | |
| 21:18:43 | mriedem | the user doesn't see services | |
| 21:18:53 | mnaser | not but for vms running in that host | |
| 21:18:56 | mriedem | the guests on the host are unaffected if you disable the compute service | |
| 21:18:59 | mnaser | in /servers/<uuid> | |
| 21:19:16 | mriedem | you just can't schedule or migrate new instances to that host while the service is disabled | |
| 21:19:57 | mriedem | you can force the service status to 'down' to evacuate guests from it | |
| 21:20:08 | mriedem | but if it's up and normal otherwise just live migrate | |
| 21:20:42 | mnaser | eandersson: and for customer facing stuff, the nova instance has field 'host_status' which will be 'MAINTENANCE' if the vm is ona disabled host | |
| 21:20:49 | mnaser | so that might be a way to warn consumers that something will happen | |
| 21:20:55 | mnaser | if you wanna say disable it for a few days before | |
| 21:21:09 | mnaser | https://developer.openstack.org/api-ref/compute/#show-server-details | |
| 21:21:13 | mnaser | i swear it's there >:( | |
| 21:21:29 | mriedem | oh right, | |
| 21:21:32 | mriedem | admin only by default | |
| 21:21:35 | mriedem | configurable via policy | |
| 21:21:48 | mnaser | ah yeah, so maybe that might be something you want to mess with policy then | |