| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-23 | |||
| 20:50:29 | melwitt | okay, yeah so that's what dansmith was saying if the notify failed (or if nova-conductor somehow was restarted at the exact moment) we'd create in cell0 and not update the instance mapping | |
| 20:53:43 | mnaser | yeah | |
| 20:53:49 | mnaser | rabbit was having a field day at that point in time | |
| 20:54:06 | dansmith | but the task/vm states should be updated, as mriedem was saying | |
| 20:54:08 | dansmith | and they're not | |
| 20:54:16 | mnaser | Unexpected error during heartbeart thread processing, retrying...: error: [Errno 104] ECONNRESET | |
| 20:54:40 | mriedem | mnaser: do you see this in the logs for that instance? https://github.com/openstack/nova/blob/master/nova/scheduler/utils.py#L581 | |
| 20:54:58 | mnaser | in conductor right? | |
| 20:55:00 | mriedem | yeah | |
| 20:55:06 | mriedem | dansmith: oh btw, instance.save() sends it's own notification | |
| 20:55:11 | mriedem | i always forget about that | |
| 20:55:16 | dansmith | true story | |
| 20:55:19 | mriedem | should be after the db update | |
| 20:56:23 | mnaser | not a single reference to that instance id in the logs | |
| 20:56:24 | mnaser | :\ | |
| 20:56:34 | mnaser | in that range of time at least | |
| 20:56:41 | mnaser | unless zgrep is failing me but i dont think so | |
| 20:56:49 | dansmith | well, that's good | |
| 20:57:03 | dansmith | mnaser: whatabout https://github.com/openstack/nova/blob/master/nova/scheduler/utils.py#L567 | |
| 20:57:15 | mnaser | i see other references to it but this instance odes not have one | |
| 20:57:59 | mriedem | Failed to compute_task_build_instances | |
| 20:58:16 | mnaser | let me get the tenant id of this vm | |
| 20:58:55 | mnaser | just no valid hosts | |
| 20:59:06 | openstackgerrit | Jim Rollenhagen proposed openstack/nova master: ironic: add instance_uuid before any other spawn activity https://review.openstack.org/563722 | |
| 20:59:06 | dansmith | and this if you have debug on: https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1098 | |
| 20:59:10 | mnaser | hmm | |
| 20:59:19 | mnaser | "Failed to get aggregates for resource provider with UUID 04ae18f3-7226-4d0f-81f3-74a73eb1722a" | |
| 20:59:25 | mnaser | for some failures in the similar timeline | |
| 20:59:37 | dansmith | mnaser: and I guess you've looked for any stack traces in this instance's build time window yeah? | |
| 21:00:06 | mnaser | if an instance tries scheduling several times and fails does it end up in cell0 as well? | |
| 21:00:07 | jroll | melwitt: jaypipes: you were both +2 on https://review.openstack.org/#/c/563722/10, had a trivial rebase conflict | |
| 21:00:13 | dansmith | mnaser: yeah | |
| 21:00:23 | dansmith | novalidhost -> cell0 | |
| 21:00:27 | mriedem | tries scheduling several times? | |
| 21:00:34 | mnaser | well as in | |
| 21:00:38 | mnaser | retrying build on another host | |
| 21:00:38 | mriedem | reschedules? | |
| 21:00:45 | mnaser | i guess thats the term | |
| 21:00:49 | mriedem | not in this case | |
| 21:00:55 | mriedem | reschedules happen within a cell | |
| 21:01:07 | mriedem | cell0 is when you don't get a host to start with | |
| 21:01:17 | dansmith | oh I guess that's right, once you've picked your first host, you're stuck in there | |
| 21:01:25 | mnaser | oh | |
| 21:01:26 | mnaser | uhoh | |
| 21:01:38 | mnaser | y'all are never gonna listen to me again | |
| 21:01:38 | dansmith | I was writing off a lot of that given that he's flat, but I guess that ain't right | |
| 21:01:41 | melwitt | jroll: +W | |
| 21:01:46 | mnaser | i just noticed a failure with | |
| 21:01:47 | mnaser | "WSREP has not yet prepared node for application use" | |
| 21:01:54 | jroll | melwitt: thanks | |
| 21:02:29 | mnaser | that is quite possibly the cause why | |
| 21:02:40 | dansmith | I don't know what that menas | |
| 21:02:41 | mnaser | it might have happened exactly in between the update for cells? | |
| 21:02:54 | mnaser | that message comes up if something happens with the galera cluster and it refuses to accept mysql requests | |
| 21:03:04 | mnaser | aka "i am bootstrapping and syncing up with other masters, leave me alone till im ready" | |
| 21:03:08 | dansmith | ah | |
| 21:03:14 | mnaser | so it refuses to complete read/writes | |
| 21:03:16 | openstackgerrit | Eric Fried proposed openstack/nova master: [placement] Add /reshaper handler for POST https://review.openstack.org/576927 | |
| 21:03:17 | openstackgerrit | Eric Fried proposed openstack/nova master: Make get_allocations_for_resource_provider sane https://review.openstack.org/584598 | |
| 21:03:17 | dansmith | well, that would explain a dropped update | |
| 21:03:18 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: Real get_allocs_for_consumer https://review.openstack.org/584599 | |
| 21:03:19 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: get_allocations_for_provider_tree https://review.openstack.org/584648 | |
| 21:03:20 | openstackgerrit | Eric Fried proposed openstack/nova master: reshaper: Look up provider if not in inventories https://review.openstack.org/585033 | |
| 21:03:21 | openstackgerrit | Eric Fried proposed openstack/nova master: Report client: _reshape helper, placement min bump https://review.openstack.org/585034 | |
| 21:03:27 | mnaser | urgh, sorry for the noise :( | |
| 21:03:41 | mriedem | mnaser: so you're thinking that happened during the save to make the instance in error state in cell0? | |
| 21:03:47 | mnaser | somehow galera manages to find more people than me to bother! | |
| 21:03:55 | mnaser | mriedem: my guess.. i cant imagine any other scenario | |
| 21:04:03 | mnaser | though i'm still so confused how like | |
| 21:04:08 | mriedem | btw, we could just put those updates on the instance before calling create() here so it's a single operation https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1155 | |
| 21:04:10 | mnaser | i cant manage to find a SINGLE log entry for that instance uuid | |
| 21:04:51 | dansmith | mriedem: the state->error stuff you mean? | |
| 21:04:57 | mriedem | dansmith: right | |
| 21:05:02 | dansmith | yeah | |
| 21:05:03 | mriedem | saves us the other instance.save() later | |
| 21:05:09 | dansmith | although | |
| 21:05:14 | dansmith | I was wondering if that would mess up the notificaitons | |
| 21:05:23 | dansmith | so you'd get a create notification in error state, | |
| 21:05:34 | dansmith | instead of a create of scheduling and then another updated one with error, then deleted, etc | |
| 21:06:34 | mriedem | i don't know if that's a big problem | |
| 21:07:03 | mriedem | depends on the consumer i guess | |
| 21:07:19 | dansmith | well, if you're currently watching for instance.updated,vm_state=ERROR for your logging thing | |
| 21:07:20 | mriedem | but i'm pretty sure we are not good about notifications as a proper state machine | |
| 21:07:23 | dansmith | and now it's instance.created | |
| 21:07:24 | dansmith | yeah | |
| 21:07:28 | dansmith | obvi | |
| 21:07:35 | mriedem | still thinking this is a bad idea https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1178 | |
| 21:07:59 | dansmith | we have to pass there or we won't clean up the BRs | |
| 21:08:32 | dansmith | log it maybe if you want | |
| 21:08:47 | mriedem | mnaser: did you say the instance mapping existed for this instance but the cell_mapping was NULL? | |
| 21:08:56 | dansmith | yes | |
| 21:09:08 | mriedem | which could be the case if the instance.save() in _set_vm_state_and_notify blew up | |
| 21:10:09 | mnaser | yes ^ | |
| 21:10:20 | mriedem | dansmith: also, doesn't it seem like we should bury in cell0 here? https://github.com/openstack/nova/blob/master/nova/conductor/manager.py#L1244 | |
| 21:10:25 | mriedem | seems like a glaring omission | |
| 21:11:01 | mriedem | well, | |
| 21:11:07 | dansmith | I dunno, | |
| 21:11:10 | dansmith | we've picked a host/cell | |
| 21:11:11 | mriedem | i guess the idea is that if the user deleted the build request, and we don't create it in a cell, | |
| 21:11:13 | mriedem | then you can't list/show it | |