| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-02 | |||
| 13:41:13 | lyarwood | sean-k-mooney: ack, so ManagedSave doesn't pause the domain, fun. | |
| 13:41:29 | mriedem | oh nvm, | |
| 13:41:38 | mriedem | i guess that's the exception handling from the yield on the context manager, | |
| 13:41:48 | mriedem | here https://github.com/openstack/nova/blob/7125dcb9cb821faf3c68526ac34365a28141e480/nova/compute/manager.py#L2092 | |
| 13:41:54 | mriedem | which is what calls driver.spawn() | |
| 13:41:57 | mriedem | god this flow is confusing | |
| 13:42:26 | lyarwood | mriedem: yeah indeed, it's awkward. | |
| 13:43:34 | mriedem | ok and the bdm.attachment_id in this flow is created in the API right? | |
| 13:43:45 | lyarwood | mriedem: yes | |
| 13:43:49 | sean-k-mooney | lyarwood: i have to go into the libvirt python binding to check but no code in nova does. anyway ill keep digging and see what i find | |
| 13:43:53 | mriedem | so api creates the attachment record, we hit hostA, fail to spawn, delete the attachment, reshcedule to hostB, try to attach and the attachment is gone | |
| 13:44:07 | lyarwood | mriedem: yup that's it | |
| 13:57:13 | mdbooth | mriedem: spent a bunch of time thinking about your rollback failed evacuated guest patch. TL;DR I think we need to clean that up in the driver, and if we get out of the driver with a running instance we shouldn't be talking about rollbacks any more. | |
| 13:57:51 | mdbooth | mriedem: Detail in review. | |
| 14:00:03 | mriedem | mdbooth: thanks; i asked in the bug report if they had details on what the actual failure was after the guest was spawned on the dest - like if it was a db error updating the instance status or something | |
| 14:00:12 | mriedem | i'm not in love with this patch as noted in the commit message | |
| 14:00:41 | mdbooth | mriedem: ack. I got that. | |
| 14:00:48 | openstackgerrit | Merged openstack/nova stable/queens: Call generate_image_url only for legacy notification https://review.openstack.org/584969 | |
| 14:06:25 | mriedem | the bug reporter replied with what they failed on the first time they hit the dest host, | |
| 14:06:29 | mriedem | and apparently it was driver.spawn() | |
| 14:06:59 | mriedem | message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken." | |
| 14:06:59 | mriedem | "libvirtError: Did not receive a reply. Possible causes include: the remote application did not send a reply, the | |
| 14:13:28 | stephenfin | gibi: If I run 'tox -e api-samples', I see three new files in doc/api_samples/. Is that expected? | |
| 14:13:52 | stephenfin | gibi: http://paste.openstack.org/show/727146/ | |
| 14:15:37 | gibi | stephenfin: actually I don't know | |
| 14:15:58 | gibi | the notification sample handling is pretty different from the api sample handling | |
| 14:16:16 | gibi | for example the test run never generates notificaton samples | |
| 14:16:22 | gibi | on the file system | |
| 14:16:30 | stephenfin | gibi: Hmm, I wonder who would know? gmann? | |
| 14:16:51 | gibi | sdauge was the mastermind but alex_xu or gmann could know | |
| 14:18:15 | mdbooth | mriedem: That sounds like a bug in the libvirt driver and/or libvirt to me. | |
| 14:21:05 | mriedem | mdbooth: agree | |
| 14:21:11 | mriedem | this was also mitaka so shrug | |
| 14:23:35 | mriedem | lyarwood: hmm, am i just not seeing it, but with the old style attach flow during bfv, if we failed to attach the volume, i don't see that we ever unreserve the volume from the instance | |
| 14:23:51 | mriedem | the api reserves the volume, but compute never unreserves it on failure | |
| 14:23:55 | mriedem | unlike if attach_volume fails | |
| 14:24:38 | mriedem | maybe that's just always been the way it is and expected b/c you can delete the instance in ERROR state which should unreserve the volume then | |
| 14:25:30 | mriedem | i guess _shutdown_instance calls the os-detach api in cinder but i don't know if that rolls back the reserved status | |
| 14:26:23 | lyarwood | mriedem: yeah I don't think we did unreserve in the old flow when we hit this error | |
| 14:26:44 | lyarwood | mriedem: I thought we had talked about updating attachments in the new flow with a None connector in this case | |
| 14:27:15 | lyarwood | mriedem: so the new compute only has to come along and update again with the correct connector | |
| 14:27:20 | mriedem | i think the os-detach call to cinder for the old flow in _shutdown_instance will make the volume available again | |
| 14:27:46 | lyarwood | kk then our removal of the attachment is fine | |
| 14:28:16 | mriedem | here is i think we're we'd get in the old flow prior to reschedule https://github.com/openstack/cinder/blob/b0e9ee1d501fb83b7ebbc59584aa6255dbaec086/cinder/volume/manager.py#L1314 | |
| 14:28:21 | mriedem | *where we'd | |
| 14:31:05 | openstackgerrit | Jay Pipes proposed openstack/nova master: DNM - example https://review.openstack.org/588295 | |
| 14:32:23 | openstackgerrit | Konstantinos Samaras-Tsakiris proposed openstack/nova master: Hide hypervisor id on windows guests https://review.openstack.org/579897 | |
| 14:32:25 | mriedem | lyarwood: ok comments in your series | |
| 14:32:35 | mriedem | lyarwood: lots of internal debate on this one, but i think what you're doing is likely the best | |
| 14:33:00 | lyarwood | mriedem: ack thanks | |
| 14:34:55 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP/PoC: safe_connect shouldn't hide failures https://review.openstack.org/584593 | |
| 14:41:39 | gmann | stephenfin: gibi api-samples tox run sample file with GENERATE_SAMPLES=True | |
| 14:42:03 | gmann | stephenfin: there can be chance that few file missing in doc/api_samples let me check those | |
| 14:45:26 | mriedem | lyarwood: looks like we've probably always created another volume on a reschedule if the source_type!='volume', i don't see anything that deletes a volume that nova created prior to rescheduling - unrelated to your issue really, but a super latent bug | |
| 14:46:16 | mriedem | most likely another reason we should move volume creation to api or conductor so that compute doesn't have to manage that | |
| 14:46:37 | mriedem | compute would just get a bdm and assume the volume already was created | |
| 14:49:54 | lyarwood | mriedem: wonderful, happy to look at that as well if you wouldn't mind writing that up in a bug? | |
| 14:50:37 | mriedem | i'd need to recreate it first, and don't have a 2 node system handy | |
| 14:51:13 | mriedem | i just know we do a better job of tracking ports to know which we've created ourselves and which we haven't, and cleaning those up properly before reschedule (delete the ports we created, unbind the ports we didn't) | |
| 14:51:29 | mriedem | we don't really do anything like that for volumes | |
| 14:59:55 | openstackgerrit | Merged openstack/nova master: tox: Ensure reused envdirs share the same deps https://review.openstack.org/588207 | |
| 15:16:40 | cfriesen | this is kind of a basic question but I'm having a hard time finding an answer. the nova docs suggest we default to using durable AMQP queues with persistent messages. The oslo.messaging code suggests the default is non-durable queues. which is it? | |
| 15:18:21 | openstackgerrit | Merged openstack/nova master: Fix a typo in comment in resource_provider.py https://review.openstack.org/588145 | |
| 15:20:42 | mriedem | cfriesen: where do the nova docs say that? | |
| 15:21:08 | mriedem | it's clearly not durable by default https://docs.openstack.org/nova/latest/configuration/config.html#oslo_messaging_rabbit.amqp_durable_queues | |
| 15:21:18 | mriedem | anything in the nova docs is likely way out of date | |
| 15:21:39 | mriedem | https://docs.openstack.org/nova/latest/reference/rpc.html?highlight=durable | |
| 15:21:40 | cfriesen | doc/source/reference/rpc.rst: | |
| 15:21:41 | mriedem | that? ^ | |
| 15:22:00 | mriedem | yeah that's all super duper old | |
| 15:22:03 | mriedem | probably predates oslo.messaging | |
| 15:22:06 | mriedem | open a bug | |
| 15:23:31 | cfriesen | okay. what about message persistance? are we always transient? | |
| 15:24:54 | mriedem | let the code be your guide young chris | |
| 15:26:08 | mriedem | i thought durable == persistent | |
| 15:28:01 | cfriesen | just to make things interesting, rabbitmq can have durable or transient exchanges and queues, and then the message itself can be persistent or transient. | |
| 15:29:39 | mriedem | idk then, ask kgiusti in #openstack-oslo? | |
| 15:29:46 | mriedem | or sileht? | |
| 15:30:11 | cfriesen | yeah, will do. I suspect we're just always transient. | |
| 15:30:17 | tssurya | mriedem: probably this has come up zillion times, but I couldn't find the right reasoning on why the power synchronization is asymmetric in nova.. why does nova not acknowledge that the vm is back up again ? | |
| 15:31:36 | tssurya | isnt' the driver the state of truth ? | |
| 15:32:08 | mriedem | tssurya: what thing are you specifically talking about? the sync_instance_power_states periodic in compute | |
| 15:32:09 | mriedem | ? | |
| 15:32:12 | tssurya | yes | |
| 15:32:18 | mriedem | that shuts down your server if the db says it's down but the driver says it's up? | |
| 15:32:27 | tssurya | exactly | |
| 15:32:36 | mriedem | because you could be getting charged for one when you told nova to shut it down | |
| 15:32:59 | tssurya | ah okay.. | |
| 15:33:11 | mriedem | it's been years since i've had to load that thing into memory | |
| 15:33:17 | mriedem | and it's always a disaster when i do | |
| 15:33:45 | tssurya | and for the ironic cases where the users may interact through the ipmi interface.. | |
| 15:33:56 | tssurya | the only solution we have is to switch the power sync off ? | |
| 15:34:21 | mriedem | nova doesn't support monkeying with the guests out of band | |
| 15:34:27 | tssurya | is there is a way we could make this behaviour configurable (a choice for deployments) | |
| 15:34:39 | tssurya | hmm | |
| 15:34:44 | mriedem | i'm sure vmware and powervc have had this same argument because the user started up the vm in vcenter or the hmc | |
| 15:34:56 | mriedem | and then nova shut it down | |
| 15:35:03 | tssurya | yes, we have the same stuff | |
| 15:35:19 | stephenfin | mriedem: Remind me: can I +W code? We've branched and everything, right? | |
| 15:35:26 | tssurya | we were just thinking if we could make this configurable.. as in tell nova not to shut it back down | |