| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-08-02 | |||
| 14:23:51 | mriedem | the api reserves the volume, but compute never unreserves it on failure | |
| 14:23:55 | mriedem | unlike if attach_volume fails | |
| 14:24:38 | mriedem | maybe that's just always been the way it is and expected b/c you can delete the instance in ERROR state which should unreserve the volume then | |
| 14:25:30 | mriedem | i guess _shutdown_instance calls the os-detach api in cinder but i don't know if that rolls back the reserved status | |
| 14:26:23 | lyarwood | mriedem: yeah I don't think we did unreserve in the old flow when we hit this error | |
| 14:26:44 | lyarwood | mriedem: I thought we had talked about updating attachments in the new flow with a None connector in this case | |
| 14:27:15 | lyarwood | mriedem: so the new compute only has to come along and update again with the correct connector | |
| 14:27:20 | mriedem | i think the os-detach call to cinder for the old flow in _shutdown_instance will make the volume available again | |
| 14:27:46 | lyarwood | kk then our removal of the attachment is fine | |
| 14:28:16 | mriedem | here is i think we're we'd get in the old flow prior to reschedule https://github.com/openstack/cinder/blob/b0e9ee1d501fb83b7ebbc59584aa6255dbaec086/cinder/volume/manager.py#L1314 | |
| 14:28:21 | mriedem | *where we'd | |
| 14:31:05 | openstackgerrit | Jay Pipes proposed openstack/nova master: DNM - example https://review.openstack.org/588295 | |
| 14:32:23 | openstackgerrit | Konstantinos Samaras-Tsakiris proposed openstack/nova master: Hide hypervisor id on windows guests https://review.openstack.org/579897 | |
| 14:32:25 | mriedem | lyarwood: ok comments in your series | |
| 14:32:35 | mriedem | lyarwood: lots of internal debate on this one, but i think what you're doing is likely the best | |
| 14:33:00 | lyarwood | mriedem: ack thanks | |
| 14:34:55 | openstackgerrit | Eric Fried proposed openstack/nova master: WIP/PoC: safe_connect shouldn't hide failures https://review.openstack.org/584593 | |
| 14:41:39 | gmann | stephenfin: gibi api-samples tox run sample file with GENERATE_SAMPLES=True | |
| 14:42:03 | gmann | stephenfin: there can be chance that few file missing in doc/api_samples let me check those | |
| 14:45:26 | mriedem | lyarwood: looks like we've probably always created another volume on a reschedule if the source_type!='volume', i don't see anything that deletes a volume that nova created prior to rescheduling - unrelated to your issue really, but a super latent bug | |
| 14:46:16 | mriedem | most likely another reason we should move volume creation to api or conductor so that compute doesn't have to manage that | |
| 14:46:37 | mriedem | compute would just get a bdm and assume the volume already was created | |
| 14:49:54 | lyarwood | mriedem: wonderful, happy to look at that as well if you wouldn't mind writing that up in a bug? | |
| 14:50:37 | mriedem | i'd need to recreate it first, and don't have a 2 node system handy | |
| 14:51:13 | mriedem | i just know we do a better job of tracking ports to know which we've created ourselves and which we haven't, and cleaning those up properly before reschedule (delete the ports we created, unbind the ports we didn't) | |
| 14:51:29 | mriedem | we don't really do anything like that for volumes | |
| 14:59:55 | openstackgerrit | Merged openstack/nova master: tox: Ensure reused envdirs share the same deps https://review.openstack.org/588207 | |
| 15:16:40 | cfriesen | this is kind of a basic question but I'm having a hard time finding an answer. the nova docs suggest we default to using durable AMQP queues with persistent messages. The oslo.messaging code suggests the default is non-durable queues. which is it? | |
| 15:18:21 | openstackgerrit | Merged openstack/nova master: Fix a typo in comment in resource_provider.py https://review.openstack.org/588145 | |
| 15:20:42 | mriedem | cfriesen: where do the nova docs say that? | |
| 15:21:08 | mriedem | it's clearly not durable by default https://docs.openstack.org/nova/latest/configuration/config.html#oslo_messaging_rabbit.amqp_durable_queues | |
| 15:21:18 | mriedem | anything in the nova docs is likely way out of date | |
| 15:21:39 | mriedem | https://docs.openstack.org/nova/latest/reference/rpc.html?highlight=durable | |
| 15:21:40 | cfriesen | doc/source/reference/rpc.rst: | |
| 15:21:41 | mriedem | that? ^ | |
| 15:22:00 | mriedem | yeah that's all super duper old | |
| 15:22:03 | mriedem | probably predates oslo.messaging | |
| 15:22:06 | mriedem | open a bug | |
| 15:23:31 | cfriesen | okay. what about message persistance? are we always transient? | |
| 15:24:54 | mriedem | let the code be your guide young chris | |
| 15:26:08 | mriedem | i thought durable == persistent | |
| 15:28:01 | cfriesen | just to make things interesting, rabbitmq can have durable or transient exchanges and queues, and then the message itself can be persistent or transient. | |
| 15:29:39 | mriedem | idk then, ask kgiusti in #openstack-oslo? | |
| 15:29:46 | mriedem | or sileht? | |
| 15:30:11 | cfriesen | yeah, will do. I suspect we're just always transient. | |
| 15:30:17 | tssurya | mriedem: probably this has come up zillion times, but I couldn't find the right reasoning on why the power synchronization is asymmetric in nova.. why does nova not acknowledge that the vm is back up again ? | |
| 15:31:36 | tssurya | isnt' the driver the state of truth ? | |
| 15:32:08 | mriedem | tssurya: what thing are you specifically talking about? the sync_instance_power_states periodic in compute | |
| 15:32:09 | mriedem | ? | |
| 15:32:12 | tssurya | yes | |
| 15:32:18 | mriedem | that shuts down your server if the db says it's down but the driver says it's up? | |
| 15:32:27 | tssurya | exactly | |
| 15:32:36 | mriedem | because you could be getting charged for one when you told nova to shut it down | |
| 15:32:59 | tssurya | ah okay.. | |
| 15:33:11 | mriedem | it's been years since i've had to load that thing into memory | |
| 15:33:17 | mriedem | and it's always a disaster when i do | |
| 15:33:45 | tssurya | and for the ironic cases where the users may interact through the ipmi interface.. | |
| 15:33:56 | tssurya | the only solution we have is to switch the power sync off ? | |
| 15:34:21 | mriedem | nova doesn't support monkeying with the guests out of band | |
| 15:34:27 | tssurya | is there is a way we could make this behaviour configurable (a choice for deployments) | |
| 15:34:39 | tssurya | hmm | |
| 15:34:44 | mriedem | i'm sure vmware and powervc have had this same argument because the user started up the vm in vcenter or the hmc | |
| 15:34:56 | mriedem | and then nova shut it down | |
| 15:35:03 | tssurya | yes, we have the same stuff | |
| 15:35:19 | stephenfin | mriedem: Remind me: can I +W code? We've branched and everything, right? | |
| 15:35:26 | tssurya | we were just thinking if we could make this configurable.. as in tell nova not to shut it back down | |
| 15:35:32 | mriedem | stephenfin: we have not branched | |
| 15:35:38 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: guest: introduce blockStats instead of domain.blockStats https://review.openstack.org/526833 | |
| 15:35:41 | tssurya | the default could be what nova does today of course | |
| 15:35:43 | stephenfin | Well then :) | |
| 15:35:50 | mriedem | stephenfin: +W depends on the change | |
| 15:35:58 | mriedem | if it's a bug fix, then yeah maybe | |
| 15:36:02 | mriedem | if it's docs, sure | |
| 15:36:10 | stephenfin | It's a refactor so nope, it should wait | |
| 15:36:12 | mriedem | if it's a big ass refactor or something risky, likely not a great idea right now | |
| 15:36:28 | stephenfin | Yup, I'll just leave it til branch. No panic on it | |
| 15:37:02 | mriedem | tssurya: likely a better question for the ML to get wider input, from both -dev and ops lists | |
| 15:37:19 | mriedem | i'm able to devote about 5% brain to your question atm | |
| 15:37:30 | tssurya | mriedem: ack, sorry for the bad timing then :) | |
| 15:37:35 | mriedem | np | |
| 15:43:20 | gmann | stephenfin: yes, these were missed and they would not fail as they all are req files which are not verified on sample tests. let me know if you can or want me to add them and good to ref those in api-ref also. | |
| 15:51:57 | stephenfin | gmann: I don't mind. Is it a big issue? I guess it just means the api-ref will be incomplete? | |
| 15:52:20 | stephenfin | gmann: If you _do_ have time to work on them, I'll happily review it | |
| 15:59:55 | mriedem | tssurya: note you can disable that sync power state task | |
| 16:00:21 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L7439 | |
| 16:00:30 | melwitt | dangit, was just pulling that up | |
| 16:01:04 | melwitt | you win...again | |
| 16:04:48 | melwitt | https://github.com/openstack/nova/blob/master/nova/conf/compute.py#L753-L777 | |
| 16:17:20 | rdiggz | good day everyone. Could someone help me understand what this means? "A sys-admin privsep daemon has been added and needs to be included in your | |
| 16:17:22 | rdiggz | rootwrap configuration. | |
| 16:18:35 | rdiggz | Its from the release notes of pike and queens. Im seeing an issue deploying a snapshot of a CentOS7 instance that points to sys-admin and im thinking they are related. | |
| 16:26:33 | mriedem | cfriesen: is this a bug that should be upstreamed or at least reported to nova? https://github.com/starlingx-staging/stx-nova/commit/fe0c0617be857161b6bb66d632b2b35887c08772 | |
| 16:27:27 | mriedem | oh nvm it's already in nova | |
| 16:27:38 | mriedem | https://github.com/openstack/nova/commit/ce8bf6734e554b116e82e924cbe81a5968441926 | |
| 16:29:42 | cfriesen | they should've pointed to upstream if it was a backport. grr. | |
| 16:30:05 | mriedem | it's not a backport | |
| 16:42:47 | mriedem | dansmith: looks like one of your earlier is_bfv attempts made it in https://github.com/starlingx-staging/stx-nova/commit/71acfeae0d1c59fdc77704527d763bd85a276f9a#diff-b839034e35c154b8c3a1c65bf7791eefR114 | |
| 16:43:16 | dansmith | mriedem: lol | |
| 16:43:16 | mriedem | cfriesen: what are offline_cpus? | |
| 16:43:59 | melwitt | I think that's part of the second iteration of my famous root_gb=0 workaround patches | |