| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-09-29 | |||
| 16:41:53 | melwitt | yeah... | |
| 16:42:13 | dansmith | so if stop and delete arrive in the right order, but each call out to conductor, we hit any conductor in the cluster, each one tries to create records in the db, who knows which one will finish first, grab the lock, etc | |
| 16:42:35 | dansmith | that's what I mean by assuming this sort of super tight ordering is unsafe, even though we *think* we're on the compute and largely single-threaded | |
| 16:42:46 | bauzas | I see | |
| 16:43:03 | bauzas | stupid idea, can't we rely on the state of the instance ? | |
| 16:43:13 | dansmith | delete is anything-goes I think | |
| 16:43:27 | bauzas | yeah, that's the original problem | |
| 16:43:32 | dansmith | because that's what we want... | |
| 16:43:41 | bauzas | yup... | |
| 16:44:15 | dansmith | and anything else we build into there is going to be pretty obscure.. "delete always deletes, except stop but not shelve... and either waits or refuses or ..." | |
| 16:44:24 | dansmith | makes the "delete always works" contract a little less clear | |
| 16:44:27 | bauzas | honestly I don't know how to move on with this ask | |
| 16:44:38 | dansmith | tell them that delete means pulling the plug, | |
| 16:44:41 | bauzas | "ask your orchestration to be smarter ?" | |
| 16:44:41 | lyarwood | how about making stop more graceful just for the libvirt driver? | |
| 16:44:46 | gibi | can we add a flag to the delete api saying I-want-a-shutdown-first? | |
| 16:45:00 | dansmith | and you wouldn't do that while waiting for start->shutdown on a physical server, so they should wait for vm_state=STOPPED before delete | |
| 16:45:24 | dansmith | gibi: that makes it a little less obscure, but doesn't eliminate the need to make that orchestration bit work, of course | |
| 16:45:37 | bauzas | yeah | |
| 16:45:39 | lyarwood | artom: there main issue with stop was that it eventually destroys the instance right? | |
| 16:45:47 | bauzas | that doesn't solve the ordering problem | |
| 16:46:07 | lyarwood | artom: and in your change you've suggested that we call shutdown on the domain as an initial step to make this more graceful | |
| 16:46:35 | dansmith | lyarwood: can we call quiesce or something else constant-time before libvirt destroy? | |
| 16:46:43 | dansmith | instead of shutdown, which the guest can block or ignore? | |
| 16:47:31 | bauzas | the root problem is the I/O flushes, right? | |
| 16:47:40 | dansmith | novaclient has a --poll option for some things. "nova stop server --poll; nova delete server" would solve this pretty easy :) | |
| 16:49:01 | melwitt | they're using tripleo/heat but I'm pretty sure heat has dependency or waiting ability | |
| 16:49:09 | dansmith | surely | |
| 16:49:33 | lyarwood | dansmith: quiesce before $domain.shutdown() might help flush things if that's where the guestOS is getting hung up | |
| 16:49:52 | melwitt | their argument has been that stop + insta delete should not break the volume, IIUC | |
| 16:50:00 | lyarwood | waiting assumes we don't kill it before it's finished shutting down | |
| 16:50:14 | dansmith | lyarwood: I meant to make sure journal buffers are written before we nuke the guest | |
| 16:50:36 | dansmith | melwitt: that argument is fine as long as you wait between stop and delete :) | |
| 16:51:01 | dansmith | melwitt: because again, start->shutdown and then pulling the plug before it finishes yields a corrupted disk :) | |
| 16:51:21 | melwitt | dansmith: that was the first thing I said on the bug report but it's gone on for a long time now and gotten into the weeds | |
| 16:52:04 | bauzas | melwitt: I haven't seen any bug report against artom's patch | |
| 16:52:12 | bauzas | I guess you're talking internally | |
| 16:52:19 | melwitt | bauzas: yes internally | |
| 16:52:42 | bauzas | yeah, because that's what worried me originally | |
| 16:52:51 | bauzas | technically, destroy works like expected | |
| 16:52:52 | dansmith | we also really need to do a better job of making this a sanitized bug externally if we're going to claim this is a bug in nova | |
| 16:53:25 | bauzas | I'm not happy with claiming this as an upstream bug | |
| 16:53:37 | dansmith | me either, fwiw :) | |
| 16:53:40 | bauzas | a blueprint or a wishlist bug | |
| 16:54:08 | dansmith | we could add a feature as gibi said, but destroy is doing the right thing here | |
| 16:54:21 | bauzas | (18:52:51) bauzas: technically, destroy works like expected | |
| 16:54:28 | melwitt | +1 to all of that | |
| 16:54:30 | dansmith | an alternative to gibi's idea, would be a delete flag that says "assuming task_state=None" meaning "delete this if nothing else is going on" | |
| 16:54:30 | bauzas | strong agreement here | |
| 16:54:47 | dansmith | but it would require the client to retry, which they could currently do by just waiting for the stop to finish | |
| 16:54:50 | dansmith | so, meh | |
| 16:55:22 | bauzas | dansmith: yeah, that's why I was considering the vm state or the task state | |
| 16:55:38 | bauzas | if we really want to do *something* | |
| 16:55:39 | bauzas | but, | |
| 16:55:49 | bauzas | this can't be done with the current destroy API | |
| 16:55:52 | dansmith | it doesn't make it do what they want, and honestly it's kinda weird since they could just wait for the stop just as well, but it's less new orchestration stuff | |
| 16:56:09 | dansmith | [09:47:37] <dansmith> novaclient has a --poll option for some things. "nova stop server --poll; nova delete server" would solve this pretty easy :) | |
| 16:56:11 | dansmith | ^ :) | |
| 16:56:12 | bauzas | it would be a 'destroy++" API | |
| 16:56:34 | bauzas | heh | |
| 16:56:43 | dansmith | also, if melwitt is right and they're using heat, then FFS, get heat to wait or something | |
| 16:56:43 | bauzas | (18:44:41) bauzas: "ask your orchestration to be smarter ?" | |
| 16:56:47 | dansmith | lol | |
| 16:56:53 | lyarwood | we keep saying they could wait for the stop to finish but wasn't that part of the issue here? Even if they did wait the libvirt driver would kill the instance prematurely before it had finished shutting down? | |
| 16:57:01 | dansmith | if we're just re-quoting ourselves, are we done here? :P | |
| 16:57:19 | gibi | if currently stop + wait for STOPPED + delete works, then a new delete-with-gracefull-shutdown could also be orchestrated from the conductor but it is obviously an orchestration and adds complexity to the already complext delete codepaths | |
| 16:57:42 | bauzas | gibi: and again, this can't be the straight delete | |
| 16:57:53 | dansmith | gibi: we don't go through conductor directly for stop or delete right now, AFAIK, so it would really get confusing to add another whole path like that, IMHO | |
| 16:57:54 | gibi | bauzas: yepp this is delete++ :D | |
| 16:58:25 | gibi | dansmith: then we either need the make the stop RCP sync or do a polling from the nova-api | |
| 16:58:33 | gibi | RPC | |
| 16:58:35 | melwitt | lyarwood: I didn't think so? | |
| 16:58:53 | bauzas | can't we just have as dansmith suggested a "destroy++" API return some 40x when the task state is not None ? | |
| 16:59:15 | lyarwood | that's fine if they aren't waiting | |
| 16:59:18 | melwitt | if that's the case I missed it | |
| 16:59:21 | dansmith | gibi: we'd need a cast to conductor, a new method there, and a sync stop operation to compute from conductor followed by delete.. waiting from the api is not reasonable because stop can take a *long* time | |
| 16:59:35 | gibi | yeah | |
| 16:59:41 | lyarwood | melwitt: I was sure we had talked about them waiting and libvirt still killing the domain before things had shutdown sorry | |
| 16:59:45 | lyarwood | I'll look at the bug again | |
| 16:59:47 | dansmith | in addition to the other four ways we can delete things :) | |
| 16:59:48 | melwitt | lyarwood: my understanding (and I could be wrong) is that they have never tried waiting to stopped | |
| 17:00:00 | lyarwood | I was sure we suggested that early on | |
| 17:00:01 | dansmith | melwitt: right that seems like it to me | |
| 17:00:03 | melwitt | lyarwood: ok same, maybe I'm way off | |
| 17:00:07 | bauzas | I'd rather prefer this destroy++ API to stop synchronously at the API level if some conditions aren't met | |
| 17:00:36 | bauzas | rather than us pursuing the idea we could achieve some distributed locking mechanism | |
| 17:00:56 | dansmith | gibi: also delete returns "I will eventually do this" to the client, which it would no longer be able to guarantee | |
| 17:01:32 | dansmith | bauzas: fwiw, I don't think that is really going to do what this person wants, so we should make sure they think it would help before we do that work | |
| 17:01:37 | gibi | dansmith: ohh, that is correct and bad :/ | |
| 17:02:05 | gibi | delete++ is just too complex | |
| 17:02:08 | bauzas | dansmith: welp, good point | |
| 17:07:07 | bauzas | either way, I think we need a proper tracking | |
| 17:07:41 | bauzas | artom: I guess you need to fill a blueprint and honestly given the brainstorm efforts we made over last hour, you need to write a bit of a spec | |
| 17:08:16 | bauzas | mostly because the current destroy action can't be used for this and we need to consider a new action parameter (or any change in our API) | |
| 17:08:41 | bauzas | artom: feel also free to write a PTG proposal for this one, we could continue the talk there | |
| 17:57:36 | spatel | Any idea how to do VM nic bonding using two VF with SRIOV ? | |
| 17:58:39 | spatel | i am looking for redundancy with sriov implementation and only solution is to do bonding inside vm | |
| 18:04:32 | opendevreview | Ivan Kolodyazhny proposed openstack/nova master: Add release note which descrube NVMe attach issue https://review.opendev.org/c/openstack/nova/+/811447 | |
| 19:15:02 | opendevreview | melanie witt proposed openstack/nova stable/train: Reject open redirection in the console proxy https://review.opendev.org/c/openstack/nova/+/791807 | |
| 19:15:03 | opendevreview | melanie witt proposed openstack/nova stable/train: address open redirect with 3 forward slashes https://review.opendev.org/c/openstack/nova/+/806629 | |