| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-06-13 | |||
| 18:08:15 | mriedem | artom: before you raise HR alarms downstream, i was joking | |
| 18:08:27 | melwitt | mriedem: this is a bit old and has a +2 from dansmith now, you're probably the best person to review since it's about nova-manage cells stuff https://review.opendev.org/651947 | |
| 18:08:57 | artom | mriedem, you haven't touched me inappropriately (yet?), so no HR alarms | |
| 18:09:01 | artom | ;) | |
| 18:09:29 | mriedem | dansmith: you know how we have this warning about stopping nova-compute before deleting it? https://developer.openstack.org/api-ref/compute/?expanded=delete-compute-service-detail#delete-compute-service otherwise dragons? it strikes me now that the delete compute service API should just be checking if the service is up and if so, fail the request with a 409 | |
| 18:10:06 | sean-k-mooney | artom: a +1 from zuul, how did you apise the ci gods to make that happen | |
| 18:10:15 | dansmith | mriedem: maybe, unless there's a reason to delete and have it recreate or something | |
| 18:10:23 | artom | sean-k-mooney, which patch? | |
| 18:10:35 | dansmith | mriedem: the other thing would be that unless you also force down, you can't stop and delete inside of a minute or whatever the service timeout is | |
| 18:11:01 | sean-k-mooney | artom: you have more then one with a +1 :P the hybrid_plug events one | |
| 18:11:04 | mriedem | yeah you'd have to force down | |
| 18:11:30 | mriedem | dansmith: the problem with the recreate is the new provider it will create in placement with no allocations for the instances on that host, which will f up scheduling | |
| 18:11:32 | artom | sean-k-mooney, except for the double words thing (that was fixed) I'm not aware of any Zuul issues | |
| 18:11:59 | mriedem | you can heal that with the heal_allocations CLI of course... | |
| 18:12:08 | mriedem | well, for non-nested / sharing allocations | |
| 18:12:29 | dansmith | mriedem: sure | |
| 18:12:45 | sean-k-mooney | artom: it was more the fact its taken a few hours for zuul to notice and run on your patch for the last few days then it being flaky | |
| 18:13:06 | artom | sean-k-mooney, oh, yeah, it's slow | |
| 18:13:13 | mriedem | i'm just down a hole on how we're orphaning providers today and how best to shore that up | |
| 18:16:30 | sean-k-mooney | im following the thread on the ml bug trying to stay out of it to a degree to leave ops speak up | |
| 18:16:59 | sean-k-mooney | although it looks like the last few mails are back and forth between you erric and chris | |
| 18:24:02 | mriedem | i think likely the best thing to do is a phased conservative approach, | |
| 18:24:18 | mriedem | 1. land my change that fixes it to delete all providers, not just the first (the ironic case), | |
| 18:24:34 | mriedem | 2. if deleting any provider fails, don't delete the compute service and bail (option 1 in the email), | |
| 18:25:02 | mriedem | 3. check for and cleanup allocations for evacuations - something we can be pretty sure we can get rid of safely | |
| 18:25:47 | mriedem | that leaves in-progress or unconfirmed migrations, and in that case i think we likely just should be saying "you can't delete the compute service until these migrations are done" | |
| 18:26:46 | sean-k-mooney | ya that seam sane. the issue that im not sure how to solve/if we shoul solv eis if you had in progeress migration and the host failed/went offline | |
| 18:27:17 | sean-k-mooney | can you evaucate an instance that was migrating | |
| 18:27:25 | sean-k-mooney | when the host went off line | |
| 18:27:54 | bnemec | aspiers: +2. Thanks for taking care of this! | |
| 18:28:03 | aspiers | bnemec: yw | |
| 18:28:20 | mriedem | sean-k-mooney: don't think so | |
| 18:28:27 | mriedem | unless you reset the instance state to ERROR | |
| 18:28:29 | sean-k-mooney | im fine to leave that up to the operator to clean up too, but im just wondering about the case where the reason their are pending migration or evacuation is because of a host failure | |
| 18:28:51 | sean-k-mooney | or reset it ot active | |
| 18:28:57 | sean-k-mooney | but ok | |
| 18:29:58 | sean-k-mooney | if it was in resize_verify or whatever the state is when we havent confirmed yet but was successafaly migrated we would proably want to reset it to active | |
| 18:30:04 | mriedem | if you're in the middle of a migration and the source host went offline, i don't see why you'd delete the service - just bring the service back up if possible | |
| 18:31:31 | sean-k-mooney | ya im somewhat unclear why the compute service is being deleted with outstanding migrations too | |
| 18:32:14 | sean-k-mooney | also reset state to active instead of confirming might not be right as it likely would not have updated the db correctly for the new host | |
| 18:32:46 | sean-k-mooney | so you would want to confirm. let it error out and then rest to active/hardreboot to fix it right | |
| 18:33:54 | mriedem | well the good news is a contracted google recruiter has contacted me about opportunities in the bay area of new york so if i want to avoid this problem i can just follow that lead and up-root my entire life | |
| 18:34:03 | mriedem | delete compute service problems be damned! | |
| 18:34:43 | sean-k-mooney | haha well that is one way to solve it yes then you can work on kubernetes problems | |
| 18:35:28 | sean-k-mooney | its really a seperate issue but should we allow you to confim a resize/migration if the souce host is down without erroring out? | |
| 18:35:29 | mriedem | then after 6 months you get there and they're like, "ok we have this resource leak issue when you delete a pod" | |
| 18:36:33 | mriedem | sean-k-mooney: i'm not sure what happens in that case, i haven't tried it, but i'd think the cast to the source compute would blow up and you'd get a 500 response from the confirmResize API | |
| 18:37:04 | sean-k-mooney | well if its a cast it a fire and forget if its a call then yes | |
| 18:37:37 | sean-k-mooney | do we start the confrim form the souce node so the api will jsut time out the request | |
| 18:37:47 | sean-k-mooney | e.g. because notheing handels it. | |
| 18:38:10 | melwitt | efried: do you want a follow up patch for your comments on https://review.opendev.org/651947 | |
| 18:38:23 | efried | melwitt: meh, they're so minor. | |
| 18:38:29 | melwitt | ack | |
| 18:38:31 | efried | melwitt: If you want to spin a new PS I can fast approve | |
| 18:38:39 | sean-k-mooney | i need to get a better undestand of how to create nova functional test to test this kind fo stuff without actully depolying devstack | |
| 18:38:41 | efried | or we can just ignore | |
| 18:40:03 | efried | mriedem: Was about to start composing the email whining about bug queue. Unless you want to do it... <bats eyes> | |
| 18:40:09 | melwitt | efried: ok. I can respin, that's easy enough. mostly I'm thinking of the doc rendering where literal would help | |
| 18:40:53 | efried | melwitt: There's probably opportunity to add literals throughout that doc - I didn't check. If so, maybe we could scoop it all up in a fup | |
| 18:41:45 | efried | anyway, I'm around for at least another 3-4h to push whatever | |
| 18:42:12 | melwitt | ok. yeah, I see most seem literal'd but some others aren't so could be done all together | |
| 18:43:45 | mriedem | efried: i do not, writing a reply to this compute service delete thread | |
| 18:44:06 | efried | mriedem: ack. Okay if I run it by you before sending? | |
| 18:47:01 | sean-k-mooney | mriedem: i might steal your test code form https://review.opendev.org/#/c/663737/1/nova/tests/functional/wsgi/test_services.py and try and create a similar test for that migrate confim case while im waiting for the pizza to arrive | |
| 18:48:01 | dansmith | mmm pizza | |
| 18:48:16 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: api-guide: better explain scheduler hints. https://review.opendev.org/522106 | |
| 18:48:38 | sean-k-mooney | hehe one of irelands suckiest inventions is slow "fast food" | |
| 18:49:28 | sean-k-mooney | all the quality of fast food with the slowness of going to the store and cooking it your self. | |
| 18:49:37 | mriedem | COOKIES | |
| 18:49:39 | sean-k-mooney | given i havent ordered it yet i might actully do that | |
| 18:51:27 | melwitt | did somebody say pizza | |
| 18:52:10 | mriedem | dansmith: i was just thinking about that yesterday for some reason, probably b/c i made a pizza for lunch | |
| 18:53:11 | dansmith | heh | |
| 19:03:10 | dansmith | mriedem: I don't even remember what the gag was.. trying to explain to sdague how to use punctuation or something | |
| 19:04:36 | openstackgerrit | melanie witt proposed openstack/nova master: Literalize CLI options in docs https://review.opendev.org/665217 | |
| 19:05:27 | mriedem | i think it was related to commit message titles | |
| 19:05:29 | mriedem | something like that | |
| 19:05:31 | mriedem | Something like that. | |
| 19:05:40 | mriedem | one of the annual flare ups in the ML | |
| 19:05:48 | mriedem | the good old days | |
| 19:06:17 | mriedem | i think we were shitting on his commit message which was all lower case with sparse amount of periods | |
| 19:06:31 | dansmith | sounds right | |
| 19:06:38 | mriedem | mostly to poke the bear and air grievances | |
| 19:06:43 | mriedem | it wasn't even festivus | |
| 19:07:14 | mriedem | and now look at him, in front of congress and all | |
| 19:07:17 | mriedem | we really showed him | |
| 19:07:32 | dansmith | because congress has such high standads | |
| 19:07:34 | dansmith | *standards | |
| 19:08:17 | mriedem | like garth brooks, sean has friends in low places | |
| 19:08:33 | dansmith | heh | |
| 19:09:22 | melwitt | remember chris gaines | |
| 19:10:37 | mriedem | how could i forget | |
| 19:10:50 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Delete resource providers for all nodes when deleting compute service https://review.opendev.org/657016 | |
| 19:10:50 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Avoid unnecessary joins in delete_resource_provider https://review.opendev.org/657021 | |
| 19:10:51 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Optimize SchedulerReportClient.delete_resource_provider https://review.opendev.org/657070 | |
| 19:11:20 | mriedem | remember...THIS?! https://www.youtube.com/watch?v=oGpFcHTxjZs | |
| 19:12:17 | dansmith | man.. bad hair day | |
| 19:12:34 | melwitt | the intro made it look like the panpipe sound was coming from the tugboat lol | |
| 19:12:43 | mriedem | maya got her first exposure to that in the car last night, with me singing along to it | |
| 19:12:44 | melwitt | peruvian panpipes | |