| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-10 | |||
| 13:31:23 | mriedem | efried: no | |
| 13:31:29 | efried | so what happens to allocations? | |
| 13:31:29 | mriedem | efried: you mean migratoin-based allocations? | |
| 13:31:34 | efried | yeah | |
| 13:31:34 | mriedem | efried: they sit on the source | |
| 13:31:56 | efried | thanks | |
| 13:31:58 | mriedem | which is why we can't delete resource providers when deleting a compute service that has evacuated instance allocations against it | |
| 13:32:03 | mriedem | that whole thread i had in the ML | |
| 13:32:31 | mriedem | efried: related to this https://review.opendev.org/#/c/678100/ | |
| 13:32:41 | sean-k-mooney | mriedem: that has been changed recently right. we delete the allcoation if they exisits if and only if you bring back up the compute agent on the failed host | |
| 13:32:55 | sean-k-mooney | so that only help if you repair whatever the issue was | |
| 13:33:06 | mriedem | sean-k-mooney: yes the source allocations are deleted if you bring up the evacuated-from compute service | |
| 13:33:10 | sean-k-mooney | if you dont you get into the stiutation in that ml thread | |
| 13:33:42 | mriedem | i don't think it's probably uncommon to have a host failure, down the compute service, evacuate it, and then try to delete the comptue service before redeploying on that host | |
| 13:34:40 | mriedem | Sundar: tempest only covers happy path testing for the most part; unit tests are good for testing error conditions and such - exceptional cases | |
| 13:34:53 | mriedem | anyway, that's up to the cyborg core team for how they want to enforce testing standards | |
| 13:35:04 | efried | mriedem: also rebuild? | |
| 13:35:10 | mriedem | efried: a rebuild isn't a migration | |
| 13:35:17 | mriedem | and you can't rebuild on a down host | |
| 13:35:53 | mriedem | so the host stays the same and the flavor stays the same, but the image might change on a rebuild | |
| 13:36:00 | sean-k-mooney | Sundar: have the depencies been fixed in https://review.opendev.org/#/c/670999/ | |
| 13:36:12 | efried | mriedem: right, separate topic, I'm saying: does rebuild also end up with multiple sets of allocations for the same instance uuid? | |
| 13:36:28 | sean-k-mooney | Sundar: you rebased it but did not run the new job via "check experimental" | |
| 13:36:40 | sean-k-mooney | so we still dont have a run that worked | |
| 13:37:03 | sean-k-mooney | the previous run failded https://c3308e17743765936b80-6c7fec3fffbf24afb7394804bcdecfae.ssl.cf2.rackcdn.com/670999/7/experimental/cyborg-tempest/2fe52ec/testr_results.html.gz | |
| 13:37:12 | mriedem | efried: no | |
| 13:37:17 | sean-k-mooney | that said we knew the depencies were wrong so we expected that | |
| 13:38:11 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Fixing broken links https://review.opendev.org/681206 | |
| 13:39:31 | luyao | mriedem: what about in the process of rebuild | |
| 13:40:08 | luyao | mriedem: rebuild is not done, how many allocations will an instance have? | |
| 13:40:10 | efried | we destroy the instance but keep the allocations, then respawn it with the existing allocations? | |
| 13:40:23 | efried | ...and maybe a different image? | |
| 13:40:30 | sean-k-mooney | Sundar: it looks like there are 3 patches still remaining on the cyborg side. 2 for nova integration and 1 for python3 support | |
| 13:41:28 | mriedem | luyao: i don't understand your question | |
| 13:41:36 | mriedem | efried: correct | |
| 13:41:49 | efried | mriedem: luyao and I are asking about the same thing. Thanks for the help. | |
| 13:42:14 | mriedem | rebuild is basically re-spawn in place on the same host with maybe a different image but the same ports/volumes/flavor - if you're on shared storage you keep your root disk, if not your root disk is rebuilt from the specified image | |
| 13:42:28 | Sundar | sean-k-mooney: I'll rerun with 'check experimental'. Re. dependencies for https://review.opendev.org/#/c/670999/, one has already merged, and the other is the tempest code itself, which is working with the patches and should merge soon. There should be only 1 for nova-integ (i.e. Nova notification) -- I fixed the topic now. | |
| 13:42:44 | sean-k-mooney | Sundar: wait a minute | |
| 13:42:51 | mriedem | though i might be thinking of evacuate for that root disk comment | |
| 13:42:56 | sean-k-mooney | im going to fix it to not waste gate resouces and fix the depencies | |
| 13:43:00 | mriedem | anyway, root disk doesn't matter for what you're asking | |
| 13:43:22 | mriedem | sean-k-mooney: cyborg integration isn't happening in nova in train - where are we with test runs on the numa live migration series? | |
| 13:43:44 | efried | stephenfin, dansmith: btw, vpmem now has a CI passing. I haven't opened one up yet, but it's taking half an hour, so it's doing... *something* :P | |
| 13:43:53 | luyao | mriedem: I saw an instance have new and old allocations in an functional test when it's in evacuating. | |
| 13:43:55 | sean-k-mooney | mriedem: i confrimed that it is not saving the updated numa toplogy so it looks like we need instance.refresh | |
| 13:44:05 | sean-k-mooney | mriedem: the migration context has the correct numa toplogy | |
| 13:44:07 | luyao | mriedem: I thought rebuild is similar to evacuate | |
| 13:44:22 | sean-k-mooney | mriedem: so it looks liek its not marking the field as changed for some reason | |
| 13:44:36 | sean-k-mooney | mriedem: perhapes because apply migration context uses setattr | |
| 13:44:45 | mriedem | luyao: evacuate is rebuild to another host when the source host is down | |
| 13:44:49 | sean-k-mooney | mriedem: i -1'd the patch | |
| 13:45:06 | mriedem | evacuate and rebuild use the same code flows in conductor and compute services with conditionals for any differences | |
| 13:45:08 | efried | luyao, mriedem: I find all of it so confusing that I don't even try to remember, because that just makes things worse. It goes against the very fiber of my being, but this is one where I'll ask. Every. Time. | |
| 13:45:15 | mriedem | e.g. evacuate does a claim on the dest host, rebuild does not | |
| 13:45:58 | mriedem | sean-k-mooney: ok i didn't realize artom removed the instance.refresh in post live migration | |
| 13:46:03 | mriedem | but haven't looked at changes since yesterday | |
| 13:46:29 | mriedem | efried: rebuild+evacuate and resize+cold-migrate are confusing in that they share code but don't do the exact same things | |
| 13:46:40 | mriedem | but are like 90% the same | |
| 13:47:37 | mriedem | efried: do we need a "what's the difference between evacuate and rebuild" thing in the contributor docs? | |
| 13:47:41 | mriedem | b/c that's probably pretty easy to write up | |
| 13:47:58 | efried | that must be why luyao was conflating rebuild+evacuate wrt allocations. That must fall in the 10% that's different. | |
| 13:48:13 | mriedem | part of it yeah | |
| 13:48:16 | efried | mriedem: I'll admit I didn't even go looking for a doc on this. | |
| 13:48:25 | mriedem | there likely isn't one | |
| 13:48:33 | mriedem | but this isn't the first time this has come up, | |
| 13:48:34 | efried | There was that one blog post | |
| 13:48:37 | efried | "one" | |
| 13:48:41 | mriedem | so instead of explaining it each time, one could just link to a doc | |
| 13:48:45 | jangutter | efried: http://www.danplanet.com/blog/2016/03/03/evacuate-in-nova-one-command-to-confuse-us-all/ this one? | |
| 13:48:56 | mriedem | not really the same | |
| 13:49:03 | efried | that looks familiar jangutter | |
| 13:49:06 | mriedem | that's about nova client commands that live migrate all instances off a host | |
| 13:49:13 | openstackgerrit | Adam Spiers proposed openstack/nova master: Improve SEV documentation and other minor tweaks https://review.opendev.org/681254 | |
| 13:49:30 | luyao | efried: yes, actually I found evacuting instance has two groups of allocations, and I thought rebuild is the same. | |
| 13:49:45 | mriedem | "After a compute host has failed, rebuild my instance from the original image in another place, keeping my name, uuid, network addresses, and any other allocated resources that I had before." | |
| 13:50:35 | efried | anyway, I envision some kind of table with all these related operations on one axis, an things like "same/different destination host", "what happens to allocations", "instance/migration UUIDs", etc on the other. | |
| 13:51:37 | mriedem | that's a much bigger doc if you're trying to lump in all move operations, | |
| 13:51:45 | openstackgerrit | sean mooney proposed openstack/nova master: [WIP] add cyborg tempest job https://review.opendev.org/670999 | |
| 13:51:46 | mriedem | because you'd have to consider resize on the same host, which is complicated in different ways | |
| 13:51:48 | efried | mriedem: is there a todo list on which that ^ could be registered so it is not forgotten, but not started until after FF? We need you for more urgent things this week. | |
| 13:51:52 | sean-k-mooney | Sundar: that ^ should test with the correct deps. im going back to testing artoms code | |
| 13:52:07 | mriedem | efried: i'm not signing up to write a doc with all of the axis of evil, | |
| 13:52:20 | mriedem | writing up something quick and easy about "what's the difference between rebuild and evacuate" is pretty easy for me to crank out | |
| 13:52:27 | efried | even so | |
| 13:52:32 | mriedem | a bug is good enough for that | |
| 13:52:41 | mriedem | docs bug, link to the irc question that started tihs | |
| 13:52:48 | efried | ack | |
| 13:53:27 | sean-k-mooney | mriedem: im going to add back in the instnace.refresh locally and add some logging to see what teh numa toplogy blob look like before and after. | |
| 13:53:44 | sean-k-mooney | mriedem: artom is in dad taxi mode currently but he should be back soon ish | |
| 13:55:20 | sean-k-mooney | although looking at that funciton im not clear why that helps | |
| 13:55:20 | Sundar | Thanks, sean-k-mooney | |
| 13:56:21 | efried | https://bugs.launchpad.net/nova/+bug/1843439 | |
| 13:56:23 | openstack | Launchpad bug 1843439 in OpenStack Compute (nova) "doc: Rebuild vs. Evacuate (and other move-ish ops)" [Low,Confirmed] - Assigned to Matt Riedemann (mriedem) | |
| 13:56:53 | efried | mriedem: there's a 'doc' tag and a 'docs' tag. Which one is real? | |
| 13:56:57 | efried | (I used both) | |
| 13:59:30 | bauzas | efried: IIRC, it was 'docs' 3 years ago | |
| 14:00:00 | bauzas | oops, I meant 'doc' | |
| 14:00:03 | bauzas | holy shit | |
| 14:00:26 | sean-k-mooney | its doc | |