| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-10 | |||
| 19:05:22 | mriedem | clutchbeyers: the change in pike would apply to any move operation through the scheduler, | |
| 19:05:32 | mriedem | so resize, live migrate, unshelve and evacuate | |
| 19:09:04 | clutchbeyers | mriedem: thank you for the clarification! | |
| 19:10:40 | mriedem | clutchbeyers: note that we could consider this a regression in pike. it's in your favor i guess, but depending on how we handle this bug we might change it back. | |
| 19:11:02 | mriedem | dansmith: maybe something we need to put in your forum session on cells... | |
| 19:13:08 | dansmith | mriedem: I'm not sure what you're saying is a regression | |
| 19:13:11 | dansmith | or what it has to do with cells | |
| 19:13:31 | dansmith | that you can't migrate across az boundaries/ | |
| 19:13:38 | mriedem | dansmith: https://bugs.launchpad.net/nova/+bug/1768876 | |
| 19:13:38 | openstack | Launchpad bug 1768876 in OpenStack Compute (nova) queens "metadata-api fails to get availability zone for instances created before pike" [High,Triaged] | |
| 19:14:02 | clutchbeyers | mriedem: it my honest opinion, if a tenant does not specify an az on vm provision, then they arent too concnered about controlling their fault domain. So when they do decide to resize and it moves to another az, they theoretically shouldn't care | |
| 19:14:33 | clutchbeyers | But for those that specifically call out an az, they have a fault domain architecture in mind and wouldnt want to disrupt that | |
| 19:14:56 | dansmith | mriedem: okay, I think we talked about that in dublin, about potentially just forcing instance.az to whatever we chose if you didn't ask for one, keeping you in whatever az you happened to land in | |
| 19:15:26 | melwitt | I think that's what the original behavior was -- only allows migrating across AZ if none was selected deliberately | |
| 19:15:47 | dansmith | melwitt: you mean only keeps you in your az if you asked for one initially | |
| 19:15:53 | clutchbeyers | there is some assumptions being made there with how the tenant is designing their app/fault tolerance | |
| 19:16:07 | melwitt | dansmith: yeah | |
| 19:16:22 | dansmith | clutchbeyers: I totally agree that if you asked for one, you shouldn't be migrated out of it | |
| 19:17:18 | dansmith | I get that mriedem's metadata thing is a regression, | |
| 19:17:28 | dansmith | I'm missing what, if any, scheduler behavior regressed | |
| 19:17:29 | mriedem | dansmith: i don't remember talking about this specifically in dublin, | |
| 19:17:41 | mriedem | the only AZ thing I remember talking about wrt cells was the cross_az_attach volume upcall thing | |
| 19:18:06 | mriedem | dansmith: what i'm pointing out in this comment https://bugs.launchpad.net/nova/+bug/1768876/comments/4 | |
| 19:18:06 | openstack | Launchpad bug 1768876 in OpenStack Compute (nova) queens "metadata-api fails to get availability zone for instances created before pike" [High,Triaged] | |
| 19:18:12 | dansmith | I'm pretty sure it was dublin, with bauzas and I | |
| 19:18:12 | mriedem | i need to test it still to confirm it is a regression | |
| 19:19:47 | dansmith | okay, I'm confused because I thought the second part of your comment (that an instance would be confined to whatever AZ it landed in) wasn't yet the case, but we considered doing that | |
| 19:20:01 | dansmith | in the dublin discussion where we were talking about how to handle instance.az when az wasn't requested | |
| 19:21:19 | mriedem | maybe we were talking about that in the context of volume cross_az_attach? | |
| 19:21:53 | dansmith | I dunno, I thought not, but I don't see it on the etherpad | |
| 19:22:08 | dansmith | I _know_ I've had a convo with bauzas about this in the not too distant past | |
| 19:22:22 | dansmith | maybe the az discussion there is merging together in my head | |
| 19:22:45 | mriedem | maybe related to the ops list thread on not allowing deleting/renaming an AZ that has instances in it? | |
| 19:23:09 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: always create consumer records https://review.openstack.org/567678 | |
| 19:23:17 | dansmith | this was face-to-face or face-to-screen | |
| 19:23:19 | dansmith | with bazuas | |
| 19:24:10 | dansmith | mriedem: what change in pike do you think is making us keep things in the same az? | |
| 19:24:30 | mriedem | https://review.openstack.org/#/c/446053/ | |
| 19:25:04 | dansmith | ah, yeah | |
| 19:25:13 | mriedem | if the instance.availability_zone was set based on CONF.default_availability_zone when the user didn't request one specifically, before that change, and that defaults to None, i believe that means the instance could freely move between AZs | |
| 19:25:33 | mriedem | anyway, i can write a simple functional test to try this once i get this 'remove fping' patch up | |
| 19:25:49 | dansmith | yeah, so, this is the change I was talking about, | |
| 19:25:59 | dansmith | which is probably when I was talking to bauzas about it (before writing this) | |
| 19:26:06 | dansmith | I just forgot that we had made it | |
| 19:26:42 | mriedem | heh, time flies :) | |
| 19:26:48 | dansmith | apparently | |
| 19:26:50 | mriedem | i don't think i was aware of this change either until yesterday | |
| 19:26:59 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove support for /os-fping REST API https://review.openstack.org/567682 | |
| 19:26:59 | melwitt | I just checked and the default is 'nova', not None. not sure if that makes a difference | |
| 19:27:13 | mriedem | melwitt: not for CONF.default_availability_zone | |
| 19:27:20 | mriedem | there are 2 "default" AZ options | |
| 19:27:22 | dansmith | so, going back to my "up for debate" comment above, | |
| 19:27:27 | mriedem | one for the user's instance and one for the scheduler | |
| 19:27:32 | melwitt | that's what I'm looking at? | |
| 19:27:39 | dansmith | what's the actual problem with keeping it where it is? | |
| 19:27:46 | melwitt | default_schedule_zone is the None one | |
| 19:27:58 | melwitt | (has no default) | |
| 19:28:29 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: always create consumer records https://review.openstack.org/567678 | |
| 19:28:37 | mriedem | i always have to read this again https://docs.openstack.org/nova/latest/user/aggregates.html#availability-zones-azs | |
| 19:29:13 | mriedem | melwitt: oh that must have changed, | |
| 19:29:15 | mriedem | see pike | |
| 19:29:19 | mriedem | https://docs.openstack.org/nova/pike/configuration/config.html | |
| 19:29:32 | mriedem | oh nvm it didn't change https://docs.openstack.org/nova/pike/configuration/config.html#DEFAULT.default_availability_zone | |
| 19:29:35 | mriedem | wtf my eyes | |
| 19:30:23 | melwitt | tbh the only reason I remember the 'nova' availability zone being default is from using devstack | |
| 19:30:33 | dansmith | [12:27:34] <dansmith>what's the actual problem with keeping it where it is? | |
| 19:30:43 | mriedem | oh my comment in the bug is wrong | |
| 19:30:45 | mriedem | default_schedule_zone is for instances | |
| 19:30:50 | mriedem | default_availability_zone is for compute services | |
| 19:31:10 | mriedem | dansmith: i don't know that there is, | |
| 19:31:14 | mriedem | except it's a behavior change | |
| 19:31:17 | mriedem | but, | |
| 19:31:26 | mriedem | (2:11:06 PM) mriedem: dansmith: maybe something we need to put in your forum session on cells... | |
| 19:31:26 | melwitt | dansmith: the concern is if that broke the original behavior of letting an instance move across AZs if someone didn't specify an AZ to begin with, if they don't care about AZ | |
| 19:31:27 | mriedem | :) | |
| 19:31:47 | melwitt | but we're not 100% sure yet if that behavior changed, mriedem is gonna test it out in a func test | |
| 19:31:47 | mriedem | maybe it's just an FYI to operators at this point, idk | |
| 19:31:56 | dansmith | no I think it did change | |
| 19:32:01 | dansmith | but Idon't think it's user-visible | |
| 19:32:06 | dansmith | what I do think makes more sense about the new behavior, | |
| 19:32:20 | melwitt | it only would be if all the compute hosts in the same AZ are at capacity and there are free ones in another AZ, right? | |
| 19:32:24 | dansmith | is that if I boot an instance, ssh in, check the AZ that it's in, and then boot an instance in another az to make sure I have separation, | |
| 19:32:34 | dansmith | then something moves my instance into the same az and I'm surprised | |
| 19:32:39 | melwitt | and they're stuck because it won't let the AZ be crossed | |
| 19:32:41 | dansmith | which won't happen now | |
| 19:33:00 | dansmith | melwitt: yeah but they have no idea if there is space in the cloud, or in their az | |
| 19:33:59 | mriedem | one fear i have is operators will rely on the force flag for live migrate or evacuate to bypass the scheduler now to move the instance into another compute just because that other compute is in another AZ, | |
| 19:34:01 | mriedem | but idk | |
| 19:34:02 | melwitt | yeah, not until they contact support I guess, and admin can see the migration should be able to work and it doesn't | |
| 19:34:38 | mriedem | if only we could force a cold migration to bypass the scheduler.... :) | |
| 19:34:40 | dansmith | mriedem: fair point, but not super strong, IMHO | |
| 19:34:46 | mriedem | dansmith: agree it's not strong | |
| 19:35:04 | mriedem | if you have 10 computes in AZ1 and 10 in AZ2, then just pick one of the ones that's already in the same AZ | |
| 19:35:38 | mriedem | also, kablam https://review.openstack.org/567682 | |
| 19:38:19 | openstackgerrit | Arvind Nadendla proposed openstack/nova master: update scheduler to use image-traits https://review.openstack.org/560596 | |
| 19:40:36 | clutchbeyers | mriedem: cold migrations bypassing the scheduler was a joke right? :) | |
| 19:42:31 | mriedem | clutchbeyers: L645 https://etherpad.openstack.org/p/nova-ptg-rocky | |
| 19:43:01 | mriedem | oh the -5 didn't make it into the notes | |
| 19:50:48 | melwitt | dansmith: I was just reading the ML thread about the map_instances marker thing, and I think what takashin was suggesting is to print the next marker before returning 1, and then let that be passed as a starting point when the command is run again https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L1193 | |
| 19:51:03 | melwitt | would that not work for some reason? | |