| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-05-10 | |||
| 19:17:18 | dansmith | I get that mriedem's metadata thing is a regression, | |
| 19:17:28 | dansmith | I'm missing what, if any, scheduler behavior regressed | |
| 19:17:29 | mriedem | dansmith: i don't remember talking about this specifically in dublin, | |
| 19:17:41 | mriedem | the only AZ thing I remember talking about wrt cells was the cross_az_attach volume upcall thing | |
| 19:18:06 | mriedem | dansmith: what i'm pointing out in this comment https://bugs.launchpad.net/nova/+bug/1768876/comments/4 | |
| 19:18:06 | openstack | Launchpad bug 1768876 in OpenStack Compute (nova) queens "metadata-api fails to get availability zone for instances created before pike" [High,Triaged] | |
| 19:18:12 | dansmith | I'm pretty sure it was dublin, with bauzas and I | |
| 19:18:12 | mriedem | i need to test it still to confirm it is a regression | |
| 19:19:47 | dansmith | okay, I'm confused because I thought the second part of your comment (that an instance would be confined to whatever AZ it landed in) wasn't yet the case, but we considered doing that | |
| 19:20:01 | dansmith | in the dublin discussion where we were talking about how to handle instance.az when az wasn't requested | |
| 19:21:19 | mriedem | maybe we were talking about that in the context of volume cross_az_attach? | |
| 19:21:53 | dansmith | I dunno, I thought not, but I don't see it on the etherpad | |
| 19:22:08 | dansmith | I _know_ I've had a convo with bauzas about this in the not too distant past | |
| 19:22:22 | dansmith | maybe the az discussion there is merging together in my head | |
| 19:22:45 | mriedem | maybe related to the ops list thread on not allowing deleting/renaming an AZ that has instances in it? | |
| 19:23:09 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: always create consumer records https://review.openstack.org/567678 | |
| 19:23:17 | dansmith | this was face-to-face or face-to-screen | |
| 19:23:19 | dansmith | with bazuas | |
| 19:24:10 | dansmith | mriedem: what change in pike do you think is making us keep things in the same az? | |
| 19:24:30 | mriedem | https://review.openstack.org/#/c/446053/ | |
| 19:25:04 | dansmith | ah, yeah | |
| 19:25:13 | mriedem | if the instance.availability_zone was set based on CONF.default_availability_zone when the user didn't request one specifically, before that change, and that defaults to None, i believe that means the instance could freely move between AZs | |
| 19:25:33 | mriedem | anyway, i can write a simple functional test to try this once i get this 'remove fping' patch up | |
| 19:25:49 | dansmith | yeah, so, this is the change I was talking about, | |
| 19:25:59 | dansmith | which is probably when I was talking to bauzas about it (before writing this) | |
| 19:26:06 | dansmith | I just forgot that we had made it | |
| 19:26:42 | mriedem | heh, time flies :) | |
| 19:26:48 | dansmith | apparently | |
| 19:26:50 | mriedem | i don't think i was aware of this change either until yesterday | |
| 19:26:59 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove support for /os-fping REST API https://review.openstack.org/567682 | |
| 19:26:59 | melwitt | I just checked and the default is 'nova', not None. not sure if that makes a difference | |
| 19:27:13 | mriedem | melwitt: not for CONF.default_availability_zone | |
| 19:27:20 | mriedem | there are 2 "default" AZ options | |
| 19:27:22 | dansmith | so, going back to my "up for debate" comment above, | |
| 19:27:27 | mriedem | one for the user's instance and one for the scheduler | |
| 19:27:32 | melwitt | that's what I'm looking at? | |
| 19:27:39 | dansmith | what's the actual problem with keeping it where it is? | |
| 19:27:46 | melwitt | default_schedule_zone is the None one | |
| 19:27:58 | melwitt | (has no default) | |
| 19:28:29 | openstackgerrit | Jay Pipes proposed openstack/nova master: placement: always create consumer records https://review.openstack.org/567678 | |
| 19:28:37 | mriedem | i always have to read this again https://docs.openstack.org/nova/latest/user/aggregates.html#availability-zones-azs | |
| 19:29:13 | mriedem | melwitt: oh that must have changed, | |
| 19:29:15 | mriedem | see pike | |
| 19:29:19 | mriedem | https://docs.openstack.org/nova/pike/configuration/config.html | |
| 19:29:32 | mriedem | oh nvm it didn't change https://docs.openstack.org/nova/pike/configuration/config.html#DEFAULT.default_availability_zone | |
| 19:29:35 | mriedem | wtf my eyes | |
| 19:30:23 | melwitt | tbh the only reason I remember the 'nova' availability zone being default is from using devstack | |
| 19:30:33 | dansmith | [12:27:34] <dansmith>what's the actual problem with keeping it where it is? | |
| 19:30:43 | mriedem | oh my comment in the bug is wrong | |
| 19:30:45 | mriedem | default_schedule_zone is for instances | |
| 19:30:50 | mriedem | default_availability_zone is for compute services | |
| 19:31:10 | mriedem | dansmith: i don't know that there is, | |
| 19:31:14 | mriedem | except it's a behavior change | |
| 19:31:17 | mriedem | but, | |
| 19:31:26 | mriedem | (2:11:06 PM) mriedem: dansmith: maybe something we need to put in your forum session on cells... | |
| 19:31:26 | melwitt | dansmith: the concern is if that broke the original behavior of letting an instance move across AZs if someone didn't specify an AZ to begin with, if they don't care about AZ | |
| 19:31:27 | mriedem | :) | |
| 19:31:47 | melwitt | but we're not 100% sure yet if that behavior changed, mriedem is gonna test it out in a func test | |
| 19:31:47 | mriedem | maybe it's just an FYI to operators at this point, idk | |
| 19:31:56 | dansmith | no I think it did change | |
| 19:32:01 | dansmith | but Idon't think it's user-visible | |
| 19:32:06 | dansmith | what I do think makes more sense about the new behavior, | |
| 19:32:20 | melwitt | it only would be if all the compute hosts in the same AZ are at capacity and there are free ones in another AZ, right? | |
| 19:32:24 | dansmith | is that if I boot an instance, ssh in, check the AZ that it's in, and then boot an instance in another az to make sure I have separation, | |
| 19:32:34 | dansmith | then something moves my instance into the same az and I'm surprised | |
| 19:32:39 | melwitt | and they're stuck because it won't let the AZ be crossed | |
| 19:32:41 | dansmith | which won't happen now | |
| 19:33:00 | dansmith | melwitt: yeah but they have no idea if there is space in the cloud, or in their az | |
| 19:33:59 | mriedem | one fear i have is operators will rely on the force flag for live migrate or evacuate to bypass the scheduler now to move the instance into another compute just because that other compute is in another AZ, | |
| 19:34:01 | mriedem | but idk | |
| 19:34:02 | melwitt | yeah, not until they contact support I guess, and admin can see the migration should be able to work and it doesn't | |
| 19:34:38 | mriedem | if only we could force a cold migration to bypass the scheduler.... :) | |
| 19:34:40 | dansmith | mriedem: fair point, but not super strong, IMHO | |
| 19:34:46 | mriedem | dansmith: agree it's not strong | |
| 19:35:04 | mriedem | if you have 10 computes in AZ1 and 10 in AZ2, then just pick one of the ones that's already in the same AZ | |
| 19:35:38 | mriedem | also, kablam https://review.openstack.org/567682 | |
| 19:38:19 | openstackgerrit | Arvind Nadendla proposed openstack/nova master: update scheduler to use image-traits https://review.openstack.org/560596 | |
| 19:40:36 | clutchbeyers | mriedem: cold migrations bypassing the scheduler was a joke right? :) | |
| 19:42:31 | mriedem | clutchbeyers: L645 https://etherpad.openstack.org/p/nova-ptg-rocky | |
| 19:43:01 | mriedem | oh the -5 didn't make it into the notes | |
| 19:50:48 | melwitt | dansmith: I was just reading the ML thread about the map_instances marker thing, and I think what takashin was suggesting is to print the next marker before returning 1, and then let that be passed as a starting point when the command is run again https://github.com/openstack/nova/blob/master/nova/cmd/manage.py#L1193 | |
| 19:51:03 | melwitt | would that not work for some reason? | |
| 19:57:53 | edleafe | melwitt: if the batches were all done in a row, it might. But there could be long gaps of time between batches, so the marker needs to be stored | |
| 19:58:23 | melwitt | I see | |
| 20:07:08 | openstackgerrit | Arvind Nadendla proposed openstack/nova master: update scheduler to use image-traits https://review.openstack.org/560596 | |
| 20:07:38 | dansmith | melwitt: yeah I know he was talking about that and I agree with edleafe that it's not a great solution | |
| 20:07:54 | dansmith | melwitt: the bit I'm not sure about was the "it could be exposed accidentally through the api" part | |
| 20:09:20 | melwitt | okay, I think I missed the "exposed accidentally through the api" part | |
| 20:09:51 | melwitt | I didn't see anyone mention something like that | |
| 20:10:04 | dansmith | he said "in some compute rest apis, it returns the marker" | |
| 20:10:26 | melwitt | I think he means like in instance list, it returns the marker if there's additional pages available | |
| 20:11:06 | melwitt | like if you have > 1000 (api max limit) instances in the list, it will give you a "next marker" so you can query again and pass the marker to get the next page | |
| 20:11:32 | dansmith | you mean you think he's offering the REST API's pagination as an example that could be used for the cli to be user-supplied-marker behavior? | |
| 20:12:03 | melwitt | that's what I think he was saying, yeah. if doing it similar to that could avoid storing the marker | |
| 20:12:32 | dansmith | I guess I can see it being read like that, but to me it sounded like he was saying it's bad to store it in-band because there's a way you can see it externally | |
| 20:12:41 | melwitt | if map_instances hands you back the next marker, says "I did a batch but there's more" and you can call again and give the marker and it does the next batch | |
| 20:14:37 | dansmith | I understand how the user could supply a marker, yeah | |
| 20:15:00 | dansmith | I just think that's fairly terrible UX for a CLI, especially a chatty one that has all kinds of other crap going to stdio like python warnings | |
| 20:15:39 | melwitt | oh, hm. yeah, I guess if I re-read it and connect the 3rd paragraph to the 2nd, it could sound that way. but I thought he was stating it's not ideal to store the marker and then next gave a suggestion on how we could avoid storing it | |
| 20:16:07 | dansmith | I'll send another clarification mail | |