Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-12-19
16:22:03 mriedem melwitt: before adding https://blueprints.launchpad.net/nova/+spec/handling-down-cell back into the runway, we should probably know if tssurya is even around
16:22:09 mriedem because i don't think she is and there are -1s on the api change
16:22:14 mriedem so there isn't much point in it being in a runway slot
16:22:19 melwitt mriedem: oh, right
16:24:04 canori01 mriedem: so pinning my flavors to the az's like you suggested yesterday worked fine. So instances don't leave their hypervisor's az if I give them a flavor that associates them to a host aggregate. My situation for the boot volumes is that they are ceph rbd backed. However, each az is backed by a different ceph cluster (because we didn't want a ceph problem in one az to affect the others).
16:24:18 canori01 Probnlem I had is that on resize operations, the scheduler sometimes picked a host in another az and the resize would subsequently fail because that host can't access the rbd volume if it's in a different az
16:24:24 melwitt sean-k-mooney: yeah, that's a way to get visibility by putting them on open discussion agenda
16:24:29 canori01 So while the flavor pinning works, I'm wondering how come it doesn't honor the OS-EXT-AZ:availability_zone of the instance when resizing
16:27:38 sean-k-mooney Cardoe: does the instance have an az set
16:27:48 mriedem canori01: as i said before, if the instance is not created with a specific az, the scheduler does not restrict it to that az
16:28:15 sean-k-mooney Cardoe: sorry that was for canori01
16:28:16 openstackgerrit Stephen Finucane proposed openstack/nova master: Handle unbound vif plug errors on compute restart https://review.openstack.org/626228
16:28:19 mriedem you could set https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.default_schedule_zone to force a default az, but you might not want that
16:28:22 stephenfin mriedem: Thanks. Addressed
16:29:22 mriedem canori01: alternatively, if each volume is in a specific az, you could set https://docs.openstack.org/nova/latest/configuration/config.html#cinder.cross_az_attach to false and then would need https://review.openstack.org/#/c/469675/ to proxy the volume az to the instance during server create
16:29:39 mriedem cross_az_attach=false means the server and root volume have to be in the same az
16:30:32 mriedem having said that, https://review.openstack.org/#/c/469675/ is kind of fugly and i would like to work on an alternative fix that is less tightly coupled down that stack of code, but haven't found the time
16:30:37 sean-k-mooney mriedem: does that rely on the cinder az mataching the nova az
16:30:43 mriedem sean-k-mooney: yes
16:31:01 mriedem as can be seen here, it's extremely easy to break cross_az_attach=false today https://review.openstack.org/#/c/467674/
16:31:35 mriedem sorrison at nectar is the only deployer i know personally (target uses it also) that uses cross_az_attach and he said their users are just required to always specify an az when creating a server
16:31:50 mriedem b/c of bug 1694844
16:31:50 openstack bug 1694844 in OpenStack Compute (nova) "Boot from volume fails when cross_az_attach=False and volume is provided to nova without an AZ for the instance" [Medium,In progress] https://launchpad.net/bugs/1694844 - Assigned to Matt Riedemann (mriedem)
16:32:15 sean-k-mooney right
16:32:41 sean-k-mooney i assume we dont have an api policy/config option to enforce that
16:32:46 canori01 mriedem: is the OS-EXT-AZ:availability_zone attribute on the instance different than what the scheduler looks at? When, instantiate from horizon and choose any availability zone, that gets set with that of the host. So if I were to actually choose an az at that time, then it would honor the az when moving/resizing?
16:33:32 sean-k-mooney i think that is just the az it is currently schdulerd to but i think the schduler looks at the requst spec for the az when picking hosts
16:33:46 melwitt mriedem: I agree with your assessment on that bug you linked. the trace shows "(self.host_state_map[host] for host in seen_nodes)" noting self.host_state_map which is before my fix landed, but also noting that's different than what the code was _before_ my fix landed as well. the previous code was "(self.host_state_map[host] for host in seen_nodes if host in self.host_state_map)" which would also avoid the KeyError, which reminds me,
16:34:09 melwitt of another fix that they also don't have, judging from that trace https://github.com/openstack/nova/commit/d72b33b986525a9b2c7aa08b609ae386de1d0e89
16:34:37 mriedem ah yeah
16:34:56 mriedem ok maybe they just reported their version incorrectly
16:35:18 mriedem sean-k-mooney: api policy/config option for what?
16:35:25 mriedem [cinder]/cross_az_attach is read in the api
16:35:53 mriedem canori01: "When, instantiate from horizon and choose any availability zone, that gets set with that of the host. So if I were to actually choose an az at that time, then it would honor the az when moving/resizing?" correct
16:36:04 mriedem canori01: "is the OS-EXT-AZ:availability_zone attribute on the instance different than what the scheduler looks at? " also correct
16:36:32 mriedem canori01: the instance.availability_zone field in the db is set to whatever az its compute host is in
16:36:46 mriedem and instance.availability_zone gets changed as the instance moves around
16:37:01 mriedem the scheduler looks at RequestSpec.availability_zone, which is the thing the user requested when they created the server
16:37:14 mriedem so request_spec.az is immutable, but instance.az is not
16:37:51 mriedem how we decide which az to use if the instance.host is in multiple azs....idk
16:38:00 mriedem ^ is a jaypipes feature
16:38:17 mriedem oh wait the compute host can be in multiple aggregates but only 1 az right?
16:38:19 mriedem i always forget the rules
16:39:25 mriedem yeah https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#host-aggregates-and-availability-zones "Availability zones are different from host aggregates in that they are explicitly exposed to the user, and hosts can only be in a single availability zone. Administrators can configure a default availability zone where instances will be scheduled when the user fails to specify one."
16:39:54 mriedem canori01: so, you could setup a default ceph pool with a default az for users that don't specify a specific az and flavor that isn't tied to a given az
16:40:22 mriedem gets kind of hard to manage after awhile probably
16:40:28 openstackgerrit Merged openstack/nova master: Address nits on I08991796aaced2abc824f608108c0c786181eb65 https://review.openstack.org/614322
16:40:49 canori01 mriedem: Perfect, thansk!
16:40:57 canori01 sean-k-mooney: thank you as well
16:43:59 mriedem stephenfin: +2
16:44:08 stephenfin \o/
16:44:38 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Handle binding_failed vif plug errors on compute restart https://review.openstack.org/626361
16:45:34 jaypipes mriedem: ask bauzas.
16:45:46 mriedem i think bauzas is on permanent PTO
16:50:42 mriedem seriously though if anyone knows if bauzas is out the rest of the year, or what, it would be nice to know since i thought he was done with downstream fires for awhile
16:50:59 bauzas mriedem: I literrally have 2 days left :(
16:51:05 mriedem oh so you have been around
16:51:08 bauzas mriedem: but I'll commit myself on upstream reviews
16:51:25 bauzas and upstream revision of the placement spec I have
16:51:27 mriedem well i can give you a bunch of specs to just blindly approve then
16:51:55 bauzas mriedem: that's reasonable, I'm just discussing with internal folks about begging time for upstream before I leave
16:52:11 bauzas I'll just throw my downstream firehose for the next 2 days
16:52:15 mriedem bauzas: in berlin you told me you were good to go for upstream again?
16:52:23 bauzas mriedem: I was *thinking* to
16:52:59 bauzas mriedem: but then someone left us, and more customers are using our OSP12/OSP13 codebase that runs placement :)
16:53:12 bauzas which makes me dragged
16:53:53 mriedem alright well here is a list: https://review.openstack.org/#/c/393930/ https://review.openstack.org/#/c/612531/ https://review.openstack.org/#/c/616037/ https://review.openstack.org/#/c/609779/ https://review.openstack.org/#/c/603352/
16:54:08 mriedem melwitt: weren't you also working on a list of specs that looked like they could use some attention before the freeze?
16:54:23 melwitt mriedem: yes, sent it out like a minute ago
16:54:23 mriedem in general, i need reviews on the cross cell resize spec from people not named dan since he's been the only one
16:54:35 mriedem and it's a hairy gd monster and if others aren't going to review it it's DOA for stein
16:55:12 mriedem bauzas: thoughts on my email yesterday about per-instance live migration timeouts would also be nice
16:55:12 melwitt yeah, I know :( I'm reviewing it today
16:55:36 bauzas mriedem: ack
16:55:38 mriedem cfriesen: you might chime in on http://lists.openstack.org/pipermail/openstack-discuss/2018-December/001112.html as well
16:56:04 melwitt mriedem: feel free to add notes and specs that are on your radar that I missed https://etherpad.openstack.org/p/nova-stein-blueprint-spec-freeze
16:56:12 mriedem will do
17:03:40 bauzas melwitt: thanks for the etherpad
17:11:06 cfriesen mriedem: will take a look
17:20:06 mriedem tl;dr are the compromises worthwhile to move forward
17:26:10 stephenfin Bhujay: What is going on with your IRC connection?
17:27:25 openstackgerrit Merged openstack/nova master: Address nits on I1f1fa1d0f79bec5a4101e03bc2d43ba581dd35a0 https://review.openstack.org/614323
17:27:32 openstackgerrit Merged openstack/nova master: Fix a broken-link in nova doc https://review.openstack.org/626113
17:32:50 openstackgerrit Matt Riedemann proposed openstack/nova stable/ocata: Handle binding_failed vif plug errors on compute restart https://review.openstack.org/626369
17:32:52 mriedem hooray for ocata em ^
17:34:39 stephenfin melwitt: Seeing as you looked at the earlier change, fancy taking a look at https://review.openstack.org/#/c/626228 ?
17:35:18 melwitt stephenfin: sure, always up for being pinged for reviews
17:35:30 stephenfin mriedem: Thanks for reviewing that nit patch (y)
17:36:16 mriedem the docs one? i didn't really, just saw gibi was +2 and it was a rebase
17:36:24 mriedem but yw :)
17:39:31 openstackgerrit Jack Ding proposed openstack/nova-specs master: Select cpu model from a list of cpu models https://review.openstack.org/620959
17:57:02 openstackgerrit Chris Dent proposed openstack/nova master: Redirect user/placement to placement docs https://review.openstack.org/626333
18:02:02 openstackgerrit Krzysztof Opasiak proposed openstack/nova master: Fix server IPs with non-unique network names https://review.openstack.org/625371
18:02:15 cfriesen stephenfin: any chance you could take a look at the cpu models spec proposed by Jack ^ ? Basically instead of setting one model in nova.conf the operator could specify a list, and the virt driver would use the first one that provides the requested cpu features.
18:02:58 openstackgerrit Krzysztof Opasiak proposed openstack/nova master: Fix server IPs with non-unique network names https://review.openstack.org/625371
18:16:00 melwitt mriedem: I wanted to bring this review to your attention, bug about returning build requests when a marker is specified (I know you love paginating stuff). I'm +2 on it https://review.openstack.org/624870
18:22:56 openstackgerrit Merged openstack/nova master: Remove legacy RequestSpec compat code from live migrate task https://review.openstack.org/625705
18:39:52 openstackgerrit Tim Rozet proposed openstack/nova master: Fixes race condition with privsep utime https://review.openstack.org/625741
18:57:55 mriedem i saw it before, asked andrey to flesh out the commit message, haven't been back
19:16:41 melwitt ah, ok

Earlier   Later