Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-12-19
16:13:25 nicolasbock sahid: Yes, that thought occurred to me as well
16:13:26 mriedem they are similar
16:13:28 nicolasbock I'll double check
16:13:34 ShilpaSD mriedem: ok, thnaks
16:13:47 mriedem nicolasbock: key pairs are in the api db
16:14:05 mriedem key pair info per instance is in the instance_extra table in the cell db
16:14:26 nicolasbock Thanks mriedem !
16:14:37 openstack Launchpad bug 1784579 in OpenStack Compute (nova) queens "unable to live migrate instance after update to queens" [Medium,In progress] - Assigned to Matt Riedemann (mriedem)
16:14:37 mriedem stephenfin: sean-k-mooney; btw, i tried to summarize some stuff on that port binding failed bug https://bugs.launchpad.net/nova/+bug/1784579/comments/13
16:15:04 sean-k-mooney mriedem: so i made the continue change and there were not unit test failures with tox -e py27 -- "pci|PCI|hvdevs|update_devices_from_hypervisor_resources" so im going to write a new one
16:15:11 stephenfin mriedem: :D https://bugs.launchpad.net/nova/+bug/1784579/comments/14
16:15:31 sean-k-mooney ill run the full set to be sure but im guessing ther was not test code
16:15:38 stephenfin Probably should have agreed on who was doing that. Oh well
16:16:07 mriedem sean-k-mooney: i'm not at all surprised there was missing test coverage for that code
16:16:37 openstackgerrit Stephen Finucane proposed openstack/nova master: Handle unbound vif plug errors on compute restart https://review.openstack.org/626228
16:16:56 stephenfin mriedem: There's the fix for the latest issue
16:17:05 sean-k-mooney mriedem: look like ther are some test but none that assert that behavior
16:18:01 sahid nicolasbock: i just checked on my devstack i can list the keypairs, i used "nova_api" database
16:20:30 melwitt jackding: review runways are only for approved blueprint implementations, not spec reviews unfortuatnately (please see instructions on the etherpad), so I'm removing the specs from the queue FYI
16:20:52 mriedem stephenfin: comments inline
16:21:18 nicolasbock sahid: Thanks, I found them!
16:21:27 nicolasbock I was looking in the `nova` database before
16:21:41 nicolasbock Thanks for the help sahid and mriedem
16:21:43 sean-k-mooney melwitt: for blueprints/specs we are ment to list them for open discution in the nova team meeting instead right too highlight them
16:22:03 mriedem melwitt: before adding https://blueprints.launchpad.net/nova/+spec/handling-down-cell back into the runway, we should probably know if tssurya is even around
16:22:09 mriedem because i don't think she is and there are -1s on the api change
16:22:14 mriedem so there isn't much point in it being in a runway slot
16:22:19 melwitt mriedem: oh, right
16:24:04 canori01 mriedem: so pinning my flavors to the az's like you suggested yesterday worked fine. So instances don't leave their hypervisor's az if I give them a flavor that associates them to a host aggregate. My situation for the boot volumes is that they are ceph rbd backed. However, each az is backed by a different ceph cluster (because we didn't want a ceph problem in one az to affect the others).
16:24:18 canori01 Probnlem I had is that on resize operations, the scheduler sometimes picked a host in another az and the resize would subsequently fail because that host can't access the rbd volume if it's in a different az
16:24:24 melwitt sean-k-mooney: yeah, that's a way to get visibility by putting them on open discussion agenda
16:24:29 canori01 So while the flavor pinning works, I'm wondering how come it doesn't honor the OS-EXT-AZ:availability_zone of the instance when resizing
16:27:38 sean-k-mooney Cardoe: does the instance have an az set
16:27:48 mriedem canori01: as i said before, if the instance is not created with a specific az, the scheduler does not restrict it to that az
16:28:15 sean-k-mooney Cardoe: sorry that was for canori01
16:28:16 openstackgerrit Stephen Finucane proposed openstack/nova master: Handle unbound vif plug errors on compute restart https://review.openstack.org/626228
16:28:19 mriedem you could set https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.default_schedule_zone to force a default az, but you might not want that
16:28:22 stephenfin mriedem: Thanks. Addressed
16:29:22 mriedem canori01: alternatively, if each volume is in a specific az, you could set https://docs.openstack.org/nova/latest/configuration/config.html#cinder.cross_az_attach to false and then would need https://review.openstack.org/#/c/469675/ to proxy the volume az to the instance during server create
16:29:39 mriedem cross_az_attach=false means the server and root volume have to be in the same az
16:30:32 mriedem having said that, https://review.openstack.org/#/c/469675/ is kind of fugly and i would like to work on an alternative fix that is less tightly coupled down that stack of code, but haven't found the time
16:30:37 sean-k-mooney mriedem: does that rely on the cinder az mataching the nova az
16:30:43 mriedem sean-k-mooney: yes
16:31:01 mriedem as can be seen here, it's extremely easy to break cross_az_attach=false today https://review.openstack.org/#/c/467674/
16:31:35 mriedem sorrison at nectar is the only deployer i know personally (target uses it also) that uses cross_az_attach and he said their users are just required to always specify an az when creating a server
16:31:50 openstack bug 1694844 in OpenStack Compute (nova) "Boot from volume fails when cross_az_attach=False and volume is provided to nova without an AZ for the instance" [Medium,In progress] https://launchpad.net/bugs/1694844 - Assigned to Matt Riedemann (mriedem)
16:31:50 mriedem b/c of bug 1694844
16:32:15 sean-k-mooney right
16:32:41 sean-k-mooney i assume we dont have an api policy/config option to enforce that
16:32:46 canori01 mriedem: is the OS-EXT-AZ:availability_zone attribute on the instance different than what the scheduler looks at? When, instantiate from horizon and choose any availability zone, that gets set with that of the host. So if I were to actually choose an az at that time, then it would honor the az when moving/resizing?
16:33:32 sean-k-mooney i think that is just the az it is currently schdulerd to but i think the schduler looks at the requst spec for the az when picking hosts
16:33:46 melwitt mriedem: I agree with your assessment on that bug you linked. the trace shows "(self.host_state_map[host] for host in seen_nodes)" noting self.host_state_map which is before my fix landed, but also noting that's different than what the code was _before_ my fix landed as well. the previous code was "(self.host_state_map[host] for host in seen_nodes if host in self.host_state_map)" which would also avoid the KeyError, which reminds me,
16:34:09 melwitt of another fix that they also don't have, judging from that trace https://github.com/openstack/nova/commit/d72b33b986525a9b2c7aa08b609ae386de1d0e89
16:34:37 mriedem ah yeah
16:34:56 mriedem ok maybe they just reported their version incorrectly
16:35:18 mriedem sean-k-mooney: api policy/config option for what?
16:35:25 mriedem [cinder]/cross_az_attach is read in the api
16:35:53 mriedem canori01: "When, instantiate from horizon and choose any availability zone, that gets set with that of the host. So if I were to actually choose an az at that time, then it would honor the az when moving/resizing?" correct
16:36:04 mriedem canori01: "is the OS-EXT-AZ:availability_zone attribute on the instance different than what the scheduler looks at? " also correct
16:36:32 mriedem canori01: the instance.availability_zone field in the db is set to whatever az its compute host is in
16:36:46 mriedem and instance.availability_zone gets changed as the instance moves around
16:37:01 mriedem the scheduler looks at RequestSpec.availability_zone, which is the thing the user requested when they created the server
16:37:14 mriedem so request_spec.az is immutable, but instance.az is not
16:37:51 mriedem how we decide which az to use if the instance.host is in multiple azs....idk
16:38:00 mriedem ^ is a jaypipes feature
16:38:17 mriedem oh wait the compute host can be in multiple aggregates but only 1 az right?
16:38:19 mriedem i always forget the rules
16:39:25 mriedem yeah https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#host-aggregates-and-availability-zones "Availability zones are different from host aggregates in that they are explicitly exposed to the user, and hosts can only be in a single availability zone. Administrators can configure a default availability zone where instances will be scheduled when the user fails to specify one."
16:39:54 mriedem canori01: so, you could setup a default ceph pool with a default az for users that don't specify a specific az and flavor that isn't tied to a given az
16:40:22 mriedem gets kind of hard to manage after awhile probably
16:40:28 openstackgerrit Merged openstack/nova master: Address nits on I08991796aaced2abc824f608108c0c786181eb65 https://review.openstack.org/614322
16:40:49 canori01 mriedem: Perfect, thansk!
16:40:57 canori01 sean-k-mooney: thank you as well
16:43:59 mriedem stephenfin: +2
16:44:08 stephenfin \o/
16:44:38 openstackgerrit Matt Riedemann proposed openstack/nova stable/pike: Handle binding_failed vif plug errors on compute restart https://review.openstack.org/626361
16:45:34 jaypipes mriedem: ask bauzas.
16:45:46 mriedem i think bauzas is on permanent PTO
16:50:42 mriedem seriously though if anyone knows if bauzas is out the rest of the year, or what, it would be nice to know since i thought he was done with downstream fires for awhile
16:50:59 bauzas mriedem: I literrally have 2 days left :(
16:51:05 mriedem oh so you have been around
16:51:08 bauzas mriedem: but I'll commit myself on upstream reviews
16:51:25 bauzas and upstream revision of the placement spec I have
16:51:27 mriedem well i can give you a bunch of specs to just blindly approve then
16:51:55 bauzas mriedem: that's reasonable, I'm just discussing with internal folks about begging time for upstream before I leave
16:52:11 bauzas I'll just throw my downstream firehose for the next 2 days
16:52:15 mriedem bauzas: in berlin you told me you were good to go for upstream again?
16:52:23 bauzas mriedem: I was *thinking* to
16:52:59 bauzas mriedem: but then someone left us, and more customers are using our OSP12/OSP13 codebase that runs placement :)
16:53:12 bauzas which makes me dragged
16:53:53 mriedem alright well here is a list: https://review.openstack.org/#/c/393930/ https://review.openstack.org/#/c/612531/ https://review.openstack.org/#/c/616037/ https://review.openstack.org/#/c/609779/ https://review.openstack.org/#/c/603352/
16:54:08 mriedem melwitt: weren't you also working on a list of specs that looked like they could use some attention before the freeze?
16:54:23 mriedem in general, i need reviews on the cross cell resize spec from people not named dan since he's been the only one
16:54:23 melwitt mriedem: yes, sent it out like a minute ago
16:54:35 mriedem and it's a hairy gd monster and if others aren't going to review it it's DOA for stein
16:55:12 melwitt yeah, I know :( I'm reviewing it today
16:55:12 mriedem bauzas: thoughts on my email yesterday about per-instance live migration timeouts would also be nice
16:55:36 bauzas mriedem: ack
16:55:38 mriedem cfriesen: you might chime in on http://lists.openstack.org/pipermail/openstack-discuss/2018-December/001112.html as well
16:56:04 melwitt mriedem: feel free to add notes and specs that are on your radar that I missed https://etherpad.openstack.org/p/nova-stein-blueprint-spec-freeze

Earlier   Later