| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-21 | |||
| 21:00:53 | mriedem | JESUS THEY ARE SO INTERTWINED!!! | |
| 21:00:54 | artom | Cold migrate is... always to a new host? | |
| 21:01:02 | mriedem | artom: you mean operator configurable... | |
| 21:01:03 | artom | mriedem, just... let me be angry | |
| 21:01:07 | artom | mriedem, sorry, yeah | |
| 21:01:19 | artom | I meant in the code, anyways | |
| 21:01:22 | mriedem | artom: remember https://review.opendev.org/#/c/676022/ | |
| 21:01:26 | melwitt | gregwork: they can but only by way of a AZ. you have to make one and map it to the aggregate(s) you want (as admin) | |
| 21:01:36 | artom | Like, you hit a method called "_cold_migrate" in the resize flow | |
| 21:01:52 | mriedem | yeah, so docstrings ftw | |
| 21:01:52 | artom | mriedem, I do | |
| 21:02:05 | mriedem | -1 people that don't write code comments | |
| 21:02:13 | gregwork | melwit: from reading up on AZ's in openstack, these seem like a very heavy abstraction compared to a simple host aggregate to configure | |
| 21:02:19 | mriedem | be the change you want to see in the world... | |
| 21:02:48 | artom | But how do I become free limitless beer? | |
| 21:02:58 | artom | (Point taken, though) | |
| 21:03:13 | melwitt | gregwork: they're really just a tag on host aggregates. shouldn't be heavy. you put tag my_az on the host aggs you want to be part of it and then the user can say my_az | |
| 21:03:46 | mriedem | artom: so if you want to see some things in a contributor doc about resize vs cold migrate, throw those into a doc bug and maybe i can crank something out | |
| 21:16:49 | melwitt | gregwork: sorry that link was for admin user to be able to bypass the scheduler. this is the normal end user instructions for how to specify AZ https://docs.openstack.org/nova/latest/user/availability-zones.html | |
| 21:20:20 | artom | mriedem, I think that I'd really at this point is a sequence diagram for resize (to start with), like we have for live migration | |
| 21:21:02 | artom | Btw, stephenfin, who gave me https://review.opendev.org/#/c/662522/, I will forever remember that | |
| 21:21:27 | artom | My fault for opening my mouth and saying I have extra bandwidth, I suppose | |
| 21:21:50 | mriedem | i had an old todo somewhere to do a resize diagram like the live migrate one i put on a post-it and efried turned into that diagram | |
| 21:21:54 | mriedem | and then got the nickel 3 years later | |
| 21:22:24 | mriedem | artom: fwiw i do have a simple resize diagram in my last cells v2 summit presentation, you could lift the slides from that and turn it into what seqdiag stuff in sphinx | |
| 21:23:00 | mriedem | https://object-storage-ca-ymq-1.vexxhost.net/swift/v1/6e4619c416ff4bd19e1c087f27a43eea/www-assets-prod/summits/26/presentations/23287/slides/Whats-new-in-Nova-CellsV2-Denver-Summit-2019.pdf | |
| 21:23:21 | mriedem | slide 26 | |
| 21:25:22 | artom | mriedem, much thanks | |
| 21:26:06 | gregwork | hmmn "Cannot update metadata of aggregate 11: Reason: One or more hosts already in availability zones [u'nova'] | |
| 21:29:56 | mriedem | i've noticed that artom has the same disappearing pattern as bauzas | |
| 21:30:03 | mriedem | t0: complain! | |
| 21:30:06 | mriedem | t1: here you can do this | |
| 21:30:12 | mriedem | t2: thanks! but i've got to run | |
| 21:31:54 | gregwork | alright nm i think i got it .. | |
| 21:32:17 | mriedem | gregwork: hosts can be in M aggregates but 1 AZ | |
| 21:32:43 | gregwork | yeah i had created an aggregate and already tagged it as nova | |
| 21:32:47 | gregwork | for the az | |
| 21:32:59 | gregwork | so adding an additional az was failing | |
| 21:33:32 | gregwork | now to figure out what the scheduler_hint is to specify the az in OS::Nova::Server | |
| 21:33:40 | gregwork | i dont think its group: | |
| 21:34:19 | mriedem | there is a big fat warning in the docs to not ever create an az literally called "nova" | |
| 21:34:33 | mriedem | b/c that's the default schedule zone for services, | |
| 21:34:50 | mriedem | so if you create an az that users boot into called nova they can be stuck and not migrate their servers (potentially) | |
| 21:35:04 | mriedem | we should probably just block that in the api, but no one has cared enough to yet | |
| 21:36:32 | gregwork | so apparently availability_zone is a property in os::nova::server and not a scheduler hint map | |
| 21:37:40 | mriedem | correct | |
| 21:37:45 | mriedem | but let's not bring heat into this please... | |
| 21:38:05 | gregwork | well im trying to understand how it works in nova so i can figure out how to solve this in heat | |
| 21:38:47 | mriedem | ok https://docs.openstack.org/api-ref/compute/?expanded=create-server-detail#create-server | |
| 21:39:45 | gregwork | os:scheduler_hints.stuff section is very very useful | |
| 21:39:46 | gregwork | thanks | |
| 21:40:37 | mriedem | yup - thank takashin for doing the thankless work of documenting a lot of that stuff | |
| 21:40:48 | mriedem | organizing it, etc | |
| 21:53:29 | efried | sean-k-mooney: what we were talking about earlier, I guess the first step to at least acknowledge the problem would be to beef up the help for https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.host | |
| 21:53:38 | efried | it already has a bullet list of "things this is used for" | |
| 21:54:37 | efried | Adding "external services looking up the compute node resource provider; due to bug #XXXXX you will break your world if you set this when using the libvirt driver, so don't." kind of thing. | |
| 22:05:22 | efried | sean-k-mooney: Potential remedy (though it would be kind of a long road) would be to expose the compute node UUID through that hypervisors API. | |
| 22:05:42 | efried | then we instruct consumers (neutron, cyborg) to use that rather than CONF.host to discover the compute node RP. | |
| 22:05:50 | efried | um | |
| 22:05:59 | efried | for libvirt | |
| 22:06:10 | efried | This whole thing makes efried :( | |
| 22:08:37 | efried | mriedem: you may have missed this earlier, but it turns out we broke CONF.host for libvirt. | |
| 22:10:31 | efried | Because external services (neutron (problem now), cyborg (problem soon)) assume CONF.host is the name of the compute node RP | |
| 22:10:54 | efried | ...which they need to know in order to hang their nested providers (neutron: bw; cyborg: accelerator) off of | |
| 22:11:56 | efried | but the compute node RP is actually hypervisor_hostname, which for libvirt is the `hostname()` of the system. | |
| 22:12:24 | efried | which is the default for CONF.host, so you're fine... unless you actually *set* that guy. | |
| 22:12:29 | efried | dansmith: ^ | |
| 22:13:52 | mriedem | "we broke CONF.host"? | |
| 22:13:58 | mriedem | who is we and when? | |
| 22:14:18 | mriedem | if we = jay and when is ocata then... | |
| 22:14:30 | artom | mriedem, all part of my cunning plan | |
| 22:15:14 | efried | okay, "If you set set CONF.host to a non-default value, and you're using libvirt, bandwidth (and accelerator, and other future external-service-created nested) providers are broken" | |
| 22:15:51 | mriedem | oopsy daisy | |
| 22:16:28 | efried | [1] https://docs.openstack.org/api-ref/compute/?expanded=list-hypervisors-detail#id306 | |
| 22:16:28 | efried | sean-k-mooney: o hey, it looks like you were maybe getting the hypervisor ID as a short ID because you were using the default microversion. Per [1] it'll come back as a UUID starting at 2.53. | |
| 22:16:38 | efried | which phew gives us a path forward without further changes. | |
| 22:16:46 | efried | I think | |
| 22:17:12 | efried | Having to hit the /os-hypervisors API is heavier than just looking at CONF.host, but that UUID is the UUID of the provider you want. | |
| 22:17:19 | mriedem | yup | |
| 22:18:35 | efried | hm, but mriedem, how do I ask /os-hypervisors for the entry for my current node? | |
| 22:19:14 | efried | cause the qparams and output still talk in terms of hypervisor_hostname :( | |
| 22:19:18 | mriedem | GET /os-hypervisors/detail?hypervisor_hostname_pattern=<hostname> | |
| 22:19:54 | efried | right, exactly. | |
| 22:20:17 | efried | I mean, if I'm going to rely on running `gethostname()` on the host, I might as well just do that from the start. | |
| 22:20:19 | mriedem | if not that, then i guess GET /os-hypervisors/detail and you iterate all until you find the one with the service.host that matches what you care about if hypervisor_hostname is not a match | |
| 22:21:17 | mriedem | i think service.host there is CONF.host | |
| 22:21:31 | mriedem | someone had an RFE at one point to be able to filter hypervisors by service.host for ironic | |
| 22:21:35 | mriedem | it would be pretty easy to do i think | |
| 22:21:50 | mriedem | it's not clear to me if you need that though | |
| 22:22:01 | mriedem | you being neutron/cyborg | |
| 22:22:14 | efried | that would be a sure, but heavy, way to correlate CONF.host. | |
| 22:22:22 | efried | esp in a big env, that's a big payload | |
| 22:22:39 | mriedem | which thing? getting all hypervisors and then finding the service.host? | |
| 22:23:00 | mriedem | yeah it would, and likely limited to 1000 results by default so you'd have to page | |
| 22:23:03 | efried | yeah, GET /os-hypervisors/detail is going to be a big response for CERn. | |
| 22:23:13 | efried | use sdk and it pages for you, which is fin. | |
| 22:23:14 | efried | fine. | |
| 22:23:18 | mriedem | sure, so GET /os-hypervisors/detail?service_host=CONF.host | |
| 22:23:25 | mriedem | ^ is the RFE i mentioned | |
| 22:23:27 | efried | oh, that's a thing... | |