| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-21 | |||
| 20:57:15 | melwitt | lol, nah just leave em out | |
| 20:57:28 | artom | Hah, yeah, just conveniently forget step 3 | |
| 20:57:35 | artom | Or step 2, even | |
| 20:58:09 | mriedem | artom: how are rebuild and resize/cold migrate intertwined? | |
| 20:58:14 | efried | Nova meeting in 2 minutes in #openstack-meeting | |
| 20:58:24 | artom | mriedem, through sheer willpower and anger | |
| 20:58:25 | efried | mriedem: they're not, rebuild and evacuate are the same thing though. | |
| 20:58:37 | mriedem | don't make me link the doc i wrote again | |
| 20:58:41 | artom | OK, resize and cold migrate then | |
| 20:58:45 | artom | Mea culpa | |
| 20:59:14 | mriedem | ok so write a resize vs cold migrate doc like this https://docs.openstack.org/nova/latest/contributor/evacuate-vs-rebuild.html | |
| 20:59:15 | mriedem | easy | |
| 20:59:24 | mriedem | 1. resize has a new flavor, cold migrate doesn't, | |
| 20:59:31 | gregwork | does nova in queens understand how to boot instances on a particular host aggregate? ive tried tagging the host aggregate "compute = 1" and then passing scheduler_hints: compute: 1 when deploying a stack with OS::Nova::Server .. however that doesnt appear to do anything and the instances ends up on whatever compute node | |
| 20:59:36 | mriedem | 2. resize can sometimes go on the same host, cold migrate does not except if you're using vcenter | |
| 21:00:05 | mriedem | artom: what other differences are there besides those 2 things? | |
| 21:00:16 | melwitt | gregwork: has to be availability zone. host agg is a admin only hidden thing | |
| 21:00:36 | artom | mriedem, I think that's about it? | |
| 21:00:47 | artom | And resize to same host is user-configurable | |
| 21:00:52 | gregwork | melwitt: oh .. so even if the admin has defined the host agg, regular users cant reference it ? | |
| 21:00:53 | mriedem | JESUS THEY ARE SO INTERTWINED!!! | |
| 21:00:54 | artom | Cold migrate is... always to a new host? | |
| 21:01:02 | mriedem | artom: you mean operator configurable... | |
| 21:01:03 | artom | mriedem, just... let me be angry | |
| 21:01:07 | artom | mriedem, sorry, yeah | |
| 21:01:19 | artom | I meant in the code, anyways | |
| 21:01:22 | mriedem | artom: remember https://review.opendev.org/#/c/676022/ | |
| 21:01:26 | melwitt | gregwork: they can but only by way of a AZ. you have to make one and map it to the aggregate(s) you want (as admin) | |
| 21:01:36 | artom | Like, you hit a method called "_cold_migrate" in the resize flow | |
| 21:01:52 | mriedem | yeah, so docstrings ftw | |
| 21:01:52 | artom | mriedem, I do | |
| 21:02:05 | mriedem | -1 people that don't write code comments | |
| 21:02:13 | gregwork | melwit: from reading up on AZ's in openstack, these seem like a very heavy abstraction compared to a simple host aggregate to configure | |
| 21:02:19 | mriedem | be the change you want to see in the world... | |
| 21:02:48 | artom | But how do I become free limitless beer? | |
| 21:02:58 | artom | (Point taken, though) | |
| 21:03:13 | melwitt | gregwork: they're really just a tag on host aggregates. shouldn't be heavy. you put tag my_az on the host aggs you want to be part of it and then the user can say my_az | |
| 21:03:46 | mriedem | artom: so if you want to see some things in a contributor doc about resize vs cold migrate, throw those into a doc bug and maybe i can crank something out | |
| 21:16:49 | melwitt | gregwork: sorry that link was for admin user to be able to bypass the scheduler. this is the normal end user instructions for how to specify AZ https://docs.openstack.org/nova/latest/user/availability-zones.html | |
| 21:20:20 | artom | mriedem, I think that I'd really at this point is a sequence diagram for resize (to start with), like we have for live migration | |
| 21:21:02 | artom | Btw, stephenfin, who gave me https://review.opendev.org/#/c/662522/, I will forever remember that | |
| 21:21:27 | artom | My fault for opening my mouth and saying I have extra bandwidth, I suppose | |
| 21:21:50 | mriedem | i had an old todo somewhere to do a resize diagram like the live migrate one i put on a post-it and efried turned into that diagram | |
| 21:21:54 | mriedem | and then got the nickel 3 years later | |
| 21:22:24 | mriedem | artom: fwiw i do have a simple resize diagram in my last cells v2 summit presentation, you could lift the slides from that and turn it into what seqdiag stuff in sphinx | |
| 21:23:00 | mriedem | https://object-storage-ca-ymq-1.vexxhost.net/swift/v1/6e4619c416ff4bd19e1c087f27a43eea/www-assets-prod/summits/26/presentations/23287/slides/Whats-new-in-Nova-CellsV2-Denver-Summit-2019.pdf | |
| 21:23:21 | mriedem | slide 26 | |
| 21:25:22 | artom | mriedem, much thanks | |
| 21:26:06 | gregwork | hmmn "Cannot update metadata of aggregate 11: Reason: One or more hosts already in availability zones [u'nova'] | |
| 21:29:56 | mriedem | i've noticed that artom has the same disappearing pattern as bauzas | |
| 21:30:03 | mriedem | t0: complain! | |
| 21:30:06 | mriedem | t1: here you can do this | |
| 21:30:12 | mriedem | t2: thanks! but i've got to run | |
| 21:31:54 | gregwork | alright nm i think i got it .. | |
| 21:32:17 | mriedem | gregwork: hosts can be in M aggregates but 1 AZ | |
| 21:32:43 | gregwork | yeah i had created an aggregate and already tagged it as nova | |
| 21:32:47 | gregwork | for the az | |
| 21:32:59 | gregwork | so adding an additional az was failing | |
| 21:33:32 | gregwork | now to figure out what the scheduler_hint is to specify the az in OS::Nova::Server | |
| 21:33:40 | gregwork | i dont think its group: | |
| 21:34:19 | mriedem | there is a big fat warning in the docs to not ever create an az literally called "nova" | |
| 21:34:33 | mriedem | b/c that's the default schedule zone for services, | |
| 21:34:50 | mriedem | so if you create an az that users boot into called nova they can be stuck and not migrate their servers (potentially) | |
| 21:35:04 | mriedem | we should probably just block that in the api, but no one has cared enough to yet | |
| 21:36:32 | gregwork | so apparently availability_zone is a property in os::nova::server and not a scheduler hint map | |
| 21:37:40 | mriedem | correct | |
| 21:37:45 | mriedem | but let's not bring heat into this please... | |
| 21:38:05 | gregwork | well im trying to understand how it works in nova so i can figure out how to solve this in heat | |
| 21:38:47 | mriedem | ok https://docs.openstack.org/api-ref/compute/?expanded=create-server-detail#create-server | |
| 21:39:45 | gregwork | os:scheduler_hints.stuff section is very very useful | |
| 21:39:46 | gregwork | thanks | |
| 21:40:37 | mriedem | yup - thank takashin for doing the thankless work of documenting a lot of that stuff | |
| 21:40:48 | mriedem | organizing it, etc | |
| 21:53:29 | efried | sean-k-mooney: what we were talking about earlier, I guess the first step to at least acknowledge the problem would be to beef up the help for https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.host | |
| 21:53:38 | efried | it already has a bullet list of "things this is used for" | |
| 21:54:37 | efried | Adding "external services looking up the compute node resource provider; due to bug #XXXXX you will break your world if you set this when using the libvirt driver, so don't." kind of thing. | |
| 22:05:22 | efried | sean-k-mooney: Potential remedy (though it would be kind of a long road) would be to expose the compute node UUID through that hypervisors API. | |
| 22:05:42 | efried | then we instruct consumers (neutron, cyborg) to use that rather than CONF.host to discover the compute node RP. | |
| 22:05:50 | efried | um | |
| 22:05:59 | efried | for libvirt | |
| 22:06:10 | efried | This whole thing makes efried :( | |
| 22:08:37 | efried | mriedem: you may have missed this earlier, but it turns out we broke CONF.host for libvirt. | |
| 22:10:31 | efried | Because external services (neutron (problem now), cyborg (problem soon)) assume CONF.host is the name of the compute node RP | |
| 22:10:54 | efried | ...which they need to know in order to hang their nested providers (neutron: bw; cyborg: accelerator) off of | |
| 22:11:56 | efried | but the compute node RP is actually hypervisor_hostname, which for libvirt is the `hostname()` of the system. | |
| 22:12:24 | efried | which is the default for CONF.host, so you're fine... unless you actually *set* that guy. | |
| 22:12:29 | efried | dansmith: ^ | |
| 22:13:52 | mriedem | "we broke CONF.host"? | |
| 22:13:58 | mriedem | who is we and when? | |
| 22:14:18 | mriedem | if we = jay and when is ocata then... | |
| 22:14:30 | artom | mriedem, all part of my cunning plan | |
| 22:15:14 | efried | okay, "If you set set CONF.host to a non-default value, and you're using libvirt, bandwidth (and accelerator, and other future external-service-created nested) providers are broken" | |
| 22:15:51 | mriedem | oopsy daisy | |
| 22:16:28 | efried | [1] https://docs.openstack.org/api-ref/compute/?expanded=list-hypervisors-detail#id306 | |
| 22:16:28 | efried | sean-k-mooney: o hey, it looks like you were maybe getting the hypervisor ID as a short ID because you were using the default microversion. Per [1] it'll come back as a UUID starting at 2.53. | |
| 22:16:38 | efried | which phew gives us a path forward without further changes. | |
| 22:16:46 | efried | I think | |
| 22:17:12 | efried | Having to hit the /os-hypervisors API is heavier than just looking at CONF.host, but that UUID is the UUID of the provider you want. | |
| 22:17:19 | mriedem | yup | |
| 22:18:35 | efried | hm, but mriedem, how do I ask /os-hypervisors for the entry for my current node? | |