| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-03 | |||
| 09:01:12 | kashyap | (At least I see an old blueprint for it - https://blueprints.launchpad.net/nova/+spec/instance-mem-and-cpu-hot-plugging) | |
| 09:15:48 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Don't create instance_extra entry for deleted instance https://review.opendev.org/412771 | |
| 09:29:53 | openstackgerrit | Merged openstack/nova master: Add functional regression test for build part of bug 1781286 https://review.opendev.org/685998 | |
| 09:29:53 | openstack | bug 1781286 in OpenStack Compute (nova) "CantStartEngineError in cell conductor during reschedule - get_host_availability_zone up-call" [Medium,In progress] https://launchpad.net/bugs/1781286 - Assigned to Matt Riedemann (mriedem) | |
| 09:35:59 | openstackgerrit | Merged openstack/nova master: Add functional regression test for migrate part of bug 1781286 https://review.opendev.org/686017 | |
| 09:35:59 | openstack | bug 1781286 in OpenStack Compute (nova) "CantStartEngineError in cell conductor during reschedule - get_host_availability_zone up-call" [Medium,In progress] https://launchpad.net/bugs/1781286 - Assigned to Matt Riedemann (mriedem) | |
| 09:39:21 | openstackgerrit | Merged openstack/python-novaclient stable/stein: Stop silently ignoring invalid 'nova boot --hint' options https://review.opendev.org/685981 | |
| 10:03:31 | bauzas | gibi: <3 | |
| 10:03:52 | bauzas | gibi: you're totally right, I forgot to update my cell check when looking at the RPs | |
| 10:03:58 | bauzas | gibi: for the audit command | |
| 10:04:08 | bauzas | gibi: so that's why https://paste.ubuntu.com/p/J4S3Y3r4fh/ | |
| 10:38:53 | sean-k-mooney | kashyap: both memory and cpu hotplug have been reject in the past both as an api do explcitly added/remove each resource to/form an instnace and in the form of live reszie | |
| 10:39:11 | sean-k-mooney | but there has been requests for it | |
| 10:39:36 | sean-k-mooney | arbiatry hotplug was rejected as it basical breaks flavors | |
| 10:40:06 | sean-k-mooney | and how flavors are used for billing | |
| 10:41:39 | sean-k-mooney | i also suspec based on comment made by dansmith in the past that while we try to ues the embeded flavor everywhere that it maters there may be edgecase where we dont so adding hotplug support that is not flavor based could have implciations for move operations | |
| 10:41:54 | sean-k-mooney | the flavor based appach was reject for a diffrent reason | |
| 10:42:05 | sean-k-mooney | resize down if very hard | |
| 10:42:52 | sean-k-mooney | you technially can offline cpus or dimms in the linux kernel then hot unplug them but that requries guest coperation | |
| 10:43:46 | sean-k-mooney | hot adding of cpus and memory generaly requires you to forward declare a set of free dimm slots or cpu sockets | |
| 10:44:24 | sean-k-mooney | and like the pcie ports i suspect there is qemu overhead required to support that so if you were to do that by default you would waste ram | |
| 10:45:42 | sean-k-mooney | so ya im not sure it fits with the cloud model where i think the spot isntance model is much more correct. e.g. burst horizontally not vertically | |
| 10:50:03 | kashyap | sean-k-mooney: Hi, reading the scroll now | |
| 10:50:39 | kashyap | sean-k-mooney: (Yeah, I recall the perils of the resize down) | |
| 10:51:01 | kashyap | sean-k-mooney: Thanks for the input. | |
| 10:58:11 | gibi | bauzas: I'm happy that I could help | |
| 11:04:52 | bauzas | gibi: around ? I wonder how to fix the cells issue | |
| 11:05:19 | gibi | bauzas: partially, I'm on a full day internal workshop | |
| 11:05:46 | gibi | bauzas: but I think there is good example in the heal_allocations code about cell handling | |
| 11:08:42 | gibi | bauzas: https://github.com/openstack/nova/blob/caffac7bb06cf7606f7447d0f2ee37ba50dd0b48/nova/cmd/manage.py#L2448 | |
| 11:09:04 | bauzas | gibi: the problem is that I need to find the cell for a compute node UUID | |
| 11:09:24 | bauzas | gibi: the problem is that I don't know yet the hostname | |
| 11:09:53 | bauzas | anyway, /me needs to go the school for his daughters | |
| 11:09:57 | bauzas | back | |
| 11:10:07 | gibi | bauzas: talk to you later | |
| 11:54:16 | stephenfin | gibi: Replied on https://review.opendev.org/#/c/684334/ | |
| 12:07:30 | gibi | stephenfin: thanks for the explanation. | |
| 13:29:03 | mriedem | stephenfin: ping me when you get https://review.opendev.org/667133 updated and i'll take a look, don't want to lose track of that since it's huge | |
| 13:33:12 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Skip test_parallel_evacuate_with_server_group until fixed https://review.opendev.org/686402 | |
| 13:39:51 | gibi | bauzas: I think I see your compute node uuid -> cell mapping issue in https://review.opendev.org/#/c/670112 | |
| 13:40:20 | gibi | bauzas: we cannot map a compute node back to a cell easily. | |
| 13:41:04 | gibi | bauzas: if we assume that the compute node RP's name is the hostname of the compute host, then we can map to cells. But the we cannot support virt drivers using more than one node per host | |
| 13:43:05 | mriedem | you can map the compute node to a host mapping which is in a cell | |
| 13:43:28 | bauzas | gibi: the problem is that we only know the RP UUID | |
| 13:43:57 | mriedem | rp uuid == compute node uuid == get the compute node == get the host mapping == get the cell | |
| 13:44:55 | bauzas | mriedem: sure, but in the nova-manage, I can't just ask https://review.opendev.org/#/c/670112/5/nova/cmd/manage.py@2669 directly, right? | |
| 13:46:35 | mriedem | oh i see, chicken and egg | |
| 13:46:53 | mriedem | you can scatter/gather cells looking for hte compute node by uuid | |
| 13:46:54 | dansmith | scatter/gather? | |
| 13:46:55 | dansmith | yeah | |
| 13:47:01 | mriedem | that's what the api does i think | |
| 13:47:47 | gibi | yeah you have to search for the root RP UUID as a compute node id in every cell | |
| 13:47:47 | mriedem | well i guess it doesn't https://github.com/openstack/nova/blob/master/nova/compute/api.py#L5261 but it could | |
| 13:48:07 | bauzas | I can scatter/gather for sure | |
| 13:48:14 | bauzas | but it will take time | |
| 13:48:28 | bauzas | I mean, it's a performance issue | |
| 13:48:42 | dansmith | eh? | |
| 13:48:45 | gibi | bauzas: would be better scatter / gather once to get all the compute node objects from all cells? | |
| 13:48:51 | bauzas | when calling the audit command, then we would have to look at all the cells to find the one having the CN UUID | |
| 13:48:54 | mriedem | scatter/gather non-cell0 cells since you shouldn't have compute nodes in cell0, | |
| 13:49:02 | mriedem | and for most deployments that's only going to be 1 cell to query | |
| 13:49:22 | dansmith | bauzas: you need to scatter/gather a get_by_uuid() right? | |
| 13:49:23 | dansmith | that should be uber fast | |
| 13:49:44 | bauzas | I did this for PS1 https://review.opendev.org/#/c/670112/1/nova/cmd/manage.py@2309 | |
| 13:50:08 | bauzas | dansmith: okay, lemme try to do it | |
| 13:50:30 | bauzas | and then we could for example ask CERN to verify the performance | |
| 13:50:37 | bauzas | or mnaser :) | |
| 13:51:11 | mriedem | my comment wasn't about using scatter/gather, it was about starting with compute nodes and working backward to resource providers | |
| 13:51:15 | bauzas | mriedem: gibi: dansmith: thanks for the help | |
| 13:51:29 | gibi | bauzas: np | |
| 13:51:50 | bauzas | mriedem: yeah, no worries, I already understood this, I just tried to explain that I already used scatter_gather so I already know it :) | |
| 13:51:55 | mriedem | being an audit / heal style command, i don't think blazying fast performance is the primary concern | |
| 13:52:04 | mriedem | *blazing even | |
| 13:52:46 | mriedem | do you allow passing in a single uuid to audit? if not you probably should so you don't have to audit all 20K nodes just to heal one of them that is a problem. | |
| 13:53:29 | mnaser | yeah i think we should loop over RPs instead of CNs | |
| 13:53:30 | gibi | mriedem: yes, the command has a --resource_provider flag | |
| 13:53:35 | mriedem | 20K compute services + x number of ironic VMs... | |
| 13:53:42 | mriedem | *ironic BMs / instances | |
| 13:53:48 | dansmith | mriedem: you know, this has come up before.. we really should move HostMapping to use (or at least have) the compute uuid in there | |
| 13:54:04 | mriedem | dansmith: the cardinality isn't correct for that | |
| 13:54:15 | mriedem | for ironic you have 1 host mapping to x nodes | |
| 13:54:30 | mriedem | gibi: ok good - except let's drop the underscore in the option name, i hate that | |
| 13:54:33 | mriedem | e.g. --max_rows | |
| 13:54:42 | mriedem | options should use dashes | |
| 13:54:51 | dansmith | mriedem: right, which also causes us trouble and why we need thatother mechanism of hostmapping discovery... | |
| 13:54:53 | mriedem | i think stephenfin and i might agree on one thing and that's probably it | |
| 13:55:22 | dansmith | mriedem: I the point is we kinda need nodes and hosts in there, by uuid or name depending :D | |
| 13:55:39 | mriedem | dansmith: so for an ironic compute service, the HostMapping just wouldn't have uuid set? | |
| 13:55:47 | sean-k-mooney | well we can still have a 1:n mapping in the 1:1 case it jsut a specific value of n | |
| 13:56:04 | sean-k-mooney | so we could model it in the db using a mapping table | |
| 13:56:11 | mriedem | .... | |
| 13:56:12 | sean-k-mooney | and in the object as a list right? | |
| 13:56:39 | sean-k-mooney | which might be what we alreday do | |
| 13:56:45 | sean-k-mooney | i was not following the conversation | |
| 13:57:01 | mriedem | from a coding api perspective, if i'm relying on the host mapping uuid to get/match the compute node uuid, then how do i know if that host is managing one or many nodes | |
| 13:57:09 | mriedem | also, | |
| 13:57:26 | dansmith | mriedem: maybe I dunno | |
| 13:57:30 | mriedem | the host mapping name doesn't / shouldn't change (the hostname on the host would have to change which is bad), | |
| 13:57:43 | mriedem | but i can guarantee you that the compute node uuid generated by the compute service will change | |
| 13:57:55 | mriedem | and that would be an up-call or new discovery from the controller to sync those up | |