| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-10-03 | |||
| 10:50:39 | kashyap | sean-k-mooney: (Yeah, I recall the perils of the resize down) | |
| 10:51:01 | kashyap | sean-k-mooney: Thanks for the input. | |
| 10:58:11 | gibi | bauzas: I'm happy that I could help | |
| 11:04:52 | bauzas | gibi: around ? I wonder how to fix the cells issue | |
| 11:05:19 | gibi | bauzas: partially, I'm on a full day internal workshop | |
| 11:05:46 | gibi | bauzas: but I think there is good example in the heal_allocations code about cell handling | |
| 11:08:42 | gibi | bauzas: https://github.com/openstack/nova/blob/caffac7bb06cf7606f7447d0f2ee37ba50dd0b48/nova/cmd/manage.py#L2448 | |
| 11:09:04 | bauzas | gibi: the problem is that I need to find the cell for a compute node UUID | |
| 11:09:24 | bauzas | gibi: the problem is that I don't know yet the hostname | |
| 11:09:53 | bauzas | anyway, /me needs to go the school for his daughters | |
| 11:09:57 | bauzas | back | |
| 11:10:07 | gibi | bauzas: talk to you later | |
| 11:54:16 | stephenfin | gibi: Replied on https://review.opendev.org/#/c/684334/ | |
| 12:07:30 | gibi | stephenfin: thanks for the explanation. | |
| 13:29:03 | mriedem | stephenfin: ping me when you get https://review.opendev.org/667133 updated and i'll take a look, don't want to lose track of that since it's huge | |
| 13:33:12 | openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Skip test_parallel_evacuate_with_server_group until fixed https://review.opendev.org/686402 | |
| 13:39:51 | gibi | bauzas: I think I see your compute node uuid -> cell mapping issue in https://review.opendev.org/#/c/670112 | |
| 13:40:20 | gibi | bauzas: we cannot map a compute node back to a cell easily. | |
| 13:41:04 | gibi | bauzas: if we assume that the compute node RP's name is the hostname of the compute host, then we can map to cells. But the we cannot support virt drivers using more than one node per host | |
| 13:43:05 | mriedem | you can map the compute node to a host mapping which is in a cell | |
| 13:43:28 | bauzas | gibi: the problem is that we only know the RP UUID | |
| 13:43:57 | mriedem | rp uuid == compute node uuid == get the compute node == get the host mapping == get the cell | |
| 13:44:55 | bauzas | mriedem: sure, but in the nova-manage, I can't just ask https://review.opendev.org/#/c/670112/5/nova/cmd/manage.py@2669 directly, right? | |
| 13:46:35 | mriedem | oh i see, chicken and egg | |
| 13:46:53 | mriedem | you can scatter/gather cells looking for hte compute node by uuid | |
| 13:46:54 | dansmith | scatter/gather? | |
| 13:46:55 | dansmith | yeah | |
| 13:47:01 | mriedem | that's what the api does i think | |
| 13:47:47 | mriedem | well i guess it doesn't https://github.com/openstack/nova/blob/master/nova/compute/api.py#L5261 but it could | |
| 13:47:47 | gibi | yeah you have to search for the root RP UUID as a compute node id in every cell | |
| 13:48:07 | bauzas | I can scatter/gather for sure | |
| 13:48:14 | bauzas | but it will take time | |
| 13:48:28 | bauzas | I mean, it's a performance issue | |
| 13:48:42 | dansmith | eh? | |
| 13:48:45 | gibi | bauzas: would be better scatter / gather once to get all the compute node objects from all cells? | |
| 13:48:51 | bauzas | when calling the audit command, then we would have to look at all the cells to find the one having the CN UUID | |
| 13:48:54 | mriedem | scatter/gather non-cell0 cells since you shouldn't have compute nodes in cell0, | |
| 13:49:02 | mriedem | and for most deployments that's only going to be 1 cell to query | |
| 13:49:22 | dansmith | bauzas: you need to scatter/gather a get_by_uuid() right? | |
| 13:49:23 | dansmith | that should be uber fast | |
| 13:49:44 | bauzas | I did this for PS1 https://review.opendev.org/#/c/670112/1/nova/cmd/manage.py@2309 | |
| 13:50:08 | bauzas | dansmith: okay, lemme try to do it | |
| 13:50:30 | bauzas | and then we could for example ask CERN to verify the performance | |
| 13:50:37 | bauzas | or mnaser :) | |
| 13:51:11 | mriedem | my comment wasn't about using scatter/gather, it was about starting with compute nodes and working backward to resource providers | |
| 13:51:15 | bauzas | mriedem: gibi: dansmith: thanks for the help | |
| 13:51:29 | gibi | bauzas: np | |
| 13:51:50 | bauzas | mriedem: yeah, no worries, I already understood this, I just tried to explain that I already used scatter_gather so I already know it :) | |
| 13:51:55 | mriedem | being an audit / heal style command, i don't think blazying fast performance is the primary concern | |
| 13:52:04 | mriedem | *blazing even | |
| 13:52:46 | mriedem | do you allow passing in a single uuid to audit? if not you probably should so you don't have to audit all 20K nodes just to heal one of them that is a problem. | |
| 13:53:29 | mnaser | yeah i think we should loop over RPs instead of CNs | |
| 13:53:30 | gibi | mriedem: yes, the command has a --resource_provider flag | |
| 13:53:35 | mriedem | 20K compute services + x number of ironic VMs... | |
| 13:53:42 | mriedem | *ironic BMs / instances | |
| 13:53:48 | dansmith | mriedem: you know, this has come up before.. we really should move HostMapping to use (or at least have) the compute uuid in there | |
| 13:54:04 | mriedem | dansmith: the cardinality isn't correct for that | |
| 13:54:15 | mriedem | for ironic you have 1 host mapping to x nodes | |
| 13:54:30 | mriedem | gibi: ok good - except let's drop the underscore in the option name, i hate that | |
| 13:54:33 | mriedem | e.g. --max_rows | |
| 13:54:42 | mriedem | options should use dashes | |
| 13:54:51 | dansmith | mriedem: right, which also causes us trouble and why we need thatother mechanism of hostmapping discovery... | |
| 13:54:53 | mriedem | i think stephenfin and i might agree on one thing and that's probably it | |
| 13:55:22 | dansmith | mriedem: I the point is we kinda need nodes and hosts in there, by uuid or name depending :D | |
| 13:55:39 | mriedem | dansmith: so for an ironic compute service, the HostMapping just wouldn't have uuid set? | |
| 13:55:47 | sean-k-mooney | well we can still have a 1:n mapping in the 1:1 case it jsut a specific value of n | |
| 13:56:04 | sean-k-mooney | so we could model it in the db using a mapping table | |
| 13:56:11 | mriedem | .... | |
| 13:56:12 | sean-k-mooney | and in the object as a list right? | |
| 13:56:39 | sean-k-mooney | which might be what we alreday do | |
| 13:56:45 | sean-k-mooney | i was not following the conversation | |
| 13:57:01 | mriedem | from a coding api perspective, if i'm relying on the host mapping uuid to get/match the compute node uuid, then how do i know if that host is managing one or many nodes | |
| 13:57:09 | mriedem | also, | |
| 13:57:26 | dansmith | mriedem: maybe I dunno | |
| 13:57:30 | mriedem | the host mapping name doesn't / shouldn't change (the hostname on the host would have to change which is bad), | |
| 13:57:43 | mriedem | but i can guarantee you that the compute node uuid generated by the compute service will change | |
| 13:57:55 | mriedem | and that would be an up-call or new discovery from the controller to sync those up | |
| 13:58:16 | sean-k-mooney | its unfortunate that we are not usign a uuid5 based on teh host name for the compute node uuid | |
| 13:58:34 | mriedem | all you have to is (1) start the service, (2) discover it to create the host mapping, (3) delete the compute service (4) restart it and bam you have a new compute node with a new uuid | |
| 13:59:00 | mriedem | sean-k-mooney: that's not a bad idea...in the non-ironic case, | |
| 13:59:16 | mriedem | for ironic, since rocky we make the compute node == resource provider == ironic node all use the ironic node uuid | |
| 13:59:22 | mriedem | that has caused some problems... | |
| 13:59:32 | bauzas | mriedem: yup, either the audit command loops over RPs, or asks for a specific RP UUID | |
| 13:59:51 | sean-k-mooney | the ironic node id often matche the chassis uuid backed into the hardware | |
| 14:00:13 | mriedem | meeting time | |
| 14:00:41 | dansmith | mriedem: sorry I lost track of that convo because I was having two | |
| 14:01:06 | mriedem | this is the problem i'm talking about with using static uuids https://review.opendev.org/#/q/Iafba419fe86446ffe636721f523fb619f8f787b3 | |
| 14:01:12 | mriedem | sean-k-mooney: ^ | |
| 14:01:53 | mriedem | somewhat related https://review.opendev.org/#/c/684849/ | |
| 14:02:11 | sean-k-mooney | oh ya i vaguely rememebr that it was related to the phyical node reblanice to a different compute service | |
| 14:02:38 | mriedem | the first is related to maintenance mode on a node, the latter is rebalance | |
| 14:02:56 | sean-k-mooney | ack | |
| 14:03:21 | stephenfin | mriedem: <3 | |
| 14:03:47 | sean-k-mooney | i need to grab somthing to eat before another meeting in an hour so ill join the nova meeting in link 20-30 mins | |
| 14:03:49 | sean-k-mooney | brb | |
| 14:04:10 | mordred | mriedem: in microversion 2.64, os-server-group changed from accepting policies=[] to policy="" ... what happens to additional policies created with lower microversion? | |
| 14:04:39 | mriedem | it was unpossible on lower microversions | |
| 14:04:46 | mriedem | shitty api design allowed a list but the backing db model didn't | |
| 14:04:59 | mriedem | something like that - the json schema only allowed a list with an entry of exactly 1 | |
| 14:05:05 | mordred | ah - cool | |