| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-01-14 | |||
| 21:35:22 | lyarwood | ade_lee: as horrid as that is it's expected the first time we connect to an empty encrypted volume | |
| 21:35:43 | ade_lee | lyarwood, ok :) | |
| 21:35:44 | lyarwood | ade_lee: as c-vol hasn't had to write and thus encrypt any data to the volume | |
| 21:35:45 | mriedem | efried: should | |
| 21:35:49 | mriedem | efried: are you on master? | |
| 21:35:56 | mriedem | and is the api configured to talk to placement? | |
| 21:36:05 | lyarwood | ade_lee: so n-cpu comes along and ensures the LUKs / crypt headers are written to the volume before we use it | |
| 21:36:18 | efried | mriedem: I found the code. It looks like for ironic deployments, it'll only delete the "first" compute node (whatever that means in practice). | |
| 21:36:46 | efried | mriedem: cf https://github.com/openstack/nova/blob/da98f4ba4554139b3901103aa0d26876b11e1d9a/nova/api/openstack/compute/services.py#L244-L247 and https://github.com/openstack/nova/blob/da98f4ba4554139b3901103aa0d26876b11e1d9a/nova/objects/service.py#L308-L311 | |
| 21:37:35 | mriedem | efried: sure enough | |
| 21:37:39 | mriedem | bugify it | |
| 21:38:40 | efried | mriedem: Well, a) I can't prove it (not having an ironic setup); but b) as jaypipes points out here, we might actually not want to do that: https://review.openstack.org/#/c/615677/17//COMMIT_MSG@30 | |
| 21:39:12 | efried | but I can open a bug anyway :) | |
| 21:40:26 | mriedem | efried: well, the "has instances on them" is caught here https://github.com/openstack/nova/blob/da98f4ba4554139b3901103aa0d26876b11e1d9a/nova/api/openstack/compute/services.py#L232 | |
| 21:41:34 | efried | mriedem: Yeah, I didn't see that pre-check, but knew placement would bounce the delete in any case if we got that far. | |
| 21:41:57 | mriedem | yeah that's also a relatively recent validation | |
| 21:42:02 | efried | mriedem: But do we really want to delete "potentially thousands" of compute node records from placement, assuming they're non-spawned? | |
| 21:42:42 | mriedem | ironic is special since we use the ironic node uuid to set the compute node uuid | |
| 21:42:45 | mriedem | so we can actually link those | |
| 21:42:48 | mriedem | which is also relatively recent... | |
| 21:42:50 | mriedem | rocky or queens | |
| 21:43:17 | mriedem | otherwise if you delete the service record and the compute_nodes records, the providers in placement would be orphaned if you restart the compute service which creates new compute nodes records with new uuids | |
| 21:43:34 | efried | mriedem: I saw code elsewhere to delete those orphans. | |
| 21:43:36 | mriedem | or just simply forget to stop the nova-compute service... | |
| 21:43:43 | efried | seems like a bass-ackwards way off doing it. | |
| 21:43:53 | efried | s/ff/f/ | |
| 21:44:12 | mriedem | says the guy that +2d the change https://review.openstack.org/#/c/554920/ | |
| 21:46:15 | mriedem | so let's say we delete a compute service hosting no instances and we delete 1000 ironic compute nodes records and related resource providers, | |
| 21:46:42 | mriedem | the next time you restart that compute service (or update_available_resources runs), nova-compute will just create 1000 new compute nodes records with uuids that match the ironic nodes that service is hosting, | |
| 21:46:50 | mriedem | and create 1000 new resource providers with those same uuids | |
| 21:47:19 | mriedem | if we don't delete the resource providers, the compute_nodes records are gone either way i think | |
| 21:47:24 | mriedem | when the services table record is deleted | |
| 21:48:02 | mriedem | because of this https://github.com/openstack/nova/blob/da98f4ba4554139b3901103aa0d26876b11e1d9a/nova/db/sqlalchemy/api.py#L406 | |
| 21:48:21 | efried | If I'm following, you're saying that deleting the service is going to cause deletion-and-recreation of all the ironic nodes even today? Just with different-than-optimal timing? | |
| 21:48:45 | mriedem | now if for some reason that service is only hosting 800 of those ironic nodes, n-cpu will create 800 compute_nodes records and you'll have orphaned 200 resource providers in placement | |
| 21:49:03 | mriedem | it will delete all of the compute_nodes table records for that service, yes | |
| 21:49:09 | mriedem | and always did | |
| 21:49:35 | mriedem | if some other compute service starts managing that other 200 nodes, then cool - their providers are already in placement | |
| 21:49:46 | mriedem | but if not, you're reporting things to the scheduler that might not be managed anywhere | |
| 21:50:25 | mriedem | the scheduler should filter those out but you could get NoValidHost if you're like CERN and have a real low config for how many placement allocation candidates you want to get back | |
| 21:51:37 | efried | I will try to express this in the bug report. | |
| 21:52:03 | mriedem | i could be wrong of course, i don't have a multinode ironic deployment to play with | |
| 21:52:14 | mriedem | but we could easily simulate this in functional tests | |
| 21:52:43 | mriedem | i guess the bug is, in the case of ironic, we should either delete 0 or all providers in placement for the related compute nodes | |
| 21:53:26 | mriedem | sure is fun needing nova to mirror everything to placement isn't it o-) | |
| 21:58:37 | efried | mriedem: https://bugs.launchpad.net/nova/+bug/1811726 -- please see if I expressed this correctly, kthx | |
| 21:58:38 | openstack | Launchpad bug 1811726 in OpenStack Compute (nova) "Deleting compute service only deletes "first" ironic node from placement" [Undecided,New] | |
| 22:03:22 | mriedem | commented | |
| 22:03:34 | mriedem | but i don't think that cleanup orphan code you pointed out is going to clean up orphans | |
| 22:04:17 | mriedem | that code in update_available_resource is for nodes that the driver no longer reports (user deleted the ironic node in the ironic api) but nova was still tracking it | |
| 22:07:44 | openstackgerrit | Jack Ding proposed openstack/nova master: [WIP] Flavor extra spec and image properties validation https://review.openstack.org/620706 | |
| 22:52:58 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Share snapshot image membership with instance owner https://review.openstack.org/630769 | |
| 22:55:04 | jaypipes | mriedem: unless I'm badly mistaken, if you attempt to delete a resource provider and the provider has allocations against it, that will fail. | |
| 22:55:21 | efried | mriedem: When you get a chance, would you mind having a quick look at https://review.openstack.org/#/c/615677/17/nova/compute/resource_tracker.py@824 ? | |
| 22:55:21 | efried | Perhaps you can explain a circumstance under which is_new_compute_node==True but the report client's provider tree cache has an entry for the "new" node. | |
| 22:55:27 | mriedem | jaypipes: yes i believe that's a 409 | |
| 22:55:48 | edleafe | git ls | |
| 22:55:52 | jaypipes | mriedem: and we are catching that 409 in the DELETE /os-services/{id} API? | |
| 22:55:52 | edleafe | doh! | |
| 22:56:09 | efried | cause I'm sure I put that code there for a reason. In fact, I want to say it was the/a main thing in this patch. | |
| 22:56:14 | mriedem | jaypipes: we don't attempt to delete services/computenodes/resoruce providers if the service host has instances on it | |
| 22:56:48 | mriedem | jaypipes: https://github.com/openstack/nova/blob/da98f4ba4554139b3901103aa0d26876b11e1d9a/nova/api/openstack/compute/services.py#L232 | |
| 22:57:28 | jaypipes | mriedem: ack. | |
| 22:57:29 | mriedem | we don't explicitly catch that 409 though no, | |
| 22:57:34 | mriedem | it'd be a 500 if we raced there | |
| 22:57:37 | jaypipes | right. | |
| 22:57:58 | mriedem | but we also wouldn't delete the service record or compute nodes table entries, so you could retry | |
| 22:58:28 | jaypipes | mriedem: which I'm fine with, really, since it's a data corruption error if that happened IMHO (no InstanceList.get_by_service() results but there ARE allocations is a corruption in my book) | |
| 23:00:52 | mriedem | efried: replied inline | |
| 23:03:46 | efried | thanks | |
| #openstack-nova - 2019-01-15 | |||
| 01:48:16 | openstackgerrit | Dakshina Ilangovan proposed openstack/nova-specs master: Nova changes to support Local Resource Management that uses Resource Management Daemon https://review.openstack.org/630817 | |
| 02:20:15 | openstackgerrit | Merged openstack/nova master: Allow run metadata api per cell https://review.openstack.org/624612 | |
| 03:04:31 | openstackgerrit | Merged openstack/nova master: Move some linux network helpers to use privsep. https://review.openstack.org/621398 | |
| 06:42:58 | openstackgerrit | melanie witt proposed openstack/nova master: Propagate exception message from _prep_block_device https://review.openstack.org/624832 | |
| 06:42:59 | openstackgerrit | melanie witt proposed openstack/nova master: Add method to generate device names universally https://review.openstack.org/573066 | |
| 06:42:59 | openstackgerrit | melanie witt proposed openstack/nova master: Add configuration of maximum disk devices to attach https://review.openstack.org/616777 | |
| 07:39:03 | openstackgerrit | Shilpa Devharakar proposed openstack/nova master: Ignore root_gb for BFV in simple tenant usage API https://review.openstack.org/612626 | |
| 08:42:11 | openstackgerrit | Jan Gutter proposed openstack/nova master: Convert vrouter legacy plugging to os-vif https://review.openstack.org/571325 | |
| 09:01:00 | openstackgerrit | Adrian Chiris proposed openstack/nova master: SR-IOV Live migration indirect port support https://review.openstack.org/620115 | |
| 09:30:51 | brinzhang | +lib/placement:start_placement_api:179 die 179 'placement-api did not start' | |
| 09:31:35 | brinzhang | When I execute ./stack.sh report this error, anyone can give some help? | |
| 09:32:01 | brinzhang | this is the local.conf file on my devstack http://paste.openstack.org/show/742619/ | |
| 10:45:36 | openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Per-instance serial number https://review.openstack.org/619953 | |
| 10:45:43 | Kevin_Zheng | the unstack script did not work correctly | |
| 10:45:48 | Kevin_Zheng | brinzhang | |
| 10:46:15 | Kevin_Zheng | delete all your devstack related files from site-enabled and site-availables | |
| 10:46:39 | Kevin_Zheng | after unstack | |
| 10:46:46 | Kevin_Zheng | then stack again | |
| 10:52:30 | brinzhang | Kevin_Zheng: I was unstack more than once, it always has this issue :( | |
| 10:52:51 | Kevin_Zheng | yeah, you should clean up those files manually | |
| 10:52:57 | Kevin_Zheng | before you call stack again | |
| 10:53:09 | Kevin_Zheng | files in site-enabled and site-avialable | |
| 10:53:35 | brinzhang | I don't particularly understand which files are included? | |
| 10:54:10 | brinzhang | For example the devstack folder? | |
| 10:54:13 | Kevin_Zheng | and systemd files | |
| 10:54:29 | Kevin_Zheng | check your apache2/site-enabled | |
| 11:26:28 | kashyap | time tox -e docs | |
| 11:26:53 | kashyap | Oeps, false window | |
| 11:33:21 | kashyap | stephenfin: Before I waste too much time on this, the `below <`Prerequisites`>__ hyperlink to a section in rST doesn't seem to fly :-( | |
| 11:33:27 | kashyap | The result: http://logs.openstack.org/27/629627/5/check/openstack-tox-docs/bde0494/html/admin/Prerequisites | |