| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-12-07 | |||
| 17:38:13 | sean-k-mooney | bauzas: i think ovh and i woudl both be happy if we fully remove the shadow tables | |
| 17:39:29 | bauzas | I still think we're ending into a weird state | |
| 17:39:53 | bauzas | we said in the past "this is expected behaviour, as actions table is here for recording user actions" | |
| 17:40:23 | bauzas | now we're about to publicly express that the actions API is just usable by default for non-deleted instances | |
| 17:40:33 | bauzas | (if we change the default) | |
| 17:40:40 | sean-k-mooney | bauzas: this is what they use https://github.com/ovh/osarchiver/ | |
| 17:41:01 | bauzas | if we don't change the default, we just provide a new microversion that won't change the behaviour by default, which is also weird | |
| 17:41:23 | sean-k-mooney | we often add microverion that dont change default behavior | |
| 17:41:28 | bauzas | and why ? because we consider the current behaviour is enough good to not touch it | |
| 17:41:43 | bauzas | sean-k-mooney: we used microversions for "signaling", I know | |
| 17:41:51 | sean-k-mooney | no | |
| 17:42:05 | sean-k-mooney | the feature this would be interocudeing is the ablity to filter on the soft delete status | |
| 17:42:17 | sean-k-mooney | that does not require use to change the defualt behavior | |
| 17:42:28 | sean-k-mooney | so its not jsut signaling | |
| 17:44:19 | gmann | yeah, adding new filtering which is asked in current specless BP | |
| 17:44:52 | gmann | currently there is no way to filter soft deleted instance actions | |
| 17:45:39 | gmann | with new microversion, we solve both use case 1. existing keep working 2. way to filter the soft deleted one | |
| 17:45:45 | sean-k-mooney | so i think we agree that the current behaiovr is not broken or incorrect, that we could start marking the records as soft deleted when the instance is soft delete but only if we maintain the currnt behavior of returning both soft and not soft deleted records and we woudl also want to allow filtering if we made this change | |
| 17:47:56 | sean-k-mooney | the filtering is an api change as we are adding a new query arg even if the default behvior does not change. | |
| 17:48:21 | pslestang | sean-k-mooney: +1 for your proposition | |
| 17:49:28 | sean-k-mooney | pslestang: by the way what happend with your upstreaming efforts http://lists.openstack.org/pipermail/openstack-discuss/2021-February/020383.html | |
| 17:49:49 | sean-k-mooney | that was the last mail i could find on that trhead was there a tc desicion made | |
| 17:50:59 | sean-k-mooney | i dont see it in https://opendev.org/openstack/osops/src/branch/master so i guess it was not added to osops | |
| 17:51:15 | sean-k-mooney | and i dont see a new repo for https://github.com/ovh/osarchiver/ | |
| 17:51:27 | sean-k-mooney | did ye decide to just keep it in github in the end | |
| 17:53:33 | sean-k-mooney | pslestang: i think the prefernce was option 3 "Move it under its own repository under opendev and propose it as a new official OpenStack project" but i dont think that happened | |
| 17:54:46 | sean-k-mooney | if that was to happen i coudl see a day where nova could discontinue our current shadown tables eventually and rely on OSArchiver after a deprecation period | |
| 17:54:50 | pslestang | sean-k-mooney:this is what I was looking for, exact option 3 prefered but nothing done | |
| 17:57:18 | pslestang | We prefer to put it under opendev than keeping it in github, we really thing that other operators could take benefit of this tool | |
| 17:57:48 | sean-k-mooney | yep and honestly i think its a better approch then we have today in nova | |
| 17:58:17 | sean-k-mooney | so if it was mature enouch and operators started to use it i think we coudl eventually depercante and remove nova archiveal supprot | |
| 17:58:45 | sean-k-mooney | and replace it with your external soltuion once we had time to update the installer tools and work with the differnt sake holders to adopt it | |
| 17:59:31 | sean-k-mooney | the fact it can archive the data to an entirly differnt db or a csv file can signifcalty help with db load on the main db | |
| 17:59:46 | sean-k-mooney | and help keep the main db small | |
| 17:59:49 | pslestang | We use it on a weekly basis (crontab) to clean our DB, and we do not have any issue (except with instance_actions_* ;-)) | |
| 18:00:54 | sean-k-mooney | oh i know it works well for you but for use to ever remove the native support in nova we woudl need all the installaiton tools or at least most of them to supprot your alternivie and we would need an upgrade stragy for exsing clouds | |
| 18:01:14 | sean-k-mooney | so its more maturing that process that would need work | |
| 18:01:40 | sean-k-mooney | cern were intersted in this too if i rememebr the mail thread | |
| 18:01:41 | pslestang | sure I understand | |
| 18:03:03 | pslestang | yes and I know also that ubisoft is using it | |
| 18:06:06 | bauzas | sean-k-mooney: distros would have to agree on it | |
| 18:06:16 | bauzas | this is shipping yet another dep | |
| 18:11:44 | sean-k-mooney | bauzas: it would be yes although we have talked about removing shadow tables in the past and not providing a replacement too | |
| 18:11:56 | sean-k-mooney | this would just be a way to provide a replacement | |
| 19:08:33 | opendevreview | mitya-eremeev-2 proposed openstack/nova master: Delete bogus attachments. https://review.opendev.org/c/openstack/nova/+/820935 | |
| 22:31:59 | opendevreview | Merged openstack/nova stable/wallaby: Add functional regression test for bug 1853009 https://review.opendev.org/c/openstack/nova/+/811805 | |
| #openstack-nova - 2021-12-08 | |||
| 05:06:03 | opendevreview | Merged openstack/nova master: Extend the reproducer for 1953359 and 1952915 https://review.opendev.org/c/openstack/nova/+/820859 | |
| 06:42:19 | opendevreview | mitya-eremeev-2 proposed openstack/nova master: Delete bogus attachments. https://review.opendev.org/c/openstack/nova/+/820935 | |
| 09:56:52 | opendevreview | Pierre Libeau proposed openstack/nova master: Nova resize don't extend disk in one specific case https://review.opendev.org/c/openstack/nova/+/820531 | |
| 10:45:03 | gibi | sean-k-mooney, slaweq: I looked at the gate bug https://bugs.launchpad.net/nova/+bug/1953478 and it seems to me that neutron sends the vif-plugged event at bind time instead of plug time in this case causing the timeout | |
| 10:45:47 | gibi | so far I thought that with ml2/ovs neutron always sends that event at plug time | |
| 10:47:24 | slaweq | gibi yes, afaik when You boot vm it will be sent when neutron will finish provisioning that port | |
| 10:47:41 | gibi | slaweq: it is an unshelve after shelve_offload | |
| 10:47:45 | slaweq | but during e.g. live migration it may be differently | |
| 10:47:55 | slaweq | I don't know about shelve and unshelve | |
| 10:47:56 | gibi | that is almost like a new boot | |
| 10:48:02 | gibi | but from a previous snapshot | |
| 10:48:25 | gibi | there is a new scheduling, a new port biding a new vif plug and spawn | |
| 10:48:52 | slaweq | so it should be sent when plug is completed | |
| 10:49:04 | slaweq | in such case | |
| 10:49:06 | gibi | yes, that is how I would expect it (and how nova expects it) | |
| 10:49:27 | slaweq | please move that bug to neutron then, I will check in our logs what happened there | |
| 10:49:31 | opendevreview | mitya-eremeev-2 proposed openstack/nova master: Delete bogus attachments. https://review.opendev.org/c/openstack/nova/+/820935 | |
| 10:49:43 | slaweq | and thx for checking that | |
| 10:49:45 | gibi | slaweq: ok, I summarized this in the bug so you have the logs with timestamps from nova | |
| 10:49:51 | slaweq | ++ | |
| 10:51:36 | gibi | I haven't looked at the other case in the same bug the tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_resize_server_revert | |
| 10:51:45 | gibi | but I can do that too | |
| 12:12:59 | sean-k-mooney | slaweq: unshelve should be the same as first boot | |
| 12:13:00 | lyarwood | does anyone know when/where during a spawn that we would expect the metadata service to have details on a given instance ready to serve up? | |
| 12:14:34 | lyarwood | I'm looking at a CI failure downstream where cloud-init gets Connection refused everytime it tries to pull from the api but I can't see errors in the actual service logs | |
| 12:19:12 | gibi | lyarwood: I have limited knowledge too, but the request is goes from the guest, goes to the neutron-metadata service via the guest's network and the neutron forwards the request to nova-metadata | |
| 12:23:13 | lyarwood | thanks I think I was missing the neutron part | |
| 12:24:12 | sean-k-mooney | ill get the code one sec | |
| 12:24:32 | sean-k-mooney | lyarwood: it will always be ready before we call libvirt | |
| 12:25:20 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/virt/netutils.py#L168 | |
| 12:25:28 | sean-k-mooney | that is where we generate the network metadata | |
| 12:26:15 | sean-k-mooney | which is called form the init of InstanceMetadata https://github.com/openstack/nova/blob/052cf963583ab7c6bbe4fcbf7bfe69f8f6733bdb/nova/api/metadata/base.py#L176 | |
| 12:27:49 | lyarwood | ACK thanks | |
| 12:28:14 | lyarwood | looks like an issue before that as the neutron-metadata-agent on the compute isn't even seeing the requests | |
| 12:28:29 | lyarwood | I guess with plugging but AFAICT from the compute logs that worked | |
| 12:29:53 | sean-k-mooney | the neutorn-metadata-agent is typeiclay not on the compute but on the contler | |
| 12:30:09 | sean-k-mooney | it runs in either the router or dhcp agent network namespaces | |
| 12:30:14 | sean-k-mooney | depening on your config | |
| 12:30:27 | sean-k-mooney | ovn works slightly differntly and i think it does it on each compute | |
| 12:30:43 | sean-k-mooney | but the agent there is jsut calling the nova metadta api | |
| 12:31:06 | sean-k-mooney | the api i belive generates the instnace mentadta directly itself and caches it | |
| 12:32:48 | sean-k-mooney | for libvirt we generate it here by the way as part of creating the config drive https://github.com/openstack/nova/blob/e537d90d6fc0977742f7126c3f8cfef6bf8b2a15/nova/virt/libvirt/driver.py#L4782 | |
| 12:36:10 | sean-k-mooney | lyarwood: are we using memcache downstream | |
| 12:36:25 | sean-k-mooney | we had to enable it upstream to make the ci stable | |
| 12:36:40 | sean-k-mooney | without it differrnt request could go to differnt api workers | |
| 12:36:57 | sean-k-mooney | cloud-init will only retry the first request | |
| 12:37:09 | sean-k-mooney | and after that it assuem the data is avaiable and does not retry them | |
| 12:37:28 | sean-k-mooney | if you hit a different worker and the node is under powered that can lead to timeouts | |
| 12:37:47 | sean-k-mooney | or failure to retive the data as genertatd the metadata is actully quite expensive | |
| 12:56:39 | lyarwood | sorry had to drop quickly for lunch | |
| 12:57:51 | lyarwood | this is part of a queens to train upgrade job downstream and it looks like the neutron metadata services are running on the computes | |
| 12:58:04 | lyarwood | I think this is failing post upgrade to train but it's super confusing | |
| 12:58:08 | lyarwood | jenkins-- | |
| 13:00:45 | lyarwood | ah no it's pre upgrade on queens | |