| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-11-22 | |||
| 17:39:46 | sean-k-mooney | ok but if its just for upgrade im not sure we shoudl do more then run multinode grenade | |
| 17:40:49 | gmann | multinode grenade does not run complete tests, if testing runtime have requirement of testing Focal I feel we should run integrated tests on that not just grenade small set of tests | |
| 17:41:09 | sean-k-mooney | i disagree | |
| 17:41:22 | sean-k-mooney | your are correct on the tests that are run https://github.com/openstack/nova/blob/master/.zuul.yaml#L512 | |
| 17:41:36 | sean-k-mooney | """In this release, we are adding the testing of Ubuntu new version 22.04. For smooth upgrade, we will continue the minimum testing for previously supported Ubuntu version in this release.""" | |
| 17:42:04 | sean-k-mooney | it specificaly say we will continue the minium testing for previous supported ubuntu verions | |
| 17:42:08 | sean-k-mooney | that is not tempest-full | |
| 17:42:31 | gmann | minimum testing means at least single job running the tests and we knew grenade job is there but adding this explicitly to make sure things does not get broken on old distro | |
| 17:42:43 | gmann | not all the jobs | |
| 17:43:04 | gmann | "run all jobs on latest distro, have at least one job to run on old distro" | |
| 17:43:28 | gmann | https://review.opendev.org/c/openstack/governance/+/860599/5/reference/project-testing-interface.rst#191 | |
| 17:43:40 | sean-k-mooney | are we going to have to do the same for debian 11 when 12 comes out | |
| 17:44:19 | gmann | we should do but we will see where all we are running debian jobs and there we should make sure there same | |
| 17:44:23 | gmann | the same | |
| 17:44:47 | sean-k-mooney | we dont have ci capsity for that | |
| 17:45:28 | sean-k-mooney | fine lets just stick with this for now but i dont think we can reasonable run all LTS distros for 2 releases | |
| 17:45:40 | sean-k-mooney | in the B cycle we shoudl drop the job | |
| 17:45:40 | gmann | debian jobs are not run on projects side (there might be few) but say if devstack run it then we should do both when we change the debian version in testing | |
| 17:46:04 | sean-k-mooney | and only test with 22.04 | |
| 17:46:12 | gmann | sean-k-mooney: true, that will be removed in next cycle of change in distro version, so for Focal yes in B | |
| 17:46:51 | gmann | it is only the cycle changing the testing to new version will make sure old version is also tested. | |
| 17:48:26 | gmann | we can do a lot of upgrade testing with distro things but we cannot do as per our CI capacity so this single job for changing version cycle is good way to accommodate something and make sure upgrade will be more smooth | |
| 18:18:27 | opendevreview | Merged openstack/nova stable/wallaby: [stable-only] Use os-brick from source in wallaby https://review.opendev.org/c/openstack/nova/+/865134 | |
| 19:44:57 | opendevreview | Ghanshyam proposed openstack/nova master: Update gate jobs as per the 2023.1 cycle testing runtime https://review.opendev.org/c/openstack/nova/+/861111 | |
| 19:47:10 | opendevreview | Ghanshyam proposed openstack/osc-placement master: Update gate jobs as per the 2023.1 cycle testing runtime https://review.opendev.org/c/openstack/osc-placement/+/861470 | |
| 19:48:15 | opendevreview | Ghanshyam proposed openstack/os-traits master: Update python classifier for python 3.10 https://review.opendev.org/c/openstack/os-traits/+/861466 | |
| 19:48:30 | opendevreview | Ghanshyam proposed openstack/python-novaclient master: Update python classifier for python 3.10 https://review.opendev.org/c/openstack/python-novaclient/+/861469 | |
| 19:54:31 | gmann | sean-k-mooney: os-vif functional sudo job failing on Jammy, I think you mentioned you have some idea to fix that?, https://zuul.opendev.org/t/openstack/build/b40811601a614064a480a02e368c2f72 | |
| 20:41:40 | sean-k-mooney | gmann: i looked at it breifly but did not try to fix it yet | |
| 20:42:15 | sean-k-mooney | gmann: debian used to have a pathced version of distutils that did weired things with the python path | |
| 20:42:35 | sean-k-mooney | the funtional tests use sudo and we use -E to preserve teh env | |
| 20:42:59 | sean-k-mooney | there is a weired interaction between sudo and virutalenv and -E | |
| 20:43:12 | sean-k-mooney | so my guess was that has changed on 22.04 | |
| 20:43:40 | sean-k-mooney | and the pythonpath in a sudo context is not finding os-vif | |
| 20:43:52 | sean-k-mooney | possibly because it might be doing a user install or similar | |
| 20:44:20 | sean-k-mooney | ill need to try and repoduce it locally to figure out whats going on exactly i just have not had time to look at it | |
| 20:44:54 | sean-k-mooney | but thats what i think is happening we are runnign privsep with sudo -E and its not finding the installed os-vif for some reason | |
| 20:59:57 | opendevreview | sean mooney proposed openstack/nova stable/wallaby: refactor: remove duplicated logic https://review.opendev.org/c/openstack/nova/+/865334 | |
| 20:59:58 | opendevreview | sean mooney proposed openstack/nova stable/wallaby: Detect port-resource-request-groups neutron API extension https://review.opendev.org/c/openstack/nova/+/865335 | |
| 20:59:59 | opendevreview | sean mooney proposed openstack/nova stable/wallaby: Record SRIOV PF MAC in the binding profile https://review.opendev.org/c/openstack/nova/+/865336 | |
| 21:08:17 | opendevreview | Anton Kurbatov proposed openstack/nova master: Fix VMs sorting fail in case of comparison with None https://review.opendev.org/c/openstack/nova/+/865037 | |
| #openstack-nova - 2022-11-23 | |||
| 02:31:54 | opendevreview | yangzhipeng proposed openstack/nova master: add delete Signed-off-by: yangzhipeng |
|
| 02:33:31 | opendevreview | yangzhipeng proposed openstack/nova master: Remove all tag if instance has beed hard deleted. Signed-off-by: yangzhipeng |
|
| 02:42:59 | opendevreview | yangzhipeng proposed openstack/nova master: Remove all tag if instance has beed hard deleted. https://review.opendev.org/c/openstack/nova/+/865362 | |
| 03:31:35 | gmann | dansmith: gibi: bauzas: need one more review on the placement RBAC spec, please check https://review.opendev.org/c/openstack/placement/+/864385 | |
| 06:59:51 | gokhani | situation ? | |
| 06:59:51 | gokhani | Good Morning Folks, When I try to reboot my instance, It destroys my instance and I am getting error "glanceclient.exc.HTTPNotFound: HTTP 404 Not Found: No image found with ID xxxx.." after reboot action I didn't find my instance with "virsh list --all" command. I didn't understand why nova tries to find glance image and why nova destroyed my instance ? Logs are in https://paste.openstack.org/show/by8oCDVLo29Wt612Z1tT/. what cause happen this | |
| 09:02:39 | sean-k-mooney | gokhani: when you reboot an instance we undefine the domain and recreated it however we will not lookup the glance image as part of a normal hard or soft reboot | |
| 09:03:14 | sean-k-mooney | so that implies that the instance was in an inconsitent state before the reboot | |
| 09:04:08 | sean-k-mooney | ile "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 9930, in _create_images_and_backing | |
| 09:07:27 | sean-k-mooney | so this should only be invokded if the vm is started via due to the resume_state_on_host_boot config option after a host reboot | |
| 09:07:37 | sean-k-mooney | https://github.com/openstack/nova/blob/stable/victoria/nova/virt/libvirt/driver.py#L3368-L3377 | |
| 09:08:35 | sean-k-mooney | gokhani: based on the trace back the instnace disk does not exist anymore | |
| 09:08:43 | sean-k-mooney | since it tokk this branch https://github.com/openstack/nova/blob/3224ceb3fffc57d2375e5163d8ffbbb77529bc38/nova/virt/libvirt/driver.py#L9949-L9951 | |
| 09:10:34 | sean-k-mooney | actullly no sorry https://github.com/openstack/nova/blob/3224ceb3fffc57d2375e5163d8ffbbb77529bc38/nova/virt/libvirt/driver.py#L9980-L9986 is the path its taking | |
| 09:11:11 | sean-k-mooney | so info['backing_file'] is equivalent to true | |
| 09:11:37 | sean-k-mooney | and to take the else branch that means that this si not the swap or ephmeral disk | |
| 09:16:49 | sean-k-mooney | anyway as the comment suggest https://github.com/openstack/nova/blob/stable/victoria/nova/virt/libvirt/driver.py#L3368-L3377 | |
| 09:17:15 | sean-k-mooney | that code path is there to endure that any backing files that are shared between teh vms are still present on the host | |
| 09:17:49 | sean-k-mooney | so this implies taht the backing file is not presnt on the host and it has been deleted from glance | |
| 09:18:41 | sean-k-mooney | in a normal hard reboot triggered by a human that code path shoudl not be taken and we will just redefine the domain | |
| 09:36:20 | kashyap | sean-k-mooney: Morning. Do we document any of this anywhere, at a high-level? | |
| 09:40:07 | sean-k-mooney | that hard reboot does not redownload the image | |
| 09:40:33 | sean-k-mooney | i dont think so but thats more of a impemation detail | |
| 09:41:07 | sean-k-mooney | it should not be required in a normal workflow sincew we are simulating rebooting a physical server | |
| 09:41:23 | sean-k-mooney | and you would not reinstall the os on a phsyical server | |
| 09:41:36 | sean-k-mooney | the fact we have code to do it after a host reboot is a little odd | |
| 09:41:52 | sean-k-mooney | but i would guess it is tehre because of a bug | |
| 09:42:09 | kashyap | Yeah, that's fair. (I'm not saying impl detail should be be documented. Just maybe somewhere in a debugging guide. I admit I can't find an appropriate place for it, though) | |
| 09:42:22 | sean-k-mooney | i.e. to work around a bug where perhaps at some point in the past the image casche got lost or something | |
| 09:42:41 | kashyap | "Image cache" ... /me runs for the hills | |
| 09:42:54 | kashyap | (Except no hills here in this part of the low lands :P) | |
| 09:43:30 | sean-k-mooney | ya ill admit im kind of surprised we have that code path to try and heal broken/missing backing files | |
| 09:43:56 | kashyap | TIL, too | |
| 09:44:04 | sean-k-mooney | the fallback to using it form the image cache if its not in glance is interesting... | |
| 09:44:13 | sean-k-mooney | it makes sense i guess | |
| 09:44:32 | sean-k-mooney | but this is and edgecase of an edgecase | |
| 09:44:36 | kashyap | Yeah, it definitely does to me. | |
| 09:45:13 | sean-k-mooney | i.e. after a host reboot the backing file need to have been delted and the image need to have been delted form glance but still exist in the host image cache for that to work | |
| 09:46:11 | sean-k-mooney | im sure thats the exact case someone hit and they added this to fix it however | |
| 09:57:33 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: Refactor volume connection cleanup out of _post_live_migration https://review.opendev.org/c/openstack/nova/+/864670 | |
| 09:57:34 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: Move pre-3.44 Cinder post live migration test to test_compute_mgr https://review.opendev.org/c/openstack/nova/+/864671 | |
| 09:57:35 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: Adds a repoducer for post live migration fail https://review.opendev.org/c/openstack/nova/+/863806 | |
| 09:57:36 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: [compute] always set instance.host in post_livemigration https://review.opendev.org/c/openstack/nova/+/864055 | |
| 09:57:38 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: func: Add _live_migrate helper to InstanceHelperMixin https://review.opendev.org/c/openstack/nova/+/865381 | |
| 09:57:40 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: func: Introduce a server_expected_state kwarg to InstanceHelperMixin._live_migrate https://review.opendev.org/c/openstack/nova/+/865382 | |
| 10:20:48 | auniyal | Hi sean-k-mooney | |
| 10:21:06 | auniyal | can you please review these patches - https://review.opendev.org/c/openstack/nova/+/864055 | |
| 10:23:14 | opendevreview | Anton Kurbatov proposed openstack/nova master: Fix VMs sorting fail in case of comparison with None https://review.opendev.org/c/openstack/nova/+/865037 | |
| 10:23:15 | sean-k-mooney | i can look quickly but i dont really have time for nova work this week outside of the vdpa rebases. | |
| 10:23:53 | sean-k-mooney | oh that the train backport | |
| 10:24:54 | auniyal | yeah, the last patch is failing before migration | |
| 10:42:02 | gokhani | sean-k-mooney, you are right, instance is in unconsistent state and I tried reboot it. Glance image is already deleted and so base image is also deleted. I didn't activate resume_state_on_host_boot option, it is false. | |
| 10:43:32 | sean-k-mooney | this code path is only taken if you dont have an authtoken | |
| 10:43:42 | sean-k-mooney | so you should not be able to get here manually | |
| 10:43:56 | sean-k-mooney | gokhani: how did you trigger the hard reboot | |
| 10:46:14 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: Refactor volume connection cleanup out of _post_live_migration https://review.opendev.org/c/openstack/nova/+/864670 | |
| 10:46:15 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: Move pre-3.44 Cinder post live migration test to test_compute_mgr https://review.opendev.org/c/openstack/nova/+/864671 | |
| 10:46:16 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: Adds a repoducer for post live migration fail https://review.opendev.org/c/openstack/nova/+/863806 | |
| 10:46:17 | opendevreview | Amit Uniyal proposed openstack/nova stable/train: [compute] always set instance.host in post_livemigration https://review.opendev.org/c/openstack/nova/+/864055 | |