| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-06-16 | |||
| 17:41:02 | sean-k-mooney | so normally we avoid that but its good to know its an option when you really need to move soemthing and is still friendlier then a cold migrate | |
| 17:41:50 | sean-k-mooney | why is that in story board | |
| 17:42:04 | sean-k-mooney | and ya we had a downstream issue to the same effect recently | |
| 17:42:45 | sean-k-mooney | melwitt: i wonder if we should actully consier blocking that in the api | |
| 17:43:11 | sean-k-mooney | its never valide to reset state form shelve offloaded to any other state then error | |
| 17:45:16 | sean-k-mooney | lyarwood: stephenfin melwitt by the way we still have a todo item somehwer to extend the reset state api to allow more states and to allow settign the task_state too | |
| 17:46:13 | sean-k-mooney | if we do that we should harden the allowed state transations IMO | |
| 17:46:49 | melwitt | sean-k-mooney: yeah... conflicted about that because reset state is kind of a wild west api call to make but on the other hand if we can block something that's guaranteed to be invalid, that makes sense too | |
| 17:46:51 | sean-k-mooney | e.g. ERROR we woudl allow to go to any sate as an escape hatch. but any none ERROR state should not be changable arbitrailly | |
| 17:47:57 | sean-k-mooney | melwitt: im pretty sure it can result in data lose | |
| 17:48:24 | sean-k-mooney | if you do resetstat form shelve offloaded and then do either hard reboot or a migration i cant rememebr which | |
| 17:48:29 | melwitt | ? | |
| 17:48:36 | sean-k-mooney | we had a customer do that with ceph recently | |
| 17:48:42 | sean-k-mooney | and they lost the snapshot | |
| 17:49:15 | melwitt | I guess I'm not realizing how that happens. if it's shelved offloaded there's no guest to reboot | |
| 17:49:59 | sean-k-mooney | yes so if i rememebr correctly unshleve failed so the reset state then hard rebooted | |
| 17:50:03 | sean-k-mooney | that failed because no host | |
| 17:50:42 | sean-k-mooney | the then reset state again and tired to shleve i think? this is where my memory is blank | |
| 17:51:02 | sean-k-mooney | when we got the bz they had tried some cobination of that several times | |
| 17:51:14 | sean-k-mooney | and the rbd volume was nolonger preset for the vm root disk | |
| 17:52:05 | melwitt | ah ok, so you're saying if we prevent that path from even starting, it can avoid whatever future thing they managed to do to lose the data | |
| 17:52:14 | sean-k-mooney | yes | |
| 17:52:58 | melwitt | yeah ok | |
| 17:58:56 | opendevreview | Rodrigo Barbieri proposed openstack/nova stable/ussuri: Error anti-affinity violation on migrations https://review.opendev.org/c/openstack/nova/+/796719 | |
| #openstack-nova - 2021-06-17 | |||
| 01:46:55 | opendevreview | melanie witt proposed openstack/nova master: Add func test for nova-manage db archive_deleted_rows --before https://review.opendev.org/c/openstack/nova/+/796744 | |
| 01:57:32 | opendevreview | melanie witt proposed openstack/nova master: Add --task-log option to nova-manage db archive_deleted_rows https://review.opendev.org/c/openstack/nova/+/780395 | |
| 02:00:14 | melwitt | lyarwood, elodilles: it's funny passing CI \o/ (but recheck-a-thon) https://review.opendev.org/c/openstack/nova/+/795432 | |
| 02:00:21 | melwitt | s/funny/finally/ | |
| 07:11:07 | gibi | lyarwood: hi! regarding https://review.opendev.org/c/openstack/nova/+/796523 I'm sure I asked this before but forgot. Where do we have now the evacuation test coverage? | |
| 07:18:13 | opendevreview | Merged openstack/nova stable/rocky: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available https://review.opendev.org/c/openstack/nova/+/747612 | |
| 07:18:37 | opendevreview | Merged openstack/nova stable/wallaby: Neutron fixture: don't clobber profile and vif_details if empty https://review.opendev.org/c/openstack/nova/+/792233 | |
| 07:18:50 | opendevreview | Merged openstack/nova stable/wallaby: Test SRIOV port move operations with PCI conflicts https://review.opendev.org/c/openstack/nova/+/790710 | |
| 07:31:09 | opendevreview | Yongli He proposed openstack/nova master: smartnic support https://review.opendev.org/c/openstack/nova/+/758944 | |
| 07:31:10 | opendevreview | Yongli He proposed openstack/nova master: smartnic support - functional tests https://review.opendev.org/c/openstack/nova/+/780147 | |
| 07:31:10 | opendevreview | Yongli He proposed openstack/nova master: smartnic support - reject server move and suspend https://review.opendev.org/c/openstack/nova/+/779913 | |
| 07:43:53 | lyarwood | melwitt: awesome :) I'll +1 only as I modified it | |
| 07:44:21 | lyarwood | gibi: it's part of the live migration jobs | |
| 07:44:32 | lyarwood | gibi: runs in the post playbook | |
| 07:45:32 | lyarwood | gibi: https://github.com/openstack/nova/tree/master/roles/run-evacuate-hook is the role we use | |
| 07:45:46 | lyarwood | gibi: https://github.com/openstack/nova/blob/master/playbooks/nova-live-migration/post-run.yaml is where it's called | |
| 07:46:10 | lyarwood | gibi: the logic being that we didn't want to stand up another multinode env every run to test evacuation | |
| 07:46:39 | lyarwood | gibi: doing it in post was easier as we didn't need to copy and paste any of the tempest playbook logic into Nova | |
| 07:50:13 | lyarwood | gibi: so for that review evacuation is tested from here https://zuul.opendev.org/t/openstack/build/057093756ca64ef994584e2cae50f537/log/job-output.txt#64392 | |
| 07:50:41 | opendevreview | Merged openstack/nova stable/victoria: Reproduce bug 1897528 https://review.opendev.org/c/openstack/nova/+/791767 | |
| 08:03:14 | gibi | lyarwood: thanks | |
| 08:03:22 | gibi | I hope I will not foget this again :) | |
| 08:14:00 | lyarwood | ^_^ | |
| 09:01:07 | opendevreview | Yongli He proposed openstack/nova master: smartnic support - new vnic type https://review.opendev.org/c/openstack/nova/+/771363 | |
| 09:01:07 | opendevreview | Yongli He proposed openstack/nova master: Smartnic support - cyborg drive https://review.opendev.org/c/openstack/nova/+/771362 | |
| 09:01:08 | opendevreview | Yongli He proposed openstack/nova master: smartnic support https://review.opendev.org/c/openstack/nova/+/758944 | |
| 09:01:09 | opendevreview | Yongli He proposed openstack/nova master: smartnic support - reject server move and suspend https://review.opendev.org/c/openstack/nova/+/779913 | |
| 09:01:10 | opendevreview | Yongli He proposed openstack/nova master: smartnic support - functional tests https://review.opendev.org/c/openstack/nova/+/780147 | |
| 09:02:40 | opendevreview | Merged openstack/nova stable/victoria: Ignore PCI devices with 32bit domain https://review.opendev.org/c/openstack/nova/+/791768 | |
| 09:03:42 | yonglihe | rebase to fix dependency problem, that's weird. | |
| 09:19:39 | lyarwood | yonglihe: A pip dependency problem? We've seen loads that make no sense recently. | |
| 09:21:25 | stephenfin | lyarwood: it's a cache issue, I think | |
| 09:21:41 | lyarwood | oh the limestone thing? | |
| 09:21:44 | stephenfin | yeah, I think so | |
| 09:22:51 | lyarwood | wonderful | |
| 09:22:59 | stephenfin | it's failing with e.g. dep a requesting >=1.2 and upper constraints requesting == 3.0, which would pass unless 3.0 wasn't available | |
| 09:24:48 | stephenfin | hmm, maybe not actually - the error message I get locally is different | |
| 09:25:08 | lyarwood | ack thanks | |
| 09:25:19 | lyarwood | that makes sense now if the cache is borked | |
| 09:26:24 | stephenfin | yonglihe: the failure on https://review.opendev.org/c/openstack/nova/+/758944/ looks real? | |
| 09:26:33 | stephenfin | AttributeError: module 'nova.network.model' has no attribute 'VNIC_TYPES_ACCELERATOR' | |
| 09:26:33 | stephenfin | if vnic_type in network_model.VNIC_TYPES_ACCELERATOR: | |
| 09:26:37 | stephenfin | (from https://zuul.opendev.org/t/openstack/build/e08dc74546d34d9a8ee67e597ade8fb2) | |
| 09:28:37 | stephenfin | elodilles: lyarwood: Care to keep working through this backport series? The victoria patches have landed now and this is another clean backport https://review.opendev.org/q/topic:%2522bug/1897528%2522+branch:stable/ussuri | |
| 09:29:37 | lyarwood | ack looking | |
| 09:30:21 | lyarwood | elodilles: https://review.opendev.org/c/openstack/nova/+/796626 - can you also take a look at this on master if you get a chance, moving the cherry-pick script out of pep8. | |
| 09:32:55 | elodilles | sure, looking at the patches :) | |
| 09:42:49 | gibi | lyarwood, stephenfin: yesterday infra turned off limestone due to the pip cache issue | |
| 09:43:09 | gibi | so we should not see these nonsensical req conflicts | |
| 09:43:13 | gibi | any more today | |
| 09:45:51 | yonglihe | stephenfin, that's because that patch lost the decency to second patch, fixed. | |
| 09:45:53 | lyarwood | wonderful | |
| 09:46:26 | lyarwood | gibi: https://bugs.launchpad.net/cinder/+bug/1932287 just caught this if you see any random volume creation failures today | |
| 09:47:30 | gibi | lyarwood: thanks, I haven't seen that issue yet | |
| 09:47:31 | opendevreview | Merged openstack/nova stable/rocky: Remove allocations before setting vm_status to SHELVED_OFFLOADED https://review.opendev.org/c/openstack/nova/+/771985 | |
| 09:48:35 | stephenfin | elodilles: Yeah, as lyarwood said, we need to move the cherry-pick change out of the pep8 job. I hadn't seen that failure | |
| 09:50:31 | gibi | lyarwood: with the exit code 139 lvs complains about missing devices and that I saw before | |
| 09:50:56 | lyarwood | yeah https://review.opendev.org/c/openstack/cinder/+/783660 fixed it elsewhere | |
| 09:51:01 | lyarwood | just not in this path | |
| 09:51:44 | gibi | lyarwood: cool, then we have a way forward | |
| 09:51:55 | opendevreview | Stephen Finucane proposed openstack/nova stable/ussuri: Reproduce bug 1897528 https://review.opendev.org/c/openstack/nova/+/791770 | |
| 09:51:56 | opendevreview | Stephen Finucane proposed openstack/nova stable/ussuri: Ignore PCI devices with 32bit domain https://review.opendev.org/c/openstack/nova/+/791771 | |
| 09:52:06 | stephenfin | elodilles: lyarwood: fixed the pep8 failure ^ | |
| 09:52:58 | gibi | lyarwood: I'm hitting https://bugs.launchpad.net/nova/+bug/1912310 many times now and almost always in the test_volume_backed_live_migration tempest test. Wondering if it worth to disable that test until ovsdbapp fix lands | |
| 09:53:06 | opendevreview | Stephen Finucane proposed openstack/nova stable/train: Reproduce bug 1897528 https://review.opendev.org/c/openstack/nova/+/792116 | |
| 09:53:07 | opendevreview | Stephen Finucane proposed openstack/nova stable/train: Ignore PCI devices with 32bit domain https://review.opendev.org/c/openstack/nova/+/792117 | |
| 09:53:22 | stephenfin | and the train ones are updated now too | |
| 09:57:33 | lyarwood | gibi: ack lets do it, I'll disable them now | |
| 10:04:22 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Skip volume backed LM tests until bug #1912310 is resolved https://review.opendev.org/c/openstack/nova/+/796813 | |
| 10:04:29 | lyarwood | gibi: ^ hopefully that's enough, if it isn't then we might want to move the LM jobs to non-voting | |
| 10:11:15 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Reintroduce validation of shadow table schema https://review.opendev.org/c/openstack/nova/+/796814 | |
| 10:11:39 | stephenfin | lyarwood: gibi: one final one, as requested ^ | |
| 10:12:38 | gibi | lyarwood: thanks | |
| 10:13:05 | elodilles | stephenfin: actually i was surprised that pep8 is failing in ussuri because of py27/six problem as py27 should be supported only up until train :-o | |
| 10:13:43 | stephenfin | elodilles: yeah, we simply weren't aggressive enough in dropping the no-longer relevant hacking checks | |
| 10:13:54 | elodilles | oh, i see | |