| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-06-17 | |||
| 11:12:34 | sean-k-mooney | by the way i have found an interestign ceph issue that well i know how to fix but dont know how to detect | |
| 11:13:00 | sean-k-mooney | lyarwood: are you familar with EC pools in ceph | |
| 11:13:57 | sean-k-mooney | i was following https://docs.ceph.com/en/latest/rbd/rbd-openstack/ to configure ceph for openstack in general and https://themeanti.me/technology/2018/08/23/ceph_erasure_openstack.html for the ec pools | |
| 11:14:13 | sean-k-mooney | and i missed a step kind of | |
| 11:14:43 | sean-k-mooney | since i have a vms pool for nova and a vms_data pool | |
| 11:15:14 | sean-k-mooney | wehn i was doing the cephx user caps configuration i need to list both the vms pool and vms_data pool | |
| 11:15:19 | sean-k-mooney | only listed vms | |
| 11:15:37 | sean-k-mooney | the result of which is that nova booted a vm and it went into the active state | |
| 11:15:48 | sean-k-mooney | but it could not reade or write its root disk | |
| 11:16:34 | sean-k-mooney | the root disk has all the data present but it was inaccessable to qemu | |
| 11:17:20 | sean-k-mooney | lyarwood: due you think that qemu might be abel to detect that and create a warnig/error or could we detech that somehow and create an error | |
| 11:17:55 | sean-k-mooney | nova is calling ceph directly to get the avaibale storage | |
| 11:18:39 | sean-k-mooney | im debating if woudl make sense for nova to try and create a volume and read form it on the host or something on startup of the agent | |
| 11:22:52 | lyarwood | I'm not sure how QEMU could catch that tbh | |
| 11:23:11 | lyarwood | tbh that smells more like a deployment tooling validation? | |
| 11:23:19 | lyarwood | I wouldn't ask Nova to check it | |
| 11:23:33 | sean-k-mooney | ok its just annoying to debug | |
| 11:23:44 | sean-k-mooney | there is no error in qemu or nova or ceph | |
| 11:24:00 | sean-k-mooney | the vm just cant find a bootable disk | |
| 11:24:26 | sean-k-mooney | lyarwood: the deployment tool im using does not technially support this anymore which is why i messed it up | |
| 11:24:36 | sean-k-mooney | kolla-ansibel now just has extrenal ceph support | |
| 11:25:00 | lyarwood | that could still be a validation for external ceph | |
| 11:25:02 | sean-k-mooney | so you predeply ceph with your favor tool and then pass it a few files like the keyrings and it does the rest | |
| 11:25:06 | lyarwood | that the keyring has r/w access | |
| 11:25:15 | sean-k-mooney | it could yes | |
| 11:25:25 | sean-k-mooney | there are post run check which i did not run | |
| 11:25:48 | sean-k-mooney | i might add one for this | |
| 11:26:31 | sean-k-mooney | i was going to try and update there docs later anyway to document some of the more advanced customisation that im doing | |
| 11:26:57 | sean-k-mooney | for example running all fo the opnestack servics on the same port but with different subdomains | |
| 11:54:54 | sean-k-mooney | lyarwood: stephenfin https://review.opendev.org/c/openstack/devstack/+/796826 | |
| 11:55:00 | sean-k-mooney | i think that will do the right thing | |
| 12:09:46 | lyarwood | LGTM but I'll wait for CI to run before I vote | |
| 12:12:26 | sean-k-mooney | i have not had time to test that so that is proably a good idea :) | |
| 12:14:19 | lyarwood | Small nit in the commit message btw, you called out the wrong bug. | |
| 12:14:37 | sean-k-mooney | oh | |
| 12:14:51 | sean-k-mooney | i can fix it but might wait for the ci to finish | |
| 12:14:56 | lyarwood | yeah no issues | |
| 12:15:12 | sean-k-mooney | ah i di | |
| 12:15:33 | sean-k-mooney | it should be https://bugs.launchpad.net/nova/+bug/1929446 | |
| 12:15:40 | sean-k-mooney | not https://bugs.launchpad.net/ubuntu/+source/grub-installer/+bug/1929466 | |
| 12:15:51 | sean-k-mooney | 446 no 466 | |
| 12:17:18 | sean-k-mooney | lyarwood: its going to fail | |
| 12:17:55 | sean-k-mooney | opt/stack/devstack/lib/os-vif: line 12: return: False: numeric argument required | |
| 12:18:14 | sean-k-mooney | i forgot you cant return sting in bash | |
| 12:18:19 | sean-k-mooney | you echo them | |
| 12:19:06 | opendevreview | Merged openstack/nova master: db: Remove dead code https://review.opendev.org/c/openstack/nova/+/786291 | |
| 12:19:33 | opendevreview | Merged openstack/nova master: gate: Remove test_evacuate.sh https://review.opendev.org/c/openstack/nova/+/796523 | |
| 12:50:01 | opendevreview | Rodrigo Barbieri proposed openstack/nova stable/ussuri: Error anti-affinity violation on migrations https://review.opendev.org/c/openstack/nova/+/796719 | |
| 13:06:38 | opendevreview | Merged openstack/nova stable/stein: Improve error log when snapshot fails https://review.opendev.org/c/openstack/nova/+/782962 | |
| 13:06:59 | opendevreview | Merged openstack/nova stable/ussuri: Reproduce bug 1897528 https://review.opendev.org/c/openstack/nova/+/791770 | |
| 13:13:00 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Add nova-tox-functional-centos8-py36 job https://review.opendev.org/c/openstack/nova/+/796684 | |
| 13:18:27 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Add nova-tox-functional-centos8-py36 job https://review.opendev.org/c/openstack/nova/+/796684 | |
| 13:19:13 | lyarwood | gah! | |
| 13:19:23 | opendevreview | Lee Yarwood proposed openstack/nova master: zuul: Add nova-tox-functional-centos8-py36 job https://review.opendev.org/c/openstack/nova/+/796684 | |
| 13:50:38 | lyarwood | gibi: sean-k-mooney is working on https://review.opendev.org/c/openstack/devstack/+/796826 to hopefully resolve lots of instability | |
| 13:51:25 | sean-k-mooney | i wonder why we are hitting this so much more often recently | |
| 13:53:21 | lyarwood | maybe we are just noticing it more recently, it's an awkward one. | |
| 13:53:54 | sean-k-mooney | ya we also kind of mentally filter out those lines in the log | |
| 13:54:01 | sean-k-mooney | at least i do most of the time | |
| 13:59:16 | lyarwood | right takes some processing of timestamps to even see the issue but most of the time the ultimate test failure is miles away from that | |
| 13:59:40 | lyarwood | sometimes I wish I worked on an easier stack :) | |
| 13:59:45 | sean-k-mooney | lyarwood: gibi its almost finished the check run by the way the current version seams to be working | |
| 13:59:53 | lyarwood | ack yeah I've been watching | |
| 13:59:59 | lyarwood | looking good thus far | |
| 14:04:09 | noonedeadpunk | o/ | |
| 14:04:39 | noonedeadpunk | folks we noticed weird behaviour that you're probably aware about | |
| 14:05:25 | opendevreview | Mohammed Naser proposed openstack/nova stable/wallaby: Allow X-OpenStack-Nova-API-Version header in CORS https://review.opendev.org/c/openstack/nova/+/796860 | |
| 14:06:08 | opendevreview | Mohammed Naser proposed openstack/nova stable/victoria: Allow X-OpenStack-Nova-API-Version header in CORS https://review.opendev.org/c/openstack/nova/+/796861 | |
| 14:06:41 | opendevreview | Mohammed Naser proposed openstack/nova stable/ussuri: Allow X-OpenStack-Nova-API-Version header in CORS https://review.opendev.org/c/openstack/nova/+/796862 | |
| 14:07:19 | opendevreview | Mohammed Naser proposed openstack/nova stable/train: Allow X-OpenStack-Nova-API-Version header in CORS https://review.opendev.org/c/openstack/nova/+/796863 | |
| 14:07:38 | noonedeadpunk | So, algorithm is kind of the following: 1. HV goes down. 2. VM is sent Shutdown (or any other request). 3. Then VM is in `powering-off` state, but it needs to be evacuated. So reset-state is issued and evacuate is processed. Now VM is running on another HV. 4 When original HV goes up it process messages that were issued while it was down and powers off VM that was evacuated and owned by another HV atm | |
| 14:08:22 | noonedeadpunk | I have a feeling that if node is not owning VM it should not have ability to influence it even if it has some commands in queue? | |
| 14:08:46 | noonedeadpunk | and maybe you have some guess where in code worth looking for this? | |
| 14:10:15 | lyarwood | so the compute manager that gets the cast in this case isn't doing any checks to ensure the instance is still on that host | |
| 14:10:41 | lyarwood | I guess it's a valid thing to do for any operations using casts | |
| 14:10:50 | noonedeadpunk | yeah, I expect smth like that is happening. But not super familiar with codebase :( | |
| 14:11:02 | sean-k-mooney | noonedeadpunk: why are you doing reset-state in your evacuate workflow | |
| 14:11:11 | sean-k-mooney | noonedeadpunk: you should not be doing reset state first | |
| 14:11:35 | noonedeadpunk | well, otherwise it can't be evacuated with `ERROR (Conflict): Cannot 'evacuate' instance e46404b1-e6e1-4d22-9f8f-12d6f51b55ae while it is in task_state powering-off` | |
| 14:11:57 | sean-k-mooney | hum | |
| 14:12:01 | sean-k-mooney | i see | |
| 14:12:07 | noonedeadpunk | Is there any other proper way to do evacuate? | |
| 14:12:21 | gibi | lyarwood, sean-k-mooney thanks. I'm happy to see that this week a lot of us focused on stabilizing the gate. | |
| 14:12:29 | noonedeadpunk | I mean technicaly we could wait until node goes up, but it might be days theoretically? | |
| 14:12:49 | lyarwood | tbh I think we should allow evacuate if the instance is powering-off | |
| 14:12:55 | lyarwood | either way the src compute is dead | |
| 14:13:07 | sean-k-mooney | yep i ws thinking the same | |
| 14:13:10 | noonedeadpunk | but it won't resolve original issue though | |
| 14:13:29 | noonedeadpunk | as then evacuated instance would be shot anyway | |
| 14:13:42 | lyarwood | well it shouldn't kill the instance on the dest | |
| 14:13:46 | noonedeadpunk | (but agree it's super valid to allow evacuate) | |
| 14:13:56 | lyarwood | the cast to shutdown the original instance on the original host should fail | |
| 14:14:12 | noonedeadpunk | fwiw it's on Victoria | |
| 14:14:22 | lyarwood | but that has the potential of moving the instance into an ERROR state | |
| 14:14:54 | lyarwood | a simple decorator to check that instance.host points at the current host would work here | |
| 14:15:00 | lyarwood | it might not work everywhere we cast | |
| 14:15:07 | lyarwood | but in this example it's fine | |
| 14:15:26 | lyarwood | noonedeadpunk: did you have a bug for this? | |
| 14:15:53 | noonedeadpunk | nope, not yet, but will submit one :) | |
| 14:16:10 | lyarwood | awesome thanks | |