Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-06-17
10:18:02 lyarwood no reason to leave it just on >=stable/xena
10:21:52 elodilles stephenfin: well, it formally violates, as it is a blueprint o:) ... anyway, I would stick to backporting only bug fixes... but... given that py27 is not supported in ussuri anymore... anyway I'm a bit unsure... o:)
10:24:12 lyarwood oh sorry I thought we were talking about the cherry-pick script
10:35:35 elodilles lyarwood: actually I've missed that discussion :X Are you planning to move out the cherry-pick-check from pep8?
10:36:19 elodilles lyarwood: nevermind, i'm just a bit slow today :D
10:36:31 lyarwood elodilles: https://review.opendev.org/c/openstack/nova/+/796626 yeah that's the idea, make it non-voting in check and only voting in the gate
10:36:47 elodilles lyarwood: yeah, sorry :X
10:45:11 sean-k-mooney lyarwood: instead of skiping the test https://review.opendev.org/c/openstack/nova/+/796813 why not just use the old driver
10:45:32 sean-k-mooney the stalling issue only happens if you use the native driver
10:45:44 sean-k-mooney the vsctl one wont have that problem
10:50:34 lyarwood We could but unless that's done in devstack across all jobs we end up with a mixed set of jobs
10:51:07 sean-k-mooney is that a bad thing
10:51:42 sean-k-mooney with the current patch your just reducing coverage
10:51:56 sean-k-mooney but the issue can still happen
10:52:21 sean-k-mooney im fine with makeing that change in devstack tempeorally
10:52:56 lyarwood kk if you could post that we can yank or revert this
10:53:40 sean-k-mooney i left a -1 on the patch already but would you like me to do the devstack change
10:54:12 sean-k-mooney jsut finishing an email but i can do it then
10:56:23 lyarwood sean-k-mooney: ack
10:56:30 lyarwood stephenfin: can you yank the +W on https://review.opendev.org/c/openstack/nova/+/796813
10:56:45 lyarwood stephenfin: sean-k-mooney is going to work around this in devstack
10:56:52 stephenfin sure, done
10:56:55 lyarwood ta
11:05:05 lyarwood noice, the FIPS fallout doesn't look that bad
11:05:06 lyarwood https://6cbf38d10f57b850b36e-212ab268b5e4bbb4b3348f98a2a831ee.ssl.cf5.rackcdn.com/790519/6/check/nova-fips/914f344/testr_results.html
11:06:16 lyarwood paramiko as expected and a server create timeout
11:07:57 lyarwood and that smells like the ovs locking up issue
11:12:00 sean-k-mooney lyarwood: cool
11:12:34 sean-k-mooney by the way i have found an interestign ceph issue that well i know how to fix but dont know how to detect
11:13:00 sean-k-mooney lyarwood: are you familar with EC pools in ceph
11:13:57 sean-k-mooney i was following https://docs.ceph.com/en/latest/rbd/rbd-openstack/ to configure ceph for openstack in general and https://themeanti.me/technology/2018/08/23/ceph_erasure_openstack.html for the ec pools
11:14:13 sean-k-mooney and i missed a step kind of
11:14:43 sean-k-mooney since i have a vms pool for nova and a vms_data pool
11:15:14 sean-k-mooney wehn i was doing the cephx user caps configuration i need to list both the vms pool and vms_data pool
11:15:19 sean-k-mooney only listed vms
11:15:37 sean-k-mooney the result of which is that nova booted a vm and it went into the active state
11:15:48 sean-k-mooney but it could not reade or write its root disk
11:16:34 sean-k-mooney the root disk has all the data present but it was inaccessable to qemu
11:17:20 sean-k-mooney lyarwood: due you think that qemu might be abel to detect that and create a warnig/error or could we detech that somehow and create an error
11:17:55 sean-k-mooney nova is calling ceph directly to get the avaibale storage
11:18:39 sean-k-mooney im debating if woudl make sense for nova to try and create a volume and read form it on the host or something on startup of the agent
11:22:52 lyarwood I'm not sure how QEMU could catch that tbh
11:23:11 lyarwood tbh that smells more like a deployment tooling validation?
11:23:19 lyarwood I wouldn't ask Nova to check it
11:23:33 sean-k-mooney ok its just annoying to debug
11:23:44 sean-k-mooney there is no error in qemu or nova or ceph
11:24:00 sean-k-mooney the vm just cant find a bootable disk
11:24:26 sean-k-mooney lyarwood: the deployment tool im using does not technially support this anymore which is why i messed it up
11:24:36 sean-k-mooney kolla-ansibel now just has extrenal ceph support
11:25:00 lyarwood that could still be a validation for external ceph
11:25:02 sean-k-mooney so you predeply ceph with your favor tool and then pass it a few files like the keyrings and it does the rest
11:25:06 lyarwood that the keyring has r/w access
11:25:15 sean-k-mooney it could yes
11:25:25 sean-k-mooney there are post run check which i did not run
11:25:48 sean-k-mooney i might add one for this
11:26:31 sean-k-mooney i was going to try and update there docs later anyway to document some of the more advanced customisation that im doing
11:26:57 sean-k-mooney for example running all fo the opnestack servics on the same port but with different subdomains
11:54:54 sean-k-mooney lyarwood: stephenfin https://review.opendev.org/c/openstack/devstack/+/796826
11:55:00 sean-k-mooney i think that will do the right thing
12:09:46 lyarwood LGTM but I'll wait for CI to run before I vote
12:12:26 sean-k-mooney i have not had time to test that so that is proably a good idea :)
12:14:19 lyarwood Small nit in the commit message btw, you called out the wrong bug.
12:14:37 sean-k-mooney oh
12:14:51 sean-k-mooney i can fix it but might wait for the ci to finish
12:14:56 lyarwood yeah no issues
12:15:12 sean-k-mooney ah i di
12:15:33 sean-k-mooney it should be https://bugs.launchpad.net/nova/+bug/1929446
12:15:40 sean-k-mooney not https://bugs.launchpad.net/ubuntu/+source/grub-installer/+bug/1929466
12:15:51 sean-k-mooney 446 no 466
12:17:18 sean-k-mooney lyarwood: its going to fail
12:17:55 sean-k-mooney opt/stack/devstack/lib/os-vif: line 12: return: False: numeric argument required
12:18:14 sean-k-mooney i forgot you cant return sting in bash
12:18:19 sean-k-mooney you echo them
12:19:06 opendevreview Merged openstack/nova master: db: Remove dead code https://review.opendev.org/c/openstack/nova/+/786291
12:19:33 opendevreview Merged openstack/nova master: gate: Remove test_evacuate.sh https://review.opendev.org/c/openstack/nova/+/796523
12:50:01 opendevreview Rodrigo Barbieri proposed openstack/nova stable/ussuri: Error anti-affinity violation on migrations https://review.opendev.org/c/openstack/nova/+/796719
13:06:38 opendevreview Merged openstack/nova stable/stein: Improve error log when snapshot fails https://review.opendev.org/c/openstack/nova/+/782962
13:06:59 opendevreview Merged openstack/nova stable/ussuri: Reproduce bug 1897528 https://review.opendev.org/c/openstack/nova/+/791770
13:13:00 opendevreview Lee Yarwood proposed openstack/nova master: zuul: Add nova-tox-functional-centos8-py36 job https://review.opendev.org/c/openstack/nova/+/796684
13:18:27 opendevreview Lee Yarwood proposed openstack/nova master: zuul: Add nova-tox-functional-centos8-py36 job https://review.opendev.org/c/openstack/nova/+/796684
13:19:13 lyarwood gah!
13:19:23 opendevreview Lee Yarwood proposed openstack/nova master: zuul: Add nova-tox-functional-centos8-py36 job https://review.opendev.org/c/openstack/nova/+/796684
13:50:38 lyarwood gibi: sean-k-mooney is working on https://review.opendev.org/c/openstack/devstack/+/796826 to hopefully resolve lots of instability
13:51:25 sean-k-mooney i wonder why we are hitting this so much more often recently
13:53:21 lyarwood maybe we are just noticing it more recently, it's an awkward one.
13:53:54 sean-k-mooney ya we also kind of mentally filter out those lines in the log
13:54:01 sean-k-mooney at least i do most of the time
13:59:16 lyarwood right takes some processing of timestamps to even see the issue but most of the time the ultimate test failure is miles away from that
13:59:40 lyarwood sometimes I wish I worked on an easier stack :)
13:59:45 sean-k-mooney lyarwood: gibi its almost finished the check run by the way the current version seams to be working
13:59:53 lyarwood ack yeah I've been watching
13:59:59 lyarwood looking good thus far
14:04:09 noonedeadpunk o/
14:04:39 noonedeadpunk folks we noticed weird behaviour that you're probably aware about
14:05:25 opendevreview Mohammed Naser proposed openstack/nova stable/wallaby: Allow X-OpenStack-Nova-API-Version header in CORS https://review.opendev.org/c/openstack/nova/+/796860
14:06:08 opendevreview Mohammed Naser proposed openstack/nova stable/victoria: Allow X-OpenStack-Nova-API-Version header in CORS https://review.opendev.org/c/openstack/nova/+/796861
14:06:41 opendevreview Mohammed Naser proposed openstack/nova stable/ussuri: Allow X-OpenStack-Nova-API-Version header in CORS https://review.opendev.org/c/openstack/nova/+/796862
14:07:19 opendevreview Mohammed Naser proposed openstack/nova stable/train: Allow X-OpenStack-Nova-API-Version header in CORS https://review.opendev.org/c/openstack/nova/+/796863
14:07:38 noonedeadpunk So, algorithm is kind of the following: 1. HV goes down. 2. VM is sent Shutdown (or any other request). 3. Then VM is in `powering-off` state, but it needs to be evacuated. So reset-state is issued and evacuate is processed. Now VM is running on another HV. 4 When original HV goes up it process messages that were issued while it was down and powers off VM that was evacuated and owned by another HV atm
14:08:22 noonedeadpunk I have a feeling that if node is not owning VM it should not have ability to influence it even if it has some commands in queue?

Earlier   Later