Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-08-30
12:46:42 gibi cool, good ide
12:46:44 gibi idea
12:46:53 sean-k-mooney the other alternitive would be to move form ubuntu 20.04 to 21.04 or to centos 8 on the affected jobs
12:49:01 sean-k-mooney well there is another alternitive too which is complie libvirt/qemu form source which i have a devstack plugin to do but i would prefer to avoid that mainly due to extra job time. its not hard to do but if we can just use distro pacakages in this case its nicer
12:49:17 gibi as we declare our supported distros beforhand of the release I would go with trying to fix ubuntu 20.04 https://governance.openstack.org/tc/reference/runtimes/xena.html
12:50:26 sean-k-mooney yes although centos 8 stream is vaild too. but ya ill see if i can look into this a little later today. ill propose a couple of different patches for different options.
12:51:20 sean-k-mooney enableing "sudo add-apt-repository ppa:jacob/virtualisation" in a pre playbook is simple as is changing the base os to centos 8 stream
12:51:45 sean-k-mooney the other options are more complicated but doable
13:06:59 opendevreview Merged openstack/nova master: Functional tests removed direct post call https://review.opendev.org/c/openstack/nova/+/766068
13:22:30 sean-k-mooney gibi: by the way we maintain a tempest plugin called whitebox that looks at some of the internals of how nova works and assert that it does the right thing. would you have any object to me enableing that for a subset of nova changes at least in a non voting capasity initaly?
13:22:48 sean-k-mooney gibi: i was thinking of making it run on change to the libvirt driver and hardware.py
13:23:20 gibi sean-k-mooney: I have no problem with it if it is actively maintained and won't take up much of the CI resources
13:25:11 sean-k-mooney yes its maintianed and runs downstream we also maintin the devstack support upstream
13:25:33 sean-k-mooney gibi: upstream many of the test are disabled because we dont have the hardware https://opendev.org/openstack/whitebox-tempest-plugin/src/branch/master/whitebox_tempest_plugin/api/compute
13:26:01 sean-k-mooney i.e. we can run the pmem, sriov or vgpu test in the gate
13:26:29 sean-k-mooney i know we were lucking to see if we coudl use this for third party ci but we still are having problems finding hardware internally to run it
13:27:46 opendevreview Balazs Gibizer proposed openstack/nova master: Add two new hacking rules https://review.opendev.org/c/openstack/nova/+/805668
13:28:11 gibi just based on the test file names even without special hardware this plugin has useful coverage
13:28:45 sean-k-mooney yep it has all the test that were orginailly don by the intel thridpary nfv ci in it but updated
13:28:51 sean-k-mooney and some other test coverage
13:31:10 gibi then lets enable it
13:32:40 sean-k-mooney gibi: when the qe member of the comptue team downstream writes test automation that is not suitable for upstream tempest becasue it depens on speicic configuration of the serives this is where we try to add the test coverage.
13:33:07 sean-k-mooney like testing adding cpu flags which we can do https://opendev.org/openstack/whitebox-tempest-plugin/src/branch/master/.zuul.yaml#L53-L55 in the ci like this
13:34:22 gibi I agree to have that coverage in our upstream gate
13:34:41 sean-k-mooney thanks ill let artom know and see if he wants me to wait for the jobs to be split or not first ill start on the WIP patch in anycase
13:43:21 artom Huh, happy coincidence, I was pondering proposing a periodic whitebox job for Nova
13:43:31 artom So, I think it's not yet stable enough for that, actually
13:43:52 artom We think we know the issue, and we're working on it, but until then I'm not sure it's ready yet
13:45:13 artom Every so often, depending on which order tests end up being executed, what we think happens is we attempt to reshape from cpu_dedicated_set to vcpu_pin_set, and that's not allowed, so there's a cascading failure. There are also around how we use admin clients and clean up after ourselves, that can also cause cascading failures
13:45:17 gibi artom: nothing is urgent from upstream perspective. If upstream feedback helps then I'm OK to enable a non voting job
13:46:01 artom gibi, I think even that's premature, as the solution to ^^^ is to change whitebox's own job a bit, so until that's done, let's not add it to nova
13:46:50 sean-k-mooney artom: ok the reason i was bring this up was we did at one point plan to enabel white box for wallaby
13:47:04 sean-k-mooney then we did not have time to actully get it stable in time
13:47:15 sean-k-mooney so i was hoping we could do thatbefore the end of xena
13:47:24 sean-k-mooney if you think its not ready however we can hold off
13:47:24 artom Ah, probably not before the end of Xena
13:47:31 artom ... well, does end == FF?
13:47:33 artom Or release?
13:48:15 sean-k-mooney well i guess i twas thinking before RC1 when stable branch is created
13:48:34 sean-k-mooney although if we were ok with backporting enableing the testing on the stable branch end could be anytime before eol i guess
13:49:27 sean-k-mooney if we dont think it ready however no need to rush
13:50:00 sean-k-mooney i would just like to keep making progress on getting this test coverage enable eithe firstparty or third party
13:51:11 artom Third party I still haven't solved the hardware problem :)
13:51:13 artom Err :(
13:51:20 sean-k-mooney artom: is bauzas back today or is he retruning tomorow
13:51:31 sean-k-mooney artom: yep i know :)
13:52:19 artom Still on PTO today, according to Workday
13:52:42 gibi I personally OK with enabling new jobs on stable but I guess elodilles or lyarwood has more authority about that :)
13:53:10 gibi as per landing it on master, this is not a feature so RC1 is the cut of date due to branching
13:53:39 artom I can try to hurry it up, especially as jparker seems to have more time for this right about now, too
13:54:58 sean-k-mooney gibi: before i recheck are there any gate blockers i should hold off for
13:55:24 sean-k-mooney i was just looking at the failure in bauzas mdev series which dont seam related
13:56:04 gibi sean-k-mooney: the "Add force kwarg to delete_allocation_for_instance" not landed yet that kills at least 1/4 of the tempest jobs all around the gate
13:56:16 sean-k-mooney ah right
13:56:20 gibi I don't know about any full blocker
13:56:49 sean-k-mooney ok i was seeing the nova-ceph-multistore job fail in several patch but have not dug in to see if its the same issue
13:57:29 sean-k-mooney oh "'Failed to delete allocations for consumer 2064788c-9fa0-474e-a66c-72cf97b45922. .."
13:57:34 sean-k-mooney ya so its just that
13:58:55 gibi yes
13:59:10 sean-k-mooney any idea why it would hit the multistore job more often
13:59:36 sean-k-mooney it looks like that is mostly the failure so ill hold off until the force patch lands
14:00:04 sean-k-mooney they dont have +w anyway so they can wait
14:00:46 elodilles well, i don't exactly followed which job you were talking about but I am less concerned enabling new CI jobs on stable than disabling one o:)
14:00:47 gibi Ive no ide about the increased frequency of multistore failure
14:01:07 gibi elodilles: it would be a job running https://opendev.org/openstack/whitebox-tempest-plugin/src/branch/master/whitebox_tempest_plugin/api/compute
14:01:35 sean-k-mooney elodilles: which is defiedn here https://opendev.org/openstack/whitebox-tempest-plugin/src/branch/master/.zuul.yaml#L31-L81
14:02:13 sean-k-mooney gibi: artom is currntly actully spliting it into two jobs one that use the old cpu pinning config and the main one will only use the new way
14:02:24 gibi ack
14:02:35 sean-k-mooney gibi: right now if the reshap happens at the wong time the job breaks
14:02:45 sean-k-mooney so we are just going to split it to aovid that
14:02:57 gibi sure make sense
14:03:17 elodilles gibi sean-k-mooney : I guess these would land on master and then backported on the most recent stable branch, am I right?
14:05:06 elodilles (hmmm, it looks quite heavy, according to its parent: tempest-multinode-full-py3)
14:07:10 gibi personally I would take it on master first
14:16:28 sean-k-mooney elodilles: well it need 2 nodes but it does not run run all the tempest test becasue we use the regex to limit it
14:16:59 sean-k-mooney tox_envlist: all
14:17:01 sean-k-mooney tempest_concurrency: 1
14:17:03 sean-k-mooney tempest_test_regex: ^whitebox_tempest_plugin\.
14:17:30 sean-k-mooney so i just ues that job to set up 2 node devstack with tempest then we just run the test from the plugin
14:20:58 elodilles oh, i see, i missed that
14:21:22 sean-k-mooney we proably could inherit form something better to make that more obvious
14:26:12 elodilles well, when someone reviews it thoroughly i think it'll be obvious o:) but that's true that at first glance the tempest-multinode-*full*-py3 suggests some time and resource heavy test job o:)
14:26:57 sean-k-mooney we proably can just use devstack-tempest ill see if there is a better job we can use in the future. that is a simple fix
15:20:12 ganso elodilles, lyarwood: Hi! If you have a spare minute could you please take a quick look at the backport now for victoria? it is clean and same as the one for wallaby from last week. Thanks in advance! https://review.opendev.org/c/openstack/nova/+/806004
15:22:44 opendevreview Ghanshyam proposed openstack/nova master: Convert features not supported error to HTTPBadRequest https://review.opendev.org/c/openstack/nova/+/806294
15:46:03 elodilles ganso: +2'd. Thanks for the backport! (fyi, lyarwood is on holiday today)
15:47:09 ganso elodilles: thanks! I will ping him tomorrow =)
15:50:11 elodilles no problem :)
16:06:20 opendevreview Merged openstack/nova master: tests: Validate AZ values https://review.opendev.org/c/openstack/nova/+/801523
17:05:36 opendevreview Merged openstack/nova master: Add force kwarg to delete_allocation_for_instance https://review.opendev.org/c/openstack/nova/+/688802
17:05:45 sean-k-mooney :)
17:15:39 opendevreview Merged openstack/nova master: Prevent deletion of a compute node belonging to another host https://review.opendev.org/c/openstack/nova/+/694802
17:15:55 opendevreview Merged openstack/nova master: Fix inactive session error in compute node creation https://review.opendev.org/c/openstack/nova/+/695189
17:16:17 opendevreview Merged openstack/nova master: Reduce mocking in test_reject_open_redirect for compat https://review.opendev.org/c/openstack/nova/+/803091
17:16:25 opendevreview Merged openstack/nova master: extend_volume of libvirt/volume/iscsi should not use device_path https://review.opendev.org/c/openstack/nova/+/801003
17:38:03 opendevreview sean mooney proposed openstack/nova stable/victoria: address open redirect with 3 forward slashes https://review.opendev.org/c/openstack/nova/+/806626
17:39:28 sean-k-mooney gibi: elodilles by the way are we goign to backport https://review.opendev.org/c/openstack/nova/+/688802
17:56:40 opendevreview sean mooney proposed openstack/nova stable/ussuri: address open redirect with 3 forward slashes https://review.opendev.org/c/openstack/nova/+/806628
18:03:58 opendevreview sean mooney proposed openstack/nova stable/train: address open redirect with 3 forward slashes https://review.opendev.org/c/openstack/nova/+/806629
19:51:45 gibi sean-k-mooney: I don't konw. The consumer types feature is only on master so we have a smaller issue on stable. And that smaller issue there since stein if I remember correclty.

Earlier   Later