| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-08-30 | |||
| 08:39:45 | abhishekk | gibi, py38 post failure for https://review.opendev.org/c/openstack/nova/+/688802, could you please dd recheck ? | |
| 10:01:54 | elodilles | gibi: could you please have a quick look at this placement release patch for stable/ussuri? (it's a generated patch to avoid release rush around EM transition): https://review.opendev.org/c/openstack/releases/+/802110 | |
| 10:07:12 | gibi | elodilles: ack I will check | |
| 10:09:25 | gibi | abhishekk feel free to recheck next time | |
| 10:11:15 | gibi | elodilles: done and thanks | |
| 10:12:18 | elodilles | gibi: thanks \o/ | |
| 12:21:37 | gibi | the force kwargs patch https://review.opendev.org/c/openstack/nova/+/688802 bounced from the gate due to bug 1912310, I've requeued it | |
| 12:24:06 | sean-k-mooney | what causes https://bugs.launchpad.net/nova/+bug/1912310 | |
| 12:25:16 | gibi | I saw libvirt internal errors like | |
| 12:25:17 | gibi | 2021-07-30 08:56:25.528+0000: 57632: error : virProcessRunInFork:1159 : internal error: child reported (status=125): unable to open /dev/sda: No such device or address | |
| 12:26:10 | sean-k-mooney | ok so it looks like its actully libvirt that is having issue not nova connecting to it | |
| 12:26:16 | gibi | yepp | |
| 12:26:21 | gibi | as far as I understand | |
| 12:27:04 | gibi | there are also occasions with | |
| 12:27:05 | gibi | virKeepAliveTimerInternal:137 : internal error: connection closed due to keepalive timeout | |
| 12:27:14 | sean-k-mooney | we are not seeing any OOM events or anythying else strange on the node at the time are we | |
| 12:27:53 | gibi | I just like two occurence and found no such thing | |
| 12:27:58 | gibi | I just checked like | |
| 12:28:20 | gibi | the nova-live-migration job set to non-voting due to this | |
| 12:28:29 | gibi | but it seems we can hit the same in nova-next too | |
| 12:28:34 | gibi | but a lot less frequently | |
| 12:29:14 | sean-k-mooney | yep if failind in nova-net in this case | |
| 12:29:16 | sean-k-mooney | https://zuul.opendev.org/t/openstack/build/f888b58ca23f49fc8f9046e9c2ad18a0/log/controller/logs/screen-dstat.txt | |
| 12:29:37 | gibi | yes | |
| 12:29:48 | gibi | that is basically a first time I see it in nova-next | |
| 12:29:51 | sean-k-mooney | we got donw to 120MB a few times but i dont see any really evidence of memory issues so likely not the kernel randomly killing things | |
| 12:32:07 | gibi | around the time of the failure we were floating around 300MB free | |
| 12:33:52 | sean-k-mooney | ya its unlikely to be the cause but we have seen OOM issue break libvirt and other process in weird ways before. | |
| 12:34:20 | gibi | true, oom can cause weird thing | |
| 12:34:22 | gibi | s | |
| 12:35:09 | sean-k-mooney | ill quickly check the cloud archive | |
| 12:35:19 | sean-k-mooney | perhaps there is a newer libvirt avaiable we coudl use instead | |
| 12:36:07 | gibi | didn't we use the max available? | |
| 12:37:00 | sean-k-mooney | well im not sure we are using the xena cloud archive currently | |
| 12:37:14 | sean-k-mooney | but looking at it they are not shiping libvirt/qemu in the cloud archive currently | |
| 12:37:41 | sean-k-mooney | we tyically dont use the most recent cloud archive version | |
| 12:39:46 | sean-k-mooney | so ya looks like we are using 6.x for ubuntu "libvirt0:amd64 6.0.0-0ubuntu8.12" | |
| 12:40:15 | sean-k-mooney | on centos stream with the advance virt modulee we would be useing 7.x.y | |
| 12:43:18 | sean-k-mooney | its a long shot but we could enable this ppa as a test to see if that would resolve it. its the one i use when i need newer libvirt on ubutu but dont want to build from source | |
| 12:43:22 | sean-k-mooney | https://launchpad.net/~jacob/+archive/ubuntu/virtualisation | |
| 12:43:57 | sean-k-mooney | althopugh that still only provides 6.6.0-1ubuntu2~ppa0 | |
| 12:44:04 | sean-k-mooney | not 7.x | |
| 12:45:09 | gibi | I'm not sure how can we enable this in infra but feel free to go ahead. We can use the nova-live-migration job as canary as that is now non-voting but still run for almost all of our patches | |
| 12:45:49 | sean-k-mooney | ya i might porpose a DNM patch just to see if that works. if it does it means we need to talk to canonical about a missing backport | |
| 12:46:04 | sean-k-mooney | proablem is i have no idea what is missing | |
| 12:46:42 | gibi | cool, good ide | |
| 12:46:44 | gibi | idea | |
| 12:46:53 | sean-k-mooney | the other alternitive would be to move form ubuntu 20.04 to 21.04 or to centos 8 on the affected jobs | |
| 12:49:01 | sean-k-mooney | well there is another alternitive too which is complie libvirt/qemu form source which i have a devstack plugin to do but i would prefer to avoid that mainly due to extra job time. its not hard to do but if we can just use distro pacakages in this case its nicer | |
| 12:49:17 | gibi | as we declare our supported distros beforhand of the release I would go with trying to fix ubuntu 20.04 https://governance.openstack.org/tc/reference/runtimes/xena.html | |
| 12:50:26 | sean-k-mooney | yes although centos 8 stream is vaild too. but ya ill see if i can look into this a little later today. ill propose a couple of different patches for different options. | |
| 12:51:20 | sean-k-mooney | enableing "sudo add-apt-repository ppa:jacob/virtualisation" in a pre playbook is simple as is changing the base os to centos 8 stream | |
| 12:51:45 | sean-k-mooney | the other options are more complicated but doable | |
| 13:06:59 | opendevreview | Merged openstack/nova master: Functional tests removed direct post call https://review.opendev.org/c/openstack/nova/+/766068 | |
| 13:22:30 | sean-k-mooney | gibi: by the way we maintain a tempest plugin called whitebox that looks at some of the internals of how nova works and assert that it does the right thing. would you have any object to me enableing that for a subset of nova changes at least in a non voting capasity initaly? | |
| 13:22:48 | sean-k-mooney | gibi: i was thinking of making it run on change to the libvirt driver and hardware.py | |
| 13:23:20 | gibi | sean-k-mooney: I have no problem with it if it is actively maintained and won't take up much of the CI resources | |
| 13:25:11 | sean-k-mooney | yes its maintianed and runs downstream we also maintin the devstack support upstream | |
| 13:25:33 | sean-k-mooney | gibi: upstream many of the test are disabled because we dont have the hardware https://opendev.org/openstack/whitebox-tempest-plugin/src/branch/master/whitebox_tempest_plugin/api/compute | |
| 13:26:01 | sean-k-mooney | i.e. we can run the pmem, sriov or vgpu test in the gate | |
| 13:26:29 | sean-k-mooney | i know we were lucking to see if we coudl use this for third party ci but we still are having problems finding hardware internally to run it | |
| 13:27:46 | opendevreview | Balazs Gibizer proposed openstack/nova master: Add two new hacking rules https://review.opendev.org/c/openstack/nova/+/805668 | |
| 13:28:11 | gibi | just based on the test file names even without special hardware this plugin has useful coverage | |
| 13:28:45 | sean-k-mooney | yep it has all the test that were orginailly don by the intel thridpary nfv ci in it but updated | |
| 13:28:51 | sean-k-mooney | and some other test coverage | |
| 13:31:10 | gibi | then lets enable it | |
| 13:32:40 | sean-k-mooney | gibi: when the qe member of the comptue team downstream writes test automation that is not suitable for upstream tempest becasue it depens on speicic configuration of the serives this is where we try to add the test coverage. | |
| 13:33:07 | sean-k-mooney | like testing adding cpu flags which we can do https://opendev.org/openstack/whitebox-tempest-plugin/src/branch/master/.zuul.yaml#L53-L55 in the ci like this | |
| 13:34:22 | gibi | I agree to have that coverage in our upstream gate | |
| 13:34:41 | sean-k-mooney | thanks ill let artom know and see if he wants me to wait for the jobs to be split or not first ill start on the WIP patch in anycase | |
| 13:43:21 | artom | Huh, happy coincidence, I was pondering proposing a periodic whitebox job for Nova | |
| 13:43:31 | artom | So, I think it's not yet stable enough for that, actually | |
| 13:43:52 | artom | We think we know the issue, and we're working on it, but until then I'm not sure it's ready yet | |
| 13:45:13 | artom | Every so often, depending on which order tests end up being executed, what we think happens is we attempt to reshape from cpu_dedicated_set to vcpu_pin_set, and that's not allowed, so there's a cascading failure. There are also around how we use admin clients and clean up after ourselves, that can also cause cascading failures | |
| 13:45:17 | gibi | artom: nothing is urgent from upstream perspective. If upstream feedback helps then I'm OK to enable a non voting job | |
| 13:46:01 | artom | gibi, I think even that's premature, as the solution to ^^^ is to change whitebox's own job a bit, so until that's done, let's not add it to nova | |
| 13:46:50 | sean-k-mooney | artom: ok the reason i was bring this up was we did at one point plan to enabel white box for wallaby | |
| 13:47:04 | sean-k-mooney | then we did not have time to actully get it stable in time | |
| 13:47:15 | sean-k-mooney | so i was hoping we could do thatbefore the end of xena | |
| 13:47:24 | sean-k-mooney | if you think its not ready however we can hold off | |
| 13:47:24 | artom | Ah, probably not before the end of Xena | |
| 13:47:31 | artom | ... well, does end == FF? | |
| 13:47:33 | artom | Or release? | |
| 13:48:15 | sean-k-mooney | well i guess i twas thinking before RC1 when stable branch is created | |
| 13:48:34 | sean-k-mooney | although if we were ok with backporting enableing the testing on the stable branch end could be anytime before eol i guess | |
| 13:49:27 | sean-k-mooney | if we dont think it ready however no need to rush | |
| 13:50:00 | sean-k-mooney | i would just like to keep making progress on getting this test coverage enable eithe firstparty or third party | |
| 13:51:11 | artom | Third party I still haven't solved the hardware problem :) | |
| 13:51:13 | artom | Err :( | |
| 13:51:20 | sean-k-mooney | artom: is bauzas back today or is he retruning tomorow | |
| 13:51:31 | sean-k-mooney | artom: yep i know :) | |
| 13:52:19 | artom | Still on PTO today, according to Workday | |
| 13:52:42 | gibi | I personally OK with enabling new jobs on stable but I guess elodilles or lyarwood has more authority about that :) | |
| 13:53:10 | gibi | as per landing it on master, this is not a feature so RC1 is the cut of date due to branching | |
| 13:53:39 | artom | I can try to hurry it up, especially as jparker seems to have more time for this right about now, too | |
| 13:54:58 | sean-k-mooney | gibi: before i recheck are there any gate blockers i should hold off for | |
| 13:55:24 | sean-k-mooney | i was just looking at the failure in bauzas mdev series which dont seam related | |
| 13:56:04 | gibi | sean-k-mooney: the "Add force kwarg to delete_allocation_for_instance" not landed yet that kills at least 1/4 of the tempest jobs all around the gate | |
| 13:56:16 | sean-k-mooney | ah right | |
| 13:56:20 | gibi | I don't know about any full blocker | |