| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-05-05 | |||
| 10:20:00 | sean-k-mooney | since we dont really use eventlets there the same way | |
| 10:20:04 | sean-k-mooney | just an fyi | |
| 10:20:37 | sean-k-mooney | deian are in the process of moving debian testing to 3.10 and have been having issue with this lately | |
| 10:22:13 | kashyap | sean-k-mooney: Thanks; so it's not just me. | |
| 10:24:41 | sean-k-mooney | kashyap: i think our unit test could pass but i know that there are some issue so i have not looked at your spcific failure but if its a devstack run i woudl expect it to fail currenlty | |
| 10:41:15 | gibi | but the current failue is somewhere in the job setup as it fails on mysql :) | |
| 10:50:35 | frickler | gibi: ah, these nice "unit" tests with external dependencies. py310 currently runs on f35, maybe your setup is ubuntu specific? | |
| 10:51:00 | gibi | frickler: yeah that seems to be the issue | |
| 10:51:05 | frickler | devstack on 22.04 runs the base services mostly without issues except for horizon | |
| 10:51:42 | frickler | https://review.opendev.org/q/topic:add-jammy2 | |
| 11:26:20 | ralonsoh | sean-k-mooney, hi! I'm playing again with live-migration and ceph | |
| 11:26:30 | ralonsoh | and, of course, during the compute installation I had this | |
| 11:26:31 | ralonsoh | 2022-05-05T11:18:22.104+0000 7fcd2a59c700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] | |
| 11:26:47 | ralonsoh | I've copied the ceph ID from the controller (that is also a compute) | |
| 11:28:14 | ralonsoh | sean-k-mooney, never mind! what I was missing was the keyring files | |
| 11:28:24 | ralonsoh | I've copied them and now it works heheheh | |
| 11:31:53 | sean-k-mooney | cool | |
| 11:32:19 | opendevreview | Merged openstack/nova master: Fix segment-aware scheduling permissions error https://review.opendev.org/c/openstack/nova/+/839361 | |
| 11:32:22 | sean-k-mooney | by the way you might find https://github.com/SeanMooney/ansible_role_devstack interesting/helpful | |
| 11:33:27 | sean-k-mooney | the ceph support curerntly only works on ubuntu since the devstack pluging currently does not have c9s support | |
| 13:16:25 | sean-k-mooney | dansmith: by the way did anything come of the excessivly high keystone and neutron load on the db | |
| 13:16:58 | sean-k-mooney | from your perfoamcne data patches | |
| 13:34:48 | opendevreview | Merged openstack/nova master: Allow claiming PCI PF if child VF is unavailable https://review.opendev.org/c/openstack/nova/+/838555 | |
| 14:03:58 | dansmith | sean-k-mooney: I brought it up to the keystone people a couple days ago, and it was agreed that it was too high, but nothing more than that (yet) | |
| 14:04:43 | sean-k-mooney | ack so they did not see any smoking gun or anything else obviously wrong | |
| 14:11:08 | dansmith | well, nothing obvious off the top of their head, I dunno that anyone has done any real digging yet | |
| 14:41:55 | ralonsoh | sean-k-mooney, sorry again. Did you see something like this https://paste.opendev.org/show/bEMWSLCst6bhd4Ir3zwl/? | |
| 14:42:15 | ralonsoh | one of the nova-compute services is exiting | |
| 14:42:25 | ralonsoh | in a non very elegant way | |
| 14:42:28 | sean-k-mooney | ew segfault | |
| 14:42:46 | ralonsoh | this is the controller-compute node | |
| 14:42:47 | sean-k-mooney | no that is new to me | |
| 14:42:50 | ralonsoh | ok | |
| 14:43:01 | ralonsoh | I'll try to re install everything | |
| 14:43:03 | sean-k-mooney | where was that thrown | |
| 14:43:12 | ralonsoh | during, of course, the live migration | |
| 14:43:13 | sean-k-mooney | was it ci or a local vm/server | |
| 14:43:20 | ralonsoh | when I'm bringing the VM to this host | |
| 14:43:37 | ralonsoh | when the VM evacuates this host, all is OK | |
| 14:43:52 | ralonsoh | no sorry, it is when the VM evacuates the host | |
| 14:44:05 | sean-k-mooney | evacuate or live migrate | |
| 14:44:11 | sean-k-mooney | they are very differnt things | |
| 14:44:17 | ralonsoh | yeah sorry | |
| 14:44:19 | ralonsoh | live-migrate | |
| 14:44:36 | ralonsoh | when the VM is leaving the host | |
| 14:44:37 | sean-k-mooney | so you live migrate and then the source host segfaults? | |
| 14:44:56 | sean-k-mooney | are there any OOM errors | |
| 14:45:03 | sean-k-mooney | or other detail | |
| 14:45:09 | ralonsoh | let me check | |
| 14:45:22 | ralonsoh | nope, I have still 13GB free | |
| 14:45:32 | sean-k-mooney | like the segfault appears to by in python but im wondering if an allcoation failed because you ran out of memory | |
| 14:45:34 | sean-k-mooney | ok | |
| 14:52:14 | opendevreview | Balazs Gibizer proposed openstack/nova master: Adapt tools/test-setup to Fedora 35 https://review.opendev.org/c/openstack/nova/+/840684 | |
| 14:52:51 | gibi | frickler: I try to unblock openstack-tox-py310 job | |
| 14:52:57 | gibi | with ^^ | |
| 14:56:11 | kashyap | gibi: Thank you! | |
| 14:58:22 | opendevreview | Rico Lin proposed openstack/nova-specs master: Add vIOMMU device support for libvirt driver https://review.opendev.org/c/openstack/nova-specs/+/840310 | |
| 15:59:15 | opendevreview | Takashi Natsume proposed openstack/python-novaclient master: Replace old URLs with new ones https://review.opendev.org/c/openstack/python-novaclient/+/840693 | |
| 16:10:22 | gibi | frickler, kashyap: my google foo failed me to figure out why the mysql password change fails on fedora 35 in openstack-tox-py310 So if you have ideas please shoot https://review.opendev.org/c/openstack/nova/+/840684/1#message-ffd5ac00ca235cfaebd12988b65dbc210c2b9ec8 | |
| 16:15:41 | clarkb | gibi: seems like your mysqladmin tool isn't compatible with mariadb | |
| 16:15:47 | clarkb | since it is generating the sql that fails | |
| 16:20:37 | mnaser | hrm | |
| 16:20:52 | gibi | hm, mariadb is on version 10.5 but mysqlclient and mysqladmin is on 8.0.28 but I'm not sure how to map these verison | |
| 16:21:16 | mnaser | i've got a really weird situation where a hypervisor stops getting vms scheduled to it | |
| 16:21:34 | mnaser | i checked `openstack resource provider inventory list 9fe525d9-df51-41c2-8ca7-8344ed0eee39` and that shows the resources available, and `openstack allocation candidate list --resource VCPU=4 --resource DISK_GB=64 --resource MEMORY_MB=2048 | grep 9fe525d9-df51-41c2-8ca7-8344ed0eee39` even returns that | |
| 16:21:51 | mnaser | so its not placement | |
| 16:21:56 | clarkb | gibi: mysqlclient and mysqladmin are probably the mysql tools and not the mariadb tools? Possible that mariadb has alternatives | |
| 16:22:47 | gibi | clarkb: yeah that make sense... try to figure out where are those alternatives | |
| 16:22:58 | mnaser | the filters in use are: "ComputeFilter, AggregateTypeAffinityFilter, ComputeCapabilitiesFilter, PciPassthroughFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter" -- i dont think the rest are relevant in this scenario | |
| 16:24:38 | mnaser | hrm, a bunch of exceptions with relation to libvirt before it fully stopped to deploy new systems | |
| 16:24:48 | mnaser | https://www.irccloud.com/pastebin/n7Ezs7aB/ | |
| 16:25:49 | mnaser | and if i try to provision an instance on them manually (by using `--host` .. it goes up fine) | |
| 16:26:22 | mnaser | and now that i've actually provisioned an instance, the vms have started to flow in the hyperivsor agian | |
| 16:26:24 | clarkb | gibi: side note: https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/840545 | |
| 16:27:14 | gibi | clarkb: yeah that can be a way out :) | |
| 16:28:44 | opendevreview | melanie witt proposed openstack/placement stable/wallaby: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840701 | |
| 16:29:51 | opendevreview | melanie witt proposed openstack/placement stable/victoria: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840702 | |
| 16:30:18 | opendevreview | melanie witt proposed openstack/placement stable/ussuri: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840703 | |
| 16:30:45 | opendevreview | melanie witt proposed openstack/placement stable/train: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840704 | |
| 16:40:56 | opendevreview | Balazs Gibizer proposed openstack/nova master: Adapt tools/test-setup to Fedora 35 https://review.opendev.org/c/openstack/nova/+/840684 | |
| 16:45:16 | sean-k-mooney | oh we have 22.04 in nodepool now | |
| 16:45:19 | sean-k-mooney | cool | |
| 16:45:46 | sean-k-mooney | i think the mysql/mariadb changes were also in devstack | |
| 16:45:55 | sean-k-mooney | we use mariadb on most distos i think now | |
| 16:48:45 | clarkb | sean-k-mooney: it is a bit of a slwo rollout while we work through various things, but ya the images are up and mostly work. The last thing we ran into was phased package updates not making sense for us | |
| 16:49:31 | sean-k-mooney | phased package updates? | |
| 16:49:39 | sean-k-mooney | as in replication to mirrors or something else | |
| 16:50:36 | clarkb | something else. Its new functionality in apt that hashes something about your host and then modulo's that against the percentage of users they want to install the package | |
| 16:51:00 | clarkb | which means they can say things like 10% of users get this package update. Then next week change it to 50% and so on until it is 100% | |
| 16:51:30 | clarkb | but reprepro doesn't understand it and it is disabled by default in chroots (which dib uses to make the images) which means you get the latest available packges as if phases didn't exist at all | |
| 16:57:36 | Uggla | sean-k-mooney, can you have a look at my comment on https://review.opendev.org/c/openstack/nova-specs/+/831506/2/specs/zed/approved/unshelve-to-host.rst#42 ? Please let me know what you think about it. | |
| 16:59:04 | sean-k-mooney | clarkb: oh ok | |
| 16:59:18 | sean-k-mooney | clarkb: ya i think we would want to turn that off | |
| 16:59:45 | mnaser | ok, this is most def a thread leak | |
| 17:00:15 | sean-k-mooney | Uggla i saw that breifly | |
| 17:00:34 | mnaser | kill -USR2 <nova-pid> | |
| 17:00:35 | sean-k-mooney | Uggla: so for unshelve ot az it ends up updating the request spec to the requested az | |
| 17:00:40 | mnaser | grep -i 'Green Thread' /tmp/gar | wc -l => 6094 | |
| 17:01:04 | sean-k-mooney | mnaser: those are the userland trhead not real os thread | |
| 17:01:21 | mnaser | a lot of them are this: | |