Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2022-05-05
10:22:13 kashyap sean-k-mooney: Thanks; so it's not just me.
10:24:41 sean-k-mooney kashyap: i think our unit test could pass but i know that there are some issue so i have not looked at your spcific failure but if its a devstack run i woudl expect it to fail currenlty
10:41:15 gibi but the current failue is somewhere in the job setup as it fails on mysql :)
10:50:35 frickler gibi: ah, these nice "unit" tests with external dependencies. py310 currently runs on f35, maybe your setup is ubuntu specific?
10:51:00 gibi frickler: yeah that seems to be the issue
10:51:05 frickler devstack on 22.04 runs the base services mostly without issues except for horizon
10:51:42 frickler https://review.opendev.org/q/topic:add-jammy2
11:26:20 ralonsoh sean-k-mooney, hi! I'm playing again with live-migration and ceph
11:26:30 ralonsoh and, of course, during the compute installation I had this
11:26:31 ralonsoh 2022-05-05T11:18:22.104+0000 7fcd2a59c700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2]
11:26:47 ralonsoh I've copied the ceph ID from the controller (that is also a compute)
11:28:14 ralonsoh sean-k-mooney, never mind! what I was missing was the keyring files
11:28:24 ralonsoh I've copied them and now it works heheheh
11:31:53 sean-k-mooney cool
11:32:19 opendevreview Merged openstack/nova master: Fix segment-aware scheduling permissions error https://review.opendev.org/c/openstack/nova/+/839361
11:32:22 sean-k-mooney by the way you might find https://github.com/SeanMooney/ansible_role_devstack interesting/helpful
11:33:27 sean-k-mooney the ceph support curerntly only works on ubuntu since the devstack pluging currently does not have c9s support
13:16:25 sean-k-mooney dansmith: by the way did anything come of the excessivly high keystone and neutron load on the db
13:16:58 sean-k-mooney from your perfoamcne data patches
13:34:48 opendevreview Merged openstack/nova master: Allow claiming PCI PF if child VF is unavailable https://review.opendev.org/c/openstack/nova/+/838555
14:03:58 dansmith sean-k-mooney: I brought it up to the keystone people a couple days ago, and it was agreed that it was too high, but nothing more than that (yet)
14:04:43 sean-k-mooney ack so they did not see any smoking gun or anything else obviously wrong
14:11:08 dansmith well, nothing obvious off the top of their head, I dunno that anyone has done any real digging yet
14:41:55 ralonsoh sean-k-mooney, sorry again. Did you see something like this https://paste.opendev.org/show/bEMWSLCst6bhd4Ir3zwl/?
14:42:15 ralonsoh one of the nova-compute services is exiting
14:42:25 ralonsoh in a non very elegant way
14:42:28 sean-k-mooney ew segfault
14:42:46 ralonsoh this is the controller-compute node
14:42:47 sean-k-mooney no that is new to me
14:42:50 ralonsoh ok
14:43:01 ralonsoh I'll try to re install everything
14:43:03 sean-k-mooney where was that thrown
14:43:12 ralonsoh during, of course, the live migration
14:43:13 sean-k-mooney was it ci or a local vm/server
14:43:20 ralonsoh when I'm bringing the VM to this host
14:43:37 ralonsoh when the VM evacuates this host, all is OK
14:43:52 ralonsoh no sorry, it is when the VM evacuates the host
14:44:05 sean-k-mooney evacuate or live migrate
14:44:11 sean-k-mooney they are very differnt things
14:44:17 ralonsoh yeah sorry
14:44:19 ralonsoh live-migrate
14:44:36 ralonsoh when the VM is leaving the host
14:44:37 sean-k-mooney so you live migrate and then the source host segfaults?
14:44:56 sean-k-mooney are there any OOM errors
14:45:03 sean-k-mooney or other detail
14:45:09 ralonsoh let me check
14:45:22 ralonsoh nope, I have still 13GB free
14:45:32 sean-k-mooney like the segfault appears to by in python but im wondering if an allcoation failed because you ran out of memory
14:45:34 sean-k-mooney ok
14:52:14 opendevreview Balazs Gibizer proposed openstack/nova master: Adapt tools/test-setup to Fedora 35 https://review.opendev.org/c/openstack/nova/+/840684
14:52:51 gibi frickler: I try to unblock openstack-tox-py310 job
14:52:57 gibi with ^^
14:56:11 kashyap gibi: Thank you!
14:58:22 opendevreview Rico Lin proposed openstack/nova-specs master: Add vIOMMU device support for libvirt driver https://review.opendev.org/c/openstack/nova-specs/+/840310
15:59:15 opendevreview Takashi Natsume proposed openstack/python-novaclient master: Replace old URLs with new ones https://review.opendev.org/c/openstack/python-novaclient/+/840693
16:10:22 gibi frickler, kashyap: my google foo failed me to figure out why the mysql password change fails on fedora 35 in openstack-tox-py310 So if you have ideas please shoot https://review.opendev.org/c/openstack/nova/+/840684/1#message-ffd5ac00ca235cfaebd12988b65dbc210c2b9ec8
16:15:41 clarkb gibi: seems like your mysqladmin tool isn't compatible with mariadb
16:15:47 clarkb since it is generating the sql that fails
16:20:37 mnaser hrm
16:20:52 gibi hm, mariadb is on version 10.5 but mysqlclient and mysqladmin is on 8.0.28 but I'm not sure how to map these verison
16:21:16 mnaser i've got a really weird situation where a hypervisor stops getting vms scheduled to it
16:21:34 mnaser i checked `openstack resource provider inventory list 9fe525d9-df51-41c2-8ca7-8344ed0eee39` and that shows the resources available, and `openstack allocation candidate list --resource VCPU=4 --resource DISK_GB=64 --resource MEMORY_MB=2048 | grep 9fe525d9-df51-41c2-8ca7-8344ed0eee39` even returns that
16:21:51 mnaser so its not placement
16:21:56 clarkb gibi: mysqlclient and mysqladmin are probably the mysql tools and not the mariadb tools? Possible that mariadb has alternatives
16:22:47 gibi clarkb: yeah that make sense... try to figure out where are those alternatives
16:22:58 mnaser the filters in use are: "ComputeFilter, AggregateTypeAffinityFilter, ComputeCapabilitiesFilter, PciPassthroughFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter" -- i dont think the rest are relevant in this scenario
16:24:38 mnaser hrm, a bunch of exceptions with relation to libvirt before it fully stopped to deploy new systems
16:24:48 mnaser https://www.irccloud.com/pastebin/n7Ezs7aB/
16:25:49 mnaser and if i try to provision an instance on them manually (by using `--host` .. it goes up fine)
16:26:22 mnaser and now that i've actually provisioned an instance, the vms have started to flow in the hyperivsor agian
16:26:24 clarkb gibi: side note: https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/840545
16:27:14 gibi clarkb: yeah that can be a way out :)
16:28:44 opendevreview melanie witt proposed openstack/placement stable/wallaby: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840701
16:29:51 opendevreview melanie witt proposed openstack/placement stable/victoria: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840702
16:30:18 opendevreview melanie witt proposed openstack/placement stable/ussuri: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840703
16:30:45 opendevreview melanie witt proposed openstack/placement stable/train: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840704
16:40:56 opendevreview Balazs Gibizer proposed openstack/nova master: Adapt tools/test-setup to Fedora 35 https://review.opendev.org/c/openstack/nova/+/840684
16:45:16 sean-k-mooney oh we have 22.04 in nodepool now
16:45:19 sean-k-mooney cool
16:45:46 sean-k-mooney i think the mysql/mariadb changes were also in devstack
16:45:55 sean-k-mooney we use mariadb on most distos i think now
16:48:45 clarkb sean-k-mooney: it is a bit of a slwo rollout while we work through various things, but ya the images are up and mostly work. The last thing we ran into was phased package updates not making sense for us
16:49:31 sean-k-mooney phased package updates?
16:49:39 sean-k-mooney as in replication to mirrors or something else
16:50:36 clarkb something else. Its new functionality in apt that hashes something about your host and then modulo's that against the percentage of users they want to install the package
16:51:00 clarkb which means they can say things like 10% of users get this package update. Then next week change it to 50% and so on until it is 100%
16:51:30 clarkb but reprepro doesn't understand it and it is disabled by default in chroots (which dib uses to make the images) which means you get the latest available packges as if phases didn't exist at all
16:57:36 Uggla sean-k-mooney, can you have a look at my comment on https://review.opendev.org/c/openstack/nova-specs/+/831506/2/specs/zed/approved/unshelve-to-host.rst#42 ? Please let me know what you think about it.
16:59:04 sean-k-mooney clarkb: oh ok
16:59:18 sean-k-mooney clarkb: ya i think we would want to turn that off
16:59:45 mnaser ok, this is most def a thread leak
17:00:15 sean-k-mooney Uggla i saw that breifly
17:00:34 mnaser kill -USR2 <nova-pid>
17:00:35 sean-k-mooney Uggla: so for unshelve ot az it ends up updating the request spec to the requested az
17:00:40 mnaser grep -i 'Green Thread' /tmp/gar | wc -l => 6094
17:01:04 sean-k-mooney mnaser: those are the userland trhead not real os thread
17:01:21 mnaser a lot of them are this:
17:01:25 mnaser https://www.irccloud.com/pastebin/YBiYPlrp/
17:01:31 mnaser and i think the root cause is from https://www.irccloud.com/pastebin/n7Ezs7aB/
17:02:22 sean-k-mooney Uggla: so for unshelve ot host i think it shoudl also update it to the AZ of the host if and only if the request spec is not none

Earlier   Later