Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-11-06
15:44:50 openstackgerrit Matt Riedemann proposed openstack/nova stable/stein: Add --dry-run option to heal_allocations CLI https://review.opendev.org/693199
15:44:51 openstackgerrit Matt Riedemann proposed openstack/nova stable/stein: Add BFV wrinkle to TestNovaManagePlacementHealAllocations https://review.opendev.org/693201
15:44:51 openstackgerrit Matt Riedemann proposed openstack/nova stable/stein: Add --instance option to heal_allocations https://review.opendev.org/693200
15:46:28 sean-k-mooney well now i know where to look and how that works. im off to lunch brb
15:53:27 openstackgerrit Boris Bobrov proposed openstack/nova master: Also enable iommu for virtio controllers and video in libvirt https://review.opendev.org/684825
15:53:28 openstackgerrit Boris Bobrov proposed openstack/nova master: Create a controller for qga when SEV is used https://review.opendev.org/693072
16:04:06 openstackgerrit Boris Bobrov proposed openstack/nova master: Move rng device checks to the appropriate method https://review.opendev.org/693210
16:07:13 efried Nov 06 09:22:55 breakfast dnsmasq[27199]: exiting on receipt of SIGTERM
16:07:13 efried Nov 06 09:19:25 breakfast dnsmasq-dhcp[27199]: read /var/lib/libvirt/dnsmasq/default.hostsfile
16:07:13 efried The libvirt-bin service is inactive. There's no journal, but systemctl shows these as the last two lines:
16:07:13 efried '/var/run/libvirt/libvirt-sock': No such file or directory.
16:07:13 efried sean-k-mooney: back to n-cpu failing to start because of
16:07:14 efried I don't see anything in the stack logs that looks like it ought to have shut the service down.
16:08:22 efried On both stacks the systemctl status shows that ~3.5min gap between loading default.hostsfile and SIGTERM
16:13:43 efried sean-k-mooney: I was able to manually start libvirt-bin and it hasn't stopped itself, so clearly something else was hitting it...
16:14:50 efried ...but with the service running, n-cpu still fails to come up in the same way.
16:21:40 openstackgerrit Boris Bobrov proposed openstack/nova master: Create a controller for qga when SEV is used https://review.opendev.org/693072
16:21:40 openstackgerrit Boris Bobrov proposed openstack/nova master: Also enable iommu for virtio controllers and video in libvirt https://review.opendev.org/684825
16:22:26 mriedem amodi: melwitt: looks like we can keep the iso file without changes https://review.opendev.org/#/c/693065/7
16:22:41 mriedem it was already fixed in the manuals repo before i imported that doc https://github.com/openstack/openstack-manuals/commit/1206b9b727594ae04f0d2a4818f183e95253b464
16:22:43 openstackgerrit Merged openstack/nova master: Pass exception through TaskBase.rollback https://review.opendev.org/692689
16:25:55 openstackgerrit Matt Riedemann proposed openstack/nova master: Revert "openstack server create" to "nova boot" in nova docs https://review.opendev.org/693065
16:27:00 mriedem this is why i'm not as free willy on accepting any old docs change since it's a pain in the butt to fix them if they are wrong
16:30:06 melwitt haha free willy
16:31:28 melwitt and yeah, understood. thanks for pointing out those previous fixes and updating it
16:34:12 dansmith mriedem: query
16:34:27 dansmith was just looking at your heal_instance_mappings thing
16:34:32 dansmith and was thinking.
16:35:57 dansmith I wonder if it wouldn't make sense for us to have another totally optional service with the sole purpose of looking for things that need healing.. instance mappings, maybe allocations, host mappings, maybe even archive and purge
16:36:31 dansmith we could make it a daemon, or something you could run from cron single-shot if you just wanted to run it at midnight or something
16:36:41 melwitt I think operators would love that
16:37:15 dansmith instead of a ton of cron entries for all those things, and not knowing you need new cron entries to run new things on a new release,
16:37:25 dansmith you just run one nova-audit in one place, no HA, just low and slow burn of audity things
16:37:40 melwitt yeah, would be nice
16:38:43 mriedem i'm not opposed
16:39:40 mriedem would that also run the discover host mappings thing that the scheduler can run today?
16:40:09 mriedem i remember glance has several things like that https://github.com/openstack/glance/blob/master/setup.cfg#L34
16:40:17 mriedem though i'm not sure if they are all daemons
16:40:22 dansmith mriedem: yeah, hence the host mappings in my list
16:41:41 mriedem i will admit the number of heal/discover/sync things is getting a bit overwhelming
16:41:49 dansmith right
16:42:00 dansmith impossible to track for the average operator I'm sure
16:42:03 mriedem and we have talked about a periodic archive/purge thing in the past
16:42:25 dansmith and we put the host mappings thing in scheduler mostly because the cardinality was small
16:42:38 dansmith (and then I subsequently solved their problem a different way)
16:42:59 mriedem and i think b/c api and conductor didn't have periodic tasks already but scheduler did
16:43:53 dansmith maybe, but the point was to run one or few of those since they'd step all over each other, and there'd be too many conductors and apis even if they did have periodics
16:43:54 mriedem maybe you want to kick that idea out into the mailing list tagged with [nova][ops] to get initial feedback? and/or just start a spec and kick that into the ML to get attention from operators.
16:44:23 mriedem we do run nproc/2 scheduler works by default now i think..
16:44:24 mriedem but yeah
16:44:28 dansmith yeah, I could start with a spec
16:44:38 dansmith I was kinda thinking of just starting with code
16:45:04 mriedem now that you've gotten a taste from the image cache stuff you're addicted!
16:45:55 dansmith JUST ONE MORE HIT MAN
16:48:44 efried Nov 06 10:45:03 breakfast nova-compute[15654]: ERROR nova.virt.libvirt.host File "/usr/local/lib/python3.6/dist-packages/libvirt.py", line 4190, in getVersion
16:48:44 efried If I start n-cpu in that window, I run into the qemu version check. I added some debug and find it's coming from:
16:48:44 efried sean-k-mooney: The libvirt-bin service seems to not want to stay alive. When I start it manually, it runs for a couple minutes and then dies, but doesn't log anything.
16:48:45 efried Nov 06 10:45:03 breakfast nova-compute[15654]: ERROR nova.virt.libvirt.host libvirt.libvirtError: internal error: Cannot find suitable emulator for x86_64
16:48:45 efried Nov 06 10:45:03 breakfast nova-compute[15654]: ERROR nova.virt.libvirt.host if ret == -1: raise libvirtError ('virConnectGetVersion() failed', conn=self)
16:50:16 sean-k-mooney is qemu-system-x86_64
16:50:19 sean-k-mooney in your path
16:50:40 sean-k-mooney also do you have permission denied errors in dmesg
16:51:12 openstackgerrit Merged openstack/nova master: Revert "openstack server create" to "nova boot" in nova docs https://review.opendev.org/693065
16:51:28 efried [73117.887250] audit: type=1400 audit(1573058703.103:556): apparmor="DENIED" operation="exec" profile="/usr/sbin/libvirtd" name="/usr/local/bin/qemu-system-microblazeel" pid=15694 comm="libvirtd" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
16:51:28 efried you mean like this
16:51:35 sean-k-mooney yes
16:51:46 sean-k-mooney so apparmor is blocking it
16:51:48 efried (yes, qemu-[system-]x86_64 is in /usr/local/bin, which is in my PATH, at least on the CLI)
16:52:07 efried oh, I thought that libvirtd.conf hack was supposed to get around apparmor
16:52:08 sean-k-mooney ya so if we fix the apparmor issue it shoudl be working
16:52:30 sean-k-mooney no it fixs the fact libvirt expect polkit to be configure to use apparmor
16:52:43 sean-k-mooney i really should just add teh compile flag
16:52:53 efried is that easy?
16:53:35 sean-k-mooney i think its just https://review.opendev.org/#/c/693094/3/devstack/libs/libvirt@93
16:53:54 sean-k-mooney add --with-apparmor_profiles after --system
16:54:16 sean-k-mooney how im working around it at the moment was https://review.opendev.org/#/c/693094/3/README.rst
16:54:29 sean-k-mooney sudo ln -s /etc/apparmor.d/usr.sbin.libvirtd /etc/apparmor.d/disable/usr.sbin.libvirtd
16:54:30 sean-k-mooney then reboot for it to take effect.
16:54:40 efried ah
16:54:45 efried I'll try it both ways...
16:55:49 sean-k-mooney apparmor is the security sandboxing tool ubuntu use. its not happy that the binaries are in a different location then normal e.g. /usr/local/
16:55:57 sean-k-mooney hence teh error
16:56:52 efried can't we `make install` with the right location?
16:57:09 sean-k-mooney yes we can just change the prefix to /usr
16:57:18 sean-k-mooney i have done that for libvirt actully
16:57:25 sean-k-mooney just not qemu
17:00:25 mriedem huh, i just saw the 'timed out waiting for response from cell' error in a functional test https://zuul.opendev.org/t/openstack/build/63001bbd58c244cea70c995f1ebf61fb/log/job-output.txt#3092
17:00:28 sean-k-mooney https://review.opendev.org/#/c/693094/3/devstack/libs/libvirt@113 this would have to be updated to "./configure --prefix=/usr"
17:01:46 sean-k-mooney in test_evacuate_with_soft_anti_affinity
17:02:14 mriedem i don't think that matters,
17:02:31 mriedem we're just calling GET /servers/server_id to get the status probably
17:02:32 mriedem WARNING [oslo.service.loopingcall] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 59.77 sec
17:02:47 sean-k-mooney likely not but just was wondering which one
17:02:56 mriedem chances are this is another infra node that hit some crazy io issue
17:03:28 openstack Launchpad bug 1851497 in OpenStack Compute (nova) "rng is only enabled if qemu guest agent is enabled" [Medium,Triaged] - Assigned to sean mooney (sean-k-mooney)
17:03:28 sean-k-mooney mriedem: oh that reminds me https://bugs.launchpad.net/nova/+bug/1851497
17:03:57 sean-k-mooney the random number generator code has apparently been partly broken since queens ...
17:04:27 sean-k-mooney mriedem: did ye nail down the gate issue to io
17:04:46 mriedem sean-k-mooney: https://review.opendev.org/#/c/129325/ was kilo
17:04:50 sean-k-mooney if the vm runs out of entrapy it will have issue connecting to ssl sockets

Earlier   Later