Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-11-06
16:37:15 dansmith instead of a ton of cron entries for all those things, and not knowing you need new cron entries to run new things on a new release,
16:37:25 dansmith you just run one nova-audit in one place, no HA, just low and slow burn of audity things
16:37:40 melwitt yeah, would be nice
16:38:43 mriedem i'm not opposed
16:39:40 mriedem would that also run the discover host mappings thing that the scheduler can run today?
16:40:09 mriedem i remember glance has several things like that https://github.com/openstack/glance/blob/master/setup.cfg#L34
16:40:17 mriedem though i'm not sure if they are all daemons
16:40:22 dansmith mriedem: yeah, hence the host mappings in my list
16:41:41 mriedem i will admit the number of heal/discover/sync things is getting a bit overwhelming
16:41:49 dansmith right
16:42:00 dansmith impossible to track for the average operator I'm sure
16:42:03 mriedem and we have talked about a periodic archive/purge thing in the past
16:42:25 dansmith and we put the host mappings thing in scheduler mostly because the cardinality was small
16:42:38 dansmith (and then I subsequently solved their problem a different way)
16:42:59 mriedem and i think b/c api and conductor didn't have periodic tasks already but scheduler did
16:43:53 dansmith maybe, but the point was to run one or few of those since they'd step all over each other, and there'd be too many conductors and apis even if they did have periodics
16:43:54 mriedem maybe you want to kick that idea out into the mailing list tagged with [nova][ops] to get initial feedback? and/or just start a spec and kick that into the ML to get attention from operators.
16:44:23 mriedem we do run nproc/2 scheduler works by default now i think..
16:44:24 mriedem but yeah
16:44:28 dansmith yeah, I could start with a spec
16:44:38 dansmith I was kinda thinking of just starting with code
16:45:04 mriedem now that you've gotten a taste from the image cache stuff you're addicted!
16:45:55 dansmith JUST ONE MORE HIT MAN
16:48:44 efried sean-k-mooney: The libvirt-bin service seems to not want to stay alive. When I start it manually, it runs for a couple minutes and then dies, but doesn't log anything.
16:48:44 efried If I start n-cpu in that window, I run into the qemu version check. I added some debug and find it's coming from:
16:48:44 efried Nov 06 10:45:03 breakfast nova-compute[15654]: ERROR nova.virt.libvirt.host File "/usr/local/lib/python3.6/dist-packages/libvirt.py", line 4190, in getVersion
16:48:45 efried Nov 06 10:45:03 breakfast nova-compute[15654]: ERROR nova.virt.libvirt.host if ret == -1: raise libvirtError ('virConnectGetVersion() failed', conn=self)
16:48:45 efried Nov 06 10:45:03 breakfast nova-compute[15654]: ERROR nova.virt.libvirt.host libvirt.libvirtError: internal error: Cannot find suitable emulator for x86_64
16:50:16 sean-k-mooney is qemu-system-x86_64
16:50:19 sean-k-mooney in your path
16:50:40 sean-k-mooney also do you have permission denied errors in dmesg
16:51:12 openstackgerrit Merged openstack/nova master: Revert "openstack server create" to "nova boot" in nova docs https://review.opendev.org/693065
16:51:28 efried you mean like this
16:51:28 efried [73117.887250] audit: type=1400 audit(1573058703.103:556): apparmor="DENIED" operation="exec" profile="/usr/sbin/libvirtd" name="/usr/local/bin/qemu-system-microblazeel" pid=15694 comm="libvirtd" requested_mask="x" denied_mask="x" fsuid=0 ouid=0
16:51:35 sean-k-mooney yes
16:51:46 sean-k-mooney so apparmor is blocking it
16:51:48 efried (yes, qemu-[system-]x86_64 is in /usr/local/bin, which is in my PATH, at least on the CLI)
16:52:07 efried oh, I thought that libvirtd.conf hack was supposed to get around apparmor
16:52:08 sean-k-mooney ya so if we fix the apparmor issue it shoudl be working
16:52:30 sean-k-mooney no it fixs the fact libvirt expect polkit to be configure to use apparmor
16:52:43 sean-k-mooney i really should just add teh compile flag
16:52:53 efried is that easy?
16:53:35 sean-k-mooney i think its just https://review.opendev.org/#/c/693094/3/devstack/libs/libvirt@93
16:53:54 sean-k-mooney add --with-apparmor_profiles after --system
16:54:16 sean-k-mooney how im working around it at the moment was https://review.opendev.org/#/c/693094/3/README.rst
16:54:29 sean-k-mooney sudo ln -s /etc/apparmor.d/usr.sbin.libvirtd /etc/apparmor.d/disable/usr.sbin.libvirtd
16:54:30 sean-k-mooney then reboot for it to take effect.
16:54:40 efried ah
16:54:45 efried I'll try it both ways...
16:55:49 sean-k-mooney apparmor is the security sandboxing tool ubuntu use. its not happy that the binaries are in a different location then normal e.g. /usr/local/
16:55:57 sean-k-mooney hence teh error
16:56:52 efried can't we `make install` with the right location?
16:57:09 sean-k-mooney yes we can just change the prefix to /usr
16:57:18 sean-k-mooney i have done that for libvirt actully
16:57:25 sean-k-mooney just not qemu
17:00:25 mriedem huh, i just saw the 'timed out waiting for response from cell' error in a functional test https://zuul.opendev.org/t/openstack/build/63001bbd58c244cea70c995f1ebf61fb/log/job-output.txt#3092
17:00:28 sean-k-mooney https://review.opendev.org/#/c/693094/3/devstack/libs/libvirt@113 this would have to be updated to "./configure --prefix=/usr"
17:01:46 sean-k-mooney in test_evacuate_with_soft_anti_affinity
17:02:14 mriedem i don't think that matters,
17:02:31 mriedem we're just calling GET /servers/server_id to get the status probably
17:02:32 mriedem WARNING [oslo.service.loopingcall] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 59.77 sec
17:02:47 sean-k-mooney likely not but just was wondering which one
17:02:56 mriedem chances are this is another infra node that hit some crazy io issue
17:03:28 sean-k-mooney mriedem: oh that reminds me https://bugs.launchpad.net/nova/+bug/1851497
17:03:28 openstack Launchpad bug 1851497 in OpenStack Compute (nova) "rng is only enabled if qemu guest agent is enabled" [Medium,Triaged] - Assigned to sean mooney (sean-k-mooney)
17:03:57 sean-k-mooney the random number generator code has apparently been partly broken since queens ...
17:04:27 sean-k-mooney mriedem: did ye nail down the gate issue to io
17:04:46 mriedem sean-k-mooney: https://review.opendev.org/#/c/129325/ was kilo
17:04:50 sean-k-mooney if the vm runs out of entrapy it will have issue connecting to ssl sockets
17:04:53 mriedem oh wait
17:04:55 mriedem wrong change
17:05:09 mriedem oh no that's right
17:05:11 mriedem yeah kilo
17:05:31 mriedem who would have guessed that a 373 LOC refactor would break something....
17:06:03 sean-k-mooney well no one noticed till now so ya
17:06:53 sean-k-mooney i was review the sev virtio viommu fix and found it odd that the qemu guest agent was adding a rng
17:07:35 melwitt mriedem: that reminds me, I was chatting with penick about the gate io issues and he suggested infra/tempest using a flavor where the iops qos/limit has been set. are you familiar with that? (I'm not)
17:10:09 mriedem the trend here is kind of interesting http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Function%20'nova.servicegroup.drivers.db.DbDriver._report_state'%20run%20outlasted%20interval%20by%5C%22&from=7d
17:10:14 mriedem noon and midnight
17:10:17 mriedem mostly on ovh
17:10:35 efried noisy cron job?
17:10:41 mriedem melwitt: i think we were talking about that in here the other day a bit
17:14:04 amodi mriedem++ melwitt++ thanks both!!
17:14:23 mriedem amodi: happy backporting
17:14:48 amodi lol sure :)
17:15:08 mriedem i posted to the ML about the ovh + spike cron
17:15:14 mriedem i was wondering about the dip there, but that's the weekend
17:16:13 mriedem so that query is kind of annoying because there are a lot that are just a couple of seconds, but those 50+ ones are indicative of something really bad on the node
17:16:43 mriedem mehinks it would be good to maybe have some config in oslo that sets a threshold for that and if it goes above the threshold then it's a WARNING but below it's just INFO?
17:16:52 mriedem right now everything is WARNING which is kind of just noise
17:16:59 mriedem bnemec: ^ what do you think about that?
17:17:26 mriedem like i'd like to make that a warning if something outlasts 30 seconds in devstack/gate and then we can trace it in e-r
17:18:27 dansmith mriedem: hmm, do we still need null_instance_uuid_scan? wasn't that like kilo and we have a constraint now?
17:19:25 mriedem heh yeah i think we can deprecate and remove that https://review.opendev.org/#/c/97946/
17:19:32 mriedem https://review.opendev.org/#/q/topic:bp/db2-database+(status:open+OR+status:merged) !!!
17:19:46 mriedem classic riderman
17:37:55 openstackgerrit Artom Lifshitz proposed openstack/nova master: DNM: break live migration to trigger bug 1788014 https://review.opendev.org/693045
17:37:55 openstack bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] https://launchpad.net/bugs/1788014 - Assigned to Matt Riedemann (mriedem)
17:38:40 jgwentworth is there any way to discover whether force_config_drive=True in nova config from tempest? asking because of this tempest change https://review.opendev.org/693096
17:43:23 openstackgerrit Dan Smith proposed openstack/nova-specs master: Add nova-audit spec https://review.opendev.org/693226

Earlier   Later