| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-06 | |||
| 16:31:28 | melwitt | and yeah, understood. thanks for pointing out those previous fixes and updating it | |
| 16:34:12 | dansmith | mriedem: query | |
| 16:34:27 | dansmith | was just looking at your heal_instance_mappings thing | |
| 16:34:32 | dansmith | and was thinking. | |
| 16:35:57 | dansmith | I wonder if it wouldn't make sense for us to have another totally optional service with the sole purpose of looking for things that need healing.. instance mappings, maybe allocations, host mappings, maybe even archive and purge | |
| 16:36:31 | dansmith | we could make it a daemon, or something you could run from cron single-shot if you just wanted to run it at midnight or something | |
| 16:36:41 | melwitt | I think operators would love that | |
| 16:37:15 | dansmith | instead of a ton of cron entries for all those things, and not knowing you need new cron entries to run new things on a new release, | |
| 16:37:25 | dansmith | you just run one nova-audit in one place, no HA, just low and slow burn of audity things | |
| 16:37:40 | melwitt | yeah, would be nice | |
| 16:38:43 | mriedem | i'm not opposed | |
| 16:39:40 | mriedem | would that also run the discover host mappings thing that the scheduler can run today? | |
| 16:40:09 | mriedem | i remember glance has several things like that https://github.com/openstack/glance/blob/master/setup.cfg#L34 | |
| 16:40:17 | mriedem | though i'm not sure if they are all daemons | |
| 16:40:22 | dansmith | mriedem: yeah, hence the host mappings in my list | |
| 16:41:41 | mriedem | i will admit the number of heal/discover/sync things is getting a bit overwhelming | |
| 16:41:49 | dansmith | right | |
| 16:42:00 | dansmith | impossible to track for the average operator I'm sure | |
| 16:42:03 | mriedem | and we have talked about a periodic archive/purge thing in the past | |
| 16:42:25 | dansmith | and we put the host mappings thing in scheduler mostly because the cardinality was small | |
| 16:42:38 | dansmith | (and then I subsequently solved their problem a different way) | |
| 16:42:59 | mriedem | and i think b/c api and conductor didn't have periodic tasks already but scheduler did | |
| 16:43:53 | dansmith | maybe, but the point was to run one or few of those since they'd step all over each other, and there'd be too many conductors and apis even if they did have periodics | |
| 16:43:54 | mriedem | maybe you want to kick that idea out into the mailing list tagged with [nova][ops] to get initial feedback? and/or just start a spec and kick that into the ML to get attention from operators. | |
| 16:44:23 | mriedem | we do run nproc/2 scheduler works by default now i think.. | |
| 16:44:24 | mriedem | but yeah | |
| 16:44:28 | dansmith | yeah, I could start with a spec | |
| 16:44:38 | dansmith | I was kinda thinking of just starting with code | |
| 16:45:04 | mriedem | now that you've gotten a taste from the image cache stuff you're addicted! | |
| 16:45:55 | dansmith | JUST ONE MORE HIT MAN | |
| 16:48:44 | efried | sean-k-mooney: The libvirt-bin service seems to not want to stay alive. When I start it manually, it runs for a couple minutes and then dies, but doesn't log anything. | |
| 16:48:44 | efried | If I start n-cpu in that window, I run into the qemu version check. I added some debug and find it's coming from: | |
| 16:48:44 | efried | Nov 06 10:45:03 breakfast nova-compute[15654]: ERROR nova.virt.libvirt.host File "/usr/local/lib/python3.6/dist-packages/libvirt.py", line 4190, in getVersion | |
| 16:48:45 | efried | Nov 06 10:45:03 breakfast nova-compute[15654]: ERROR nova.virt.libvirt.host if ret == -1: raise libvirtError ('virConnectGetVersion() failed', conn=self) | |
| 16:48:45 | efried | Nov 06 10:45:03 breakfast nova-compute[15654]: ERROR nova.virt.libvirt.host libvirt.libvirtError: internal error: Cannot find suitable emulator for x86_64 | |
| 16:50:16 | sean-k-mooney | is qemu-system-x86_64 | |
| 16:50:19 | sean-k-mooney | in your path | |
| 16:50:40 | sean-k-mooney | also do you have permission denied errors in dmesg | |
| 16:51:12 | openstackgerrit | Merged openstack/nova master: Revert "openstack server create" to "nova boot" in nova docs https://review.opendev.org/693065 | |
| 16:51:28 | efried | you mean like this | |
| 16:51:28 | efried | [73117.887250] audit: type=1400 audit(1573058703.103:556): apparmor="DENIED" operation="exec" profile="/usr/sbin/libvirtd" name="/usr/local/bin/qemu-system-microblazeel" pid=15694 comm="libvirtd" requested_mask="x" denied_mask="x" fsuid=0 ouid=0 | |
| 16:51:35 | sean-k-mooney | yes | |
| 16:51:46 | sean-k-mooney | so apparmor is blocking it | |
| 16:51:48 | efried | (yes, qemu-[system-]x86_64 is in /usr/local/bin, which is in my PATH, at least on the CLI) | |
| 16:52:07 | efried | oh, I thought that libvirtd.conf hack was supposed to get around apparmor | |
| 16:52:08 | sean-k-mooney | ya so if we fix the apparmor issue it shoudl be working | |
| 16:52:30 | sean-k-mooney | no it fixs the fact libvirt expect polkit to be configure to use apparmor | |
| 16:52:43 | sean-k-mooney | i really should just add teh compile flag | |
| 16:52:53 | efried | is that easy? | |
| 16:53:35 | sean-k-mooney | i think its just https://review.opendev.org/#/c/693094/3/devstack/libs/libvirt@93 | |
| 16:53:54 | sean-k-mooney | add --with-apparmor_profiles after --system | |
| 16:54:16 | sean-k-mooney | how im working around it at the moment was https://review.opendev.org/#/c/693094/3/README.rst | |
| 16:54:29 | sean-k-mooney | sudo ln -s /etc/apparmor.d/usr.sbin.libvirtd /etc/apparmor.d/disable/usr.sbin.libvirtd | |
| 16:54:30 | sean-k-mooney | then reboot for it to take effect. | |
| 16:54:40 | efried | ah | |
| 16:54:45 | efried | I'll try it both ways... | |
| 16:55:49 | sean-k-mooney | apparmor is the security sandboxing tool ubuntu use. its not happy that the binaries are in a different location then normal e.g. /usr/local/ | |
| 16:55:57 | sean-k-mooney | hence teh error | |
| 16:56:52 | efried | can't we `make install` with the right location? | |
| 16:57:09 | sean-k-mooney | yes we can just change the prefix to /usr | |
| 16:57:18 | sean-k-mooney | i have done that for libvirt actully | |
| 16:57:25 | sean-k-mooney | just not qemu | |
| 17:00:25 | mriedem | huh, i just saw the 'timed out waiting for response from cell' error in a functional test https://zuul.opendev.org/t/openstack/build/63001bbd58c244cea70c995f1ebf61fb/log/job-output.txt#3092 | |
| 17:00:28 | sean-k-mooney | https://review.opendev.org/#/c/693094/3/devstack/libs/libvirt@113 this would have to be updated to "./configure --prefix=/usr" | |
| 17:01:46 | sean-k-mooney | in test_evacuate_with_soft_anti_affinity | |
| 17:02:14 | mriedem | i don't think that matters, | |
| 17:02:31 | mriedem | we're just calling GET /servers/server_id to get the status probably | |
| 17:02:32 | mriedem | WARNING [oslo.service.loopingcall] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 59.77 sec | |
| 17:02:47 | sean-k-mooney | likely not but just was wondering which one | |
| 17:02:56 | mriedem | chances are this is another infra node that hit some crazy io issue | |
| 17:03:28 | sean-k-mooney | mriedem: oh that reminds me https://bugs.launchpad.net/nova/+bug/1851497 | |
| 17:03:28 | openstack | Launchpad bug 1851497 in OpenStack Compute (nova) "rng is only enabled if qemu guest agent is enabled" [Medium,Triaged] - Assigned to sean mooney (sean-k-mooney) | |
| 17:03:57 | sean-k-mooney | the random number generator code has apparently been partly broken since queens ... | |
| 17:04:27 | sean-k-mooney | mriedem: did ye nail down the gate issue to io | |
| 17:04:46 | mriedem | sean-k-mooney: https://review.opendev.org/#/c/129325/ was kilo | |
| 17:04:50 | sean-k-mooney | if the vm runs out of entrapy it will have issue connecting to ssl sockets | |
| 17:04:53 | mriedem | oh wait | |
| 17:04:55 | mriedem | wrong change | |
| 17:05:09 | mriedem | oh no that's right | |
| 17:05:11 | mriedem | yeah kilo | |
| 17:05:31 | mriedem | who would have guessed that a 373 LOC refactor would break something.... | |
| 17:06:03 | sean-k-mooney | well no one noticed till now so ya | |
| 17:06:53 | sean-k-mooney | i was review the sev virtio viommu fix and found it odd that the qemu guest agent was adding a rng | |
| 17:07:35 | melwitt | mriedem: that reminds me, I was chatting with penick about the gate io issues and he suggested infra/tempest using a flavor where the iops qos/limit has been set. are you familiar with that? (I'm not) | |
| 17:10:09 | mriedem | the trend here is kind of interesting http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Function%20'nova.servicegroup.drivers.db.DbDriver._report_state'%20run%20outlasted%20interval%20by%5C%22&from=7d | |
| 17:10:14 | mriedem | noon and midnight | |
| 17:10:17 | mriedem | mostly on ovh | |
| 17:10:35 | efried | noisy cron job? | |
| 17:10:41 | mriedem | melwitt: i think we were talking about that in here the other day a bit | |
| 17:14:04 | amodi | mriedem++ melwitt++ thanks both!! | |
| 17:14:23 | mriedem | amodi: happy backporting | |
| 17:14:48 | amodi | lol sure :) | |
| 17:15:08 | mriedem | i posted to the ML about the ovh + spike cron | |
| 17:15:14 | mriedem | i was wondering about the dip there, but that's the weekend | |
| 17:16:13 | mriedem | so that query is kind of annoying because there are a lot that are just a couple of seconds, but those 50+ ones are indicative of something really bad on the node | |
| 17:16:43 | mriedem | mehinks it would be good to maybe have some config in oslo that sets a threshold for that and if it goes above the threshold then it's a WARNING but below it's just INFO? | |
| 17:16:52 | mriedem | right now everything is WARNING which is kind of just noise | |
| 17:16:59 | mriedem | bnemec: ^ what do you think about that? | |
| 17:17:26 | mriedem | like i'd like to make that a warning if something outlasts 30 seconds in devstack/gate and then we can trace it in e-r | |
| 17:18:27 | dansmith | mriedem: hmm, do we still need null_instance_uuid_scan? wasn't that like kilo and we have a constraint now? | |