| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-11-06 | |||
| 16:48:44 | efried | Nov 06 10:45:03 breakfast nova-compute[15654]: ERROR nova.virt.libvirt.host File "/usr/local/lib/python3.6/dist-packages/libvirt.py", line 4190, in getVersion | |
| 16:48:44 | efried | If I start n-cpu in that window, I run into the qemu version check. I added some debug and find it's coming from: | |
| 16:48:44 | efried | sean-k-mooney: The libvirt-bin service seems to not want to stay alive. When I start it manually, it runs for a couple minutes and then dies, but doesn't log anything. | |
| 16:48:45 | efried | Nov 06 10:45:03 breakfast nova-compute[15654]: ERROR nova.virt.libvirt.host libvirt.libvirtError: internal error: Cannot find suitable emulator for x86_64 | |
| 16:48:45 | efried | Nov 06 10:45:03 breakfast nova-compute[15654]: ERROR nova.virt.libvirt.host if ret == -1: raise libvirtError ('virConnectGetVersion() failed', conn=self) | |
| 16:50:16 | sean-k-mooney | is qemu-system-x86_64 | |
| 16:50:19 | sean-k-mooney | in your path | |
| 16:50:40 | sean-k-mooney | also do you have permission denied errors in dmesg | |
| 16:51:12 | openstackgerrit | Merged openstack/nova master: Revert "openstack server create" to "nova boot" in nova docs https://review.opendev.org/693065 | |
| 16:51:28 | efried | [73117.887250] audit: type=1400 audit(1573058703.103:556): apparmor="DENIED" operation="exec" profile="/usr/sbin/libvirtd" name="/usr/local/bin/qemu-system-microblazeel" pid=15694 comm="libvirtd" requested_mask="x" denied_mask="x" fsuid=0 ouid=0 | |
| 16:51:28 | efried | you mean like this | |
| 16:51:35 | sean-k-mooney | yes | |
| 16:51:46 | sean-k-mooney | so apparmor is blocking it | |
| 16:51:48 | efried | (yes, qemu-[system-]x86_64 is in /usr/local/bin, which is in my PATH, at least on the CLI) | |
| 16:52:07 | efried | oh, I thought that libvirtd.conf hack was supposed to get around apparmor | |
| 16:52:08 | sean-k-mooney | ya so if we fix the apparmor issue it shoudl be working | |
| 16:52:30 | sean-k-mooney | no it fixs the fact libvirt expect polkit to be configure to use apparmor | |
| 16:52:43 | sean-k-mooney | i really should just add teh compile flag | |
| 16:52:53 | efried | is that easy? | |
| 16:53:35 | sean-k-mooney | i think its just https://review.opendev.org/#/c/693094/3/devstack/libs/libvirt@93 | |
| 16:53:54 | sean-k-mooney | add --with-apparmor_profiles after --system | |
| 16:54:16 | sean-k-mooney | how im working around it at the moment was https://review.opendev.org/#/c/693094/3/README.rst | |
| 16:54:29 | sean-k-mooney | sudo ln -s /etc/apparmor.d/usr.sbin.libvirtd /etc/apparmor.d/disable/usr.sbin.libvirtd | |
| 16:54:30 | sean-k-mooney | then reboot for it to take effect. | |
| 16:54:40 | efried | ah | |
| 16:54:45 | efried | I'll try it both ways... | |
| 16:55:49 | sean-k-mooney | apparmor is the security sandboxing tool ubuntu use. its not happy that the binaries are in a different location then normal e.g. /usr/local/ | |
| 16:55:57 | sean-k-mooney | hence teh error | |
| 16:56:52 | efried | can't we `make install` with the right location? | |
| 16:57:09 | sean-k-mooney | yes we can just change the prefix to /usr | |
| 16:57:18 | sean-k-mooney | i have done that for libvirt actully | |
| 16:57:25 | sean-k-mooney | just not qemu | |
| 17:00:25 | mriedem | huh, i just saw the 'timed out waiting for response from cell' error in a functional test https://zuul.opendev.org/t/openstack/build/63001bbd58c244cea70c995f1ebf61fb/log/job-output.txt#3092 | |
| 17:00:28 | sean-k-mooney | https://review.opendev.org/#/c/693094/3/devstack/libs/libvirt@113 this would have to be updated to "./configure --prefix=/usr" | |
| 17:01:46 | sean-k-mooney | in test_evacuate_with_soft_anti_affinity | |
| 17:02:14 | mriedem | i don't think that matters, | |
| 17:02:31 | mriedem | we're just calling GET /servers/server_id to get the status probably | |
| 17:02:32 | mriedem | WARNING [oslo.service.loopingcall] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 59.77 sec | |
| 17:02:47 | sean-k-mooney | likely not but just was wondering which one | |
| 17:02:56 | mriedem | chances are this is another infra node that hit some crazy io issue | |
| 17:03:28 | openstack | Launchpad bug 1851497 in OpenStack Compute (nova) "rng is only enabled if qemu guest agent is enabled" [Medium,Triaged] - Assigned to sean mooney (sean-k-mooney) | |
| 17:03:28 | sean-k-mooney | mriedem: oh that reminds me https://bugs.launchpad.net/nova/+bug/1851497 | |
| 17:03:57 | sean-k-mooney | the random number generator code has apparently been partly broken since queens ... | |
| 17:04:27 | sean-k-mooney | mriedem: did ye nail down the gate issue to io | |
| 17:04:46 | mriedem | sean-k-mooney: https://review.opendev.org/#/c/129325/ was kilo | |
| 17:04:50 | sean-k-mooney | if the vm runs out of entrapy it will have issue connecting to ssl sockets | |
| 17:04:53 | mriedem | oh wait | |
| 17:04:55 | mriedem | wrong change | |
| 17:05:09 | mriedem | oh no that's right | |
| 17:05:11 | mriedem | yeah kilo | |
| 17:05:31 | mriedem | who would have guessed that a 373 LOC refactor would break something.... | |
| 17:06:03 | sean-k-mooney | well no one noticed till now so ya | |
| 17:06:53 | sean-k-mooney | i was review the sev virtio viommu fix and found it odd that the qemu guest agent was adding a rng | |
| 17:07:35 | melwitt | mriedem: that reminds me, I was chatting with penick about the gate io issues and he suggested infra/tempest using a flavor where the iops qos/limit has been set. are you familiar with that? (I'm not) | |
| 17:10:09 | mriedem | the trend here is kind of interesting http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Function%20'nova.servicegroup.drivers.db.DbDriver._report_state'%20run%20outlasted%20interval%20by%5C%22&from=7d | |
| 17:10:14 | mriedem | noon and midnight | |
| 17:10:17 | mriedem | mostly on ovh | |
| 17:10:35 | efried | noisy cron job? | |
| 17:10:41 | mriedem | melwitt: i think we were talking about that in here the other day a bit | |
| 17:14:04 | amodi | mriedem++ melwitt++ thanks both!! | |
| 17:14:23 | mriedem | amodi: happy backporting | |
| 17:14:48 | amodi | lol sure :) | |
| 17:15:08 | mriedem | i posted to the ML about the ovh + spike cron | |
| 17:15:14 | mriedem | i was wondering about the dip there, but that's the weekend | |
| 17:16:13 | mriedem | so that query is kind of annoying because there are a lot that are just a couple of seconds, but those 50+ ones are indicative of something really bad on the node | |
| 17:16:43 | mriedem | mehinks it would be good to maybe have some config in oslo that sets a threshold for that and if it goes above the threshold then it's a WARNING but below it's just INFO? | |
| 17:16:52 | mriedem | right now everything is WARNING which is kind of just noise | |
| 17:16:59 | mriedem | bnemec: ^ what do you think about that? | |
| 17:17:26 | mriedem | like i'd like to make that a warning if something outlasts 30 seconds in devstack/gate and then we can trace it in e-r | |
| 17:18:27 | dansmith | mriedem: hmm, do we still need null_instance_uuid_scan? wasn't that like kilo and we have a constraint now? | |
| 17:19:25 | mriedem | heh yeah i think we can deprecate and remove that https://review.opendev.org/#/c/97946/ | |
| 17:19:32 | mriedem | https://review.opendev.org/#/q/topic:bp/db2-database+(status:open+OR+status:merged) !!! | |
| 17:19:46 | mriedem | classic riderman | |
| 17:37:55 | openstack | bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] https://launchpad.net/bugs/1788014 - Assigned to Matt Riedemann (mriedem) | |
| 17:37:55 | openstackgerrit | Artom Lifshitz proposed openstack/nova master: DNM: break live migration to trigger bug 1788014 https://review.opendev.org/693045 | |
| 17:38:40 | jgwentworth | is there any way to discover whether force_config_drive=True in nova config from tempest? asking because of this tempest change https://review.opendev.org/693096 | |
| 17:43:23 | openstackgerrit | Dan Smith proposed openstack/nova-specs master: Add nova-audit spec https://review.opendev.org/693226 | |
| 17:43:34 | artom | jgwentworth, nothing's discoverable in Tempest, it's all configured | |
| 17:43:46 | dansmith | mriedem: jgwentworth ^ wanna skim that for obvious booboos before I blast to the ML? | |
| 17:44:31 | jgwentworth | artom: yeah, I was thinking whether there's some kinda api discoverability from nova that I'm not aware of | |
| 17:44:33 | artom | jgwentworth, in this case, the [compute-features-enabled]/config_drive option | |
| 17:44:41 | jgwentworth | right | |
| 17:44:52 | artom | jgwentworth, ah, doesn't ring a bell | |
| 17:44:59 | dansmith | jgwentworth: isn't that per compute node? | |
| 17:45:05 | jgwentworth | yeah, I'm pretty sure there's not | |
| 17:45:15 | jgwentworth | dansmith: oh, yeah I guess it can be | |
| 17:45:28 | artom | jgwentworth, I suppose you could always boot a VM, ssh to it, and check | |
| 17:45:30 | artom | But... | |
| 17:45:45 | artom | You're meant to be using the config option :) | |
| 17:45:49 | jgwentworth | sigh. ok. just thinking, the test could be useful if force_config_drive=False but it's not in the gate so it would be a dupe test | |
| 17:45:56 | artom | And skip tests that need config drive if it's not set | |
| 17:46:07 | jgwentworth | and is only not a dupe in envs where force_config_drive=False | |
| 17:48:09 | jgwentworth | so at least in the gate, it should be disabled by default. maybe it should actually check the opposite, only run if [compute-features-enabled]/config_drive=False | |
| 17:48:39 | jgwentworth | because tempest seems to be using that to mean whether config drive is being forced | |
| 17:49:19 | artom | What does passing config_drive=True on server creation even do? | |
| 17:49:44 | artom | "Indicates whether a config drive enables metadata injection." | |
| 17:49:53 | artom | That's... no the same as "a config drive will be present" | |
| 17:49:55 | artom | Is it? | |
| 17:50:03 | larsks | artom: attaches a config drive to your instance that will be used for getting instance metadata instead of the network metadata service. | |
| 17:50:19 | artom | larsks, see, I had no idea that was user-controllable in that way | |