| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-02-18 | |||
| 16:03:08 | sean-k-mooney | oh ya :) | |
| 16:03:08 | gibi | if you want to mention something | |
| 16:58:07 | sean-k-mooney | actully looks liek we have a few others that were not on the list | |
| 16:58:31 | sean-k-mooney | like https://bugs.launchpad.net/nova/+bug/1861893 | |
| 16:58:32 | openstack | Launchpad bug 1861893 in OpenStack Compute (nova) "os-assisted-volume-snapshots passes unsanitised file path to the libvirt driver" [Medium,Confirmed] | |
| 16:59:08 | sean-k-mooney | we could not actuly exploit ^ so its just a security hardening case | |
| 16:59:29 | openstackgerrit | Vishal Manchanda proposed openstack/nova master: Docs: Correct ``Password injection using the dashboard`` Explanation https://review.opendev.org/c/openstack/nova/+/775084 | |
| 16:59:44 | sean-k-mooney | specificly class c1 "Not considered a practical vulnerability (but some people might assign a CVE for it)" | |
| 17:00:48 | openstack | Launchpad bug 1861893 in OpenStack Compute (nova) "os-assisted-volume-snapshots passes unsanitised file path to the libvirt driver" [Medium,Confirmed] | |
| 17:05:49 | gibi | stephenfin, dansmith: I have extra logs for the slowness of the db migration tests (even after the compaction) https://review.opendev.org/c/openstack/nova/+/775094 This result confirms for me that it is simply an extremely slow test node with noisy neighbors. I have not got new ideas how to move forward with this | |
| 17:13:49 | sean-k-mooney | can we increase the timeout wait again | |
| 17:14:16 | sean-k-mooney | you tweaked it previously if im remebring corectly | |
| 17:14:41 | sean-k-mooney | if its really just a slow node retrying it like the pytest flaky decorator does wont help | |
| 17:15:14 | sean-k-mooney | gibi: alternitivly i dont know if there is a way to do the db walk in multiple tests | |
| 17:15:49 | sean-k-mooney | e.g decompose it to do it in severnal batches of verions over 2+ tests | |
| 17:16:11 | sean-k-mooney | but ya not really sure how to workaroud this | |
| 17:16:46 | sean-k-mooney | at least nothing to adress the underlying issue | |
| 17:16:50 | dansmith | gibi: what provider? | |
| 17:17:30 | dansmith | gibi: tripleo was having some IO problems on a vexx node, but we did a bunch of digging into the node stats and it seemed like it was just too many noisy neighbors | |
| 17:18:36 | dansmith | gibi: oh that's an arm64 run | |
| 17:19:17 | sean-k-mooney | in that case ya | |
| 17:20:03 | sean-k-mooney | so Provider: linaro-us | |
| 17:20:45 | sean-k-mooney | although the py39 verion passed in hlaf the time | |
| 17:21:17 | sean-k-mooney | so its not a genral problem with the provider but there might be too high a contention on some hosts | |
| 17:21:23 | gibi | sean-k-mooney: I can increase the timeout yes | |
| 17:21:41 | gibi | dansmith: I can do some more recheck to get a non arm failure if that helps | |
| 17:22:13 | dansmith | gibi: well, I'm just saying, I don't know much about that environment.. could be raspberry pis and slow SD cards for all I know :P | |
| 17:22:29 | gibi | dansmith: OK :) | |
| 17:22:32 | sean-k-mooney | dansmith: they are realy arm server systems | |
| 17:23:15 | dansmith | sean-k-mooney: I know :P | |
| 17:24:15 | stephenfin | I think they're emulated | |
| 17:24:32 | stephenfin | Assuming it's the same machines that you linked me from the kolla jobs yesterday, sean-k-mooney | |
| 17:24:34 | dansmith | really? I thought these were actual arm boxes | |
| 17:24:35 | sean-k-mooney | dansmith: https://zuul.openstack.org/build/b038b4112ef34b15b885f204da27c808/log/primary/logs/kolla/nova/nova-compute.txt#843 | |
| 17:24:50 | sean-k-mooney | stephenfin: its is and its not emulated | |
| 17:25:18 | sean-k-mooney | well they are vms if that is what you mean | |
| 17:25:44 | stephenfin | I mean it's using <domain>qemu</domain> | |
| 17:25:45 | sean-k-mooney | but this i arm hardware provide by linaro to do real testing on arm | |
| 17:25:49 | stephenfin | not <domain>kvm</domain> | |
| 17:26:03 | stephenfin | so it's entirely software emulation, right? | |
| 17:26:03 | sean-k-mooney | stephenfin: right but thats becasue we disabel nested vert | |
| 17:26:29 | sean-k-mooney | the vms created by it will use qemu without kvm in the kolla job | |
| 17:26:29 | stephenfin | okay, cool, that's all I was saying | |
| 17:26:47 | stephenfin | the PPC jobs you linked are running on bare metal (with KVM) | |
| 17:26:56 | sean-k-mooney | but the vms we are runing the test in are on real arm hardware | |
| 17:27:10 | sean-k-mooney | stephenfin: yep its a third party ci | |
| 17:27:37 | sean-k-mooney | stephenfin: nested vrit was disabled by policy in the first party ci | |
| 17:28:13 | sean-k-mooney | that has been relaxed slightly more recnetly but its still not gernarally avaliable | |
| 17:28:41 | stephenfin | yup | |
| 17:28:54 | stephenfin | fwiw, the PPC jobs seem to be running on bare metal or nested virt | |
| 17:28:59 | stephenfin | I'm not sure which | |
| 17:29:03 | stephenfin | but KVM is enabled | |
| 17:29:12 | sean-k-mooney | i would guess nested virt | |
| 17:29:29 | sean-k-mooney | did they only have 8 cores? | |
| 17:29:31 | stephenfin | as would I. Letting arbitrary jobs loose on bare metal doesn't sound so good | |
| 17:29:36 | sean-k-mooney | and 8 G of ram | |
| 17:30:00 | stephenfin | Oh, I don't recall and I don't have the links handy | |
| 17:30:13 | sean-k-mooney | the arm jobs are 8 cores and 8g of ram like or standard ci flavors | |
| 17:31:21 | sean-k-mooney | stephenfin: not important but htat a good way to ficure out if its nested virt or not even with host passthough | |
| 17:31:33 | stephenfin | yup, good point that | |
| 17:33:44 | sean-k-mooney | that takes over an hour if your doing a remote instal like i am | |
| 19:47:25 | spatel | sean-k-mooney any idea i am seeing this error on nova - ERROR stevedore.extension [req-249d0078-a423-4815-9c32-589f1814209a 3ffb7b8d55e1471aa56407e5a583037f 584651a0fe954d9480435678ea3ce5ba - default default] Could not load 'oslo_cache.etcd3gw': No module named 'etcd3gw': ModuleNotFoundError: No module named 'etcd3gw' | |
| 19:59:14 | sean-k-mooney | am its realted to how oslo.cache is loading all the backend drivers even if you are not using it | |
| 19:59:25 | sean-k-mooney | it wont break anything but it should be fixed at some point | |
| 20:00:49 | spatel | thank you for confirmation. | |
| 20:01:30 | sean-k-mooney | when it import the backeds to see which one it shoudl use its uncondtionaly trying to import etcd3gw package | |
| 20:01:34 | spatel | After rebooting my rabbitMQ cluster look like everything slowed down, VM creation process and VM delete process are very slow | |
| 20:02:02 | spatel | I have noticed this in nova logs - failed to perform operation on queue 'notifications_designate.info' in vhost '/nova' due to timeout | |
| 20:02:33 | spatel | when i checked designate.info queue on rabbitMQ and it doesn't exist | |
| 20:02:36 | sean-k-mooney | well if you restarted it its going to cause all the compute to have to reconnect | |
| 20:02:59 | sean-k-mooney | depending on how it happend you might need to restart the nove comptue agents too | |
| 20:03:29 | spatel | hmm! i did rolling restart node-1 then node-2 ... node-3 | |
| 20:04:07 | sean-k-mooney | depending on how you have the clustering set up and loadbalnce it may or may not help | |
| 20:04:33 | spatel | i don't mind to restart all compute agent or nova-api | |
| 20:05:03 | spatel | i am wondering about 'notifications_designate.info' queue, who create that queue ? | |
| 20:05:40 | spatel | does nova create that queue when it receive designate notification? | |
| 20:06:10 | sean-k-mooney | its the other way around | |
| 20:06:19 | sean-k-mooney | designate subscipes to nova notifiction | |
| 20:06:29 | sean-k-mooney | nova does not listten for notificaiton | |
| 20:07:31 | spatel | how do i get that queue back? 'notifications_designate.info' | |
| 20:08:27 | sean-k-mooney | posibly restart the designate server but i dont really know | |
| 20:08:31 | spatel | what if i tell nova.conf driver = noop | |
| 20:08:51 | spatel | I did restart designate but no luck | |
| 20:14:22 | johnsom | spatel Did you upgrade designate? Newer versions don't rely on rabbit or the sink for nova/neutron/designate messaging. | |
| 20:14:57 | spatel | I didn't upgrade designate, i am running victoria one | |
| 20:15:32 | spatel | johnsom it was working before but after rabbitMQ rolling restart everything looks bad | |
| 20:15:54 | spatel | i may try to re-install designate | |
| 20:16:21 | johnsom | Hmm, that seems unnecessary. | |
| 20:17:03 | johnsom | I really haven't poked too much at the notifications, so don't have a lot of experience there. You could ask in the #openstack-dns channel though. Others there may have more details on the notifications. | |
| 20:18:39 | spatel | johnsom agreed, i am seeing when i create vm it takes almost 5 min to spun up, getting stuck in scheduling... | |
| 20:18:57 | spatel | if i delete vm it get stuck in deleting stat for 5 min.. | |
| 20:19:19 | spatel | all i am seeing in error related designate.info queue not found | |
| 21:19:12 | spatel | johnsom after restarting glance i found VM creating process speed up but deleting still taking 5 min to delete vm | |
| 21:19:53 | johnsom | So maybe the designate messages are not related | |
| 21:21:00 | spatel | possible | |
| 21:21:23 | spatel | my rabbitMQ cluster also looking health so not sure what holding back to delete vm | |
| 21:21:32 | spatel | not seeing any error also.. | |
| 21:22:32 | spatel | let me restart all service except rabbitMQ and see | |
| 23:04:01 | spatel | johnsom does image-caching is disable when using Ceph storage for glance? | |
| 23:04:24 | johnsom | spatel Sorry, I have no idea. | |