| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-02-18 | |||
| 17:16:50 | dansmith | gibi: what provider? | |
| 17:17:30 | dansmith | gibi: tripleo was having some IO problems on a vexx node, but we did a bunch of digging into the node stats and it seemed like it was just too many noisy neighbors | |
| 17:18:36 | dansmith | gibi: oh that's an arm64 run | |
| 17:19:17 | sean-k-mooney | in that case ya | |
| 17:20:03 | sean-k-mooney | so Provider: linaro-us | |
| 17:20:45 | sean-k-mooney | although the py39 verion passed in hlaf the time | |
| 17:21:17 | sean-k-mooney | so its not a genral problem with the provider but there might be too high a contention on some hosts | |
| 17:21:23 | gibi | sean-k-mooney: I can increase the timeout yes | |
| 17:21:41 | gibi | dansmith: I can do some more recheck to get a non arm failure if that helps | |
| 17:22:13 | dansmith | gibi: well, I'm just saying, I don't know much about that environment.. could be raspberry pis and slow SD cards for all I know :P | |
| 17:22:29 | gibi | dansmith: OK :) | |
| 17:22:32 | sean-k-mooney | dansmith: they are realy arm server systems | |
| 17:23:15 | dansmith | sean-k-mooney: I know :P | |
| 17:24:15 | stephenfin | I think they're emulated | |
| 17:24:32 | stephenfin | Assuming it's the same machines that you linked me from the kolla jobs yesterday, sean-k-mooney | |
| 17:24:34 | dansmith | really? I thought these were actual arm boxes | |
| 17:24:35 | sean-k-mooney | dansmith: https://zuul.openstack.org/build/b038b4112ef34b15b885f204da27c808/log/primary/logs/kolla/nova/nova-compute.txt#843 | |
| 17:24:50 | sean-k-mooney | stephenfin: its is and its not emulated | |
| 17:25:18 | sean-k-mooney | well they are vms if that is what you mean | |
| 17:25:44 | stephenfin | I mean it's using <domain>qemu</domain> | |
| 17:25:45 | sean-k-mooney | but this i arm hardware provide by linaro to do real testing on arm | |
| 17:25:49 | stephenfin | not <domain>kvm</domain> | |
| 17:26:03 | sean-k-mooney | stephenfin: right but thats becasue we disabel nested vert | |
| 17:26:03 | stephenfin | so it's entirely software emulation, right? | |
| 17:26:29 | stephenfin | okay, cool, that's all I was saying | |
| 17:26:29 | sean-k-mooney | the vms created by it will use qemu without kvm in the kolla job | |
| 17:26:47 | stephenfin | the PPC jobs you linked are running on bare metal (with KVM) | |
| 17:26:56 | sean-k-mooney | but the vms we are runing the test in are on real arm hardware | |
| 17:27:10 | sean-k-mooney | stephenfin: yep its a third party ci | |
| 17:27:37 | sean-k-mooney | stephenfin: nested vrit was disabled by policy in the first party ci | |
| 17:28:13 | sean-k-mooney | that has been relaxed slightly more recnetly but its still not gernarally avaliable | |
| 17:28:41 | stephenfin | yup | |
| 17:28:54 | stephenfin | fwiw, the PPC jobs seem to be running on bare metal or nested virt | |
| 17:28:59 | stephenfin | I'm not sure which | |
| 17:29:03 | stephenfin | but KVM is enabled | |
| 17:29:12 | sean-k-mooney | i would guess nested virt | |
| 17:29:29 | sean-k-mooney | did they only have 8 cores? | |
| 17:29:31 | stephenfin | as would I. Letting arbitrary jobs loose on bare metal doesn't sound so good | |
| 17:29:36 | sean-k-mooney | and 8 G of ram | |
| 17:30:00 | stephenfin | Oh, I don't recall and I don't have the links handy | |
| 17:30:13 | sean-k-mooney | the arm jobs are 8 cores and 8g of ram like or standard ci flavors | |
| 17:31:21 | sean-k-mooney | stephenfin: not important but htat a good way to ficure out if its nested virt or not even with host passthough | |
| 17:31:33 | stephenfin | yup, good point that | |
| 17:33:44 | sean-k-mooney | that takes over an hour if your doing a remote instal like i am | |
| 19:47:25 | spatel | sean-k-mooney any idea i am seeing this error on nova - ERROR stevedore.extension [req-249d0078-a423-4815-9c32-589f1814209a 3ffb7b8d55e1471aa56407e5a583037f 584651a0fe954d9480435678ea3ce5ba - default default] Could not load 'oslo_cache.etcd3gw': No module named 'etcd3gw': ModuleNotFoundError: No module named 'etcd3gw' | |
| 19:59:14 | sean-k-mooney | am its realted to how oslo.cache is loading all the backend drivers even if you are not using it | |
| 19:59:25 | sean-k-mooney | it wont break anything but it should be fixed at some point | |
| 20:00:49 | spatel | thank you for confirmation. | |
| 20:01:30 | sean-k-mooney | when it import the backeds to see which one it shoudl use its uncondtionaly trying to import etcd3gw package | |
| 20:01:34 | spatel | After rebooting my rabbitMQ cluster look like everything slowed down, VM creation process and VM delete process are very slow | |
| 20:02:02 | spatel | I have noticed this in nova logs - failed to perform operation on queue 'notifications_designate.info' in vhost '/nova' due to timeout | |
| 20:02:33 | spatel | when i checked designate.info queue on rabbitMQ and it doesn't exist | |
| 20:02:36 | sean-k-mooney | well if you restarted it its going to cause all the compute to have to reconnect | |
| 20:02:59 | sean-k-mooney | depending on how it happend you might need to restart the nove comptue agents too | |
| 20:03:29 | spatel | hmm! i did rolling restart node-1 then node-2 ... node-3 | |
| 20:04:07 | sean-k-mooney | depending on how you have the clustering set up and loadbalnce it may or may not help | |
| 20:04:33 | spatel | i don't mind to restart all compute agent or nova-api | |
| 20:05:03 | spatel | i am wondering about 'notifications_designate.info' queue, who create that queue ? | |
| 20:05:40 | spatel | does nova create that queue when it receive designate notification? | |
| 20:06:10 | sean-k-mooney | its the other way around | |
| 20:06:19 | sean-k-mooney | designate subscipes to nova notifiction | |
| 20:06:29 | sean-k-mooney | nova does not listten for notificaiton | |
| 20:07:31 | spatel | how do i get that queue back? 'notifications_designate.info' | |
| 20:08:27 | sean-k-mooney | posibly restart the designate server but i dont really know | |
| 20:08:31 | spatel | what if i tell nova.conf driver = noop | |
| 20:08:51 | spatel | I did restart designate but no luck | |
| 20:14:22 | johnsom | spatel Did you upgrade designate? Newer versions don't rely on rabbit or the sink for nova/neutron/designate messaging. | |
| 20:14:57 | spatel | I didn't upgrade designate, i am running victoria one | |
| 20:15:32 | spatel | johnsom it was working before but after rabbitMQ rolling restart everything looks bad | |
| 20:15:54 | spatel | i may try to re-install designate | |
| 20:16:21 | johnsom | Hmm, that seems unnecessary. | |
| 20:17:03 | johnsom | I really haven't poked too much at the notifications, so don't have a lot of experience there. You could ask in the #openstack-dns channel though. Others there may have more details on the notifications. | |
| 20:18:39 | spatel | johnsom agreed, i am seeing when i create vm it takes almost 5 min to spun up, getting stuck in scheduling... | |
| 20:18:57 | spatel | if i delete vm it get stuck in deleting stat for 5 min.. | |
| 20:19:19 | spatel | all i am seeing in error related designate.info queue not found | |
| 21:19:12 | spatel | johnsom after restarting glance i found VM creating process speed up but deleting still taking 5 min to delete vm | |
| 21:19:53 | johnsom | So maybe the designate messages are not related | |
| 21:21:00 | spatel | possible | |
| 21:21:23 | spatel | my rabbitMQ cluster also looking health so not sure what holding back to delete vm | |
| 21:21:32 | spatel | not seeing any error also.. | |
| 21:22:32 | spatel | let me restart all service except rabbitMQ and see | |
| 23:04:01 | spatel | johnsom does image-caching is disable when using Ceph storage for glance? | |
| 23:04:24 | johnsom | spatel Sorry, I have no idea. | |
| 23:04:31 | spatel | I wonder that was my issue because recently i migrated glance to ceph rbd | |
| 23:05:07 | spatel | may be that is why its very slow to create VM (earlier when i was using local disk with glance talking few second to spun up vm) | |
| 23:05:26 | spatel | taking* | |
| 23:46:12 | spatel | johnsom all my problem solved :) | |
| 23:46:25 | johnsom | spatel Oh good | |
| 23:46:38 | spatel | after setting driver = noop in nova-api and nova-compute* nodes | |
| 23:47:20 | spatel | my issue is nova still looking for designate.info queue which doesn't exist and that was causing all chaos | |
| 23:48:02 | spatel | I need to work on designate to find out what is going on but i believe interim solution is to tell nova don't look for notification queue | |
| 23:49:19 | spatel | anyway! time for dinner! see you tomorrow.. thank you for your support. | |
| #openstack-nova - 2021-02-19 | |||
| 00:47:27 | openstackgerrit | Jorhson Deng proposed openstack/nova master: Reject to attach and detach volume while task_state is not None https://review.opendev.org/c/openstack/nova/+/758918 | |
| 02:18:05 | openstackgerrit | Merged openstack/nova master: Add regression test for bug 1914777 https://review.opendev.org/c/openstack/nova/+/775307 | |
| 02:18:07 | openstack | bug 1914777 in OpenStack Compute (nova) "Possible race condition between n-cpu and n-api when deleting a building instance" [High,In progress] https://launchpad.net/bugs/1914777 - Assigned to melanie witt (melwitt) | |
| 02:18:25 | openstackgerrit | Merged openstack/nova stable/ussuri: tools: Allow check-cherry-picks.sh to be disabled by an env var https://review.opendev.org/c/openstack/nova/+/766024 | |
| 04:47:35 | openstackgerrit | melanie witt proposed openstack/nova master: Handle instance = None in _local_delete_cleanup https://review.opendev.org/c/openstack/nova/+/775308 | |
| 07:02:05 | openstackgerrit | Merged openstack/nova stable/ussuri: libvirt: Only ask tpool.Proxy to autowrap vir* classes https://review.opendev.org/c/openstack/nova/+/761223 | |
| 07:41:04 | openstackgerrit | Yongli He proposed openstack/nova master: smartnic support https://review.opendev.org/c/openstack/nova/+/758944 | |
| 07:57:26 | yonglihe | sean-k-mooney: review invite, thanks: https://review.opendev.org/q/topic:%22bp%252Fsriov-smartnic-support%22+(status:open%20OR%20status:merged) | |