| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-02-18 | |||
| 17:29:29 | sean-k-mooney | did they only have 8 cores? | |
| 17:29:31 | stephenfin | as would I. Letting arbitrary jobs loose on bare metal doesn't sound so good | |
| 17:29:36 | sean-k-mooney | and 8 G of ram | |
| 17:30:00 | stephenfin | Oh, I don't recall and I don't have the links handy | |
| 17:30:13 | sean-k-mooney | the arm jobs are 8 cores and 8g of ram like or standard ci flavors | |
| 17:31:21 | sean-k-mooney | stephenfin: not important but htat a good way to ficure out if its nested virt or not even with host passthough | |
| 17:31:33 | stephenfin | yup, good point that | |
| 17:33:44 | sean-k-mooney | that takes over an hour if your doing a remote instal like i am | |
| 19:47:25 | spatel | sean-k-mooney any idea i am seeing this error on nova - ERROR stevedore.extension [req-249d0078-a423-4815-9c32-589f1814209a 3ffb7b8d55e1471aa56407e5a583037f 584651a0fe954d9480435678ea3ce5ba - default default] Could not load 'oslo_cache.etcd3gw': No module named 'etcd3gw': ModuleNotFoundError: No module named 'etcd3gw' | |
| 19:59:14 | sean-k-mooney | am its realted to how oslo.cache is loading all the backend drivers even if you are not using it | |
| 19:59:25 | sean-k-mooney | it wont break anything but it should be fixed at some point | |
| 20:00:49 | spatel | thank you for confirmation. | |
| 20:01:30 | sean-k-mooney | when it import the backeds to see which one it shoudl use its uncondtionaly trying to import etcd3gw package | |
| 20:01:34 | spatel | After rebooting my rabbitMQ cluster look like everything slowed down, VM creation process and VM delete process are very slow | |
| 20:02:02 | spatel | I have noticed this in nova logs - failed to perform operation on queue 'notifications_designate.info' in vhost '/nova' due to timeout | |
| 20:02:33 | spatel | when i checked designate.info queue on rabbitMQ and it doesn't exist | |
| 20:02:36 | sean-k-mooney | well if you restarted it its going to cause all the compute to have to reconnect | |
| 20:02:59 | sean-k-mooney | depending on how it happend you might need to restart the nove comptue agents too | |
| 20:03:29 | spatel | hmm! i did rolling restart node-1 then node-2 ... node-3 | |
| 20:04:07 | sean-k-mooney | depending on how you have the clustering set up and loadbalnce it may or may not help | |
| 20:04:33 | spatel | i don't mind to restart all compute agent or nova-api | |
| 20:05:03 | spatel | i am wondering about 'notifications_designate.info' queue, who create that queue ? | |
| 20:05:40 | spatel | does nova create that queue when it receive designate notification? | |
| 20:06:10 | sean-k-mooney | its the other way around | |
| 20:06:19 | sean-k-mooney | designate subscipes to nova notifiction | |
| 20:06:29 | sean-k-mooney | nova does not listten for notificaiton | |
| 20:07:31 | spatel | how do i get that queue back? 'notifications_designate.info' | |
| 20:08:27 | sean-k-mooney | posibly restart the designate server but i dont really know | |
| 20:08:31 | spatel | what if i tell nova.conf driver = noop | |
| 20:08:51 | spatel | I did restart designate but no luck | |
| 20:14:22 | johnsom | spatel Did you upgrade designate? Newer versions don't rely on rabbit or the sink for nova/neutron/designate messaging. | |
| 20:14:57 | spatel | I didn't upgrade designate, i am running victoria one | |
| 20:15:32 | spatel | johnsom it was working before but after rabbitMQ rolling restart everything looks bad | |
| 20:15:54 | spatel | i may try to re-install designate | |
| 20:16:21 | johnsom | Hmm, that seems unnecessary. | |
| 20:17:03 | johnsom | I really haven't poked too much at the notifications, so don't have a lot of experience there. You could ask in the #openstack-dns channel though. Others there may have more details on the notifications. | |
| 20:18:39 | spatel | johnsom agreed, i am seeing when i create vm it takes almost 5 min to spun up, getting stuck in scheduling... | |
| 20:18:57 | spatel | if i delete vm it get stuck in deleting stat for 5 min.. | |
| 20:19:19 | spatel | all i am seeing in error related designate.info queue not found | |
| 21:19:12 | spatel | johnsom after restarting glance i found VM creating process speed up but deleting still taking 5 min to delete vm | |
| 21:19:53 | johnsom | So maybe the designate messages are not related | |
| 21:21:00 | spatel | possible | |
| 21:21:23 | spatel | my rabbitMQ cluster also looking health so not sure what holding back to delete vm | |
| 21:21:32 | spatel | not seeing any error also.. | |
| 21:22:32 | spatel | let me restart all service except rabbitMQ and see | |
| 23:04:01 | spatel | johnsom does image-caching is disable when using Ceph storage for glance? | |
| 23:04:24 | johnsom | spatel Sorry, I have no idea. | |
| 23:04:31 | spatel | I wonder that was my issue because recently i migrated glance to ceph rbd | |
| 23:05:07 | spatel | may be that is why its very slow to create VM (earlier when i was using local disk with glance talking few second to spun up vm) | |
| 23:05:26 | spatel | taking* | |
| 23:46:12 | spatel | johnsom all my problem solved :) | |
| 23:46:25 | johnsom | spatel Oh good | |
| 23:46:38 | spatel | after setting driver = noop in nova-api and nova-compute* nodes | |
| 23:47:20 | spatel | my issue is nova still looking for designate.info queue which doesn't exist and that was causing all chaos | |
| 23:48:02 | spatel | I need to work on designate to find out what is going on but i believe interim solution is to tell nova don't look for notification queue | |
| 23:49:19 | spatel | anyway! time for dinner! see you tomorrow.. thank you for your support. | |
| #openstack-nova - 2021-02-19 | |||
| 00:47:27 | openstackgerrit | Jorhson Deng proposed openstack/nova master: Reject to attach and detach volume while task_state is not None https://review.opendev.org/c/openstack/nova/+/758918 | |
| 02:18:05 | openstackgerrit | Merged openstack/nova master: Add regression test for bug 1914777 https://review.opendev.org/c/openstack/nova/+/775307 | |
| 02:18:07 | openstack | bug 1914777 in OpenStack Compute (nova) "Possible race condition between n-cpu and n-api when deleting a building instance" [High,In progress] https://launchpad.net/bugs/1914777 - Assigned to melanie witt (melwitt) | |
| 02:18:25 | openstackgerrit | Merged openstack/nova stable/ussuri: tools: Allow check-cherry-picks.sh to be disabled by an env var https://review.opendev.org/c/openstack/nova/+/766024 | |
| 04:47:35 | openstackgerrit | melanie witt proposed openstack/nova master: Handle instance = None in _local_delete_cleanup https://review.opendev.org/c/openstack/nova/+/775308 | |
| 07:02:05 | openstackgerrit | Merged openstack/nova stable/ussuri: libvirt: Only ask tpool.Proxy to autowrap vir* classes https://review.opendev.org/c/openstack/nova/+/761223 | |
| 07:41:04 | openstackgerrit | Yongli He proposed openstack/nova master: smartnic support https://review.opendev.org/c/openstack/nova/+/758944 | |
| 07:57:26 | yonglihe | sean-k-mooney: review invite, thanks: https://review.opendev.org/q/topic:%22bp%252Fsriov-smartnic-support%22+(status:open%20OR%20status:merged) | |
| 08:02:13 | yonglihe | gibi: review invite, thanks. https://review.opendev.org/q/topic:%22bp%252Fsriov-smartnic-support%22+(status:open%20OR%20status:merged) | |
| 08:05:11 | gibi | yonglihe: I will try | |
| 08:06:38 | yonglihe | thanks, it's kind late for this release . lots of depend-on items, good news is that's all got merged(neutron and cyborg). | |
| 08:10:22 | gibi | yonglihe: ack, honestly I have a lot of thing to review so I might not be able to get to it this week | |
| 08:12:25 | yonglihe | ack -:) | |
| 09:07:51 | lyarwood | elod: https://review.opendev.org/c/openstack/nova/+/761224 - would you mind hitting this today, downstream finally ran into it so I'm getting chased around with a pitchfork. ^_^ | |
| 09:14:56 | elod | lyarwood: sure, looking :) | |
| 09:23:58 | openstackgerrit | Jorhson Deng proposed openstack/nova master: Reject to attach and detach volume while task_state is not None https://review.opendev.org/c/openstack/nova/+/758918 | |
| 09:24:07 | bauzas | gibi: stephenfin: eek, now I added nova/network/neutron.py in the mypy files, looks to me I need to add *all* the type hints for all the methods /o\ | |
| 09:24:21 | bauzas | gibi: stephenfin: context, Zuul is -1 for pep8 https://review.opendev.org/c/openstack/nova/+/773976 | |
| 09:26:32 | elod | lyarwood: patch looks good, +2+W'd | |
| 09:27:29 | melwitt | bauzas: mypy is now your life | |
| 09:28:02 | bauzas | :( | |
| 09:28:13 | bauzas | I was already loud about mypy | |
| 09:28:18 | bauzas | so i won't tell more | |
| 09:28:39 | melwitt | it's ok, I was just trying to be funny | |
| 09:28:40 | bauzas | but next time I'm asked to provide type hints for a module, I'll lookup whether the file is already checked | |
| 09:29:05 | bauzas | melwitt: no worries, I prefer american pies | |
| 09:29:55 | lyarwood | bauzas: -1 nit did you mean bauzas(self) -> 'SadPanda': ? | |
| 09:30:04 | melwitt | lol | |
| 09:30:09 | bauzas | excellent | |
| 09:30:28 | bauzas | AttibutError : object 'SadPanda' does not exist | |
| 09:31:56 | bauzas | but yeah, mypy is like a cold sea, you just step into it by the toe first | |
| 09:32:15 | bauzas | or you'll get cold | |
| 09:35:31 | bauzas | but at least I can reproduce it locally \o/ | |
| 09:41:18 | swp20 | stephenfin: morning | |
| 09:42:30 | swp20 | for the noVNC patch, can we add TLS in the follow change? as the bp's goal is add the password to get vnc url. | |
| 09:51:48 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Add net & utils methods for routed nets & segments https://review.opendev.org/c/openstack/nova/+/773976 | |
| 09:51:49 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Add a routed networks scheduler pre-filter https://review.opendev.org/c/openstack/nova/+/749068 | |
| 09:52:04 | bauzas | stephenfin: gibi: eventually gave up on fixing neutron issues | |
| 09:52:07 | bauzas | (for mypy) | |
| 09:56:46 | openstackgerrit | Wenping Song proposed openstack/nova master: Nova supports password encrypted VNC https://review.opendev.org/c/openstack/nova/+/622336 | |
| 10:00:17 | openstackgerrit | Lucian Petrut proposed openstack/nova master: hyper-v rbd volume support https://review.opendev.org/c/openstack/nova/+/763550 | |
| 10:06:02 | brinzhang | stephenfin, swp20: yes, this spec we just want to add the vnc authenticate | |
| 10:06:02 | brinzhang | by password, if we want to add the x509vnc support, we can do that later, rather than in this spec | |
| 10:08:00 | brinzhang | stephenfin, swp20: as your point in https://review.opendev.org/c/openstack/nova/+/622336/30/nova/console/rfb/auths.py#30 | |