| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-02-18 | |||
| 15:51:56 | openstackgerrit | Lucas Alvares Gomes proposed openstack/nova master: [OVN] Adapt the live-migration job scripts to work with OVN https://review.opendev.org/c/openstack/nova/+/776419 | |
| 15:51:59 | sean-k-mooney | and we called that out in the spec by the way | |
| 15:52:02 | sean-k-mooney | " The point of this microversion is really to signal that users should not be using this legacy part of the compute API, and to set a timer on when it could be removed if nova ever starts requiring a higher minimum supported microversion in the distant future." | |
| 15:52:46 | sean-k-mooney | gibi: is there an etherpad for the xena ptg? | |
| 15:53:09 | sean-k-mooney | gibi: part of me really wants to declare xena a tech debt removal cycle | |
| 15:54:02 | sean-k-mooney | but i do think discussing raising the min microverion may be somethign we shoudl discuss or a differnt path to removing long deperecated apis | |
| 15:54:16 | gibi | sean-k-mooney: good point I should have start a xena ptg pad | |
| 15:54:40 | sean-k-mooney | gibi: ya that getting worringly close... | |
| 15:55:07 | gibi | sean-k-mooney: I've just read back, if you want to discuss the unembargoed security bug on the meeting then please bring it up :) either in the bug section or in the OpenDiscussion | |
| 15:56:02 | sean-k-mooney | an sure i thin the networking one im goign to fix via other work so ill just add it to the things i test | |
| 15:56:17 | gibi | ok | |
| 16:00:55 | gibi | sean-k-mooney: xena ptg etherpad https://etherpad.opendev.org/p/nova-xena-ptg | |
| 16:01:19 | sean-k-mooney | gibi: thanks im going to try and avoid adding too much to it if i can :) | |
| 16:01:43 | gibi | sean-k-mooney: do not limit yourself, we can always remove things later :) | |
| 16:01:52 | sean-k-mooney | oh you already have the do not translate message nice | |
| 16:02:41 | gibi | carried over from the wallaby pad :) | |
| 16:02:45 | sean-k-mooney | ya i guess i just havent done my onw retro on how things have gone over the last 6-12 monts and what did/didnt get done | |
| 16:03:03 | gibi | sean-k-mooney: btw team meeting is ongoing :) | |
| 16:03:08 | gibi | if you want to mention something | |
| 16:03:08 | sean-k-mooney | oh ya :) | |
| 16:58:07 | sean-k-mooney | actully looks liek we have a few others that were not on the list | |
| 16:58:31 | sean-k-mooney | like https://bugs.launchpad.net/nova/+bug/1861893 | |
| 16:58:32 | openstack | Launchpad bug 1861893 in OpenStack Compute (nova) "os-assisted-volume-snapshots passes unsanitised file path to the libvirt driver" [Medium,Confirmed] | |
| 16:59:08 | sean-k-mooney | we could not actuly exploit ^ so its just a security hardening case | |
| 16:59:29 | openstackgerrit | Vishal Manchanda proposed openstack/nova master: Docs: Correct ``Password injection using the dashboard`` Explanation https://review.opendev.org/c/openstack/nova/+/775084 | |
| 16:59:44 | sean-k-mooney | specificly class c1 "Not considered a practical vulnerability (but some people might assign a CVE for it)" | |
| 17:00:48 | openstack | Launchpad bug 1861893 in OpenStack Compute (nova) "os-assisted-volume-snapshots passes unsanitised file path to the libvirt driver" [Medium,Confirmed] | |
| 17:05:49 | gibi | stephenfin, dansmith: I have extra logs for the slowness of the db migration tests (even after the compaction) https://review.opendev.org/c/openstack/nova/+/775094 This result confirms for me that it is simply an extremely slow test node with noisy neighbors. I have not got new ideas how to move forward with this | |
| 17:13:49 | sean-k-mooney | can we increase the timeout wait again | |
| 17:14:16 | sean-k-mooney | you tweaked it previously if im remebring corectly | |
| 17:14:41 | sean-k-mooney | if its really just a slow node retrying it like the pytest flaky decorator does wont help | |
| 17:15:14 | sean-k-mooney | gibi: alternitivly i dont know if there is a way to do the db walk in multiple tests | |
| 17:15:49 | sean-k-mooney | e.g decompose it to do it in severnal batches of verions over 2+ tests | |
| 17:16:11 | sean-k-mooney | but ya not really sure how to workaroud this | |
| 17:16:46 | sean-k-mooney | at least nothing to adress the underlying issue | |
| 17:16:50 | dansmith | gibi: what provider? | |
| 17:17:30 | dansmith | gibi: tripleo was having some IO problems on a vexx node, but we did a bunch of digging into the node stats and it seemed like it was just too many noisy neighbors | |
| 17:18:36 | dansmith | gibi: oh that's an arm64 run | |
| 17:19:17 | sean-k-mooney | in that case ya | |
| 17:20:03 | sean-k-mooney | so Provider: linaro-us | |
| 17:20:45 | sean-k-mooney | although the py39 verion passed in hlaf the time | |
| 17:21:17 | sean-k-mooney | so its not a genral problem with the provider but there might be too high a contention on some hosts | |
| 17:21:23 | gibi | sean-k-mooney: I can increase the timeout yes | |
| 17:21:41 | gibi | dansmith: I can do some more recheck to get a non arm failure if that helps | |
| 17:22:13 | dansmith | gibi: well, I'm just saying, I don't know much about that environment.. could be raspberry pis and slow SD cards for all I know :P | |
| 17:22:29 | gibi | dansmith: OK :) | |
| 17:22:32 | sean-k-mooney | dansmith: they are realy arm server systems | |
| 17:23:15 | dansmith | sean-k-mooney: I know :P | |
| 17:24:15 | stephenfin | I think they're emulated | |
| 17:24:32 | stephenfin | Assuming it's the same machines that you linked me from the kolla jobs yesterday, sean-k-mooney | |
| 17:24:34 | dansmith | really? I thought these were actual arm boxes | |
| 17:24:35 | sean-k-mooney | dansmith: https://zuul.openstack.org/build/b038b4112ef34b15b885f204da27c808/log/primary/logs/kolla/nova/nova-compute.txt#843 | |
| 17:24:50 | sean-k-mooney | stephenfin: its is and its not emulated | |
| 17:25:18 | sean-k-mooney | well they are vms if that is what you mean | |
| 17:25:44 | stephenfin | I mean it's using <domain>qemu</domain> | |
| 17:25:45 | sean-k-mooney | but this i arm hardware provide by linaro to do real testing on arm | |
| 17:25:49 | stephenfin | not <domain>kvm</domain> | |
| 17:26:03 | sean-k-mooney | stephenfin: right but thats becasue we disabel nested vert | |
| 17:26:03 | stephenfin | so it's entirely software emulation, right? | |
| 17:26:29 | stephenfin | okay, cool, that's all I was saying | |
| 17:26:29 | sean-k-mooney | the vms created by it will use qemu without kvm in the kolla job | |
| 17:26:47 | stephenfin | the PPC jobs you linked are running on bare metal (with KVM) | |
| 17:26:56 | sean-k-mooney | but the vms we are runing the test in are on real arm hardware | |
| 17:27:10 | sean-k-mooney | stephenfin: yep its a third party ci | |
| 17:27:37 | sean-k-mooney | stephenfin: nested vrit was disabled by policy in the first party ci | |
| 17:28:13 | sean-k-mooney | that has been relaxed slightly more recnetly but its still not gernarally avaliable | |
| 17:28:41 | stephenfin | yup | |
| 17:28:54 | stephenfin | fwiw, the PPC jobs seem to be running on bare metal or nested virt | |
| 17:28:59 | stephenfin | I'm not sure which | |
| 17:29:03 | stephenfin | but KVM is enabled | |
| 17:29:12 | sean-k-mooney | i would guess nested virt | |
| 17:29:29 | sean-k-mooney | did they only have 8 cores? | |
| 17:29:31 | stephenfin | as would I. Letting arbitrary jobs loose on bare metal doesn't sound so good | |
| 17:29:36 | sean-k-mooney | and 8 G of ram | |
| 17:30:00 | stephenfin | Oh, I don't recall and I don't have the links handy | |
| 17:30:13 | sean-k-mooney | the arm jobs are 8 cores and 8g of ram like or standard ci flavors | |
| 17:31:21 | sean-k-mooney | stephenfin: not important but htat a good way to ficure out if its nested virt or not even with host passthough | |
| 17:31:33 | stephenfin | yup, good point that | |
| 17:33:44 | sean-k-mooney | that takes over an hour if your doing a remote instal like i am | |
| 19:47:25 | spatel | sean-k-mooney any idea i am seeing this error on nova - ERROR stevedore.extension [req-249d0078-a423-4815-9c32-589f1814209a 3ffb7b8d55e1471aa56407e5a583037f 584651a0fe954d9480435678ea3ce5ba - default default] Could not load 'oslo_cache.etcd3gw': No module named 'etcd3gw': ModuleNotFoundError: No module named 'etcd3gw' | |
| 19:59:14 | sean-k-mooney | am its realted to how oslo.cache is loading all the backend drivers even if you are not using it | |
| 19:59:25 | sean-k-mooney | it wont break anything but it should be fixed at some point | |
| 20:00:49 | spatel | thank you for confirmation. | |
| 20:01:30 | sean-k-mooney | when it import the backeds to see which one it shoudl use its uncondtionaly trying to import etcd3gw package | |
| 20:01:34 | spatel | After rebooting my rabbitMQ cluster look like everything slowed down, VM creation process and VM delete process are very slow | |
| 20:02:02 | spatel | I have noticed this in nova logs - failed to perform operation on queue 'notifications_designate.info' in vhost '/nova' due to timeout | |
| 20:02:33 | spatel | when i checked designate.info queue on rabbitMQ and it doesn't exist | |
| 20:02:36 | sean-k-mooney | well if you restarted it its going to cause all the compute to have to reconnect | |
| 20:02:59 | sean-k-mooney | depending on how it happend you might need to restart the nove comptue agents too | |
| 20:03:29 | spatel | hmm! i did rolling restart node-1 then node-2 ... node-3 | |
| 20:04:07 | sean-k-mooney | depending on how you have the clustering set up and loadbalnce it may or may not help | |
| 20:04:33 | spatel | i don't mind to restart all compute agent or nova-api | |
| 20:05:03 | spatel | i am wondering about 'notifications_designate.info' queue, who create that queue ? | |
| 20:05:40 | spatel | does nova create that queue when it receive designate notification? | |
| 20:06:10 | sean-k-mooney | its the other way around | |
| 20:06:19 | sean-k-mooney | designate subscipes to nova notifiction | |
| 20:06:29 | sean-k-mooney | nova does not listten for notificaiton | |
| 20:07:31 | spatel | how do i get that queue back? 'notifications_designate.info' | |
| 20:08:27 | sean-k-mooney | posibly restart the designate server but i dont really know | |
| 20:08:31 | spatel | what if i tell nova.conf driver = noop | |