| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-08-30 | |||
| 11:26:06 | gibi | elodilles: could you check this backport https://review.opendev.org/q/I9a2a7614313297f11a55d99fb94916d3583a9504 , please? :) | |
| 12:04:09 | manuvakery1 | Hey.. All of sudden lot of vms shutoff automatically | |
| 12:04:15 | manuvakery1 | "libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: /usr/include/c++/11/bits/random.tcc:2667: void std::discrete_distribution<_IntType>::param_type::_M_initialize() [with _IntType = int]: Assertion '__sum > 0' failed" | |
| 12:04:27 | manuvakery1 | I see this error in the logs | |
| 12:05:10 | manuvakery1 | qemu instance log also has same error | |
| 12:05:19 | manuvakery1 | https://www.irccloud.com/pastebin/edKaN6J0/ | |
| 12:05:46 | manuvakery1 | running openstack wallaby | |
| 12:06:02 | manuvakery1 | any hints on the root cause | |
| 12:06:39 | auniyal | it says reason=crashed, so nova may be not aware of it ! | |
| 12:06:58 | auniyal | anything in nova-compute logs ? | |
| 12:07:31 | manuvakery1 | compute log also has same error | |
| 12:07:32 | manuvakery1 | 2024-08-30 11:56:08.942 22938 ERROR nova.compute.manager [instance: 9dea6796-4103-4f55-8caa-16dc6848bbf9] libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: /usr/include/c++/11/bits/random.tcc:2667: void std::discrete_distribution<_IntType>::param_type::_M_initialize() [with _IntType = int]: Assertion '__sum > 0' failed. | |
| 12:08:20 | manuvakery1 | https://www.irccloud.com/pastebin/5H4M8DvL/ | |
| 12:09:34 | auniyal | so this is during new VM creation | |
| 12:09:39 | manuvakery1 | yup | |
| 12:14:46 | elodilles | gibi: ACK, will do o/ | |
| 12:15:24 | elodilles | bauzas auniyal : if you have some time to review the python-novaclient release patch, that would be awesome: https://review.opendev.org/c/openstack/releases/+/927136 | |
| 12:25:08 | Uggla | gibi, sorry I have just noticed you comments. Good that the instance.update was due to a misconfiguration because I was not confident on that one. | |
| 12:26:07 | Uggla | gibi, I need to fix some stuff on the client patches, so I will bump to 2.97. I will tell you when it will be ready. | |
| 12:28:31 | auniyal | elodilles, done | |
| 12:30:21 | elodilles | auniyal: thanks \o/ | |
| 12:54:38 | sean-k-mooney | bauzas: gibi can we proceed with https://review.opendev.org/c/openstack/nova/+/922492 | |
| 12:55:46 | sean-k-mooney | the depend on is already meged | |
| 12:58:39 | opendevreview | Fabian Wiesel proposed openstack/nova master: Vmware: Remove uuid parameter from get_vmdk_info call https://review.opendev.org/c/openstack/nova/+/910627 | |
| 12:59:13 | gibi | sean-k-mooney: so 3.12 is a non mandatory test target for 2024.2 and ubuntu 24.04 not on the tested runtimes list. How stable this job is with 3.12? Do we risk a gate block? | |
| 13:01:49 | bauzas | I left a +2 for saying I'm OK but I agree with gibi, we need to discuss that point before merging | |
| 13:02:24 | sean-k-mooney | https://zuul.openstack.org/builds?job_name=devstack-platform-ubuntu-noble&skip=0 it looks resanobale to me on the devstack side https://zuul.openstack.org/builds?job_name=devstack-platform-ubuntu-noble&skip=0 we also moved whitebox to it a while ago | |
| 13:03:31 | sean-k-mooney | it will be a required testing runtime for 2025.1 so we will need to start moving all the jobs shortly | |
| 13:03:56 | sean-k-mooney | so i want to have this in dalmaiton too if we can | |
| 13:04:19 | gibi | sean-k-mooney: OK those job results looks acceptable to me. I think we can merge it and if we see an uptick of nova-next failures we can always revert it. | |
| 13:04:46 | sean-k-mooney | +1 to reverting if it proves unstable | |
| 13:05:16 | gibi | bauzas: any objection before I +A it? | |
| 13:06:04 | sean-k-mooney | i think the python 3.12 coverage is very valuable as its also a proxy for rhel 10 | |
| 13:06:10 | bauzas | I don't find any | |
| 13:06:11 | sean-k-mooney | until we get centos 10 enabled | |
| 13:06:32 | sean-k-mooney | to od that i or someone need to update disk image built to supprot it but that not that hard | |
| 13:06:33 | bauzas | we could revert the bump if needed | |
| 13:07:15 | gibi | approved | |
| 13:07:46 | sean-k-mooney | thanks :) im going to grab a coffee brb | |
| 13:15:13 | gibi | have a nice one | |
| 13:16:20 | opendevreview | Fabian Wiesel proposed openstack/nova master: Vmware: Remove uuid parameter from get_vmdk_info call https://review.opendev.org/c/openstack/nova/+/910627 | |
| 13:20:44 | sean-k-mooney | its some cold brew i made at the start of the week. i have not been sleeping weel the last few days so (4-5) horus most night beign awake until after 5 in the morning | |
| 13:21:03 | opendevreview | Fabian Wiesel proposed openstack/nova master: Vmware: Remove uuid parameter from get_vmdk_info call https://review.opendev.org/c/openstack/nova/+/910627 | |
| 13:21:27 | sean-k-mooney | so this is my first coffe of the day with a big dash of salted carammel, i normlaly drink it black | |
| 13:22:05 | sean-k-mooney | idd blame the lack fo sleep on the cofee but i normally only have one cup a day and i skip it 2 days thsi week | |
| 13:22:35 | gibi | sean-k-mooney: this year I did not brew cold. I try to limit my coffein intake as it disturbs my sleep and the cold one has more kick than a lightly packed mokka | |
| 13:23:33 | sean-k-mooney | ya i got an expresso machien about 3-4 years ago so i went form 2-3 nesspreo pods a day to one ammericano | |
| 13:23:49 | sean-k-mooney | turns out to be about the same amount of cafeine | |
| 13:54:06 | artom | Where's our opensearch/elasticsearch instance again? | |
| 13:55:28 | artom | https://opensearch.logs.openstack.org/_dashboards/app/ needs a login... | |
| 14:01:23 | dansmith | artom: openstack/openstack | |
| 14:01:31 | sean-k-mooney | ya that | |
| 14:01:32 | dansmith | apparently the new one can't be unauthenticated | |
| 14:03:38 | sean-k-mooney | artom: are you looking into my comment https://review.opendev.org/c/openstack/nova/+/926083/comments/ad6713dc_e6542156 | |
| 14:04:37 | artom | sean-k-mooney, the shelve thing? No, I got an openstacksdk error in the latest results | |
| 14:04:45 | sean-k-mooney | ah | |
| 14:04:55 | sean-k-mooney | ya i saw that job fail in one of my other patches | |
| 14:05:07 | sean-k-mooney | i have not reeckec but it passed on the next patch in my series | |
| 14:05:10 | artom | Really trivial fix, I was just looking how often it happened | |
| 14:05:13 | sean-k-mooney | so i assume there s an itermint failure | |
| 14:05:34 | artom | Yeah, cloud.get returns None, and we unconditionally try to [ ] that | |
| 14:06:00 | artom | Seems like just skipping the None case with an if and retrying the loop iteration would help... | |
| 14:07:14 | sean-k-mooney | manuvakery1: that feels like something i have heard fo before but its obviously a qemu bug | |
| 14:07:56 | sean-k-mooney | i guess it could be in libvirt but its defnintly not in nova | |
| 14:08:47 | sean-k-mooney | well cloud.get is presumable reading form the clouds.yaml which should nto be none | |
| 14:09:49 | artom | I think it's an API call... it's polling for stack events | |
| 14:10:17 | sean-k-mooney | i didnt look at the actual errror yet | |
| 14:10:19 | artom | Sorry, cloud.get_stack() :) | |
| 14:10:34 | artom | Which I assume calls Heat | |
| 14:10:44 | sean-k-mooney | presumeable | |
| 15:11:07 | opendevreview | Fabian Wiesel proposed openstack/nova master: Vmware: Remove uuid parameter from get_vmdk_info call https://review.opendev.org/c/openstack/nova/+/910627 | |
| 16:01:41 | dansmith | sean-k-mooney: +2 now ^ | |
| 17:04:35 | sean-k-mooney | dansmith: im out of energy now but i reveiwed that and approved. if you are going to request a FFE for the nova enablement of multiple locations support, given its size i think i woudl support that but if just want to leave that to next cycle it think thats also fine. it really depends n when the tempest stuff get resolved i guess so ill deffer to your judgement there. | |
| 17:05:10 | dansmith | well, tbh, I wasn't even going to consider it a feature | |
| 17:05:30 | dansmith | it's tiny and it's just using the glance api the right way, | |
| 17:05:39 | dansmith | but if you think it needs to be considered as such, then that's fine | |
| 17:05:52 | dansmith | not sure what that really means with bauzas gone next week though | |
| 17:06:09 | sean-k-mooney | well more i was suggeting its small enouch that we may want to consier not procedurlly blocking it | |
| 17:06:25 | sean-k-mooney | if the tempet issue can be resolved by the next team meeting | |
| 17:06:38 | sean-k-mooney | well FFE dont requrie PTL input | |
| 17:06:50 | sean-k-mooney | historically it only requried 3 cores to say yes | |
| 17:07:23 | sean-k-mooney | i only consider https://review.opendev.org/c/openstack/nova/+/891036 a feature in that we would not back port it | |
| 17:08:04 | sean-k-mooney | mainly because of the requirements bump but i agree otherwise its just using glances api the secure way | |
| 17:08:04 | dansmith | there's no point in backporting it | |
| 17:08:22 | sean-k-mooney | right because there is no supprot in older glance | |
| 17:08:27 | dansmith | I meant determining whether it was a feature or not, for the PTL buy-in.. I think that's bauzas' decision to make if it's unclear | |
| 17:08:58 | dansmith | anyway, we'll keep pushing on the tempest patch and can revisit on monday maybe | |
| 17:09:24 | sean-k-mooney | ack, either way i thin kthis could merge after rc1 if we decied to wait | |
| 17:09:29 | dansmith | the tempest patch isn't required, AFAIK, it just reduces a race that is already there just not hit because we don't actually use it yet, but it has passed without it | |
| 17:09:32 | sean-k-mooney | having it 2025.1 is the main thing | |
| 17:10:10 | sean-k-mooney | ah ok so its not strictly a dep but we would have some instablity without it | |
| 17:10:34 | dansmith | right | |
| 17:11:01 | sean-k-mooney | ok well im going to checkout for the weekend | |
| 17:11:07 | sean-k-mooney | catch you monday o/ | |
| 17:11:18 | dansmith | o/ | |
| 18:39:29 | opendevreview | Jay Faulkner proposed openstack/nova master: [ironic] Factor out metadata and send to ironic https://review.opendev.org/c/openstack/nova/+/927639 | |
| 19:38:14 | opendevreview | Merged openstack/nova stable/2023.1: Added context manager for instance lock https://review.opendev.org/c/openstack/nova/+/912511 | |
| 19:39:16 | bauzas | I'm still there, I have some slides to work on | |
| 19:59:18 | JayF | I think I'm going to need an FFE for 927639 if it's merging for C; I've been blocked on reviews and got some today and revised. I'd imagine it'll be ready to land very early next week. TBH, I'd like it in so it's off my plate, but it won't cause significant hardship if we have to push it. | |
| 19:59:29 | JayF | s/C/D/ | |