| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2024-05-07 | |||
| 16:29:54 | gibi | this is the master failure https://f267574198a517c003d1-a40b9478ae1bd073dc32f331038fe6d7.ssl.cf5.rackcdn.com/periodic-weekly/opendev.org/openstack/nova/master/nova-emulation/825cc9a/testr_results.html | |
| 16:30:30 | gibi | I did not dig deeper | |
| 16:30:45 | gibi | would be nice to see if the failures on each branch are the same or not | |
| 16:30:45 | auniyal | it only adds req-id | |
| 16:30:45 | auniyal | I added this https://review.opendev.org/c/openstack/tempest/+/918439 for same - is there a way we can run priodic depends on this ? | |
| 16:30:49 | sean-k-mooney | Instance failed to spawn: oslo_messaging.rpc.client.RemoteError: Remote error: DBConnectionError (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') | |
| 16:30:59 | sean-k-mooney | so maybe an oom kill | |
| 16:31:05 | sean-k-mooney | of the db | |
| 16:31:24 | auniyal | I saw this for antelope https://zuul.openstack.org/build/183dfe690ad44431b023ef95ec347bd9 | |
| 16:31:28 | gibi | OK, I can check the logs tomorrow for possible oom | |
| 16:31:57 | dansmith | emulation being slower, oom kill seems likely I bet | |
| 16:32:02 | sean-k-mooney | we can move on lill quickly tak a look | |
| 16:32:18 | gibi | #action gibi to check the nova-emulation periodic failures for oom kill events | |
| 16:32:31 | gibi | any other gate issue we need to discuss today? | |
| 16:33:24 | sean-k-mooney | ay 04 08:51:02 np0037439600 kernel: nova-compute invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 | |
| 16:33:32 | gibi | nice catch | |
| 16:33:46 | sean-k-mooney | oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/mysql.service,task=mysqld,pid=63525,uid=116 | |
| 16:33:56 | sean-k-mooney | so nova trigered it and it kills mysql | |
| 16:34:05 | sean-k-mooney | which is why the job was unhappy | |
| 16:34:23 | gibi | could you please file a bug for it? if you cannot the I will file it tomorrow | |
| 16:34:41 | sean-k-mooney | sure this might jut need swap or some tweak to the concurancy | |
| 16:34:46 | gibi | ack, thanks | |
| 16:34:50 | gibi | moving on | |
| 16:34:52 | gibi | #topic Release Planning | |
| 16:35:01 | gibi | #link https://releases.openstack.org/dalmatian/schedule.html | |
| 16:35:08 | gibi | #link https://review.opendev.org/c/openstack/releases/+/918422 nova specific schedule is proposed | |
| 16:35:17 | gibi | #info Dalmatian-1 in 1 week | |
| 16:35:25 | gibi | #info Spec review day is next Tuesday (14th of May) | |
| 16:36:13 | gibi | anything else about the release? | |
| 16:37:10 | gibi | #topic Review priorities | |
| 16:37:16 | gibi | #link https://etherpad.opendev.org/p/nova-dalmatian-status | |
| 16:37:30 | gibi | bauzas update the status etherpad | |
| 16:37:40 | gibi | *updated | |
| 16:37:53 | gibi | any questions about the priorities? | |
| 16:38:41 | gibi | #topic Stable Branches | |
| 16:38:45 | gibi | elodilles: your turn | |
| 16:38:52 | elodilles | yepp, so | |
| 16:38:59 | elodilles | #info no recent merges, but stable/2024.1 gate should be OK | |
| 16:39:09 | elodilles | #info stable/2023.2 gate is broken due to grenade-skip-level job (fix: https://review.opendev.org/c/openstack/grenade/+/918452 ) | |
| 16:39:42 | elodilles | and it seems meanwhile, that the fix is not enough, as e.g. keystone haven't transitioned to unmaintained yet :S | |
| 16:40:16 | elodilles | #info stable/2023.1 gate is probably also broken due to grenade job, probably same issue | |
| 16:40:27 | elodilles | i'll try to look further tomorrow | |
| 16:40:56 | gibi | thanks | |
| 16:40:59 | elodilles | i've updated the stable gate pad: | |
| 16:41:02 | elodilles | #info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci | |
| 16:41:23 | elodilles | will continue to update that, too | |
| 16:41:31 | elodilles | that's all from me | |
| 16:41:38 | gibi | thanks elodilles | |
| 16:41:46 | elodilles | np | |
| 16:41:56 | gibi | any other stable topic from others? | |
| 16:42:43 | gibi | #topic vmwareapi 3rd-party CI efforts Highlights | |
| 16:42:55 | gibi | from the agenda | |
| 16:42:56 | gibi | #info Fixing some "lost" changes | |
| 16:43:16 | fwiesel | Hi, that's me... | |
| 16:43:21 | gibi | sure go ahead | |
| 16:43:45 | fwiesel | So, the way I get the changes are via ssh stream, if that gets restarted and there is a new change, that one would get lost. | |
| 16:43:56 | opendevreview | Merged openstack/os-traits master: Bump min versions to exclude known bad versions https://review.opendev.org/c/openstack/os-traits/+/917713 | |
| 16:43:59 | fwiesel | I work on polling all past changes to fix that. | |
| 16:44:13 | fwiesel | That's from my side, unless there are any questions? | |
| 16:44:42 | gibi | fwiesel: thanks for the update | |
| 16:44:48 | gibi | and the continued effort on the CI | |
| 16:45:52 | gibi | if no question from others then we move to our last topic today | |
| 16:46:17 | gibi | #topic Open discussion | |
| 16:46:22 | gibi | nothing on the agenda | |
| 16:46:34 | gibi | does anybody has anything else for today? | |
| 16:47:19 | tkajinam | nothing from me. | |
| 16:47:35 | gibi | It seem we are done for today. | |
| 16:47:38 | gibi | thanks for joining | |
| 16:47:46 | gibi | have a nice rest of the day | |
| 16:47:50 | opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2024/nova.2024-05-07-16.00.log.html | |
| 16:47:50 | opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2024/nova.2024-05-07-16.00.txt | |
| 16:47:50 | opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2024/nova.2024-05-07-16.00.html | |
| 16:47:50 | opendevmeet | Meeting ended Tue May 7 16:47:50 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | |
| 16:47:50 | gibi | #endmeeting | |
| 16:48:01 | elodilles | thanks o/ | |
| 16:48:03 | fwiesel | Thanks | |
| 16:48:57 | opendevreview | sean mooney proposed openstack/nova master: tweak emulation job to avoid OOM errors https://review.opendev.org/c/openstack/nova/+/918464 | |
| 16:49:08 | sean-k-mooney | gibi: i think ^ will fix the emulation job | |
| 16:49:09 | gibi | ganso: as far as I see in ussuri the functional env called functional | |
| 16:49:18 | gibi | so tox -e functional should work | |
| 16:49:51 | tkajinam | I wonder if we have any core time concept for review spec days. I'll try to be available around the irc meeting time anyway. | |
| 16:50:27 | gibi | tkajinam: I think there are no core time for it | |
| 16:50:45 | opendevreview | sean mooney proposed openstack/nova master: [DNM] test nova-emulation https://review.opendev.org/c/openstack/nova/+/918465 | |
| 16:52:27 | tkajinam | gibi, ok, thanks. that's what I guessed. | |
| 16:52:38 | sean-k-mooney | dansmith: by the way as far as im aware for the libvirt driver virtio-scsi should be the default scsi modle when you enable it with hw_disk_bus=scsi | |
| 16:52:57 | dansmith | sean-k-mooney: it is most definitely not | |
| 16:53:08 | dansmith | defaults to lsilogic | |
| 16:53:10 | sean-k-mooney | looking at the code paths it looks like there might be a second way to get a contoler | |
| 16:53:36 | dansmith | took me a while while debugging to realize it was working, but cirros didn't have drivers for lsi | |
| 16:53:51 | sean-k-mooney | so i dont knwo fi we regressed it then | |
| 16:54:00 | sean-k-mooney | beucase at one point it was virtio-scisi | |
| 16:54:48 | sean-k-mooney | we doucmented it as vrito-scsi here https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt-image.json#L65-L69 | |
| 16:55:17 | sean-k-mooney | in libverty | |
| 16:55:59 | dansmith | just sayin' it doesn't work for me locally unless I set that | |
| 16:56:12 | sean-k-mooney | what do you have hw_disk_bus set too | |
| 16:57:27 | sean-k-mooney | it looks like we have 2 ways to get a scsi contoler either we get fone form seting hw_scsi_modle here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5794 | |
| 16:57:45 | sean-k-mooney | or we get one by setign hw_disk_bus or hw_cdrom_bus to scsi | |
| 16:58:02 | sean-k-mooney | i was aware of the latter two options but i didnt knwo about https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L5923 | |
| 16:58:20 | dansmith | I don't have disk_bus set on the image or flavor, but it _is_ set in the BDM of courtse | |
| 16:59:33 | opendevreview | Dan Smith proposed openstack/nova master: Enable virtio-scsi in nova-next https://review.opendev.org/c/openstack/nova/+/918458 | |