| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2023-04-17 | |||
| 16:08:31 | opendevreview | Stephen Finucane proposed openstack/nova master: tests: Pass parameters to sqlalchemy.text() as bindparams https://review.opendev.org/c/openstack/nova/+/880665 | |
| 16:08:31 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Remove unnecessary 'insert()' argument https://review.opendev.org/c/openstack/nova/+/880664 | |
| 16:08:32 | opendevreview | Stephen Finucane proposed openstack/nova master: db: Avoid relying on autobegin https://review.opendev.org/c/openstack/nova/+/880668 | |
| 16:08:32 | opendevreview | Stephen Finucane proposed openstack/nova master: tests: Add missing args to sqlalchemy.Table https://review.opendev.org/c/openstack/nova/+/880667 | |
| 16:08:33 | opendevreview | Stephen Finucane proposed openstack/nova master: tests: Remove test for DB special characters https://review.opendev.org/c/openstack/nova/+/880669 | |
| 16:41:10 | opendevreview | Merged openstack/placement master: Update python testing as per zed cycle testing runtime https://review.opendev.org/c/openstack/placement/+/841690 | |
| 17:11:53 | opendevreview | Dan Smith proposed openstack/nova master: Stop ignoring missing compute nodes in claims https://review.opendev.org/c/openstack/nova/+/880633 | |
| 17:11:53 | opendevreview | Dan Smith proposed openstack/nova master: Remove silent failure to find a node on rebuild https://review.opendev.org/c/openstack/nova/+/880632 | |
| 19:30:10 | dansmith | gibi: sean-k-mooney: bauzas: re our conversation this morning, I've seen this on plenty of volume tests lately as well: https://04e1a81b9004b13f1732-58d59663991bfd1d0dc3ba3167f83a98.ssl.cf1.rackcdn.com/880632/2/check/nova-ceph-multistore/f6ebaaf/testr_results.html | |
| 19:30:23 | dansmith | i.e. several kernel crashes and, obviously, a failure to detach later | |
| 19:31:00 | dansmith | it's this kind of thing that makes me highly suspect of cirros | |
| 22:15:56 | sean-k-mooney | dansmith: if its the old cirror image form the 5.2 release | |
| 22:16:07 | sean-k-mooney | actully thts a differnt crash then i have seen in the past | |
| #openstack-nova - 2023-04-18 | |||
| 04:39:02 | frickler | 5,15 kernel should be new cirros. that failure very much looks like memory corruption to me | |
| 06:38:18 | opendevreview | Merged openstack/placement master: Bugtracker link update https://review.opendev.org/c/openstack/placement/+/876768 | |
| 07:50:07 | gibi | dansmith: I agree that kernel crash points to cirros. Obviously if we see such crashes whathever happens after it (ie detach failure) is probably a consequence of the crash. | |
| 07:50:23 | gibi | dansmith: in the original detach failure case there was no kernel crash, so this is definitely different | |
| 08:01:56 | kashyap | gibi: On that device-detach patch from the QEMU upstream, do we have a way to test it upstream? -- https://www.mail-archive.com/qemu-devel@nongnu.org/msg952944.html | |
| 08:02:25 | kashyap | gibi: I can create an RPM build of QEMU with that patch for Fedora ... if we there's a Fedora job that can test it | |
| 08:02:46 | kashyap | But probably a Ubuntu package is preferred, I guess | |
| 08:11:46 | gibi | I don't think we have a Fedora based job upstream :/ | |
| 08:12:51 | gibi | kashyap: but if you can build the fix to Fedora then you can at least try my libvirt only reproduction to see if that is fixed | |
| 08:13:48 | kashyap | gibi: Oh, right indeed. I'll give it a go this week and update you on the bug | |
| 08:13:57 | gibi | cool,thanks | |
| 10:40:03 | sean-k-mooney | gibi: bauzas i now want to add a new hw_vif_model specifically igb | |
| 10:40:05 | sean-k-mooney | https://www.qemu.org/docs/master/system/devices/igb.html | |
| 10:40:38 | sean-k-mooney | qemu added sriov virtualsiation support to it last year | |
| 10:41:05 | gibi | wut? that feels super useful for testing if it does what I think it does | |
| 10:41:07 | sean-k-mooney | unfortunetly it will take a while before our cloud providers have this | |
| 10:41:18 | sean-k-mooney | gibi: yep thats why they added it | |
| 10:41:33 | sean-k-mooney | it would allow us to take a neutron port and then create VFs from it | |
| 10:41:43 | gibi | then I'm +100 | |
| 10:42:18 | sean-k-mooney | support was added in libvirt 9.3.0 | |
| 10:42:37 | sean-k-mooney | that may be qemu 9.3.0 libvirts docs kind of suck | |
| 10:42:46 | sean-k-mooney | at makeing it cleare if they mean the qemu or libvirt version | |
| 10:44:42 | sean-k-mooney | ill finish the filter stuff firts and you know actully try and do it locally before proposing anything but if it work ill bring this up in a team meeting | |
| 10:44:51 | sean-k-mooney | just not today | |
| 10:45:42 | sean-k-mooney | on a side note this is the alpine image i created https://fileshare.seanmooney.info/ | |
| 10:46:18 | sean-k-mooney | im going to update the nova job patch to skip trying to build it and just pull that and see if it works after i finish the docs chagnes on teh schduler patch | |
| 11:16:21 | zigo | How do I migrate away a VM with status SUSPENDED ?!? | |
| 11:26:54 | sean-k-mooney | zigo: i suspended means suspended to disk. i think a cold migration might work but i would have to check the api | |
| 11:28:51 | sean-k-mooney | live migration wont because its not actully running | |
| 11:29:11 | sean-k-mooney | looking at the micate api it does not say but resize says You can only resize a server when its status is ACTIVE or SHUTOFF. | |
| 11:29:33 | sean-k-mooney | zigo: so my guess is without resuming the instnace its not possible | |
| 11:45:07 | sean-k-mooney | bauzas: you have reviewd this in the past can you take alook at this backprot again https://review.opendev.org/c/openstack/nova/+/790447 | |
| 11:59:10 | zigo | sean-k-mooney: Thanks. | |
| 12:37:35 | bauzas | sean-k-mooney: me clicks | |
| 13:32:37 | dansmith | gibi: sean-k-mooney: yep, it could be totally different problems. However, we're at the point of suspecting the guest is not playing ball, and sometimes we just see no detach and other times a kernel crash | |
| 13:32:51 | dansmith | sometimes I see only one trace in the klog when that happens, other times several like this | |
| 13:33:20 | dansmith | so I'm just saying that I think expecting cirros to behave like a real guest is already suspect (in my mind) | |
| 13:44:48 | gibi | yeah, I agree | |
| 13:51:26 | sean-k-mooney | so i have the alpine image hosted at https://fileshare.seanmooney.info/alpine.qcow2 | |
| 13:51:42 | sean-k-mooney | ill see if i can just sub that in later today into a job and see what breaks | |
| 13:54:01 | dansmith | most of the guest crashes were in kmalloc things, | |
| 13:55:28 | dansmith | which makes me wonder if there's more interaction with the host there | |
| 13:58:02 | dansmith | actually they all go through load_module() in the kernel and are failing there | |
| 14:13:32 | bauzas | dansmith: sorry I didn't had time to look at how many failures were having this kind of crash | |
| 15:45:19 | bauzas | reminder : nova meeting in 15 mins | |
| 16:00:26 | dansmith | bauzas: reminder, nova meeting 24 seconds ago | |
| 16:00:33 | opendevmeet | The meeting name has been set to 'nova' | |
| 16:00:33 | opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 16:00:33 | opendevmeet | Meeting started Tue Apr 18 16:00:33 2023 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 16:00:33 | bauzas | #startmeeting nova | |
| 16:00:40 | bauzas | dansmith: you beated me on the clock | |
| 16:00:51 | bauzas | walcome falks | |
| 16:01:03 | elodilles | :D | |
| 16:01:04 | elodilles | o/ | |
| 16:01:05 | bauzas | #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting | |
| 16:01:06 | auniyal | o/ | |
| 16:01:40 | bauzas | let's try to do a quick one | |
| 16:01:56 | bauzas | so let's start | |
| 16:02:00 | bauzas | #topic Bugs (stuck/critical) | |
| 16:02:04 | bauzas | #info No Critical bug | |
| 16:02:28 | bauzas | (welp, somehow) | |
| 16:02:32 | gibi | o/ | |
| 16:02:34 | bauzas | #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 13 new untriaged bugs (-5 since the last meeting) | |
| 16:02:41 | bauzas | kudos to gibi | |
| 16:02:52 | gibi | triaged couple of one https://etherpad.opendev.org/p/nova-bug-triage-20230411 | |
| 16:02:52 | bauzas | gibi: any bug you wanna raise today ? | |
| 16:02:58 | bauzas | thanks | |
| 16:03:02 | gibi | nothing to raise today | |
| 16:04:01 | bauzas | very quickly looked at them, thanks for fishing them | |
| 16:04:11 | bauzas | #info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster | |
| 16:04:23 | bauzas | so, next in the roster is melwitt | |
| 16:04:31 | bauzas | not sure she's around | |
| 16:04:43 | bauzas | I'll ask her later if she can help | |
| 16:04:49 | bauzas | if not, I'll take the baton | |
| 16:05:00 | bauzas | #info bug baton is being passed to melwitt/bauzas | |
| 16:05:16 | bauzas | #topic Gate status | |
| 16:05:21 | bauzas | #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs | |
| 16:05:26 | bauzas | #link https://etherpad.opendev.org/p/nova-ci-failures | |
| 16:05:37 | bauzas | do people want to discuss about the recent failures we got ? | |
| 16:05:47 | bauzas | I didn't had time to look at logsearch yet | |
| 16:06:53 | Uggla | o/ | |
| 16:06:57 | bauzas | well, fair enough | |
| 16:07:22 | bauzas | let's say I haven't seen yet a lot of failures in my internal gerrit email directory | |
| 16:07:34 | bauzas | moving on so | |
| 16:08:46 | bauzas | #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status | |
| 16:09:00 | bauzas | as always, all greens | |
| 16:09:05 | bauzas | #info Please look at the gate failures and file a bug report with the gate-failure tag. | |