| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-22 | |||
| 15:23:02 | bauzas | nova meeting in 38 mins | |
| 15:23:03 | bauzas | here | |
| 15:23:11 | sean-k-mooney | ack for now i think i can accept gibis explation of what they were trying to enable | |
| 15:23:47 | gibi | cool | |
| 15:23:47 | sean-k-mooney | we can let the db handel the extra complexity for now and we can simplyfy the query either in nova or placment in the futrue if needed | |
| 15:23:54 | kashyap | gibi: When you get a minute, do you, or anyone know what's going in this test here? - https://github.com/openstack/tempest/blob/master/tempest/api/compute/images/test_list_image_filters.py#L108 | |
| 15:23:54 | gibi | thank you | |
| 15:24:32 | sean-k-mooney | gibi: im just worried that if we say "the db query optimiser will take care of it" too much we could get some nasty surprise in large deplopyments | |
| 15:24:54 | kashyap | (I'm trying to debug an upstream TripleO bug from hell that's "critical", and it's hitting the failure at line-128) | |
| 15:24:58 | gibi | sean-k-mooney: sure, I will do some performance testing to see if it matters | |
| 15:26:20 | gibi | kashyap: looking... | |
| 15:27:21 | kashyap | (That's the Tempest bug here: https://bugs.launchpad.net/tripleo/+bug/1959014/) | |
| 15:27:28 | gibi | kashyap: boot servers, then snapshot the server and upload the image to glance | |
| 15:27:49 | kashyap | Is it _really_ a live snapshot? Where "it" == create_image_from_server() | |
| 15:27:58 | kashyap | Is it a live snapshot? /me looks | |
| 15:28:53 | kashyap | create_image_from_server() --> create_image() | |
| 15:30:19 | gibi | it depends | |
| 15:31:23 | kashyap | Hm, I'm combing through the libvirt <-> QEMU logs given that it seems to fail at "live snapshot", but need more time to dig | |
| 15:31:31 | sean-k-mooney | kashyap its not always a live snapshot there is a config option that you can enabel. | |
| 15:31:35 | gibi | I will try to get to the logic nova does... | |
| 15:31:40 | sean-k-mooney | we also fall back to cold snapshots in some cases | |
| 15:31:50 | kashyap | sean-k-mooney: Yeah, I realize that | |
| 15:31:55 | sean-k-mooney | nova's api does not gurentee that its live | |
| 15:32:11 | sean-k-mooney | however if we stop it we will start the vm after the snap shot | |
| 15:32:33 | sean-k-mooney | so the pre and post state shoudl be the same | |
| 15:35:21 | gibi | kashyap: https://github.com/openstack/nova/blob/28cbcbb3737d544b7f73bd5aec55ccbd40222c95/nova/virt/libvirt/driver.py#L2893-L2922 | |
| 15:35:34 | gibi | nova logs if it is live of cold snapshot | |
| 15:36:41 | kashyap | gibi: Thank you; so this path live snapshot here. (The instance isn't in SHUTDOWN state) | |
| 15:37:01 | kashyap | s/looks/looks at/ | |
| 15:38:14 | kashyap | That's the Nova log - https://logserver.rdoproject.org/83/38983/2/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-victoria/67d64d4/logs/overcloud-novacompute-0/var/log/containers/nova/nova-compute.log.1.gz | |
| 15:38:36 | kashyap | (There's another nova-compute.log.txt.gz in the directory; maybe log rotation) | |
| 15:39:18 | kashyap | gibi: Yep, it logs it: | |
| 15:39:19 | kashyap | 2022-02-08 17:25:54.917 7 INFO nova.virt.libvirt.driver [req-5a68f755-b20c-41a7-a005-3bcc8b800b28 9afce47c724e4db7958ea7fc1eca1c27 8f305ae0fc324a41971784ef2430d8e3 - default default] [instance: 4262e009-6e83-4d8f-bd24-8a478cecda4a] Beginning live snapshot process | |
| 15:39:23 | kashyap | Thank you! | |
| 15:42:29 | gibi | kashyap: based on artom's comment in the bug there is also a log about direct snapshot it should be something like "Performing standard snapshot because direct..." | |
| 15:42:40 | gibi | direct snasphot only works with RDB I think | |
| 15:42:59 | gibi | RBD | |
| 15:43:03 | kashyap | As if people know what is a "standard snapshot" :) | |
| 15:43:30 | kashyap | I don't see such a message in the above nova-compute log | |
| 15:44:05 | kashyap | gibi: So, here it is directly hitting live snapshot process | |
| 15:44:33 | kashyap | I'm talking to the libvirt dev to see if we can reverse engineer the failure ... from the QEMU traces | |
| 15:44:40 | kashyap | (26MB: https://kashyapc.fedorapeople.org/lp_1959014/QMP_exchange.txt) | |
| 15:44:44 | gibi | then maybe artom looked at a different run her https://bugs.launchpad.net/tripleo/+bug/1959014/comments/2 | |
| 15:45:09 | kashyap | Yeah, likely. This is a different run where the (QEMU) 'blockdev-del' failure is occuring | |
| 15:50:22 | sean-k-mooney | well standard snapshot in this context means a local snapshot of a file done by qemu | |
| 15:50:37 | kashyap | I've asked Peter Krempa from libvirt about the log traces; we'll see if we get something | |
| 15:50:42 | sean-k-mooney | vs direct which means we are delegating the snapshot to the ceph cluster | |
| 15:51:13 | kashyap | sean-k-mooney: Yeah, but "standard" and "direct" are too abstract terms. We should be explicit on what it means. | |
| 15:51:30 | kashyap | (But that's a patch for a different day.) | |
| 15:52:21 | sean-k-mooney | well perhaps but thse are really intended for us as nova develpers to read | |
| 15:52:59 | kashyap | Yeah, even for developers who work on different subsystems it will be far from obvious. | |
| 15:53:04 | sean-k-mooney | so there si some expecation of knowing what the code does | |
| 15:53:24 | gibi | kashyap: looking at the req-5a68f755-b20c-41a7-a005-3bcc8b800b28 you quoted above, I don't see anything in the logs about the failure, nova logged that the snapshot is done and the image is uploaded https://review.rdoproject.org/zuul/build/67d64d4a4d584eb8be9db0fbc6e18c9d/log/logs/overcloud-novacompute-0/var/log/containers/nova/nova-compute.log.1.gz#5705 | |
| 15:54:16 | kashyap | gibi: Hmmm, odd; so the guy says the above "reproduces" the problem, not sure what is going on here | |
| 15:54:56 | kashyap | But in the libvirtd debug log, I _do_ see | |
| 15:54:59 | kashyap | : | |
| 15:54:59 | kashyap | 2022-02-08 17:32:24.369+0000: 31580: error : qemuMonitorJSONCheckErrorFull:406 : internal error: unable to execute QEMU command 'blockdev-del': Failed to find node with node-name='libvirt-4- | |
| 15:55:03 | kashyap | storage' | |
| 15:55:36 | gibi | hm based on the timestamp that is 7 mintutes later | |
| 15:55:55 | kashyap | ("blockdev-del" deletes a block device that's added by "blockdev-add") | |
| 15:56:44 | kashyap | gibi: Hmm, I see. Then I'm scratching my head here about what's going on from a failure trigger | |
| 15:57:24 | sean-k-mooney | that is likely part of the test cleanup | |
| 15:58:18 | kashyap | sean-k-mooney: No, from the traceback in the bug description, it is happening at resource_setup() here: | |
| 15:58:18 | sean-k-mooney | if its 7 minute after the snapshot | |
| 15:58:21 | kashyap | https://github.com/openstack/tempest/blob/master/tempest/api/compute/images/test_list_image_filters.py#L110 | |
| 15:58:24 | opendevreview | Alexey Stupnikov proposed openstack/nova master: Run clean up calls when queued live migration is aborted https://review.opendev.org/c/openstack/nova/+/828570 | |
| 15:59:40 | sean-k-mooney | hum why woudl dat do a blockdev-del | |
| 15:59:48 | sean-k-mooney | unless the vm coudl not boot | |
| 15:59:53 | sean-k-mooney | and it was cleaning up | |
| 16:00:58 | kashyap | Yeah, it sounds like a clean-up act somewhere | |
| 16:01:52 | gibi | bauzas: ping :) | |
| 16:01:53 | opendevreview | Elod Illes proposed openstack/nova stable/queens: [stable-only] Fix sphinx doc job https://review.opendev.org/c/openstack/nova/+/830327 | |
| 16:02:09 | bauzas | oh shit | |
| 16:02:14 | bauzas | #startmeeting nova | |
| 16:02:14 | opendevmeet | Meeting started Tue Feb 22 16:02:13 2022 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot. | |
| 16:02:14 | opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | |
| 16:02:14 | opendevmeet | The meeting name has been set to 'nova' | |
| 16:02:24 | bauzas | hey folks, sorry the delay | |
| 16:02:31 | gmann | o/ | |
| 16:02:35 | chateaulav | \o | |
| 16:02:42 | bauzas | that's its fault | |
| 16:03:00 | bauzas | #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting | |
| 16:03:10 | elodilles | o/ | |
| 16:03:20 | bauzas | let's a quick meeting since we're close to some deadline :) | |
| 16:03:24 | bauzas | let's do* | |
| 16:03:30 | bauzas | #topic Bugs (stuck/critical) | |
| 16:03:35 | bauzas | #info No Critical bug | |
| 16:03:39 | bauzas | #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 27 new untriaged bugs (-15 since the last meeting) | |
| 16:03:48 | bauzas | thanks gibi and others for having looked at the bugs | |
| 16:03:52 | bauzas | #help Nova bug triage help is appreciated https://wiki.openstack.org/wiki/Nova/BugTriage | |
| 16:03:57 | bauzas | link https://storyboard.openstack.org/#!/project/openstack/placement 27 open stories (+0 since the last meeting) in Storyboard for Placement | |
| 16:04:01 | bauzas | #link https://storyboard.openstack.org/#!/project/openstack/placement 27 open stories (+0 since the last meeting) in Storyboard for Placement | |
| 16:04:06 | bauzas | voila | |
| 16:04:10 | bauzas | any bugs to discuss ? | |
| 16:04:48 | bauzas | looks not | |
| 16:04:57 | bauzas | again, thanks for the folks who looked at the bugs | |
| 16:05:10 | bauzas | #topic Gate status | |
| 16:05:14 | bauzas | #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs | |
| 16:05:17 | bauzas | #link https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly Placement periodic job status | |
| 16:05:21 | bauzas | #info Please look at the gate failures and file a bug report with the gate-failure tag. | |