| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2022-02-22 | |||
| 15:12:39 | sean-k-mooney | well it shoudl still be rejected | |
| 15:12:49 | gibi | it is satisfyable | |
| 15:13:00 | sean-k-mooney | only if we have t1 | |
| 15:13:02 | gibi | yes | |
| 15:13:11 | sean-k-mooney | right im not sure we shoudl allow that | |
| 15:13:15 | gibi | I think we cannot expect that the client will normalize the query | |
| 15:13:35 | sean-k-mooney | i dont think any frobiden traits should over lap with any required even if we in | |
| 15:13:45 | gibi | maybe these required triat pieces are coming form different direction in nova | |
| 15:13:47 | sean-k-mooney | since we cant make forbidne traits optional | |
| 15:14:26 | gibi | one prefilter says I need either T1 or T2 another prefilter says I don't like T2 | |
| 15:14:40 | gibi | I don't want nova to do the normalization in this case | |
| 15:14:58 | sean-k-mooney | i kind of feel like nova shoudl be preprocessing that rahter then having ot have placment fiture that out | |
| 15:15:08 | sean-k-mooney | but i see why you are trying to be more lax | |
| 15:15:32 | gibi | I think it is a lot easier to accept it than forcing the client to normalize | |
| 15:16:07 | sean-k-mooney | required=in:T1,T2&required=!T2 is logically jsut required=T1 | |
| 15:16:12 | gibi | yes | |
| 15:16:26 | sean-k-mooney | if we do accept that we shoudl really preprocess that before we generate the sql queries | |
| 15:16:34 | gibi | why? | |
| 15:16:44 | gibi | I think sql can handle this | |
| 15:17:07 | sean-k-mooney | i mean it could but i think that will not scale well | |
| 15:17:07 | gibi | I even think sql should have the query planner to optimize this out | |
| 15:17:10 | bauzas | sean-k-mooney: thanks for accepting the existing SQL query from gibi | |
| 15:17:29 | gibi | * the sql server | |
| 15:17:41 | sean-k-mooney | gibi: if that was your orginal intent i guess what you have will do that | |
| 15:17:47 | bauzas | sean-k-mooney: as I knew since 10 years, in general the subqueries use the same execution plans than in the inner joins | |
| 15:18:08 | bauzas | but we could verify it with the EXPLAIN query | |
| 15:18:22 | sean-k-mooney | bauzas: my expirnce is inner joins like that tended to result in more memory usage then subquires | |
| 15:18:57 | sean-k-mooney | im not sure that sqligte will optimise to the same degree as mysql in this regard | |
| 15:19:10 | gibi | I don't believe either | |
| 15:19:23 | gibi | but nobody runs sqlite in production (hopefully :) | |
| 15:19:46 | bauzas | sean-k-mooney: well, in general it depends on the DMBS | |
| 15:19:55 | sean-k-mooney | ya i was more worreid about our func tests but | |
| 15:20:03 | bauzas | sean-k-mooney: but with mysql 5.0 I saw that it was the same | |
| 15:20:09 | sean-k-mooney | we likely wont have db quiries that will stress it enough there to matter | |
| 15:20:19 | opendevreview | Elod Illes proposed openstack/nova master: Lightbits LightOS driver https://review.opendev.org/c/openstack/nova/+/821606 | |
| 15:21:24 | sean-k-mooney | bauzas: lets revisit this in the future when we have data one way or another | |
| 15:21:32 | bauzas | ++ | |
| 15:21:36 | bauzas | agreed | |
| 15:21:58 | gibi | I promise I get back to this and gather some performance data from a mysql env | |
| 15:22:29 | sean-k-mooney | bauzas: if you woudl not mind reviwing 826492 that is the only patch that does not have +w | |
| 15:22:47 | sean-k-mooney | bauzas: gibi and i were just discussing how we detct conflicts | |
| 15:22:48 | bauzas | sean-k-mooney: I could do this after the meeting | |
| 15:22:53 | bauzas | btw. | |
| 15:23:02 | bauzas | nova meeting in 38 mins | |
| 15:23:03 | bauzas | here | |
| 15:23:11 | sean-k-mooney | ack for now i think i can accept gibis explation of what they were trying to enable | |
| 15:23:47 | sean-k-mooney | we can let the db handel the extra complexity for now and we can simplyfy the query either in nova or placment in the futrue if needed | |
| 15:23:47 | gibi | cool | |
| 15:23:54 | gibi | thank you | |
| 15:23:54 | kashyap | gibi: When you get a minute, do you, or anyone know what's going in this test here? - https://github.com/openstack/tempest/blob/master/tempest/api/compute/images/test_list_image_filters.py#L108 | |
| 15:24:32 | sean-k-mooney | gibi: im just worried that if we say "the db query optimiser will take care of it" too much we could get some nasty surprise in large deplopyments | |
| 15:24:54 | kashyap | (I'm trying to debug an upstream TripleO bug from hell that's "critical", and it's hitting the failure at line-128) | |
| 15:24:58 | gibi | sean-k-mooney: sure, I will do some performance testing to see if it matters | |
| 15:26:20 | gibi | kashyap: looking... | |
| 15:27:21 | kashyap | (That's the Tempest bug here: https://bugs.launchpad.net/tripleo/+bug/1959014/) | |
| 15:27:28 | gibi | kashyap: boot servers, then snapshot the server and upload the image to glance | |
| 15:27:49 | kashyap | Is it _really_ a live snapshot? Where "it" == create_image_from_server() | |
| 15:27:58 | kashyap | Is it a live snapshot? /me looks | |
| 15:28:53 | kashyap | create_image_from_server() --> create_image() | |
| 15:30:19 | gibi | it depends | |
| 15:31:23 | kashyap | Hm, I'm combing through the libvirt <-> QEMU logs given that it seems to fail at "live snapshot", but need more time to dig | |
| 15:31:31 | sean-k-mooney | kashyap its not always a live snapshot there is a config option that you can enabel. | |
| 15:31:35 | gibi | I will try to get to the logic nova does... | |
| 15:31:40 | sean-k-mooney | we also fall back to cold snapshots in some cases | |
| 15:31:50 | kashyap | sean-k-mooney: Yeah, I realize that | |
| 15:31:55 | sean-k-mooney | nova's api does not gurentee that its live | |
| 15:32:11 | sean-k-mooney | however if we stop it we will start the vm after the snap shot | |
| 15:32:33 | sean-k-mooney | so the pre and post state shoudl be the same | |
| 15:35:21 | gibi | kashyap: https://github.com/openstack/nova/blob/28cbcbb3737d544b7f73bd5aec55ccbd40222c95/nova/virt/libvirt/driver.py#L2893-L2922 | |
| 15:35:34 | gibi | nova logs if it is live of cold snapshot | |
| 15:36:41 | kashyap | gibi: Thank you; so this path live snapshot here. (The instance isn't in SHUTDOWN state) | |
| 15:37:01 | kashyap | s/looks/looks at/ | |
| 15:38:14 | kashyap | That's the Nova log - https://logserver.rdoproject.org/83/38983/2/check/periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-victoria/67d64d4/logs/overcloud-novacompute-0/var/log/containers/nova/nova-compute.log.1.gz | |
| 15:38:36 | kashyap | (There's another nova-compute.log.txt.gz in the directory; maybe log rotation) | |
| 15:39:18 | kashyap | gibi: Yep, it logs it: | |
| 15:39:19 | kashyap | 2022-02-08 17:25:54.917 7 INFO nova.virt.libvirt.driver [req-5a68f755-b20c-41a7-a005-3bcc8b800b28 9afce47c724e4db7958ea7fc1eca1c27 8f305ae0fc324a41971784ef2430d8e3 - default default] [instance: 4262e009-6e83-4d8f-bd24-8a478cecda4a] Beginning live snapshot process | |
| 15:39:23 | kashyap | Thank you! | |
| 15:42:29 | gibi | kashyap: based on artom's comment in the bug there is also a log about direct snapshot it should be something like "Performing standard snapshot because direct..." | |
| 15:42:40 | gibi | direct snasphot only works with RDB I think | |
| 15:42:59 | gibi | RBD | |
| 15:43:03 | kashyap | As if people know what is a "standard snapshot" :) | |
| 15:43:30 | kashyap | I don't see such a message in the above nova-compute log | |
| 15:44:05 | kashyap | gibi: So, here it is directly hitting live snapshot process | |
| 15:44:33 | kashyap | I'm talking to the libvirt dev to see if we can reverse engineer the failure ... from the QEMU traces | |
| 15:44:40 | kashyap | (26MB: https://kashyapc.fedorapeople.org/lp_1959014/QMP_exchange.txt) | |
| 15:44:44 | gibi | then maybe artom looked at a different run her https://bugs.launchpad.net/tripleo/+bug/1959014/comments/2 | |
| 15:45:09 | kashyap | Yeah, likely. This is a different run where the (QEMU) 'blockdev-del' failure is occuring | |
| 15:50:22 | sean-k-mooney | well standard snapshot in this context means a local snapshot of a file done by qemu | |
| 15:50:37 | kashyap | I've asked Peter Krempa from libvirt about the log traces; we'll see if we get something | |
| 15:50:42 | sean-k-mooney | vs direct which means we are delegating the snapshot to the ceph cluster | |
| 15:51:13 | kashyap | sean-k-mooney: Yeah, but "standard" and "direct" are too abstract terms. We should be explicit on what it means. | |
| 15:51:30 | kashyap | (But that's a patch for a different day.) | |
| 15:52:21 | sean-k-mooney | well perhaps but thse are really intended for us as nova develpers to read | |
| 15:52:59 | kashyap | Yeah, even for developers who work on different subsystems it will be far from obvious. | |
| 15:53:04 | sean-k-mooney | so there si some expecation of knowing what the code does | |
| 15:53:24 | gibi | kashyap: looking at the req-5a68f755-b20c-41a7-a005-3bcc8b800b28 you quoted above, I don't see anything in the logs about the failure, nova logged that the snapshot is done and the image is uploaded https://review.rdoproject.org/zuul/build/67d64d4a4d584eb8be9db0fbc6e18c9d/log/logs/overcloud-novacompute-0/var/log/containers/nova/nova-compute.log.1.gz#5705 | |
| 15:54:16 | kashyap | gibi: Hmmm, odd; so the guy says the above "reproduces" the problem, not sure what is going on here | |
| 15:54:56 | kashyap | But in the libvirtd debug log, I _do_ see | |
| 15:54:59 | kashyap | 2022-02-08 17:32:24.369+0000: 31580: error : qemuMonitorJSONCheckErrorFull:406 : internal error: unable to execute QEMU command 'blockdev-del': Failed to find node with node-name='libvirt-4- | |
| 15:54:59 | kashyap | : | |