| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-04-22 | |||
| 12:05:39 | kashyap | lyarwood: I also threw you under the "liaoson bus" without asking checking in first with you ... | |
| 12:17:20 | sean-k-mooney | kashyap: im not actully sure we shoudl be doing that vs the hw:profile concept | |
| 12:17:48 | kashyap | sean-k-mooney: Well, is there something up for the hw:profile thing? | |
| 12:18:00 | kashyap | sean-k-mooney: And why not? 'pc' will be dead _anyway_ | |
| 12:18:04 | sean-k-mooney | no not yet | |
| 12:18:07 | kashyap | And we'll be doing the operators a disservice | |
| 12:18:12 | kashyap | I know; not at least for 5 years | |
| 12:18:19 | sean-k-mooney | kashyap: because we can potentailly break people on upgrade | |
| 12:18:29 | sean-k-mooney | its the same reason libvirt did not change the default | |
| 12:18:58 | kashyap | Well, the discussion there is _far_ more nuanced; libvirt is different - it gives the nuts and bolts | |
| 12:19:23 | kashyap | I actually had a few paragraphs of text in the spec (but deleted), they were about: | |
| 12:19:34 | sean-k-mooney | kashyap: there are other factors but that does not demish the fact that we have many fo the same constratints | |
| 12:19:54 | kashyap | ... whether Nova shold make the 'policy' decision. I wrote an example, etc. But cut it out after I saw the recommendations from the QEMU/KVM/libvirt folks over the year | |
| 12:20:28 | kashyap | sean-k-mooney: Yeah, I'm aware of why libvirt did it; I've even linked to it in a previous edition of the commit message | |
| 12:20:36 | kashyap | (The exact change of libvirt and its reasoning) | |
| 12:20:50 | kashyap | sean-k-mooney: I'm typing fast, as I need to have lunch and head out for some air, been cooped up long | |
| 12:20:56 | kashyap | But will catch here later | |
| 12:21:46 | sean-k-mooney | ok but as it stand while i think we shoudl make q35 our default in osp in not sure if we shoudl chagne the code to do that | |
| 12:23:47 | kashyap | sean-k-mooney: Before I head out: | |
| 12:23:49 | kashyap | I'm also open for this: | |
| 12:23:56 | kashyap | - A Nova CI job with 'q35' passes | |
| 12:24:11 | kashyap | - TripleO flips the default from 'pc' --> 'q35' | |
| 12:24:12 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Fix list rendering in the accelerator support doc https://review.opendev.org/721846 | |
| 12:24:14 | sean-k-mooney | kashyap: lyarwood is working to make nova-next do that | |
| 12:24:34 | kashyap | sean-k-mooney: Yes, I investigated the failures yesterday and posted a Tempest one-liner for the failing test | |
| 12:24:39 | kashyap | All linked in the spec | |
| 12:24:42 | sean-k-mooney | the ci job and then ya the plan as far as i understood for osp was to change the default in ooo | |
| 12:25:05 | kashyap | sean-k-mooney: Yes, TripleO upstream has this posted: https://review.opendev.org/#/c/716526/ | |
| 12:25:11 | kashyap | It is waiting on the Nova CI | |
| 12:25:22 | sean-k-mooney | ah cool | |
| 12:25:40 | sean-k-mooney | im reading your spec now. go have lunch :) | |
| 12:26:43 | kashyap | sean-k-mooney: Thanks. :) Likewise | |
| 12:33:14 | bauzas | gibi: can I hold the bugs lock again ? | |
| 12:33:44 | bauzas | aarents: thanks for reporting https://bugs.launchpad.net/nova/+bug/1874032 | |
| 12:33:44 | openstack | Launchpad bug 1874032 in OpenStack Compute (nova) "nova-compute become stuck when doing IO on busy file system" [Undecided,New] | |
| 12:34:26 | gibi | bauzas: sue | |
| 12:34:28 | gibi | sure | |
| 12:35:02 | gibi | I'm swamped with a downstream stuff with a close deadline | |
| 12:35:09 | aarents | bauzas: this one is painfull.. | |
| 12:35:13 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Ussuri 21.0.0 prelude section https://review.opendev.org/721548 | |
| 12:35:31 | bauzas | gibi: <3 with love for your pain | |
| 12:36:05 | bauzas | aarents: sure, but I think it should be a Wishlist bug | |
| 12:36:29 | gibi | bauzas: thanks | |
| 12:36:59 | gibi | funny that RC1 is due tomorrow and my downstream high prio stuff due today (but was not on the radar until Monday) | |
| 12:39:38 | bauzas | gibi: in my company, we also have sometimes some downstream priorities that are in the same times than upstream yes... | |
| 12:39:50 | bauzas | I totally understand you :p | |
| 12:39:58 | bauzas | aarents: so, about your bug | |
| 12:41:08 | bauzas | aarents: you asked why we don't have compute workers, right? | |
| 12:41:20 | aarents | yep | |
| 12:41:34 | bauzas | b/c there is only one service per host | |
| 12:42:16 | bauzas | and in general, when you want to call some I/O issue, you don't do this by the nova-compute service | |
| 12:42:47 | bauzas | that's rather the nova-compute service which calls privsep | |
| 12:45:06 | bauzas | aarents: eg. https://github.com/openstack/nova/blob/master/nova/privsep/qemu.py#L37 | |
| 12:45:57 | bauzas | and then as you can see we call processutils.execute() https://github.com/openstack/nova/blob/master/nova/privsep/qemu.py#L81 | |
| 12:47:08 | aarents | bauzas: ok.. so to workaround this we have to put glance upload outside of nova-compute, in fact that what I've done to fix that on our release I made an execute(*curl).. | |
| 12:48:10 | bauzas | aarents: do you have I/O issues when snapshoting or uploading ? | |
| 12:48:22 | bauzas | from what I see, you generate some I/O calls | |
| 12:48:41 | bauzas | so I thought the problem would be around snapshoting, not uploading to glance | |
| 12:48:43 | aarents | yes during glance upload because IO is made inside nova-compute | |
| 12:49:10 | bauzas | we don't write on disk when we do glance upload, do we ? | |
| 12:49:20 | aarents | the glance upload done during instance snapshot | |
| 12:50:09 | aarents | we read an extracted file (done by qemu-ing convert)) | |
| 12:50:14 | aarents | localy | |
| 12:52:47 | bauzas | oh man this is pizza code from what I can read | |
| 12:52:57 | aarents | :) | |
| 12:53:26 | bauzas | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2252 | |
| 12:55:25 | aarents | bauzas here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2421 | |
| 12:56:20 | aarents | it then use glance api to do the PUT in glance so slow IO are running in nova-compute | |
| 12:56:33 | aarents | not in a processutils.execute() | |
| 12:58:03 | aarents | so maybe workaround processutils.execute(*curl) can be the solution to check | |
| 12:58:08 | aarents | my | |
| 12:58:14 | bauzas | aarents: that's not the glance upload issue | |
| 12:58:28 | bauzas | aarents: IMHO the problem is with the semaphore we wanna take | |
| 12:58:30 | bauzas | hence the D | |
| 12:59:38 | bauzas | the problem with reviewing this code is that nobody documented which cases raise those exceptions we catch https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2358-L2359 | |
| 13:00:10 | bauzas | aarents: because we only fall back into this semaphore lock if you are not able to do the other way | |
| 13:04:07 | aarents | I think, I'm not sure, in my case, we are in NotImplementedError because because direct_snapshot is a feature used typically with ceph, not implemented in raw/qcow2 | |
| 13:04:17 | bauzas | that's correct | |
| 13:04:25 | bauzas | I found the change in question | |
| 13:04:31 | bauzas | aarents: https://review.opendev.org/#/c/205282/ | |
| 13:04:52 | bauzas | aarents: here in your case, we're not talking of RBD snapshots, right? | |
| 13:05:14 | aarents | right, local backed instance raw or qcow2 | |
| 13:05:35 | elod | lyarwood: I've prepared the Train release patch: https://review.opendev.org/#/c/721883/ if you have time please double-check the version bump and the reasons | |
| 13:06:30 | bauzas | aarents: kk, okay so the exception catch-up clause is the default when non-RBD | |
| 13:06:37 | bauzas | (just speaking out loud) | |
| 13:06:49 | aarents | exactly | |
| 13:13:24 | bauzas | aarents: https://review.opendev.org/#/c/609180/ | |
| 13:13:55 | bauzas | aarents: fwiw, I'm about to triage it as Invalid | |
| 13:14:14 | bauzas | but you could open a blueprint for optimizing how we manage I/O concurrency | |
| 13:18:53 | sean-k-mooney | stephenfin: i have added the topic of libvirt hardware profiles to the ptg topic list https://etherpad.opendev.org/p/nova-victoria-ptg on line 60 by the way | |
| 13:19:03 | stephenfin | sweet | |
| 13:19:10 | sean-k-mooney | i will start fleshing it out more here https://etherpad.opendev.org/p/libvirt-hardware-profiles | |
| 13:19:38 | sean-k-mooney | but im not sure if i will get to it this week | |
| 13:19:58 | sean-k-mooney | the spec template is more or less the same form last cycle right? | |
| 13:21:01 | sean-k-mooney | oh there already is one https://github.com/openstack/nova-specs/blob/master/specs/victoria-template.rst | |
| 13:22:01 | bauzas | sean-k-mooney: stephenfin: if you don't mind MHO, I'd not want to have the hardware profiles be operator-defined | |
| 13:22:10 | bauzas | this defeats interop | |
| 13:22:18 | sean-k-mooney | bauzas: yep its out of scope in the first version | |
| 13:22:29 | bauzas | in *any* version tbc :) | |
| 13:22:54 | sean-k-mooney | i said we could add that after if we wanted but only if they cannot modify intree ones and if we have an api to list and show the hardware profiles | |