| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-04-22 | |||
| 11:27:05 | elod | lyarwood: and of course I can also try to prepare the release patches if needed (do we need other than that for the release?) | |
| 11:27:23 | lyarwood | looks like we have a few things in stable/stein | |
| 11:27:34 | lyarwood | elod: if you have bandwidth then please feel free to do so for train | |
| 11:27:57 | kashyap | lyarwood: Nod | |
| 11:28:47 | elod | lyarwood: sure, I'll do that now | |
| 11:28:54 | lyarwood | awesome thanks elod | |
| 11:32:35 | elod | zigo: hmmm, interesting. I have to check that. About reproduction: there are steps in the bug report but I think that not everything are needed, but haven't looked at it yet. | |
| 11:33:30 | sean-k-mooney | alex_xu: by the way i dont know why https://review.opendev.org/#/c/662264/ that show up as last updted 2 days ago i have not touch it since i abandonded it last august and there has been no update to it in the last 2 days | |
| 11:37:59 | sean-k-mooney | alex_xu: my best guess is someone removed there name form the review list or something like that. | |
| 11:45:40 | openstackgerrit | Qiu Fossen proposed openstack/nova master: The instance is volume backed and power state is PAUSED，shelve the instance failed https://review.opendev.org/711609 | |
| 11:55:29 | gibi | lyarwood, elod: thanks a lot! | |
| 12:05:02 | openstackgerrit | Kashyap Chamarthy proposed openstack/nova-specs master: Make Q35 machine type the default for x86 https://review.opendev.org/631154 | |
| 12:05:18 | kashyap | lyarwood: --^ For your "copious free time" :) | |
| 12:05:39 | kashyap | lyarwood: I also threw you under the "liaoson bus" without asking checking in first with you ... | |
| 12:17:20 | sean-k-mooney | kashyap: im not actully sure we shoudl be doing that vs the hw:profile concept | |
| 12:17:48 | kashyap | sean-k-mooney: Well, is there something up for the hw:profile thing? | |
| 12:18:00 | kashyap | sean-k-mooney: And why not? 'pc' will be dead _anyway_ | |
| 12:18:04 | sean-k-mooney | no not yet | |
| 12:18:07 | kashyap | And we'll be doing the operators a disservice | |
| 12:18:12 | kashyap | I know; not at least for 5 years | |
| 12:18:19 | sean-k-mooney | kashyap: because we can potentailly break people on upgrade | |
| 12:18:29 | sean-k-mooney | its the same reason libvirt did not change the default | |
| 12:18:58 | kashyap | Well, the discussion there is _far_ more nuanced; libvirt is different - it gives the nuts and bolts | |
| 12:19:23 | kashyap | I actually had a few paragraphs of text in the spec (but deleted), they were about: | |
| 12:19:34 | sean-k-mooney | kashyap: there are other factors but that does not demish the fact that we have many fo the same constratints | |
| 12:19:54 | kashyap | ... whether Nova shold make the 'policy' decision. I wrote an example, etc. But cut it out after I saw the recommendations from the QEMU/KVM/libvirt folks over the year | |
| 12:20:28 | kashyap | sean-k-mooney: Yeah, I'm aware of why libvirt did it; I've even linked to it in a previous edition of the commit message | |
| 12:20:36 | kashyap | (The exact change of libvirt and its reasoning) | |
| 12:20:50 | kashyap | sean-k-mooney: I'm typing fast, as I need to have lunch and head out for some air, been cooped up long | |
| 12:20:56 | kashyap | But will catch here later | |
| 12:21:46 | sean-k-mooney | ok but as it stand while i think we shoudl make q35 our default in osp in not sure if we shoudl chagne the code to do that | |
| 12:23:47 | kashyap | sean-k-mooney: Before I head out: | |
| 12:23:49 | kashyap | I'm also open for this: | |
| 12:23:56 | kashyap | - A Nova CI job with 'q35' passes | |
| 12:24:11 | kashyap | - TripleO flips the default from 'pc' --> 'q35' | |
| 12:24:12 | openstackgerrit | Takashi Natsume proposed openstack/nova master: Fix list rendering in the accelerator support doc https://review.opendev.org/721846 | |
| 12:24:14 | sean-k-mooney | kashyap: lyarwood is working to make nova-next do that | |
| 12:24:34 | kashyap | sean-k-mooney: Yes, I investigated the failures yesterday and posted a Tempest one-liner for the failing test | |
| 12:24:39 | kashyap | All linked in the spec | |
| 12:24:42 | sean-k-mooney | the ci job and then ya the plan as far as i understood for osp was to change the default in ooo | |
| 12:25:05 | kashyap | sean-k-mooney: Yes, TripleO upstream has this posted: https://review.opendev.org/#/c/716526/ | |
| 12:25:11 | kashyap | It is waiting on the Nova CI | |
| 12:25:22 | sean-k-mooney | ah cool | |
| 12:25:40 | sean-k-mooney | im reading your spec now. go have lunch :) | |
| 12:26:43 | kashyap | sean-k-mooney: Thanks. :) Likewise | |
| 12:33:14 | bauzas | gibi: can I hold the bugs lock again ? | |
| 12:33:44 | bauzas | aarents: thanks for reporting https://bugs.launchpad.net/nova/+bug/1874032 | |
| 12:33:44 | openstack | Launchpad bug 1874032 in OpenStack Compute (nova) "nova-compute become stuck when doing IO on busy file system" [Undecided,New] | |
| 12:34:26 | gibi | bauzas: sue | |
| 12:34:28 | gibi | sure | |
| 12:35:02 | gibi | I'm swamped with a downstream stuff with a close deadline | |
| 12:35:09 | aarents | bauzas: this one is painfull.. | |
| 12:35:13 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Ussuri 21.0.0 prelude section https://review.opendev.org/721548 | |
| 12:35:31 | bauzas | gibi: <3 with love for your pain | |
| 12:36:05 | bauzas | aarents: sure, but I think it should be a Wishlist bug | |
| 12:36:29 | gibi | bauzas: thanks | |
| 12:36:59 | gibi | funny that RC1 is due tomorrow and my downstream high prio stuff due today (but was not on the radar until Monday) | |
| 12:39:38 | bauzas | gibi: in my company, we also have sometimes some downstream priorities that are in the same times than upstream yes... | |
| 12:39:50 | bauzas | I totally understand you :p | |
| 12:39:58 | bauzas | aarents: so, about your bug | |
| 12:41:08 | bauzas | aarents: you asked why we don't have compute workers, right? | |
| 12:41:20 | aarents | yep | |
| 12:41:34 | bauzas | b/c there is only one service per host | |
| 12:42:16 | bauzas | and in general, when you want to call some I/O issue, you don't do this by the nova-compute service | |
| 12:42:47 | bauzas | that's rather the nova-compute service which calls privsep | |
| 12:45:06 | bauzas | aarents: eg. https://github.com/openstack/nova/blob/master/nova/privsep/qemu.py#L37 | |
| 12:45:57 | bauzas | and then as you can see we call processutils.execute() https://github.com/openstack/nova/blob/master/nova/privsep/qemu.py#L81 | |
| 12:47:08 | aarents | bauzas: ok.. so to workaround this we have to put glance upload outside of nova-compute, in fact that what I've done to fix that on our release I made an execute(*curl).. | |
| 12:48:10 | bauzas | aarents: do you have I/O issues when snapshoting or uploading ? | |
| 12:48:22 | bauzas | from what I see, you generate some I/O calls | |
| 12:48:41 | bauzas | so I thought the problem would be around snapshoting, not uploading to glance | |
| 12:48:43 | aarents | yes during glance upload because IO is made inside nova-compute | |
| 12:49:10 | bauzas | we don't write on disk when we do glance upload, do we ? | |
| 12:49:20 | aarents | the glance upload done during instance snapshot | |
| 12:50:09 | aarents | we read an extracted file (done by qemu-ing convert)) | |
| 12:50:14 | aarents | localy | |
| 12:52:47 | bauzas | oh man this is pizza code from what I can read | |
| 12:52:57 | aarents | :) | |
| 12:53:26 | bauzas | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2252 | |
| 12:55:25 | aarents | bauzas here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2421 | |
| 12:56:20 | aarents | it then use glance api to do the PUT in glance so slow IO are running in nova-compute | |
| 12:56:33 | aarents | not in a processutils.execute() | |
| 12:58:03 | aarents | so maybe workaround processutils.execute(*curl) can be the solution to check | |
| 12:58:08 | aarents | my | |
| 12:58:14 | bauzas | aarents: that's not the glance upload issue | |
| 12:58:28 | bauzas | aarents: IMHO the problem is with the semaphore we wanna take | |
| 12:58:30 | bauzas | hence the D | |
| 12:59:38 | bauzas | the problem with reviewing this code is that nobody documented which cases raise those exceptions we catch https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2358-L2359 | |
| 13:00:10 | bauzas | aarents: because we only fall back into this semaphore lock if you are not able to do the other way | |
| 13:04:07 | aarents | I think, I'm not sure, in my case, we are in NotImplementedError because because direct_snapshot is a feature used typically with ceph, not implemented in raw/qcow2 | |
| 13:04:17 | bauzas | that's correct | |
| 13:04:25 | bauzas | I found the change in question | |
| 13:04:31 | bauzas | aarents: https://review.opendev.org/#/c/205282/ | |
| 13:04:52 | bauzas | aarents: here in your case, we're not talking of RBD snapshots, right? | |
| 13:05:14 | aarents | right, local backed instance raw or qcow2 | |
| 13:05:35 | elod | lyarwood: I've prepared the Train release patch: https://review.opendev.org/#/c/721883/ if you have time please double-check the version bump and the reasons | |
| 13:06:30 | bauzas | aarents: kk, okay so the exception catch-up clause is the default when non-RBD | |
| 13:06:37 | bauzas | (just speaking out loud) | |
| 13:06:49 | aarents | exactly | |
| 13:13:24 | bauzas | aarents: https://review.opendev.org/#/c/609180/ | |