| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-04-22 | |||
| 12:33:14 | bauzas | gibi: can I hold the bugs lock again ? | |
| 12:33:44 | bauzas | aarents: thanks for reporting https://bugs.launchpad.net/nova/+bug/1874032 | |
| 12:33:44 | openstack | Launchpad bug 1874032 in OpenStack Compute (nova) "nova-compute become stuck when doing IO on busy file system" [Undecided,New] | |
| 12:34:26 | gibi | bauzas: sue | |
| 12:34:28 | gibi | sure | |
| 12:35:02 | gibi | I'm swamped with a downstream stuff with a close deadline | |
| 12:35:09 | aarents | bauzas: this one is painfull.. | |
| 12:35:13 | openstackgerrit | Sylvain Bauza proposed openstack/nova master: Ussuri 21.0.0 prelude section https://review.opendev.org/721548 | |
| 12:35:31 | bauzas | gibi: <3 with love for your pain | |
| 12:36:05 | bauzas | aarents: sure, but I think it should be a Wishlist bug | |
| 12:36:29 | gibi | bauzas: thanks | |
| 12:36:59 | gibi | funny that RC1 is due tomorrow and my downstream high prio stuff due today (but was not on the radar until Monday) | |
| 12:39:38 | bauzas | gibi: in my company, we also have sometimes some downstream priorities that are in the same times than upstream yes... | |
| 12:39:50 | bauzas | I totally understand you :p | |
| 12:39:58 | bauzas | aarents: so, about your bug | |
| 12:41:08 | bauzas | aarents: you asked why we don't have compute workers, right? | |
| 12:41:20 | aarents | yep | |
| 12:41:34 | bauzas | b/c there is only one service per host | |
| 12:42:16 | bauzas | and in general, when you want to call some I/O issue, you don't do this by the nova-compute service | |
| 12:42:47 | bauzas | that's rather the nova-compute service which calls privsep | |
| 12:45:06 | bauzas | aarents: eg. https://github.com/openstack/nova/blob/master/nova/privsep/qemu.py#L37 | |
| 12:45:57 | bauzas | and then as you can see we call processutils.execute() https://github.com/openstack/nova/blob/master/nova/privsep/qemu.py#L81 | |
| 12:47:08 | aarents | bauzas: ok.. so to workaround this we have to put glance upload outside of nova-compute, in fact that what I've done to fix that on our release I made an execute(*curl).. | |
| 12:48:10 | bauzas | aarents: do you have I/O issues when snapshoting or uploading ? | |
| 12:48:22 | bauzas | from what I see, you generate some I/O calls | |
| 12:48:41 | bauzas | so I thought the problem would be around snapshoting, not uploading to glance | |
| 12:48:43 | aarents | yes during glance upload because IO is made inside nova-compute | |
| 12:49:10 | bauzas | we don't write on disk when we do glance upload, do we ? | |
| 12:49:20 | aarents | the glance upload done during instance snapshot | |
| 12:50:09 | aarents | we read an extracted file (done by qemu-ing convert)) | |
| 12:50:14 | aarents | localy | |
| 12:52:47 | bauzas | oh man this is pizza code from what I can read | |
| 12:52:57 | aarents | :) | |
| 12:53:26 | bauzas | https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2252 | |
| 12:55:25 | aarents | bauzas here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2421 | |
| 12:56:20 | aarents | it then use glance api to do the PUT in glance so slow IO are running in nova-compute | |
| 12:56:33 | aarents | not in a processutils.execute() | |
| 12:58:03 | aarents | so maybe workaround processutils.execute(*curl) can be the solution to check | |
| 12:58:08 | aarents | my | |
| 12:58:14 | bauzas | aarents: that's not the glance upload issue | |
| 12:58:28 | bauzas | aarents: IMHO the problem is with the semaphore we wanna take | |
| 12:58:30 | bauzas | hence the D | |
| 12:59:38 | bauzas | the problem with reviewing this code is that nobody documented which cases raise those exceptions we catch https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2358-L2359 | |
| 13:00:10 | bauzas | aarents: because we only fall back into this semaphore lock if you are not able to do the other way | |
| 13:04:07 | aarents | I think, I'm not sure, in my case, we are in NotImplementedError because because direct_snapshot is a feature used typically with ceph, not implemented in raw/qcow2 | |
| 13:04:17 | bauzas | that's correct | |
| 13:04:25 | bauzas | I found the change in question | |
| 13:04:31 | bauzas | aarents: https://review.opendev.org/#/c/205282/ | |
| 13:04:52 | bauzas | aarents: here in your case, we're not talking of RBD snapshots, right? | |
| 13:05:14 | aarents | right, local backed instance raw or qcow2 | |
| 13:05:35 | elod | lyarwood: I've prepared the Train release patch: https://review.opendev.org/#/c/721883/ if you have time please double-check the version bump and the reasons | |
| 13:06:30 | bauzas | aarents: kk, okay so the exception catch-up clause is the default when non-RBD | |
| 13:06:37 | bauzas | (just speaking out loud) | |
| 13:06:49 | aarents | exactly | |
| 13:13:24 | bauzas | aarents: https://review.opendev.org/#/c/609180/ | |
| 13:13:55 | bauzas | aarents: fwiw, I'm about to triage it as Invalid | |
| 13:14:14 | bauzas | but you could open a blueprint for optimizing how we manage I/O concurrency | |
| 13:18:53 | sean-k-mooney | stephenfin: i have added the topic of libvirt hardware profiles to the ptg topic list https://etherpad.opendev.org/p/nova-victoria-ptg on line 60 by the way | |
| 13:19:03 | stephenfin | sweet | |
| 13:19:10 | sean-k-mooney | i will start fleshing it out more here https://etherpad.opendev.org/p/libvirt-hardware-profiles | |
| 13:19:38 | sean-k-mooney | but im not sure if i will get to it this week | |
| 13:19:58 | sean-k-mooney | the spec template is more or less the same form last cycle right? | |
| 13:21:01 | sean-k-mooney | oh there already is one https://github.com/openstack/nova-specs/blob/master/specs/victoria-template.rst | |
| 13:22:01 | bauzas | sean-k-mooney: stephenfin: if you don't mind MHO, I'd not want to have the hardware profiles be operator-defined | |
| 13:22:10 | bauzas | this defeats interop | |
| 13:22:18 | sean-k-mooney | bauzas: yep its out of scope in the first version | |
| 13:22:29 | bauzas | in *any* version tbc :) | |
| 13:22:54 | sean-k-mooney | i said we could add that after if we wanted but only if they cannot modify intree ones and if we have an api to list and show the hardware profiles | |
| 13:23:08 | bauzas | if we go this way, we absolutely have to be pedantic and provide some abstraction that'd be versioned and defined by us | |
| 13:23:24 | sean-k-mooney | ill state teh same in the spec but i dont want to enable that in victoria | |
| 13:23:44 | bauzas | I think this absolutely needs to be discussed in the V spec | |
| 13:23:58 | bauzas | b/c this could be a terrible knob | |
| 13:24:03 | sean-k-mooney | bauzas: yes we would im planning for them to be version even if they are in tree only | |
| 13:24:21 | bauzas | or we would allow *custom* profiles | |
| 13:24:33 | sean-k-mooney | bauzas: well it wont provide anything that can be done with flavor extrra specs or image properties today | |
| 13:24:36 | bauzas | which would be yet another Far-West in Novaz | |
| 13:24:44 | bauzas | correct | |
| 13:24:52 | sean-k-mooney | *cannot | |
| 13:25:21 | bauzas | for this exact reason, I think we need to state in the spec that we WON'T support operator-defined profiles ever | |
| 13:25:35 | sean-k-mooney | e.g. its just a static set of profiles that change the default of existing tunables. | |
| 13:25:51 | sean-k-mooney | bauzas: if you want me to make that the starting point im ok with that | |
| 13:26:40 | sean-k-mooney | if someone else want to chagne that after Victoria we can have that discussion then but im totaly fine with only nova as a project being able to define these profiles | |
| 13:26:52 | bauzas | shoot, but again, I just want to be clear on the fact that I'm absolutely opposed to any attempt of delivering custom profiles unless they're explicitely stated "custom" | |
| 13:27:17 | bauzas | sean-k-mooney: this sounds reasonable | |
| 13:27:36 | sean-k-mooney | ok im fine with that too. i understnad the interop concern and i share that | |
| 13:27:46 | bauzas | leave others the pain to propose some customizable profiles in another spec | |
| 13:29:15 | sean-k-mooney | bauzas: yep, i intened to have no api or db changes and to store the profiles as yaml files that are packaged as data files as part of nova | |
| 13:29:35 | bauzas | sean-k-mooney: that's design discussion for sure | |
| 13:29:42 | bauzas | (but fwiw, I'll -1 this :D) | |
| 13:30:01 | bauzas | don't provide guns to kids | |
| 13:30:20 | sean-k-mooney | well i just dont want ot have a bunch of dicts in a module but my other appoch was to mirror the flavor validation work | |
| 13:30:26 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add docs and releasenotes for BP policy-defaults-refresh https://review.opendev.org/720129 | |
| 13:30:34 | bauzas | sean-k-mooney: the latter sounds good | |
| 13:30:56 | bauzas | but let's see what other ppl think | |
| 13:31:03 | sean-k-mooney | bauzas: it makes verioning harder which is the only reason i dont want to defautl to that but we can discuss in the spec later | |
| 13:31:20 | bauzas | sean-k-mooney: we have a couple of versioned objects you know | |
| 13:31:26 | bauzas | and we have versioned libs | |
| 13:31:58 | sean-k-mooney | yes but i want you to be able to say give me hw:profile=realtime-vicoria in W or Z | |
| 13:32:03 | bauzas | but this sounds a rathole to discuss this now :D | |
| 13:32:36 | bauzas | let's draft the UX in the spec ;) | |