| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-31 | |||
| 15:47:46 | cfriesen | mdbooth: kashyap: thanks guys, I've got options to try out now. :) | |
| 15:48:43 | mdbooth | cfriesen: Sadly, I don't think this change will cause existing downloads to break | |
| 15:48:57 | mdbooth | I say sadly, because we do format inspection on backing files :( | |
| 15:49:20 | dansmith | mriedem: you use cast-as-call in your test, for a reason I assume.. do you know why I'm getting NotFound and need that poll loop? | |
| 15:49:32 | kashyap | mdbooth: BTW, do you really mean two users trying to simultaneously to a qcow2 file? | |
| 15:49:37 | mdbooth | I don't know of any current security issues relating to this, but it's still a danger | |
| 15:49:49 | kashyap | s/users/processes/ | |
| 15:50:27 | mdbooth | kashyap: So, you create a qcow2 file with a remote backing file and COR, lets call this cache.qcow | |
| 15:50:42 | mdbooth | kashyap: You then create instance_a.qcow2 which has cache.qcow2 as a backing file | |
| 15:51:03 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Address comments when moving volume detach to block_device.py https://review.openstack.org/452252 | |
| 15:51:04 | mriedem | lyarwood: ^ | |
| 15:51:17 | mriedem | dansmith: i used cast as call because that's what the test did that i copied from :) | |
| 15:51:23 | dansmith | heh, yeah okay | |
| 15:51:25 | mriedem | dansmith: as for the 404 while it's building... | |
| 15:51:26 | mdbooth | When you read a block from instance_a.qcow2, it'll be looked up in cache.qcow2, miss, be downloaded, and stored in cache.qcow before being returned to instance_a.qcow2 | |
| 15:51:55 | mdbooth | So basically, the qemu process writing to instance_a.qcow2 is also writing to cache.qcow2 | |
| 15:51:58 | mriedem | dansmith: could be hitting some window between when the build request and instance exist for the get() method in the compute API? | |
| 15:52:06 | mriedem | dansmith: i thought we closed that in ocata thoguh | |
| 15:52:07 | mriedem | *though | |
| 15:52:08 | mdbooth | Now simultaneously do the same thing with instance_b.qcow2 | |
| 15:52:17 | lyarwood | mriedem: ack thanks, LGTM | |
| 15:52:24 | dansmith | mriedem: well, that's my point.. we shouldn | |
| 15:52:26 | dansmith | mriedem: shouldn't have that problem | |
| 15:52:38 | mdbooth | You've now got 2 separate qemu processes writing to cache.qcow2 without coordination | |
| 15:52:46 | mdbooth | I don't believe this will end well | |
| 15:52:47 | kashyap | mdbooth: Hmm. It's not security issues (about multiple processes trying to write to a qcow2 file live), but image being corrupted -- as we speak, there are patches using POSIX / Linux file-locking to prevent that at QEMU-level | |
| 15:53:02 | mdbooth | kashyap: The security issue is image format detection | |
| 15:53:14 | mdbooth | We shouldn't rely on it, but we do | |
| 15:53:29 | kashyap | mdbooth: Oh, you were talking to Chris there, duh :-) | |
| 15:53:32 | mdbooth | But in this case it's convenient that we do, because it won't break anything if you change force_raw | |
| 15:53:54 | dansmith | mriedem: ah, I know why | |
| 15:53:55 | kashyap | mdbooth: But I agree with your design of speeding up remote boots with COR | |
| 15:54:30 | mriedem | dansmith: what i'd do, is temporarily hack your test change to change https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/servers.py#L1144 and do LOG.exception so you get the traceback | |
| 15:54:42 | dansmith | mriedem: I done already figger'd it | |
| 15:54:47 | mriedem | do tell | |
| 15:54:52 | mriedem | no instance mapping? | |
| 15:55:06 | dansmith | no, because that's not a bug | |
| 15:55:12 | openstackgerrit | Sivasathurappan Radhakrishnan proposed openstack/nova master: Raise general error message when volume attach fails https://review.openstack.org/320163 | |
| 15:55:17 | dansmith | it's service_version | |
| 15:55:22 | dansmith | because of all the switchin' during service setup | |
| 15:55:25 | dansmith | api thinks api is old | |
| 15:55:41 | dansmith | which is why we have fixtures.AllServicesCurrent() | |
| 15:56:13 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/api.py#L2386 ? | |
| 15:56:27 | dansmith | mriedem: yar | |
| 15:56:28 | mriedem | is service_version 0 or soething? | |
| 15:56:30 | mriedem | something | |
| 15:56:43 | mriedem | this reminds me of something i've been meaning to talk to you about, | |
| 15:56:46 | dansmith | it is in the db that is getting consulted | |
| 15:57:20 | mriedem | https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/servers.py#L437 | |
| 15:57:36 | dansmith | HAH | |
| 15:57:39 | mriedem | in the api we check the minimum compute service version for things like get-me-a-network and device tags | |
| 15:57:45 | mriedem | that's not going to work anymore with multicell | |
| 15:57:55 | kashyap | mdbooth: One last on that -- in your scenario of speeding up a remote boot while provisioning, it is still a single QEMU process that is handling COR, and live commit, isn't it... | |
| 15:57:59 | mriedem | https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/servers.py#L583 | |
| 15:58:09 | dansmith | mriedem: well (a) we need to remove those now anyway, but (b) there's an all_cells worker to get that | |
| 15:58:15 | mdbooth | kashyap: Not if you've got 2 instances using the same image backing file | |
| 15:58:29 | kashyap | mdbooth: Ah, right! | |
| 15:58:30 | mdbooth | Both before the backing file download has finished | |
| 15:58:46 | kashyap | mdbooth: But yes, as it stands, upstream doesn't support multiple writers | |
| 15:59:06 | mriedem | dansmith: yeah https://github.com/openstack/nova/blob/master/nova/cmd/api_os_compute.py#L43 | |
| 15:59:12 | kashyap | Probably you want to write an RFE to qemu-{devel, block} lists :-) | |
| 15:59:14 | mdbooth | COR == implicitly writeable backing file | |
| 15:59:25 | mriedem | device tagging was newton too right? | |
| 15:59:28 | mriedem | yeah, i know it wasn't ocat | |
| 15:59:29 | mriedem | *ocata | |
| 15:59:45 | dansmith | mriedem: well, it was fixed in ocata api, but yeah | |
| 15:59:56 | mriedem | however | |
| 16:00:03 | mriedem | we have something else using that now in nova.compute.api | |
| 16:00:14 | mriedem | https://github.com/openstack/nova/blob/master/nova/compute/api.py#L1404 | |
| 16:00:17 | mriedem | ^ was added in pike | |
| 16:00:34 | dansmith | ack | |
| 16:00:46 | mriedem | we could remove the checks for get me a network and device tags though probably | |
| 16:01:34 | mriedem | i can't find your nova-status patch | |
| 16:01:41 | mriedem | ah https://review.openstack.org/#/c/442787/ | |
| 16:03:25 | openstackgerrit | Dan Smith proposed openstack/nova master: Add functional test for two-cell scheduler behaviors https://review.openstack.org/452006 | |
| 16:09:21 | mriedem | dansmith: that looks much better | |
| 16:09:41 | dansmith | I'm thrilled | |
| 16:11:45 | leakypipes | _gryf: lol at Michal's April Fool's spec :) | |
| 16:12:27 | leakypipes | dulek: :) | |
| 16:33:53 | ildikov | mriedem: lyarwood: would the Cinder-Nova meeting work for you an hour earlier (1600 UTC) on Thursdays? | |
| 16:33:55 | openstackgerrit | Anusha Unnam proposed openstack/nova master: Add description to policies in server_password.py https://review.openstack.org/452259 | |
| 16:35:55 | lyarwood | ildikov: yes | |
| 16:36:53 | ildikov | lyarwood: cool, tnx | |
| 16:43:26 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/nova master: [WIP] Add Neutron port capabilities to devspec in request https://review.openstack.org/451777 | |
| 17:05:06 | openstackgerrit | Anusha Unnam proposed openstack/nova master: Add description to policies in server_external_events.py https://review.openstack.org/452263 | |
| 17:26:02 | openstackgerrit | Anusha Unnam proposed openstack/nova master: Add description to policies in server_diagnostics.py https://review.openstack.org/452267 | |
| 17:26:43 | JayF | mriedem: https://review.openstack.org/#/c/416487/ mind dropping your procedural -2 from last cycle? | |
| 17:26:59 | JayF | mriedem: we're tying all the patches together now to get it all tested up in tempest and ready to merge (in ironic, then nova) | |
| 17:49:58 | jroll | oh we should approve that blueprint | |
| 17:51:35 | openstackgerrit | Mario Villaplana proposed openstack/nova master: virt/ironic: Implement rescue and unrescue https://review.openstack.org/416487 | |
| 17:52:27 | JayF | mriedem: \o/ thanks | |
| 17:53:02 | mriedem | yw | |
| 17:53:17 | mriedem | ildikov: works for me | |
| 17:55:03 | sdague | mriedem: so... clarkb has an interesting though on the libvirt malloc crashes, which is that we start doing UCA by default | |
| 17:55:13 | sdague | which would mean jumping up to libvirt 2.5 in master | |
| 17:55:36 | sdague | it has a lot of other implications, he's going to write the whole thing up to share on the ML | |
| 17:55:46 | mriedem | ok | |
| 17:55:50 | sdague | but, I wanted to get that seed out there for everyone to ponder | |
| 17:56:18 | sdague | it would mean we'd start testing some of the pathways that aren't otherwise tested | |
| 17:56:35 | mriedem | well, some of those are things like numa/pci right? | |