| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-01-25 | |||
| 15:20:06 | jgriffith | mriedem: Oh | |
| 15:20:11 | jgriffith | yes you're correct | |
| 15:20:16 | mriedem | ok cool | |
| 15:20:19 | jgriffith | sorry, I was confusing the bootable policy | |
| 15:20:27 | mriedem | jgriffith: and does that apply to both retype and migration? | |
| 15:20:29 | jgriffith | You're correct, in-use should just return and error immediately | |
| 15:20:41 | giodeco | When trying to create a new instance on my Newton Openstack env, I have an error in /var/log/nova-conductor.api of type "Corrupt image download. Checksum was d41d8cd98f00b204e9800998ecf8427e expected f8ab98ff5e73ebab884d80c9dc9c7290". I believe it is something related to network or rabbitmq. Any suggestion? | |
| 15:21:02 | mriedem | jackie-truong: ^ | |
| 15:21:24 | jgriffith | mriedem: that's a great question :) | |
| 15:21:57 | mriedem | jgriffith: heh :) basically anything that triggers a call to nova's swap volume api | |
| 15:22:08 | jackie-truong | mriedem: which question are you pointing me to? | |
| 15:22:14 | mriedem | jackie-truong: the image checksum one | |
| 15:22:29 | mriedem | sounds like a john hopkins image signature validation question | |
| 15:23:12 | jackie-truong | it does | |
| 15:23:17 | mdbooth | mriedem: Bizarrely, we've also had users call it directly. However, I don't think that would cause a data corruption issue specifically. | |
| 15:23:19 | jackie-truong | lemme look | |
| 15:24:08 | mriedem | mdbooth: it's an admin-only api in nova by default | |
| 15:24:09 | jgriffith | mriedem: so currently anything that calls retype on a multiattach that's in use is going to return an exception | |
| 15:24:13 | mriedem | so normal end users shouldn't be able to do that | |
| 15:24:21 | jgriffith | mriedem: regardless of *how* it tries to do it | |
| 15:24:35 | mriedem | jgriffith: does volume migration go through the retype flow even if it's not a retype? | |
| 15:24:42 | jgriffith | mriedem: nope | |
| 15:24:43 | mriedem | sort of like how we call cold migration a resize in nova | |
| 15:24:59 | jgriffith | I don't know that swap ever goes through a cidner retype? | |
| 15:25:24 | jgriffith | remember it's cinder's retype with migration that calls swap and get us in that weird case | |
| 15:26:20 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: DNM: test if tempest-full is triggered or not https://review.openstack.org/537936 | |
| 15:26:32 | mriedem | jgriffith: looks like nova's swap volume is called from VolumeManager._migrate_volume_generic | |
| 15:26:37 | mdbooth | mriedem: Anyway, I only brought that up to point out that it's not without precedent. Ideally it would work, but given that it has never worked I think it's a simple enough message to users that: 'This doesn't work'. In the meantime we take the reduced complexity in Nova and wait for the upstream fix. | |
| 15:26:46 | mriedem | called from migrate_volume | |
| 15:26:59 | jgriffith | mriedem: right, the live-migration of volumes is the trigger | |
| 15:27:13 | jgriffith | either from the Cinder side or the Nova side; but not the swap call itself | |
| 15:28:13 | jgriffith | mriedem: what I was getting at is that the swap process itself takes two volumes so that's not an issue; migrate is another story | |
| 15:29:28 | mriedem | yeah ok i found in the volume manager code where retype calls migrate_volume | |
| 15:29:36 | mriedem | and then there is just the migrate_volume called directly from the api | |
| 15:29:56 | jgriffith | mriedem: my point is that I don't think there is ever a case where Nova calls cinders retype is there? | |
| 15:30:02 | mriedem | no | |
| 15:30:06 | mdbooth | No | |
| 15:30:07 | mriedem | i'm only worried about the case that cinder calls nova | |
| 15:30:11 | mriedem | to swap with a multiattach volume | |
| 15:30:35 | jgriffith | mriedem: yeah, but in that case Cinder should've already completed (or failed) to do the work of creating the destination volume etc | |
| 15:30:39 | mriedem | and this is the retype check in the API https://github.com/openstack/cinder/blob/master/cinder/volume/api.py#L1647 | |
| 15:30:41 | jgriffith | so we should be *ok* | |
| 15:31:23 | mriedem | jgriffith: isn't ^ saying, that if i'm retyping from a multiattach=True to a multiattach=True type, we don't care if it's attached or not? | |
| 15:31:26 | jgriffith | mriedem: right, no options there; it just won't let you do it with in-use | |
| 15:31:41 | mriedem | well, the in-use check is only if the src/dst multiattach values are changing | |
| 15:32:08 | mriedem | s/changing/different/ | |
| 15:32:14 | jgriffith | mriedem: yes | |
| 15:32:30 | jgriffith | mriedem: yes to the if the multiattach setting is the same we don't care | |
| 15:32:39 | jgriffith | then we behave like any other retype command | |
| 15:33:07 | mriedem | ok so we can call swap_volume in nova for a retype/migrate on a multiattach volume | |
| 15:33:09 | mriedem | mdbooth: ^ | |
| 15:33:33 | mriedem | we == cinder | |
| 15:33:55 | SpazZz | hehe when yo uassociate too closely to your code | |
| 15:34:04 | openstackgerrit | Jackie Truong proposed openstack/nova master: Add trusted_certs to instance_extra https://review.openstack.org/537897 | |
| 15:34:11 | jgriffith | hehe | |
| 15:34:19 | mdbooth | mriedem: Yeah, that would definitely cause a problem. | |
| 15:34:23 | mdbooth | SpazZz: hehe | |
| 15:35:03 | mdbooth | I think you're right though, that cinder is probably best placed to handle that. | |
| 15:36:26 | mriedem | well i'll try to write up a cinder api-ref docs patch | |
| 15:36:29 | mriedem | for all the things | |
| 15:37:54 | mdbooth | jgriffith: Incidentally, this might be handled by context, but at first glance it looks like you've got a race there even if you fix the multiattach check. | |
| 15:38:59 | mdbooth | We're checking that volume.status is available, and if it is then we're proceeding. However, I don't see any assertion that volume.status continues to be available. | |
| 15:39:22 | mdbooth | In Nova we typically do this by atomically changing task state. | |
| 15:40:09 | mriedem | mdbooth: they do that here https://github.com/openstack/cinder/blob/master/cinder/volume/api.py#L1699 | |
| 15:40:37 | jgriffith | mdbooth: We have a conditional on the volume object IIRC | |
| 15:40:37 | mdbooth | So we'd do: if instance.status == 'all good': atomic_update(instance.task_state = 'doing my thang', expected task_state = 'whatever') | |
| 15:41:11 | jgriffith | mdbooth: after that yeah we don't lock or anything if that's what you mean | |
| 15:41:26 | openstackgerrit | Jackie Truong proposed openstack/nova master: Add trusted_image_certificates to REST API https://review.openstack.org/486204 | |
| 15:41:26 | openstackgerrit | Jackie Truong proposed openstack/nova master: Implement certificate_utils https://review.openstack.org/479949 | |
| 15:41:53 | SpazZz | IBM CI looks like it's cleaning up.. can hopefully get the my +1 back and look sexier for the Core Reviewers | |
| 15:42:03 | SpazZz | Walk by and be like, "girlll look at all these +1's over here.." | |
| 15:42:50 | openstackgerrit | Jackie Truong proposed openstack/nova master: Add trusted_certs object https://review.openstack.org/489408 | |
| 15:43:12 | mdbooth | jgriffith: Ah, yes. So in this case we'd want to modify 'expected' on line 1685 to contain only available. | |
| 15:43:57 | mdbooth | Assuming conditional_update is an atomic CAS, that would completely close the race window. | |
| 15:44:13 | mdbooth | But yeah, it's currently very short. | |
| 15:47:22 | jgriffith | mdbooth: yeah, IMO there's probably going to be some additions/fixes that are more critical to this before we get to things like closing that up. | |
| 15:48:03 | mdbooth | jgriffith: Easy to close, hard to spot... | |
| 16:02:44 | openstackgerrit | Brianna Poulos proposed openstack/nova master: Reduce complexity of _from_db_object https://review.openstack.org/537728 | |
| 16:11:20 | Tengu | hello, anyone can tell me what the error "NovaException: Unsupported VIF type unbound convert '_nova_to_osvif_vif_unbound'" means? it apparently prevents launching a new instance in a specific project/tenant on openstack pike. | |
| 16:11:50 | mriedem | Tengu: probably means an unexpected vif type for the port and nova doesn't know how to handle it | |
| 16:11:53 | mriedem | sean-k-mooney: ^ | |
| 16:12:13 | Tengu | mriedem: hmm. weird thing is: this happens only in one project :/. | |
| 16:12:21 | bpoulos | giodeco: The checksum error you're seeing isn't related to image signature verification. It looks like it's a Glance Exception raised if the MD5 checksum of the image isn't as expected (signature verification doesn't use MD5). See http://git.openstack.org/cgit/openstack/python-glanceclient/tree/glanceclient/common/utils.py#n415 | |
| 16:12:27 | mriedem | Tengu: then i don't know | |
| 16:12:31 | Tengu | and I didn't change anything in the configuration :/. | |
| 16:12:41 | Tengu | might it be related to some kind of quota? | |
| 16:12:49 | mriedem | not on the nova side | |
| 16:12:54 | Tengu | hmm. | |
| 16:13:02 | Tengu | shall I ping on neutron channel? | |
| 16:13:27 | mriedem | http://codesearch.openstack.org/?q=Unsupported%20VIF%20type&i=nope&files=&repos= | |
| 16:13:57 | Tengu | hmmm, nova-network is deprecated and not in use anymore in pike, is it? | |
| 16:14:08 | mriedem | this isn't nova-network | |
| 16:14:17 | Tengu | oh. | |
| 16:14:29 | mriedem | the libvirt driver in nova needs to know how to handle the vif type to setup the guest xml properly when attaching the network interface | |
| 16:15:03 | Tengu | hmm. | |
| 16:15:28 | Tengu | if it was for all project in the openstack deploy, I'd say "due to update"… but this isn't the case. | |
| 16:15:46 | mriedem | what is the vif type? the error is saying the vif type is "unbound" for some reason | |
| 16:15:47 | mriedem | which isn't correct | |
| 16:15:48 | sean-k-mooney | mriedem: it means some how the ovs_vif_utill.py fucntion with a neutron port that we have not done port binding on yet | |
| 16:16:06 | sean-k-mooney | mriedem: e.g. vif_unbound is the state a port is in before it is added to a vm | |