| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-08-18 | |||
| 10:48:28 | lyarwood | kk | |
| 10:48:37 | admin0 | the containers are only in the controllers . but as far as i have seen the api does its job and its the hypervisor where its stuck | |
| 10:48:55 | lyarwood | yeah it's just the rbd import that's borked | |
| 10:52:55 | admin0 | is this the file which is the source of all these commands? https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/storage/rbd_utils.py | |
| 10:58:18 | lyarwood | admin0: https://opendev.org/openstack/nova/src/branch/stable/train/nova/virt/libvirt/storage/rbd_utils.py for the version you are running | |
| 10:59:39 | admin0 | i removed the configdrive option .. checking once more | |
| 11:00:15 | admin0 | and all images are raw | |
| 11:02:25 | admin0 | lyarwood, it worked | |
| 11:02:27 | admin0 | finally its active now | |
| 11:02:33 | admin0 | removing config drive did the trick | |
| 11:03:16 | admin0 | so if ceph is being used ( and cinder/glance/nova ) is in the same cluster, its recommended to use raw instead of qcow2 ? | |
| 11:04:19 | lyarwood | Yeah RAW instead of qcow2 to avoid the download and import | |
| 11:04:45 | lyarwood | but for config drives you can't avoid it, but they should be small and quick enough to import that it shouldn't be an issue | |
| 11:07:54 | lyarwood | melwitt / johnthetubaguy / dansmith ; https://review.opendev.org/#/c/699291/ - thoughts on finally landing this from mriedem? | |
| 11:08:50 | jsuchome | lyarwood: hi, me again ... any chance for that job testing direct-rbd-download? | |
| 11:09:54 | lyarwood | jsuchome: yup still on my list | |
| 11:18:33 | admin0 | so 2 observations .. if i use a qcow image, they do not boot at all .. if i change to raw images, they show active in the horizon, and i can see the console, but they are stuck on boot "booting from hard drive .. " | |
| 11:20:15 | lyarwood | admin0: and these are cirros images? | |
| 11:20:20 | admin0 | yes | |
| 11:20:33 | admin0 | one is bionic, but the output is kind of similar | |
| 11:20:37 | admin0 | they are stuck on boot | |
| 11:21:06 | lyarwood | admin0: that smells like there's something really wrong with your ceph env then | |
| 11:22:17 | lyarwood | jsuchome: I think I'm going to try to test your change locally first instead of insisting on a job to exercise it | |
| 11:22:26 | lyarwood | jsuchome: so I don't hold this up any longer | |
| 11:23:39 | jsuchome | cool | |
| 11:25:38 | lyarwood | that I expect to fail tbh | |
| 11:25:53 | lyarwood | as the rbd devstack plugin is held togther with tape and chewing gum | |
| 11:28:33 | openstackgerrit | Lee Yarwood proposed openstack/nova master: [goal] Prepare for job migration to Ubuntu Focal (20.04) https://review.opendev.org/738126 | |
| 12:01:27 | mnaser | lyarwood: we actually use it quite often for CI and it's been running quite OK for us | |
| 12:01:36 | mnaser | happy to help clean it up if need be | |
| 12:03:09 | lyarwood | mnaser: yeah apologies I didn't mean anything by that just that it has failed with a few non-standard deployment configs I've given it in the past :) | |
| 12:03:21 | lyarwood | this however worked AFAICT | |
| 12:04:03 | mnaser | Oh yeah no i agree that is is held together by tape though | |
| 12:04:13 | mnaser | I think it deploys Ceph 12.x | |
| 12:04:25 | mnaser | 3 major releases behind | |
| 12:05:03 | mnaser | Which isn’t ideal but eh, I don’t think anyone would like it if i made it implemented the latest release using cephadm which requires docker :p | |
| 12:05:47 | lyarwood | there appears to be some logic for that already in the plugin iirc | |
| 12:06:01 | lyarwood | but not for the latest and greatest release | |
| 12:06:05 | lyarwood | just the container part | |
| 13:06:05 | stephenfin | sean-k-mooney: Have you any idea what sets this field during live migration? https://github.com/openstack/nova/blob/master/nova/objects/migrate_data.py#L41 | |
| 13:09:07 | kashyap | He should definitely know ... in the not-too-distant past I've seen Sean talk about VIF types. | |
| 13:11:13 | sean-k-mooney | oh hi one sec | |
| 13:11:28 | sean-k-mooney | stephenfin: yes i responed in the code review | |
| 13:11:46 | stephenfin | code review? | |
| 13:11:47 | sean-k-mooney | its set in the conducrot live migration task | |
| 13:12:04 | sean-k-mooney | there is a patch up related to a bug | |
| 13:12:20 | sean-k-mooney | i assume you were reviewing https://review.opendev.org/#/c/742180/ | |
| 13:12:33 | sean-k-mooney | for https://bugs.launchpad.net/nova/+bug/1888395 | |
| 13:12:33 | openstack | Launchpad bug 1888395 in OpenStack Compute (nova) "shared live migration of a vm with a vif is broken in train" [Undecided,In progress] - Assigned to Kirill Egorov (kegorov-progmaticlab) | |
| 13:12:43 | sean-k-mooney | the current patch makes incorrect assumtions | |
| 13:12:51 | stephenfin | ah, no, I'm trying to add functional tests for SR-IOV live migration | |
| 13:12:57 | sean-k-mooney | ah ok | |
| 13:13:15 | sean-k-mooney | stephenfin: https://github.com/openstack/nova/blob/d4c857dfcb1ccfa5410de55671e69c722bbc990e/nova/conductor/tasks/live_migrate.py#L377-L383 | |
| 13:13:49 | sean-k-mooney | we create new vifs that have the old vif object inside them then we populate the vif_type form the dest port binding | |
| 13:14:04 | sean-k-mooney | stephenfin: so its set here https://github.com/openstack/nova/blob/d4c857dfcb1ccfa5410de55671e69c722bbc990e/nova/conductor/tasks/live_migrate.py#L433-L436 | |
| 13:14:27 | sean-k-mooney | but since that uses setattr you would find it with grep | |
| 13:14:31 | stephenfin | setattr | |
| 13:14:35 | stephenfin | gfdi | |
| 13:15:02 | stephenfin | sean-k-mooney++ Thanks. Try as I may, I couldn't find that /o\ | |
| 13:15:03 | sean-k-mooney | three are only 4 fileds you could replace that with 4 lines | |
| 13:15:30 | stephenfin | I think I might. setattr kills discoverability | |
| 13:16:24 | sean-k-mooney | if there were more filed i might argue for settattr but its just vnic_type, vif_type, profile and details | |
| 13:17:04 | sean-k-mooney | thats not going to change without a neutron api change and a new microversion which will require a nova code change to use anyway | |
| 13:20:11 | sean-k-mooney | stephenfin: if your going to refactor it might be nice to drop this too https://github.com/openstack/nova/blob/d4c857dfcb1ccfa5410de55671e69c722bbc990e/nova/conductor/tasks/live_migrate.py#L372-L379 | |
| 13:20:25 | stephenfin | agreed | |
| 13:26:18 | sean-k-mooney | lyarwood: sorry you pinged me eairlier but i didnt have irc up on my monitor this morning. did you figure it out? | |
| 13:26:53 | sean-k-mooney | oh im not connected downstream either i must not have connect to the vpn | |
| 13:41:28 | sean-k-mooney | stephenfin: https://review.opendev.org/#/c/728037/6 is in merge conflict so ill adress your nit and ping you to re review once done. | |
| 13:42:07 | sean-k-mooney | if im rebasing i might as well fix them now | |
| 13:44:24 | stephenfin | sean-k-mooney: cool | |
| 13:45:19 | stephenfin | sean-k-mooney: another question for you. The NeutronFixture isn't returning a vif_type key in the response to 'POST /v2.0/ports/{port_id}/bindings' | |
| 13:45:48 | stephenfin | Is that neutron's responsibility? Feel free to tell me to RTFM (I don't have an SR-IOV deployment to hand to experiment :() | |
| 13:46:20 | sean-k-mooney | yes the vif type can technicaly change on a migration but the vif type is set by the ml2 driver when it binds the port | |
| 13:46:37 | sean-k-mooney | stephenfin: can you show me the line | |
| 13:47:00 | stephenfin | This piece you linked earlier https://github.com/openstack/nova/blob/d4c857dfcb1ccfa5410de55671e69c722bbc990e/nova/conductor/tasks/live_migrate.py#L433-L436 | |
| 13:47:13 | sean-k-mooney | i ment the line in the fixture | |
| 13:47:22 | sean-k-mooney | i wanted to see what you have acess too | |
| 13:47:28 | stephenfin | Ohh | |
| 13:47:42 | stephenfin | Sec while I figure that out | |
| 13:47:55 | sean-k-mooney | you could copy it form the soruce vif | |
| 13:48:01 | sean-k-mooney | in the migrate vif objects | |
| 13:48:25 | sean-k-mooney | it can change but only if your migrating from like linux bridge hosts to ovs | |
| 13:48:47 | sean-k-mooney | for sriov it should not change although its techincially possible | |
| 13:48:52 | stephenfin | sean-k-mooney: https://github.com/openstack/nova/blob/master/nova/tests/fixtures.py#L1659-L1676 | |
| 13:49:01 | stephenfin | That's what the fixture is doing. It's rather dumb | |
| 13:49:06 | sean-k-mooney | if you went from say a hardware offloaded ovs to sriov host | |
| 13:49:24 | stephenfin | Okay, I have the port_id so I can use the stored details | |
| 13:49:26 | sean-k-mooney | ya it does jus tthe minium it needed too | |
| 13:49:38 | stephenfin | with a big note saying that stub doesn't allow you to switch backends | |
| 13:49:47 | stephenfin | if that sounds reasonable | |
| 13:49:56 | sean-k-mooney | self._port_bindings[port_id][host] | |
| 13:50:22 | sean-k-mooney | so you you likely can just look it up assuming [host] also has another entry for the souce node | |
| 13:51:01 | sean-k-mooney | so loop over self._port_bindings[port_id] and fine the one where the key is not euqal to host | |
| 13:51:18 | sean-k-mooney | then grab the info from that | |
| 13:51:38 | stephenfin | Good call | |
| 13:53:22 | sean-k-mooney | that wont always work but it should work in the migration case | |
| 13:54:25 | sean-k-mooney | so you will need to guard it with an if and only do it if the vif_type is not in the binding data i guess | |
| 13:56:22 | sean-k-mooney | actully it might work because reasons. | |
| 13:56:40 | sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/tests/fixtures.py#L1573 that is where the vif_type is currently coming form in the fixture | |
| 13:57:29 | sean-k-mooney | well i guess htere are other places too | |