| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-08-18 | |||
| 10:09:13 | admin0 | events: [{'event': 'compute__do_build_and_run_instance', 'start_time': '2020-08-18T09:47:36.000000', 'finish_time': None, 'result': None, 'traceback': None}] | |
| 10:20:17 | admin0 | got it | |
| 10:20:27 | admin0 | how to grep using that specific instanceid | |
| 10:28:28 | lyarwood | admin0: it's the request-id that you want | |
| 10:28:37 | lyarwood | openstack server event list $instance | grep create | awk '{ print $2 }' | |
| 10:42:18 | admin0 | i am making change to remove the config drive to retry again | |
| 10:42:23 | admin0 | so config drives are not compatible with ceph ? | |
| 10:43:26 | lyarwood | admin0: they are but there's something broken in your env with regards to importing disks into rbd | |
| 10:43:56 | admin0 | is there any manual command i can try to validate this ? | |
| 10:44:48 | lyarwood | I'm not sure how you've deployed your env but try using rbd --import as the same user as is running nova-compute | |
| 10:44:59 | admin0 | its done using openstack-ansible | |
| 10:45:12 | admin0 | so the standard osa install | |
| 10:45:44 | admin0 | which has quiet frankly worked well in all past deployments .. but in this new one, it suddenly does not work and i am unable to figure out why | |
| 10:46:38 | lyarwood | kk I've never actually used OSA but that would be my way forward here, run the command as the same user and in the same place as nova-compute is running it | |
| 10:46:52 | lyarwood | and then throw on some additional -v flags (or whatever rbd uses) to debug | |
| 10:47:22 | lyarwood | sean-k-mooney: ^ when you're about | |
| 10:47:24 | admin0 | so basically will be to change nova.conf debug = true, grep the subprocess to capture the commands and then run the same commands again .. | |
| 10:47:41 | admin0 | i will give it a shot | |
| 10:47:48 | lyarwood | admin0: looks like you're already logging in DEBUG, but yeah. | |
| 10:48:06 | admin0 | its because it failed here :D | |
| 10:48:07 | lyarwood | admin0: you just need to be sure you're running the commands under the same user and path (assuming OSA isn't deploying containers?) | |
| 10:48:16 | admin0 | it is not | |
| 10:48:28 | lyarwood | kk | |
| 10:48:37 | admin0 | the containers are only in the controllers . but as far as i have seen the api does its job and its the hypervisor where its stuck | |
| 10:48:55 | lyarwood | yeah it's just the rbd import that's borked | |
| 10:52:55 | admin0 | is this the file which is the source of all these commands? https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/storage/rbd_utils.py | |
| 10:58:18 | lyarwood | admin0: https://opendev.org/openstack/nova/src/branch/stable/train/nova/virt/libvirt/storage/rbd_utils.py for the version you are running | |
| 10:59:39 | admin0 | i removed the configdrive option .. checking once more | |
| 11:00:15 | admin0 | and all images are raw | |
| 11:02:25 | admin0 | lyarwood, it worked | |
| 11:02:27 | admin0 | finally its active now | |
| 11:02:33 | admin0 | removing config drive did the trick | |
| 11:03:16 | admin0 | so if ceph is being used ( and cinder/glance/nova ) is in the same cluster, its recommended to use raw instead of qcow2 ? | |
| 11:04:19 | lyarwood | Yeah RAW instead of qcow2 to avoid the download and import | |
| 11:04:45 | lyarwood | but for config drives you can't avoid it, but they should be small and quick enough to import that it shouldn't be an issue | |
| 11:07:54 | lyarwood | melwitt / johnthetubaguy / dansmith ; https://review.opendev.org/#/c/699291/ - thoughts on finally landing this from mriedem? | |
| 11:08:50 | jsuchome | lyarwood: hi, me again ... any chance for that job testing direct-rbd-download? | |
| 11:09:54 | lyarwood | jsuchome: yup still on my list | |
| 11:18:33 | admin0 | so 2 observations .. if i use a qcow image, they do not boot at all .. if i change to raw images, they show active in the horizon, and i can see the console, but they are stuck on boot "booting from hard drive .. " | |
| 11:20:15 | lyarwood | admin0: and these are cirros images? | |
| 11:20:20 | admin0 | yes | |
| 11:20:33 | admin0 | one is bionic, but the output is kind of similar | |
| 11:20:37 | admin0 | they are stuck on boot | |
| 11:21:06 | lyarwood | admin0: that smells like there's something really wrong with your ceph env then | |
| 11:22:17 | lyarwood | jsuchome: I think I'm going to try to test your change locally first instead of insisting on a job to exercise it | |
| 11:22:26 | lyarwood | jsuchome: so I don't hold this up any longer | |
| 11:23:39 | jsuchome | cool | |
| 11:25:38 | lyarwood | that I expect to fail tbh | |
| 11:25:53 | lyarwood | as the rbd devstack plugin is held togther with tape and chewing gum | |
| 11:28:33 | openstackgerrit | Lee Yarwood proposed openstack/nova master: [goal] Prepare for job migration to Ubuntu Focal (20.04) https://review.opendev.org/738126 | |
| 12:01:27 | mnaser | lyarwood: we actually use it quite often for CI and it's been running quite OK for us | |
| 12:01:36 | mnaser | happy to help clean it up if need be | |
| 12:03:09 | lyarwood | mnaser: yeah apologies I didn't mean anything by that just that it has failed with a few non-standard deployment configs I've given it in the past :) | |
| 12:03:21 | lyarwood | this however worked AFAICT | |
| 12:04:03 | mnaser | Oh yeah no i agree that is is held together by tape though | |
| 12:04:13 | mnaser | I think it deploys Ceph 12.x | |
| 12:04:25 | mnaser | 3 major releases behind | |
| 12:05:03 | mnaser | Which isn’t ideal but eh, I don’t think anyone would like it if i made it implemented the latest release using cephadm which requires docker :p | |
| 12:05:47 | lyarwood | there appears to be some logic for that already in the plugin iirc | |
| 12:06:01 | lyarwood | but not for the latest and greatest release | |
| 12:06:05 | lyarwood | just the container part | |
| 13:06:05 | stephenfin | sean-k-mooney: Have you any idea what sets this field during live migration? https://github.com/openstack/nova/blob/master/nova/objects/migrate_data.py#L41 | |
| 13:09:07 | kashyap | He should definitely know ... in the not-too-distant past I've seen Sean talk about VIF types. | |
| 13:11:13 | sean-k-mooney | oh hi one sec | |
| 13:11:28 | sean-k-mooney | stephenfin: yes i responed in the code review | |
| 13:11:46 | stephenfin | code review? | |
| 13:11:47 | sean-k-mooney | its set in the conducrot live migration task | |
| 13:12:04 | sean-k-mooney | there is a patch up related to a bug | |
| 13:12:20 | sean-k-mooney | i assume you were reviewing https://review.opendev.org/#/c/742180/ | |
| 13:12:33 | sean-k-mooney | for https://bugs.launchpad.net/nova/+bug/1888395 | |
| 13:12:33 | openstack | Launchpad bug 1888395 in OpenStack Compute (nova) "shared live migration of a vm with a vif is broken in train" [Undecided,In progress] - Assigned to Kirill Egorov (kegorov-progmaticlab) | |
| 13:12:43 | sean-k-mooney | the current patch makes incorrect assumtions | |
| 13:12:51 | stephenfin | ah, no, I'm trying to add functional tests for SR-IOV live migration | |
| 13:12:57 | sean-k-mooney | ah ok | |
| 13:13:15 | sean-k-mooney | stephenfin: https://github.com/openstack/nova/blob/d4c857dfcb1ccfa5410de55671e69c722bbc990e/nova/conductor/tasks/live_migrate.py#L377-L383 | |
| 13:13:49 | sean-k-mooney | we create new vifs that have the old vif object inside them then we populate the vif_type form the dest port binding | |
| 13:14:04 | sean-k-mooney | stephenfin: so its set here https://github.com/openstack/nova/blob/d4c857dfcb1ccfa5410de55671e69c722bbc990e/nova/conductor/tasks/live_migrate.py#L433-L436 | |
| 13:14:27 | sean-k-mooney | but since that uses setattr you would find it with grep | |
| 13:14:31 | stephenfin | setattr | |
| 13:14:35 | stephenfin | gfdi | |
| 13:15:02 | stephenfin | sean-k-mooney++ Thanks. Try as I may, I couldn't find that /o\ | |
| 13:15:03 | sean-k-mooney | three are only 4 fileds you could replace that with 4 lines | |
| 13:15:30 | stephenfin | I think I might. setattr kills discoverability | |
| 13:16:24 | sean-k-mooney | if there were more filed i might argue for settattr but its just vnic_type, vif_type, profile and details | |
| 13:17:04 | sean-k-mooney | thats not going to change without a neutron api change and a new microversion which will require a nova code change to use anyway | |
| 13:20:11 | sean-k-mooney | stephenfin: if your going to refactor it might be nice to drop this too https://github.com/openstack/nova/blob/d4c857dfcb1ccfa5410de55671e69c722bbc990e/nova/conductor/tasks/live_migrate.py#L372-L379 | |
| 13:20:25 | stephenfin | agreed | |
| 13:26:18 | sean-k-mooney | lyarwood: sorry you pinged me eairlier but i didnt have irc up on my monitor this morning. did you figure it out? | |
| 13:26:53 | sean-k-mooney | oh im not connected downstream either i must not have connect to the vpn | |
| 13:41:28 | sean-k-mooney | stephenfin: https://review.opendev.org/#/c/728037/6 is in merge conflict so ill adress your nit and ping you to re review once done. | |
| 13:42:07 | sean-k-mooney | if im rebasing i might as well fix them now | |
| 13:44:24 | stephenfin | sean-k-mooney: cool | |
| 13:45:19 | stephenfin | sean-k-mooney: another question for you. The NeutronFixture isn't returning a vif_type key in the response to 'POST /v2.0/ports/{port_id}/bindings' | |
| 13:45:48 | stephenfin | Is that neutron's responsibility? Feel free to tell me to RTFM (I don't have an SR-IOV deployment to hand to experiment :() | |
| 13:46:20 | sean-k-mooney | yes the vif type can technicaly change on a migration but the vif type is set by the ml2 driver when it binds the port | |
| 13:46:37 | sean-k-mooney | stephenfin: can you show me the line | |
| 13:47:00 | stephenfin | This piece you linked earlier https://github.com/openstack/nova/blob/d4c857dfcb1ccfa5410de55671e69c722bbc990e/nova/conductor/tasks/live_migrate.py#L433-L436 | |
| 13:47:13 | sean-k-mooney | i ment the line in the fixture | |
| 13:47:22 | sean-k-mooney | i wanted to see what you have acess too | |
| 13:47:28 | stephenfin | Ohh | |