Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-08-18
09:40:09 admin0 hmm..
09:40:36 admin0 ok .. let me try that way
09:40:41 admin0 i found a snippet : https://gist.github.com/winggundamth/7fe69a99ffcd1dfd14471101497b1407
09:41:24 lyarwood yup so with RAW we can clone directly between rbd pools
09:41:55 lyarwood with qcow2 in stable/train that you appear to be using we download the image to the compute and upload to the vms pool
09:46:18 openstackgerrit Mamduh proposed openstack/os-vif master: Update linux_net - Refactor code of linux_net to more cleaner and increase performace https://review.opendev.org/746673
09:54:36 admin0 lyarwood, https://gist.githubusercontent.com/a1git/78adef462d99eb88829006f36b988f6b/raw/03ffff0d17a94200f6ce4d2594c76f9c2f130890/gistfile1.txt -- this is using a 112mb cirros raw image
09:54:41 admin0 still in spawning state
09:56:41 admin0 i can confirm that from this server, if i copy a command like ceph df --format=json --id cinder --conf /etc/ceph/ceph.conf .. it works
09:59:36 lyarwood admin0: looks like the final thing it's doing again is a rbd import
09:59:48 lyarwood admin0: Aug 18 05:47:41 h1 nova-compute[74659]: 2020-08-18 05:47:41.882 74659 DEBUG oslo_concurrency.processutils [req-f72a63da-329e-45c4-b4dc-7728bb8d2b93 1be712988728425ea0060d3bf7ed7a1e 594a25f5acc346518cd6340ec8c36945 - default default] Running cmd (subprocess): rbd import --pool vms /var/lib/nova/instances/9ea7d4
10:00:24 lyarwood admin0: I think that's the config drive
10:00:41 lyarwood hmmm odd
10:00:52 admin0 so removing config drive will fix this ?
10:01:06 admin0 i can give that a try also
10:01:22 lyarwood well it will allow an instance to spawn but there's an underlying issue with rbd import to address
10:03:33 lyarwood btw to keep the log snippets sane try grepping for the request-id listed in `openstack server event list $instance` next time
10:04:08 lyarwood it should limit the logs you need to look at to that instance and request to spawn etc
10:04:25 lyarwood brb
10:07:29 admin0 i only saw delete and create in the event
10:09:13 admin0 events: [{'event': 'compute__do_build_and_run_instance', 'start_time': '2020-08-18T09:47:36.000000', 'finish_time': None, 'result': None, 'traceback': None}]
10:20:17 admin0 got it
10:20:27 admin0 how to grep using that specific instanceid
10:28:28 lyarwood admin0: it's the request-id that you want
10:28:37 lyarwood openstack server event list $instance | grep create | awk '{ print $2 }'
10:42:18 admin0 i am making change to remove the config drive to retry again
10:42:23 admin0 so config drives are not compatible with ceph ?
10:43:26 lyarwood admin0: they are but there's something broken in your env with regards to importing disks into rbd
10:43:56 admin0 is there any manual command i can try to validate this ?
10:44:48 lyarwood I'm not sure how you've deployed your env but try using rbd --import as the same user as is running nova-compute
10:44:59 admin0 its done using openstack-ansible
10:45:12 admin0 so the standard osa install
10:45:44 admin0 which has quiet frankly worked well in all past deployments .. but in this new one, it suddenly does not work and i am unable to figure out why
10:46:38 lyarwood kk I've never actually used OSA but that would be my way forward here, run the command as the same user and in the same place as nova-compute is running it
10:46:52 lyarwood and then throw on some additional -v flags (or whatever rbd uses) to debug
10:47:22 lyarwood sean-k-mooney: ^ when you're about
10:47:24 admin0 so basically will be to change nova.conf debug = true, grep the subprocess to capture the commands and then run the same commands again ..
10:47:41 admin0 i will give it a shot
10:47:48 lyarwood admin0: looks like you're already logging in DEBUG, but yeah.
10:48:06 admin0 its because it failed here :D
10:48:07 lyarwood admin0: you just need to be sure you're running the commands under the same user and path (assuming OSA isn't deploying containers?)
10:48:16 admin0 it is not
10:48:28 lyarwood kk
10:48:37 admin0 the containers are only in the controllers . but as far as i have seen the api does its job and its the hypervisor where its stuck
10:48:55 lyarwood yeah it's just the rbd import that's borked
10:52:55 admin0 is this the file which is the source of all these commands? https://opendev.org/openstack/nova/src/branch/master/nova/virt/libvirt/storage/rbd_utils.py
10:58:18 lyarwood admin0: https://opendev.org/openstack/nova/src/branch/stable/train/nova/virt/libvirt/storage/rbd_utils.py for the version you are running
10:59:39 admin0 i removed the configdrive option .. checking once more
11:00:15 admin0 and all images are raw
11:02:25 admin0 lyarwood, it worked
11:02:27 admin0 finally its active now
11:02:33 admin0 removing config drive did the trick
11:03:16 admin0 so if ceph is being used ( and cinder/glance/nova ) is in the same cluster, its recommended to use raw instead of qcow2 ?
11:04:19 lyarwood Yeah RAW instead of qcow2 to avoid the download and import
11:04:45 lyarwood but for config drives you can't avoid it, but they should be small and quick enough to import that it shouldn't be an issue
11:07:54 lyarwood melwitt / johnthetubaguy / dansmith ; https://review.opendev.org/#/c/699291/ - thoughts on finally landing this from mriedem?
11:08:50 jsuchome lyarwood: hi, me again ... any chance for that job testing direct-rbd-download?
11:09:54 lyarwood jsuchome: yup still on my list
11:18:33 admin0 so 2 observations .. if i use a qcow image, they do not boot at all .. if i change to raw images, they show active in the horizon, and i can see the console, but they are stuck on boot "booting from hard drive .. "
11:20:15 lyarwood admin0: and these are cirros images?
11:20:20 admin0 yes
11:20:33 admin0 one is bionic, but the output is kind of similar
11:20:37 admin0 they are stuck on boot
11:21:06 lyarwood admin0: that smells like there's something really wrong with your ceph env then
11:22:17 lyarwood jsuchome: I think I'm going to try to test your change locally first instead of insisting on a job to exercise it
11:22:26 lyarwood jsuchome: so I don't hold this up any longer
11:23:39 jsuchome cool
11:25:38 lyarwood that I expect to fail tbh
11:25:53 lyarwood as the rbd devstack plugin is held togther with tape and chewing gum
11:28:33 openstackgerrit Lee Yarwood proposed openstack/nova master: [goal] Prepare for job migration to Ubuntu Focal (20.04) https://review.opendev.org/738126
12:01:27 mnaser lyarwood: we actually use it quite often for CI and it's been running quite OK for us
12:01:36 mnaser happy to help clean it up if need be
12:03:09 lyarwood mnaser: yeah apologies I didn't mean anything by that just that it has failed with a few non-standard deployment configs I've given it in the past :)
12:03:21 lyarwood this however worked AFAICT
12:04:03 mnaser Oh yeah no i agree that is is held together by tape though
12:04:13 mnaser I think it deploys Ceph 12.x
12:04:25 mnaser 3 major releases behind
12:05:03 mnaser Which isn’t ideal but eh, I don’t think anyone would like it if i made it implemented the latest release using cephadm which requires docker :p
12:05:47 lyarwood there appears to be some logic for that already in the plugin iirc
12:06:01 lyarwood but not for the latest and greatest release
12:06:05 lyarwood just the container part
13:06:05 stephenfin sean-k-mooney: Have you any idea what sets this field during live migration? https://github.com/openstack/nova/blob/master/nova/objects/migrate_data.py#L41
13:09:07 kashyap He should definitely know ... in the not-too-distant past I've seen Sean talk about VIF types.
13:11:13 sean-k-mooney oh hi one sec
13:11:28 sean-k-mooney stephenfin: yes i responed in the code review
13:11:46 stephenfin code review?
13:11:47 sean-k-mooney its set in the conducrot live migration task
13:12:04 sean-k-mooney there is a patch up related to a bug
13:12:20 sean-k-mooney i assume you were reviewing https://review.opendev.org/#/c/742180/
13:12:33 sean-k-mooney for https://bugs.launchpad.net/nova/+bug/1888395
13:12:33 openstack Launchpad bug 1888395 in OpenStack Compute (nova) "shared live migration of a vm with a vif is broken in train" [Undecided,In progress] - Assigned to Kirill Egorov (kegorov-progmaticlab)
13:12:43 sean-k-mooney the current patch makes incorrect assumtions
13:12:51 stephenfin ah, no, I'm trying to add functional tests for SR-IOV live migration
13:12:57 sean-k-mooney ah ok
13:13:15 sean-k-mooney stephenfin: https://github.com/openstack/nova/blob/d4c857dfcb1ccfa5410de55671e69c722bbc990e/nova/conductor/tasks/live_migrate.py#L377-L383
13:13:49 sean-k-mooney we create new vifs that have the old vif object inside them then we populate the vif_type form the dest port binding
13:14:04 sean-k-mooney stephenfin: so its set here https://github.com/openstack/nova/blob/d4c857dfcb1ccfa5410de55671e69c722bbc990e/nova/conductor/tasks/live_migrate.py#L433-L436
13:14:27 sean-k-mooney but since that uses setattr you would find it with grep
13:14:31 stephenfin setattr
13:14:35 stephenfin gfdi

Earlier   Later