Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2019-06-12
15:22:02 mriedem sure
15:22:19 mriedem maybe you want to re-apply your +2 when doing so
15:39:57 Sundar sean-k-mooney: Re. tempest CI with a fake driver/device, I have some questions
15:41:21 Sundar sean-k-mooney: When there is no real device, the VM bringup is going to fail. Not only that, there would be no VM in 'virsh list', so we can't do 'virsh dumpxml ...' to see if the right device(s) got attached.
15:41:37 Sundar So, how would would one do any testing?
15:42:45 sean-k-mooney Sundar: there are a few ways. one we can use dummy devices
15:43:10 sean-k-mooney but the intent is to test teh end to end workflow not assert a spcific device exits
15:43:35 Sundar What would be the success metric for the test?
15:46:43 mriedem Sundar: tempest isn't going to be doing virsh list anyway
15:46:57 mriedem tempest tests should be hypervisor agnostic, like the compute api should be hypervisor agnostic
15:47:27 mriedem Sundar: the basic success metric for tempest tests are going to be that the API behaves as expected,
15:47:49 mriedem i.e. if i create a server with a flavor configured for an ARQ device profile, that once the server is ACTIVE there is an attached ARQ in cyborg
15:48:02 mriedem and when the server is deleted, the corresponding ARQ resource is also deleted
15:49:08 mriedem Sundar: whitebox integration testing is something that would likely be done in a tempest plugin / tests that are configured to only run when there is real hardware in a 3rd party CI env
15:50:32 Sundar mriedem: "once the server is ACTIVE" -- with a fake device, it never becomes active. It will go to error state.
15:51:20 mriedem Sundar: because of the event not being sent from cyborg, or because libvirt will fail to start the domain?
15:51:26 Sundar We need to ensure the device gets attached to the VM
15:51:28 sean-k-mooney Sundar: not if we add a bus type for dummy devices and handel that in nova
15:51:54 Sundar mridem: latter - libvirt will fail it
15:52:13 mriedem then i agree with sean-k-mooney that we'd need to stub something out if we know we're using a fake device,
15:52:20 sean-k-mooney Sundar: we dont we jsut need to pass info to nova to let it know its a fake device and libivirt can just not try to atach it
15:52:22 mriedem which is gross, but i'd rather have that than no api integration testing
15:52:33 Sundar sean-k-mooney: 'bus type for dumy devices' -- is this a new thing? It is not there today, right?
15:52:54 mriedem wouldn't there be some metadata on the ARQ resource that can tell us (nova) that it's a fake device?
15:52:55 sean-k-mooney sure but its useful for testing
15:53:03 mriedem similar to vif type on a port
15:53:10 mriedem or type on a volume
15:54:06 sean-k-mooney Sundar: we have many thing we can do like have the fake driver create mdev or other software device like a loop device and tell nova to use those too
15:54:30 sean-k-mooney or we can have a sentenel that just tells nova its a fake device for testing
15:55:18 Sundar mriedem, sean-k-mooney: The attach handle in the ARQ, returned by Cyborg to Nova virt driver, will identify the type of the handle, e.g. 'PCI'. I could return a special type, say, 'testPCI', and modify Nova virt driver to ignore that
15:55:37 Sundar That would be used solely for testing
15:55:47 sean-k-mooney Sundar: yes that is basically what i was thinking wew would do
15:55:48 mriedem i'd just calle it "fake" or something like that but yeah
15:56:28 Sundar Ah ok. Got it. Thanks. :)
15:56:49 sean-k-mooney Sundar: the real thing we want to test is teh end to end workflow
15:57:08 mriedem s/real/main/ for now
15:57:21 sean-k-mooney :) right
15:57:28 mriedem real low-level whitebox integration testing can be done with 3rd party CI and real hardware,
15:57:34 mriedem but let's not get the cart before the horse
15:57:38 sean-k-mooney in a third party ci we can then also validate it with real hardware
15:57:56 sean-k-mooney yep
16:01:18 stephenfin mriedem: Cool. I spotted some other things when rebasing. Comments left and I've a follow-up patch I'll post. Feel free to squash it into yours if you want
16:01:56 openstackgerrit Stephen Finucane proposed openstack/nova master: Drop pre-cinder 3.44 version compatibility https://review.opendev.org/621061
16:01:57 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove 'MultiattachNotSupportedByVirtDriver' https://review.opendev.org/651315
16:01:57 openstackgerrit Stephen Finucane proposed openstack/nova master: Follow-up for I6a777b4b7a5729488f939df8c40e49bd40aec3dd https://review.opendev.org/664967
16:03:48 openstackgerrit Stephen Finucane proposed openstack/nova master: Remove 'MultiattachSupportNotYetAvailable' exception https://review.opendev.org/651315
16:07:48 openstackgerrit Balazs Gibizer proposed openstack/nova master: prepare func test env for moving servers with bandwidth https://review.opendev.org/655109
16:07:48 openstackgerrit Balazs Gibizer proposed openstack/nova master: func test for migrate server with ports having resource request https://review.opendev.org/655113
16:14:03 Nick_A Do you guys have any tips for what we should look at to determine why cloud-init is working properly for us on lxd but not kvm?
16:18:50 sean-k-mooney if its working on lxd then the nova metadata service is working properly
16:19:41 sean-k-mooney it might be a diffrence in how netwroking works
16:20:18 sean-k-mooney you really need to try and trace the request
16:21:04 sean-k-mooney so if you can log into one of your kvm instnce try an curl the metadata api via the 169... adress
16:22:03 sean-k-mooney if you dont have a vm image with a password set for debuging then you can enable config drive to initally be able to log in to the vm and debug the issue
16:24:33 Nick_A thank you
16:34:42 openstackgerrit Lee Yarwood proposed openstack/nova master: libvirt: flatten rbd images when unshelving an instance https://review.opendev.org/457886
16:37:58 openstackgerrit Adam Spiers proposed openstack/nova master: Add extra spec parameter and image property for memory encryption https://review.opendev.org/664420
16:42:45 mriedem stephenfin: this shouldn't be stable-only i don't think but you might want to verify if the bug is legit https://review.opendev.org/#/c/664838/1
16:48:59 aspiers efried: Slightly out of my depth here. So shall I go ahead and change spawn() to persist allocations as a new field in the Instance object?
16:50:42 efried aspiers: I wasn't following whatever conversation led up to this
16:51:00 efried Is something we're doing with SEV requiring us to be able to look at the allocation some time after spawn?
16:51:10 efried or is this related to some other topic?
16:51:11 aspiers Yes
16:51:18 efried Do tell
16:51:42 aspiers http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-06-12.log.html#t2019-06-12T12:50:55
16:52:14 aspiers There are multiple places scattered through the code for building the guest config XML which need to know whether SEV is required
16:52:40 aspiers but to perform the check in the way you requested, allocations need to be accessible at those points
16:53:00 efried say wha now?
16:53:13 efried you're telling me that every time libvirt attaches a volume, it rebuilds the whole XML from scratch?
16:53:20 aspiers no
16:53:31 aspiers it needs to know whether to use iommu
16:53:32 efried you're telling me that attaching a volume needs to be SEV aware?
16:53:45 aspiers not necessarily even attaching a volume
16:54:00 efried okay, I see.
16:54:02 aspiers just block devices and stuff at launch-time
16:54:18 aspiers and also to decide whether to include <locked />
16:54:30 aspiers the latest 3 commits at the top of the series
16:55:00 efried I think long term, having allocations on the instance object is a great idea. mriedem, dansmith, how do y'all feel about that?
16:55:00 efried However, in this scenario, you could also solve by commonizing (libvirt/utils?) the code that inspects the flavor/image meta.
16:55:19 dansmith efried: why?
16:55:25 aspiers Yes, that possibility occurred to me too
16:56:42 efried dansmith: example here is SEV. We were considering doing the flavor/image parsing and validation just once, in the request filter, to add resources=MEM_ENCRYPTION_CONTEXT. Then after that, e.g. in spawn(), just looking at the allocation for MEM_ENCRYPTION_CONTEXT so we don't have to redo that parsing/validation.
16:57:12 aspiers here's sean-k-mooney's take: http://eavesdrop.openstack.org/irclogs/%23openstack-nova/%23openstack-nova.2019-06-12.log.html#t2019-06-12T13:13:16
16:57:18 efried In spawn specifically, we are getting the allocation as a param, so that's fine. But there are (aspiers tells me) other places where we need the same information but don't (currently) have access to the allocations.
16:57:21 dansmith spawn having access to the allocations makes sense, but storing them on/with the instance doesn't really
16:57:52 aspiers here's an example: https://review.opendev.org/#/c/662558/5/nova/virt/libvirt/driver.py
16:58:08 dansmith I imagine bauzas would want access to allocations for gpu stuff as well, and I'm sure we discussed making that a param to spawn in the past
16:58:15 aspiers currently this is broken because it's insufficient to pass only the flavor to _sev_required()
16:58:23 efried allocations are already passed to spawn
16:58:33 dansmith ack, okay I was sure we had discussed
16:58:36 aspiers right, the problem is that they are not passed much further on
16:58:58 aspiers sean-k-mooney's other suggestion was to pass from spawn() to _get_guest_xml() and so on
16:59:21 dansmith yeah, pass them further if you want, but storing them .. no so much
16:59:33 aspiers but that doesn't solve things like attaching volumes to an SEV instance
16:59:39 efried you could cache a map of instance:allocations on the virt driver
16:59:47 dansmith efried: dude
16:59:53 efried but if you restart the compute driver you'd have to rebuild that by querying placement
16:59:57 dansmith aspiers: so look them up and pass them in if you need
17:00:09 efried we've said we don't like virt drivers talking to placement
17:00:19 dansmith yeah

Earlier   Later