Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2017-03-16
12:32:55 lyarwood mdbooth: back now, going to attempt to eat lunch and then catch up on your review comments
12:40:15 openstackgerrit Béla Vancsics proposed openstack/nova master: Reduce code complexity - libvirt/config.py https://review.openstack.org/359879
12:40:52 alexey_weyl gibi: on my devstack which is a devstack that works with master branch, I have AZs but don't have any aggregated hosts
12:41:06 mdbooth lyarwood: You might want to start with the detach refactor, btw, because I think that requires a rethink which may affect other patches.
12:42:29 gibi alexey_weyl: but you have an empty host aggregate I assume
12:44:05 lyarwood mdbooth: kk looking now
12:44:08 gibi alexey_weyl: actually you can have more than one empty host aggregate connected to the same AZ
12:45:46 gibi alexey_weyl: and those aggregates have uuid
12:45:56 gibi alexey_weyl: if you need something unique
12:47:04 openstackgerrit John Garbutt proposed openstack/nova-specs master: Add additional-default-policy-roles spec https://review.openstack.org/427872
12:51:52 alexey_weyl gibi: I don't quite understand. I ran on my devstack the command "nova aggregate-list" and received that it is empty and then I ran "nova availability-zone-list" and I saw to AZs
12:52:27 alexey_weyl gibi: what is this empty aggregated host?
12:59:19 lyarwood hmm does any have an idea how I can reference a line in the original file of a gerrit diff?
13:00:09 rfolco cdent, ping
13:00:16 cdent hi rfolco
13:00:42 rfolco cdent, cannot find how to test concurrent update/delete. How to emulate that ?
13:01:19 cdent rfolco: the idea you have before of trying to do a DELETE ../inventories with an incorrect resource_provider_generation should be good enough
13:02:06 rfolco cdent, hmm I thought the generation would fall into a different code path, not that exception.
13:02:34 cdent the generation being wrong is the only thing that can cause ConcurrentUpdateDetected to raise
13:02:35 jaypipes sfinucan: well, there will need to be *some* os-vif complementary part to that, yes.
13:03:24 jaypipes macsz: btw, you're comment on that "i find the lack of commit message disturbing." was most excellent ;)
13:03:36 rfolco cdent, also, delete does not produce json, only post/put do. So not sure about the other suggestion on checking delete request/response with 409
13:03:47 gibi alexey_weyl: run nova aggregate-list
13:04:09 cdent rfolco: the error's response body will be json if you send an accept header of application/json
13:04:27 rfolco cdent, accept header mesans the decorator ?
13:04:32 cdent so even though a success response is empty an error response will not
13:04:51 cdent i mean in the gabbi test: request_headers:\naccept: application/json\n
13:05:14 openstackgerrit Balazs Gibizer proposed openstack/nova master: use context mgr in instance.delete https://review.openstack.org/443764
13:05:46 rfolco cdent, hmmm, ok, thanks
13:06:05 cdent rfolco: that "hmmm" makes it sound you're not quite sure what I mean or ... ?
13:06:27 rfolco cdent, haha hmm means, ok in theory I got it, lets see in practice.
13:06:58 cdent ah, okay, cool. let me know if you need more info as you go along
13:07:00 rfolco cdent, I'll bother you again if I get stuck
13:08:03 gibi alexey_weyl: the AZ is not a real entity in nova. The host aggregate is the real one and if you have an aggregate with a special metadata 'availability_zone' then the hosts in that aggregate become part of an AZ
13:08:38 alexey_weyl gibi: I have ran the nova aggregate-list and I returned empty
13:08:47 alexey_weyl gibi: although I have 2 AZs
13:24:16 gibi alexey_weyl: nova and internal?
13:25:17 gibi alexey_weyl: nova is the default AZ. host not part of any AZ will belong there
13:26:00 gibi alexey_weyl: you can ignore the internal, as far as I understand internal never contains compute hosts
13:26:53 mriedem has anyone seen this test fail before? http://logs.openstack.org/75/446175/1/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/479c7bf/console.html#_2017-03-16_01_31_33_688563
13:26:58 gibi alexey_weyl: I hope somebody correct me as I reached the limit of my knowledge
13:29:17 mriedem bauzas: can you take a look at the ocata regression fix here https://review.openstack.org/#/c/444106/ and the functional test below it? the test is the one i was talking about yesterday.
13:29:30 mriedem we need to get those in because they are hurting the ironic ci
13:45:36 openstackgerrit Rafael Folco proposed openstack/nova master: DELETE all inventory for a resource provider https://review.openstack.org/416669
13:50:44 openstackgerrit Balazs Gibizer proposed openstack/nova master: doc: configurable versioned notifications topics https://review.openstack.org/446523
13:50:58 jlvillal sfinucan, Thanks for the heads up on the flake8 patch. I commented.
13:52:50 mriedem lyarwood: mdbooth: i'm debugging a failure in the gate http://logs.openstack.org/75/446175/1/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/479c7bf/logs/screen-n-cpu.txt.gz?level=TRACE#_2017-03-16_01_01_32_897 where detach of a volume to an unshelved server fails, but it actually looks like when we did the detach on the guest, we got an error from libvirt that we usually handle (device not found) and raise that back up to
13:52:51 mriedem compute manager to basically ignore saying it's not actually attached,
13:53:01 mriedem but then it looks like we try to detach again, and get a different unhandled i/o error from libvirt
13:53:18 lyarwood looking
13:54:06 mriedem the test steps are: create a server and wait for it to be active, create a volume and wait for it to be available, shelve the server and wait for it to be offloaded, attach the volume, unshelve the server and wait for it to be active, then verify the volume is attached, and then start the teardown (detach volume and wait for it to be available, then delete the volume, then delete the server)
13:55:22 lyarwood and this is failing during the teardown right?
13:55:26 openstackgerrit Gábor Antal proposed openstack/nova master: Transform instance.volume_attach notification https://review.openstack.org/401992
13:55:30 mriedem one thing i noticed is that the test specifies /dev/vdb when attaching the volume, and since it's the libvirt driver it ignores that and the volume is actually mounted at /dev/vdc
13:55:33 mriedem lyarwood: yeah
13:55:37 mriedem the volume detach fails
13:55:55 mriedem so tempest times out waiting for the volume to go from in-use to available
13:56:33 mriedem we get here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L408
13:56:34 openstackgerrit Sean Dague proposed openstack/nova master: remove hacking rule that enforces log translation https://review.openstack.org/446452
13:57:02 openstackgerrit Gábor Antal proposed openstack/nova master: Transform instance.volume_detach notification https://review.openstack.org/408676
13:59:12 mriedem what i don't get is it looks like we hit the "no target device" error from libvirt, which we handle and reraise as DeviceNotFound, but then it looks like we try to detach again and get "libvirtError: End of file while reading data: Input/output error" which we don't handle
14:00:58 mriedem ok so the device name is different also because when attaching a volume to a shelved offloaded instance, we don't set the device on the bdm https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3678
14:01:14 mriedem the tempest test should probably not even be specifying the device name
14:02:08 lyarwood yeah sorry I was just looking for the device xml during the detach to confirm what the target device was
14:12:09 mriedem so in the attach volume to shelved offloaded server case, we actually call os-attach in the cinder api to mark the volume as in-use, since we can't do it from the compute (since the instance isn't on a host),
14:12:10 openstackgerrit Gábor Antal proposed openstack/nova master: Transform instance.volume_detach notification https://review.openstack.org/408676
14:12:40 mriedem i'm wondering how that might mess up the new create attachment flows, i guess we can just update the volume attachment later when we're unshelving on the host and have the connection info, we update the vol attachment in cinder
14:12:55 alexey_weyl gibi: so the nova AZ has no aggregated host that the nova AZ is part of.
14:13:23 alexey_weyl gibi: is it correct to devstack only? or also to openstack in production?
14:13:24 lyarwood mriedem: right, update with the connector and get the connection_info back during unshelve
14:14:27 gibi alexey_weyl: the nova AZ contains by default every host that does not belong to any other AZ.
14:15:27 gibi alexey_weyl: it is not devstack specific but the name of the default AZ is configurable in the nova.conf with default_availability_zone option
14:16:05 lyarwood mriedem: do we have a bug for the detach from unshelve issue btw? I don't think it's an issue with the target dev, we look this up correctly and call for the detach using the correct xml AFAICT
14:16:16 mriedem lyarwood: we don't yet no
14:16:27 lyarwood mriedem: kk, I'll write one up if I can reproduce
14:16:33 alexey_weyl gibi: I see. Thanks a lot
14:16:50 mriedem i think we've been seeing "libvirtError: End of file while reading data: Input/output error" in the logs so much when the libvirt connection temporarily drops, that we just haven't considered this a separate issue
14:17:01 mriedem but it clearly is related to detaching the device
14:17:01 mriedem if ret == -1: raise libvirtError ('virDomainDetachDeviceFlags() failed', dom=self)
14:19:05 openstackgerrit Balazs Gibizer proposed openstack/nova master: Transform instance.reboot notifications https://review.openstack.org/382959
14:19:10 sdague sfinucan: fixed unit tests on - https://review.openstack.org/#/c/446452/
14:19:20 mriedem lyarwood: i think i see an issue
14:19:52 mriedem lyarwood: in the normal attach volume flow, we pass do_driver_attach=True here https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L4799
14:20:02 mriedem which is used here https://github.com/openstack/nova/blob/master/nova/virt/block_device.py#L265
14:20:10 mriedem to actually attach the block device to the guest
14:20:19 gibi sdague: Hi! I left a detailed use case description in https://review.openstack.org/#/c/440580/5/specs/pike/approved/scheduler-hints-in-server-details.rst
14:20:21 dansmith gibi: replied on https://review.openstack.org/#/c/445697/2
14:20:29 gibi dansmith: checking...
14:20:34 mriedem lyarwood: in the case of unshelve, we don't pass that https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L4445
14:20:37 mriedem and it defaults to False
14:20:50 mriedem so i don't think we ever actually attach the block device to the guest during unshelve
14:21:12 lyarwood mriedem: isn't part of the test to start the instance?
14:21:25 mriedem lyarwood: the test doesn't ssh into the guest to see that the device is there
14:21:42 mriedem the test just hits the compute api asking if the volume is associated with the instance, which it is via the bdms table
14:21:51 mriedem that doesn't mean it's actually attached in the guest though :)
14:22:17 mriedem i'm wondering if it's always been this way and the test is new, or if this was regressed when we removed check_attach internally,
14:22:25 lyarwood mriedem: right, I might be mixing up the ordering here, there's debug XML listing the device when we bring the instance up
14:22:34 mriedem https://github.com/openstack/nova/commit/63805735c25a54ad1b9b97e05080c1a6153d8e22
14:23:24 mriedem nvm, i guess we just never pass do_driver_attach for unshelve
14:23:51 sfinucan sdague: Cool. Done

Earlier   Later