| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-16 | |||
| 12:06:41 | alexey_weyl | gibi: Hi | |
| 12:07:41 | alexey_weyl | gibi: I have checked it also, and i saw that notifications work for vitrage with master. but still notifications don't work on liberty (notifications about the aggregated hosts) | |
| 12:10:05 | gibi | alexey_weyl: I don't have access to Liberty test node right now. Also Liberty is already reached end of support | |
| 12:10:48 | gibi | alexey_weyl: so even if it is buggy we cannot fix it on Liberty | |
| 12:14:37 | alexey_weyl | gibi: ok, i see that, and it is ok. but I have some other issue | |
| 12:15:25 | alexey_weyl | gibi: The thing is that for example in devstack when you create it, you don't have an "aggregated host" but we have AZs. | |
| 12:16:41 | alexey_weyl | gibi: After our talk before I thought that I could use only the Aggregated hosts to get the needed data, but it seems that I would need to get the AZs by calling the availability-zone list and thus because it doesn't has id we have a problem | |
| 12:17:03 | mdbooth | lyarwood: You back? | |
| 12:19:01 | robcresswell | Quick question; someone's added a patch to Horizon that hides the Soft Reboot button if the Instance state is anything other than Active; is this correct? Had a look at the API docs but they only seem to show how to form the request, not any of the conditions around its usage. | |
| 12:19:54 | mdbooth | robcresswell: soft reboot doesn't make sense for an instance which isn't running | |
| 12:20:38 | robcresswell | mdbooth: Yeah, I'm not clued in on every specific status nova supports | |
| 12:20:44 | mdbooth | error, perhaps | |
| 12:21:30 | mdbooth | robcresswell: I mean, regardless of what the api allows, from a UI pov that restriction makes sense to me | |
| 12:22:06 | robcresswell | mdbooth: Cool. Good enough for me. | |
| 12:22:24 | robcresswell | Thanks | |
| 12:24:43 | alexey_weyl | gibi: The reason that I would need to call the availability-zone list as well is because sometimes we don't have the aggregated hosts but we have AZs | |
| 12:27:27 | gibi | alexey_weyl: if you have an AZ then you automatically have a host aggregate behind it | |
| 12:28:13 | gibi | alexey_weyl: the availability-zone list just iterate throught the host aggregates to see if there is a metadata key on the aggregate with name availability_zone | |
| 12:29:04 | openstackgerrit | John Garbutt proposed openstack/nova-specs master: Add policy-remove-scope-checks spec https://review.openstack.org/433037 | |
| 12:31:55 | gibi | alexey_weyl: what would be different for you if there would be uuid besids AZ name? | |
| 12:32:55 | lyarwood | mdbooth: back now, going to attempt to eat lunch and then catch up on your review comments | |
| 12:40:15 | openstackgerrit | Béla Vancsics proposed openstack/nova master: Reduce code complexity - libvirt/config.py https://review.openstack.org/359879 | |
| 12:40:52 | alexey_weyl | gibi: on my devstack which is a devstack that works with master branch, I have AZs but don't have any aggregated hosts | |
| 12:41:06 | mdbooth | lyarwood: You might want to start with the detach refactor, btw, because I think that requires a rethink which may affect other patches. | |
| 12:42:29 | gibi | alexey_weyl: but you have an empty host aggregate I assume | |
| 12:44:05 | lyarwood | mdbooth: kk looking now | |
| 12:44:08 | gibi | alexey_weyl: actually you can have more than one empty host aggregate connected to the same AZ | |
| 12:45:46 | gibi | alexey_weyl: and those aggregates have uuid | |
| 12:45:56 | gibi | alexey_weyl: if you need something unique | |
| 12:47:04 | openstackgerrit | John Garbutt proposed openstack/nova-specs master: Add additional-default-policy-roles spec https://review.openstack.org/427872 | |
| 12:51:52 | alexey_weyl | gibi: I don't quite understand. I ran on my devstack the command "nova aggregate-list" and received that it is empty and then I ran "nova availability-zone-list" and I saw to AZs | |
| 12:52:27 | alexey_weyl | gibi: what is this empty aggregated host? | |
| 12:59:19 | lyarwood | hmm does any have an idea how I can reference a line in the original file of a gerrit diff? | |
| 13:00:09 | rfolco | cdent, ping | |
| 13:00:16 | cdent | hi rfolco | |
| 13:00:42 | rfolco | cdent, cannot find how to test concurrent update/delete. How to emulate that ? | |
| 13:01:19 | cdent | rfolco: the idea you have before of trying to do a DELETE ../inventories with an incorrect resource_provider_generation should be good enough | |
| 13:02:06 | rfolco | cdent, hmm I thought the generation would fall into a different code path, not that exception. | |
| 13:02:34 | cdent | the generation being wrong is the only thing that can cause ConcurrentUpdateDetected to raise | |
| 13:02:35 | jaypipes | sfinucan: well, there will need to be *some* os-vif complementary part to that, yes. | |
| 13:03:24 | jaypipes | macsz: btw, you're comment on that "i find the lack of commit message disturbing." was most excellent ;) | |
| 13:03:36 | rfolco | cdent, also, delete does not produce json, only post/put do. So not sure about the other suggestion on checking delete request/response with 409 | |
| 13:03:47 | gibi | alexey_weyl: run nova aggregate-list | |
| 13:04:09 | cdent | rfolco: the error's response body will be json if you send an accept header of application/json | |
| 13:04:27 | rfolco | cdent, accept header mesans the decorator ? | |
| 13:04:32 | cdent | so even though a success response is empty an error response will not | |
| 13:04:51 | cdent | i mean in the gabbi test: request_headers:\naccept: application/json\n | |
| 13:05:14 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: use context mgr in instance.delete https://review.openstack.org/443764 | |
| 13:05:46 | rfolco | cdent, hmmm, ok, thanks | |
| 13:06:05 | cdent | rfolco: that "hmmm" makes it sound you're not quite sure what I mean or ... ? | |
| 13:06:27 | rfolco | cdent, haha hmm means, ok in theory I got it, lets see in practice. | |
| 13:06:58 | cdent | ah, okay, cool. let me know if you need more info as you go along | |
| 13:07:00 | rfolco | cdent, I'll bother you again if I get stuck | |
| 13:08:03 | gibi | alexey_weyl: the AZ is not a real entity in nova. The host aggregate is the real one and if you have an aggregate with a special metadata 'availability_zone' then the hosts in that aggregate become part of an AZ | |
| 13:08:38 | alexey_weyl | gibi: I have ran the nova aggregate-list and I returned empty | |
| 13:08:47 | alexey_weyl | gibi: although I have 2 AZs | |
| 13:24:16 | gibi | alexey_weyl: nova and internal? | |
| 13:25:17 | gibi | alexey_weyl: nova is the default AZ. host not part of any AZ will belong there | |
| 13:26:00 | gibi | alexey_weyl: you can ignore the internal, as far as I understand internal never contains compute hosts | |
| 13:26:53 | mriedem | has anyone seen this test fail before? http://logs.openstack.org/75/446175/1/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/479c7bf/console.html#_2017-03-16_01_31_33_688563 | |
| 13:26:58 | gibi | alexey_weyl: I hope somebody correct me as I reached the limit of my knowledge | |
| 13:29:17 | mriedem | bauzas: can you take a look at the ocata regression fix here https://review.openstack.org/#/c/444106/ and the functional test below it? the test is the one i was talking about yesterday. | |
| 13:29:30 | mriedem | we need to get those in because they are hurting the ironic ci | |
| 13:45:36 | openstackgerrit | Rafael Folco proposed openstack/nova master: DELETE all inventory for a resource provider https://review.openstack.org/416669 | |
| 13:50:44 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: doc: configurable versioned notifications topics https://review.openstack.org/446523 | |
| 13:50:58 | jlvillal | sfinucan, Thanks for the heads up on the flake8 patch. I commented. | |
| 13:52:50 | mriedem | lyarwood: mdbooth: i'm debugging a failure in the gate http://logs.openstack.org/75/446175/1/gate/gate-tempest-dsvm-neutron-full-ubuntu-xenial/479c7bf/logs/screen-n-cpu.txt.gz?level=TRACE#_2017-03-16_01_01_32_897 where detach of a volume to an unshelved server fails, but it actually looks like when we did the detach on the guest, we got an error from libvirt that we usually handle (device not found) and raise that back up to | |
| 13:52:51 | mriedem | compute manager to basically ignore saying it's not actually attached, | |
| 13:53:01 | mriedem | but then it looks like we try to detach again, and get a different unhandled i/o error from libvirt | |
| 13:53:18 | lyarwood | looking | |
| 13:54:06 | mriedem | the test steps are: create a server and wait for it to be active, create a volume and wait for it to be available, shelve the server and wait for it to be offloaded, attach the volume, unshelve the server and wait for it to be active, then verify the volume is attached, and then start the teardown (detach volume and wait for it to be available, then delete the volume, then delete the server) | |
| 13:55:22 | lyarwood | and this is failing during the teardown right? | |
| 13:55:26 | openstackgerrit | Gábor Antal proposed openstack/nova master: Transform instance.volume_attach notification https://review.openstack.org/401992 | |
| 13:55:30 | mriedem | one thing i noticed is that the test specifies /dev/vdb when attaching the volume, and since it's the libvirt driver it ignores that and the volume is actually mounted at /dev/vdc | |
| 13:55:33 | mriedem | lyarwood: yeah | |
| 13:55:37 | mriedem | the volume detach fails | |
| 13:55:55 | mriedem | so tempest times out waiting for the volume to go from in-use to available | |
| 13:56:33 | mriedem | we get here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/guest.py#L408 | |
| 13:56:34 | openstackgerrit | Sean Dague proposed openstack/nova master: remove hacking rule that enforces log translation https://review.openstack.org/446452 | |
| 13:57:02 | openstackgerrit | Gábor Antal proposed openstack/nova master: Transform instance.volume_detach notification https://review.openstack.org/408676 | |
| 13:59:12 | mriedem | what i don't get is it looks like we hit the "no target device" error from libvirt, which we handle and reraise as DeviceNotFound, but then it looks like we try to detach again and get "libvirtError: End of file while reading data: Input/output error" which we don't handle | |
| 14:00:58 | mriedem | ok so the device name is different also because when attaching a volume to a shelved offloaded instance, we don't set the device on the bdm https://github.com/openstack/nova/blob/master/nova/compute/api.py#L3678 | |
| 14:01:14 | mriedem | the tempest test should probably not even be specifying the device name | |
| 14:02:08 | lyarwood | yeah sorry I was just looking for the device xml during the detach to confirm what the target device was | |
| 14:12:09 | mriedem | so in the attach volume to shelved offloaded server case, we actually call os-attach in the cinder api to mark the volume as in-use, since we can't do it from the compute (since the instance isn't on a host), | |
| 14:12:10 | openstackgerrit | Gábor Antal proposed openstack/nova master: Transform instance.volume_detach notification https://review.openstack.org/408676 | |
| 14:12:40 | mriedem | i'm wondering how that might mess up the new create attachment flows, i guess we can just update the volume attachment later when we're unshelving on the host and have the connection info, we update the vol attachment in cinder | |
| 14:12:55 | alexey_weyl | gibi: so the nova AZ has no aggregated host that the nova AZ is part of. | |
| 14:13:23 | alexey_weyl | gibi: is it correct to devstack only? or also to openstack in production? | |
| 14:13:24 | lyarwood | mriedem: right, update with the connector and get the connection_info back during unshelve | |
| 14:14:27 | gibi | alexey_weyl: the nova AZ contains by default every host that does not belong to any other AZ. | |
| 14:15:27 | gibi | alexey_weyl: it is not devstack specific but the name of the default AZ is configurable in the nova.conf with default_availability_zone option | |
| 14:16:05 | lyarwood | mriedem: do we have a bug for the detach from unshelve issue btw? I don't think it's an issue with the target dev, we look this up correctly and call for the detach using the correct xml AFAICT | |
| 14:16:16 | mriedem | lyarwood: we don't yet no | |
| 14:16:27 | lyarwood | mriedem: kk, I'll write one up if I can reproduce | |
| 14:16:33 | alexey_weyl | gibi: I see. Thanks a lot | |
| 14:16:50 | mriedem | i think we've been seeing "libvirtError: End of file while reading data: Input/output error" in the logs so much when the libvirt connection temporarily drops, that we just haven't considered this a separate issue | |
| 14:17:01 | mriedem | but it clearly is related to detaching the device | |
| 14:17:01 | mriedem | if ret == -1: raise libvirtError ('virDomainDetachDeviceFlags() failed', dom=self) | |
| 14:19:05 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transform instance.reboot notifications https://review.openstack.org/382959 | |