| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2017-03-16 | |||
| 09:25:44 | gibi | bauzas, alexey_weyl: it seems so, yes. | |
| 09:25:48 | bauzas | perfect | |
| 09:25:57 | alexey_weyl | bauzas: gibi: I didn't quite understood all what you said. can you please summerize that for me? what do I need to do? | |
| 09:26:02 | bauzas | alexey_weyl: so, your problem is solved from a nova perspective | |
| 09:26:12 | gibi | alexey_weyl: sure | |
| 09:26:21 | bauzas | alexey_weyl: as gibi said, we already emit notifications on aggregate update | |
| 09:26:26 | bauzas | https://github.com/openstack/nova/blob/master/nova/objects/aggregate.py#L442 | |
| 09:26:47 | bauzas | alexey_weyl: the thing is, we emit the information on a non-versioned dict | |
| 09:27:04 | gibi | alexey_weyl: we are creating a new notification interface for nova, to have a proper interface. these are called versioned notifications. | |
| 09:27:13 | bauzas | alexey_weyl: so gibi is the lead for transforming our legacy payloads into a richer format | |
| 09:27:17 | alexey_weyl | bauzas: gibi: I am listening to all of the notifications from nova and I didn't see any notification from nova when I changed the name of the AZ or the Aggregated host | |
| 09:27:40 | gibi | alexey_weyl: do you listen to notifications topic? | |
| 09:27:45 | alexey_weyl | yes | |
| 09:28:01 | alexey_weyl | we have added to notification topics a vitrage_notificcation | |
| 09:28:02 | bauzas | https://github.com/openstack/nova/blob/master/nova/objects/aggregate.py#L443 is the notification name | |
| 09:28:10 | gibi | alexey_weyl: hm, you should get aggregate.updatemetadata.end notification | |
| 09:28:11 | alexey_weyl | and thus we can receive all the data | |
| 09:28:17 | openstackgerrit | Béla Vancsics proposed openstack/nova master: Reduce code complexity - libvirt/config.py https://review.openstack.org/359879 | |
| 09:28:41 | alexey_weyl | bauzas: gibi: I will check it again just to make sure. | |
| 09:28:44 | bauzas | in theory, that notification has a payload including the metadata that changed https://github.com/openstack/nova/blob/master/nova/objects/aggregate.py#L423 | |
| 09:29:01 | gibi | alexey_weyl: if you not see that after updating metadata of an aggregate then we have some bug | |
| 09:29:22 | gibi | alexey_weyl: I will try to verify that as well | |
| 09:30:05 | gibi | alexey_weyl: btw, do vitrage plans to move from the legacy nova notifications to the versioned ones? | |
| 09:31:48 | bauzas | gibi: AFAICS, if you do the WSGI action on os-aggregates to set_metadata, you call the Aggregate object method that emits the legacy notification | |
| 09:31:58 | bauzas | https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/aggregates.py#L196 | |
| 09:32:34 | bauzas | and https://github.com/openstack/nova/blob/master/nova/compute/api.py#L4490 | |
| 09:32:50 | alexey_weyl | gibi: I have to admit that this is the first time I hear about that, so I will need to check it out. But if this is something that all the projects have then vitrage will have to use such the new notifications | |
| 09:33:31 | gibi | alexey_weyl: we are in the process to define and implement the new interface so it is the good time to chime in from consumer side | |
| 09:33:44 | gibi | bauzas: that is my understanding of the code as well | |
| 09:34:17 | gibi | bauzas: but if alexey_weyl does not see the notification then there is something we don't see | |
| 09:34:44 | alexey_weyl | gibi: bauzas: I have checked it again, and I didn't receive any notification about AZ and Aggregated host when changed their names | |
| 09:35:14 | gibi | alexey_weyl: do you get aggregate.create when you create a new aggregate? | |
| 09:35:15 | bauzas | alexey_weyl: and what if you change a metadata key that is *not* availability_zone ? | |
| 09:36:31 | alexey_weyl | gibi: bauzas: When i add, delete, update instances I receive all the notification. | |
| 09:37:04 | alexey_weyl | gibi: bauzas: maybe it is related to the fact that I perform the changes in the horizon admin tab? | |
| 09:37:29 | alexey_weyl | gibi: bauzas: I will try to create a new aggregated host via cli | |
| 09:37:44 | gibi | alexey_weyl: I think horizon should not be the problem | |
| 09:39:03 | alexey_weyl | gibi: bauzas: Ok, have created a new aggregated using the cli and i see it in the horizon but I didn't receive any upate | |
| 09:40:52 | gibi | alexey_weyl: strange | |
| 09:41:50 | gibi | alexey_weyl: did you check the notification from vitrage logs? doesn't vitrage filter on event_type? | |
| 09:42:27 | alexey_weyl | I have checked, I have put a print before the filtering | |
| 09:42:47 | alexey_weyl | gibi: I have checked, I have put a print before the filtering | |
| 09:44:33 | alexey_weyl | gibi: bauzas: | |
| 09:44:46 | alexey_weyl | gibi: bauzas: I might know the problem | |
| 09:44:50 | gibi | alexey_weyl: I set the notification driver to log in nova.conf and created an aggregate | |
| 09:44:55 | gibi | alexey_weyl: and I got the log: | |
| 09:44:56 | gibi | alexey_weyl: /var/log/nova/nova-api.log:2017-03-16T10:43:00.616385+01:00 cic-3 nova-api[22020]: 2017-03-16 10:43:00.615 22020 INFO oslo.messaging.notification.aggregate.create.start [req-0b7f199d-8d06-4d2c-bcd6-fc5a8d03b623 ad400a8540f744b5b7041a356d51b6f3 a21d93baeb5e42c8a9d1da782279e309 - - -] {"event_type": "aggregate.create.start", "timestamp": "2017-03-16 09:43:00.615696", "payload": {"name": "gibi-aggregate"}, "priority": "INF | |
| 09:45:05 | gibi | alexey_weyl: but this is Mitaka version | |
| 09:45:12 | alexey_weyl | gibi: bauzas: The productive system I work on is liberty. This might be the problem, right? | |
| 09:45:43 | gibi | alexey_weyl: the legacy aggregate.create works for me on Mitaka | |
| 09:45:50 | gibi | alexey_weyl: let me check Liberty... | |
| 09:46:27 | bauzas | FWIW https://blueprints.launchpad.net/nova/+spec/az-block-name-update | |
| 09:46:45 | gibi | alexey_weyl: Liberty code also has the notification https://github.com/openstack/nova/blob/liberty-eol/nova/objects/aggregate.py#L126 | |
| 09:47:03 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add PCIWeigher https://review.openstack.org/379524 | |
| 09:47:19 | alexey_weyl | gibi: bauzas: ok, now it is really weird. | |
| 09:48:36 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Add PCIWeigher https://review.openstack.org/379524 | |
| 09:48:37 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Prefer non-PCI host nodes for non-PCI instances https://review.openstack.org/379625 | |
| 09:48:38 | gibi | alexey_weyl: is there a way for you to reproduce the problem closer to master ? | |
| 09:49:09 | alexey_weyl | gibi: bauzas: Yes, I am going to work on pike now, and i will check it there. | |
| 09:49:15 | alexey_weyl | gibi: bauzas: will update you later on. | |
| 09:49:27 | alexey_weyl | gibi: bauzas: anyway, thank you very much guys!!!! :)\ | |
| 09:49:37 | gibi | alexey_weyl: just ping me when you have some test results | |
| 09:50:49 | gibi | bauzas: you proposal makes sense to me. less thing to change means less confusion. So changing the AZ name would be a lot harderd after it. | |
| 09:50:58 | gibi | bauzas: deleting aggregate and creating a new one | |
| 09:51:10 | alexey_weyl | gibi: no problem. thanks :) | |
| 09:52:01 | bauzas | gibi: in theory, once you accept an instance to be created on that AZ, then your contract becomes harder | |
| 09:52:18 | bauzas | because some people trust you for caring their instances | |
| 09:52:30 | gibi | bauzas: true. | |
| 09:53:42 | gibi | bauzas: what if just the az name needs to be changed but not the meanin of it? I mean the renamed az still means the same failure domain. | |
| 09:54:21 | bauzas | gibi: what I need to understand is why operators need to update their AZ names | |
| 09:54:33 | bauzas | it's just a string | |
| 09:54:42 | gibi | bauzas: renaming the room in the building? :) | |
| 09:54:57 | bauzas | so they would expose their topology semantically? | |
| 09:55:16 | jith | Hi all, I have configured openstack kilo setup in debian jessie with one controller node and two compute nodes. I have used glusterfs for shared storage. I have mounted the shared storage in /var/lib/nova/instances on both nodes. I can do migration of vms between nodes.. But live-migration throws following error.. Pls do guide me.. http://pastebin.com/DHPWEkaZ | |
| 09:55:24 | gibi | bauzas: you are right. My point is not really valid. | |
| 09:55:37 | gibi | bauzas: anyhow I'm just playing devils advocate here. | |
| 09:55:47 | bauzas | jith: I'm sorry, this channel is focused on development questions | |
| 09:56:00 | bauzas | gibi: nah it's fine, I could be wrong | |
| 09:56:28 | gibi | bauzas: az name should not reflect real topology. It should be failure domain A , B etc | |
| 09:56:47 | gibi | bauzas: so your point seems OK to me | |
| 09:56:47 | bauzas | gibi: the only problem I see with preventing the AZ rename is that it's becoming hard to migrate instances | |
| 09:57:01 | bauzas | say I made a mistake and I want to fix that | |
| 09:57:14 | gibi | bauzas: does admin can migrate instance by ignoring AZ? | |
| 09:57:25 | bauzas | migi: he could force the migration | |
| 09:57:42 | bauzas | oops | |
| 09:57:47 | bauzas | s/migi/gibi | |
| 09:57:53 | migi | :) | |
| 09:57:56 | gibi | bauzas: force migration could work for me | |
| 09:58:00 | gibi | migi, bauzas: :) | |
| 09:58:23 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Validate huge pages https://review.openstack.org/399653 | |
| 09:58:24 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: create functional test base class https://review.openstack.org/407055 | |
| 09:58:37 | bauzas | anyway, migrating from one AZ to another is a big deal | |
| 09:59:03 | sfinucan | jaypipes: Fixed that hugepages functional test https://review.openstack.org/#/c/399653/ | |
| 10:00:06 | bauzas | gibi: since one host can't be in two AZs at the same time, I'd suggest to place the host on a specific aggregate that is not AZ-facing | |
| 10:00:29 | bauzas | gibi: and then move the host to the new aggregate with the right AZ name | |
| 10:00:59 | bauzas | rather than issuing a live-migration | |
| 10:01:49 | bauzas | I should test that in devstack | |
| 10:03:52 | mdbooth | lyarwood (or anybody else): Do you know what would consume volume.detach instance notifications? | |
| 10:04:41 | gibi | bauzas: ohh, I see | |
| 10:05:52 | lyarwood | mdbooth: no sorry | |