| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-08-26 | |||
| 16:35:01 | sean-k-mooney | that only works for libvirt | |
| 16:35:09 | melwitt | stephenfin: ah ok. thanks | |
| 16:35:11 | bauzas | stephenfin: but in the meantime, instancemixin seems better to use IMHO + those extra lines | |
| 16:35:13 | yoctozepto | sean-k-mooney: nah, that's bad naming | |
| 16:35:28 | stephenfin | bauzas: We have it. It's call '_IntegratedTestBase' | |
| 16:35:29 | yoctozepto | actually masakari uses this-called-string for corosync and nova-compute | |
| 16:35:36 | yoctozepto | and asssumes that they are the same | |
| 16:35:41 | stephenfin | bauzas: look at the superclasses for that https://github.com/openstack/nova/blob/master/nova/tests/functional/integrated_helpers.py#L1008 | |
| 16:35:47 | yoctozepto | somehow they thought it always equals hypervisors... | |
| 16:35:59 | yoctozepto | well, I would have as well probably | |
| 16:36:08 | stephenfin | *It's called | |
| 16:36:30 | bauzas | stephenfin: I do remember the pain writing functional tests that were requiring duplicate code | |
| 16:37:09 | sean-k-mooney | yoctozepto: maybe its not clear if https://docs.openstack.org/api-ref/compute/?expanded=list-compute-services-detail#list-compute-services is the hyperviour_hostname or the host value form teh cofnig | |
| 16:37:16 | sean-k-mooney | i suppect its the host value form the config | |
| 16:38:05 | yoctozepto | sean-k-mooney: it is, it's just that noonedeadpunk kept the old name, I commented on that too | |
| 16:38:27 | sean-k-mooney | ok so its the host value | |
| 16:38:31 | sean-k-mooney | not the hypervisor_hostname | |
| 16:38:36 | noonedeadpunk | At fisrt I was trying to keep changes minimal) | |
| 16:38:54 | yoctozepto | noonedeadpunk: all in all, this patch makes masakari a bit better but still kinda delays the issue that one could actually want to monitor the hypervisors, and have their names agree with those in corosync cluster | |
| 16:39:09 | bauzas | either way, it's becoming late for me, \o | |
| 16:39:12 | sean-k-mooney | noonedeadpunk: thats an good goal generaly but but names existingon the compute node recored in teh db | |
| 16:39:34 | sean-k-mooney | so we should try to use hypervisor_hostname and host name differently depending on which we mean | |
| 16:40:10 | noonedeadpunk | Yeah, I see it now. Just wanted to made it as bug fix to get backported at first :p | |
| 16:40:38 | noonedeadpunk | but you're totally right | |
| 16:41:53 | yoctozepto | noonedeadpunk: if we straighten the logic right, then you know who's the core who will accept those backports :-) | |
| 16:42:35 | sean-k-mooney | noonedeadpunk: these are teh rules libvirt uses by the way | |
| 16:42:38 | sean-k-mooney | https://github.com/libvirt/libvirt/blob/7ba4838a18afb609d900190c57b0cbfb842ccbdf/src/util/virutil.c#L470-L489 | |
| 16:45:37 | sean-k-mooney | the did modify this 8 months ago https://github.com/libvirt/libvirt/commit/26d9748ff114a060ee751959d108d062f737f5d9 | |
| 16:46:35 | noonedeadpunk | sean-k-mooney: I think that C way of gethostname may differ from python implementation.... | |
| 16:46:52 | noonedeadpunk | And I'm not C guy unfortunatelly... | |
| 16:46:52 | sean-k-mooney | they did not in the past but now they are calling a new fucntion | |
| 16:47:08 | sean-k-mooney | the python way called the c function | |
| 16:47:18 | yoctozepto | sean-k-mooney: I linked to that libvirt code just above :-) | |
| 16:47:21 | sean-k-mooney | libvirt is now using g_get_host_name(); | |
| 16:47:47 | sean-k-mooney | instead of gethostname | |
| 16:49:30 | noonedeadpunk | Yeah, so things may broke for me with some libvirt update... Anyway that would mean time to work on naming convention:) Like I have only 1 such region - the first one I've built... | |
| 16:49:43 | noonedeadpunk | but it created corner case.... | |
| 16:52:05 | sean-k-mooney | 6.0 is the default in ubuntu and centos 8 | |
| 16:52:29 | sean-k-mooney | i know ooo hardcodes the fqdn in /etc/hostname | |
| 16:52:38 | yoctozepto | noonedeadpunk: masakari needs fixing nonetheless ;p | |
| 16:52:46 | sean-k-mooney | and i think it also manually sets the [default]/host value | |
| 16:53:12 | sean-k-mooney | nova depends on teh actull hostname not changing | |
| 16:59:41 | sean-k-mooney | yoctozepto: noonedeadpunk this is where we call libvirt if you are interested https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L976 | |
| 17:00:06 | sean-k-mooney | and that is used here https://github.com/openstack/nova/blob/a7735d5e379c20c59cfb639f9f4d339bcffca2f9/nova/virt/libvirt/driver.py#L8418 | |
| 17:00:47 | yoctozepto | sean-k-mooney: thanks, I just did not realize it's libvirt telling us its name, but thanks to you it's clear for me now | |
| 17:02:03 | sean-k-mooney | ya genericly its the hypervior e.g vmware or hyperv or libvirt that provides this | |
| 17:02:59 | openstackgerrit | Merged openstack/nova stable/train: libvirt: Provide VIR_MIGRATE_PARAM_PERSIST_XML during live migration https://review.opendev.org/747973 | |
| 17:03:15 | openstackgerrit | Merged openstack/nova master: releasenotes: Detail support for server ops with vTPM https://review.opendev.org/748215 | |
| 17:04:26 | noonedeadpunk | sean-k-mooney: just one more stupid question... trying to find host-evacuate api call in https://docs.openstack.org/api-ref/compute/ but don't see for some reason (only server evacuate) | |
| 17:05:03 | yoctozepto | sean-k-mooney: https://docs.openstack.org/api-ref/compute/?expanded=evacuate-server-evacuate-action-detail#evacuate-server-evacuate-action ? | |
| 17:05:10 | yoctozepto | noonedeadpunk: ^ | |
| 17:05:28 | yoctozepto | it's server (instance/vm) that's getting evacuated | |
| 17:06:17 | noonedeadpunk | and how does https://docs.openstack.org/nova/rocky/admin/evacuate.html#evacuate-all-instances work? It gets list of instances and evacuate one by one ? | |
| 17:09:07 | yoctozepto | noonedeadpunk: that's what I would assume, could use checking the code | |
| 17:09:47 | sean-k-mooney | noonedeadpunk: host-evacuate is not an api action | |
| 17:09:55 | sean-k-mooney | noonedeadpunk: its a client command | |
| 17:09:59 | noonedeadpunk | ok, got it | |
| 17:10:00 | noonedeadpunk | yeah | |
| 17:10:24 | sean-k-mooney | noonedeadpunk: http://www.danplanet.com/blog/2016/03/03/evacuate-in-nova-one-command-to-confuse-us-all/ | |
| 17:11:14 | sean-k-mooney | dansmith's blog that is basically mandatory reading on host-evacuate and how it related to every thing else | |
| 17:11:47 | noonedeadpunk | oh, ok, now I know under what conditions our client loose all of their data from ephemeral drives... | |
| 17:11:48 | sean-k-mooney | yoctozepto: host-evacuate does not call teh evacuate api | |
| 17:11:55 | noonedeadpunk | when masakari calls instance evacuate | |
| 17:11:55 | sean-k-mooney | yoctozepto: it does cold migrate | |
| 17:12:33 | sean-k-mooney | actully no it does evacuate | |
| 17:12:39 | sean-k-mooney | i should read the blog more often | |
| 17:13:21 | noonedeadpunk | "The core of the evacuate process in nova is actually rebuild, which in many cases is a destructive operation" | |
| 17:13:31 | noonedeadpunk | so it's not evacuate and you;re right | |
| 17:13:40 | sean-k-mooney | from the blog | |
| 17:13:42 | sean-k-mooney | "The nova host-evacuate command does not translate directly to a server-side operation, but is more of a client-side macro or “meta operation.” When you call this command, you provide a hypervisor hostname, which the client uses to list and trigger evacuate operations on each instance running on that hypervisor. You would use this command post-failure (just like the single-instance evacuate | |
| 17:13:45 | sean-k-mooney | command) to trigger evacuations of all the instances on a failed compute host." | |
| 17:14:10 | sean-k-mooney | noonedeadpunk: no it is evacuate | |
| 17:14:26 | sean-k-mooney | noonedeadpunk: but evacuate does not mean what you think it does | |
| 17:14:57 | noonedeadpunk | yeah | |
| 17:15:20 | noonedeadpunk | so when masakari calls instance evacuate when this instance is not volume based - it get's wiped out | |
| 17:15:21 | sean-k-mooney | evacuate only preseved data if you use bfv or are on shared storage | |
| 17:15:51 | sean-k-mooney | noonedeadpunk: unless nova is using ceph via the rbd imageages type or /var/lib/nova is on nfs | |
| 17:15:53 | noonedeadpunk | and I guess even with rbd drive rebuild is destructove for non-bfv? | |
| 17:16:19 | noonedeadpunk | hm... | |
| 17:16:19 | sean-k-mooney | noonedeadpunk: no its not with ceph we detech its on shared storage | |
| 17:16:46 | noonedeadpunk | hm..... | |
| 17:17:23 | sean-k-mooney | dansmith: its proably in your blog post | |
| 17:17:29 | noonedeadpunk | then I still don't get why I got situations when ephemeral got lost or wiped out during node crush.... | |
| 17:17:38 | noonedeadpunk | but whatever) | |
| 17:17:45 | noonedeadpunk | I know how to disable this :p | |
| 17:18:06 | sean-k-mooney | dansmith: i keep it bookmarked as a reference document | |
| 17:18:20 | dansmith | yeah, I should charge admission | |
| 17:19:16 | sean-k-mooney | noonedeadpunk: ootnote: In the case of volume-backed instances, the root disk of the instance is usually in a common location such as on a SAN device. In this case, the root disk is not destroyed, but any other instance state is recreated (which includes memory, ephemeral disk, swap disk, etc). | |
| 17:19:47 | noonedeadpunk | yeah, so it's re-created from the image? | |
| 17:19:52 | noonedeadpunk | as rebuild do | |
| 17:20:35 | noonedeadpunk | which is destructive and all changes made on vm are gone which is eventually as intended? | |
| 17:20:36 | sean-k-mooney | the root disk shoudl not be but if you have extra ephermeral disk they proably are i dont know how addtional ephemeral disks work with ceph. i now swap is store as a ceph volume | |
| 17:20:43 | yoctozepto | sean-k-mooney, noonedeadpunk: I guess masakari should include a warning in big, red font about the necessity to have HA storage first before running masakari | |
| 17:21:00 | noonedeadpunk | I guess so... | |
| 17:21:05 | yoctozepto | it's obvious to us (well, me at least) but years of practice has proven it's not entirely that obvious to all of users | |
| 17:21:10 | sean-k-mooney | yoctozepto: if masikari automate evacuate yes | |
| 17:21:20 | sean-k-mooney | or keep all your data on a cinder data volume | |
| 17:21:29 | yoctozepto | yeah, that's what I meant | |
| 17:21:29 | noonedeadpunk | I mean I have ceph for everything.... But still non bfv instances get's wiped out during evacuations... | |
| 17:21:59 | noonedeadpunk | ok, whatever) | |