| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-08-26 | |||
| 16:42:35 | sean-k-mooney | noonedeadpunk: these are teh rules libvirt uses by the way | |
| 16:42:38 | sean-k-mooney | https://github.com/libvirt/libvirt/blob/7ba4838a18afb609d900190c57b0cbfb842ccbdf/src/util/virutil.c#L470-L489 | |
| 16:45:37 | sean-k-mooney | the did modify this 8 months ago https://github.com/libvirt/libvirt/commit/26d9748ff114a060ee751959d108d062f737f5d9 | |
| 16:46:35 | noonedeadpunk | sean-k-mooney: I think that C way of gethostname may differ from python implementation.... | |
| 16:46:52 | sean-k-mooney | they did not in the past but now they are calling a new fucntion | |
| 16:46:52 | noonedeadpunk | And I'm not C guy unfortunatelly... | |
| 16:47:08 | sean-k-mooney | the python way called the c function | |
| 16:47:18 | yoctozepto | sean-k-mooney: I linked to that libvirt code just above :-) | |
| 16:47:21 | sean-k-mooney | libvirt is now using g_get_host_name(); | |
| 16:47:47 | sean-k-mooney | instead of gethostname | |
| 16:49:30 | noonedeadpunk | Yeah, so things may broke for me with some libvirt update... Anyway that would mean time to work on naming convention:) Like I have only 1 such region - the first one I've built... | |
| 16:49:43 | noonedeadpunk | but it created corner case.... | |
| 16:52:05 | sean-k-mooney | 6.0 is the default in ubuntu and centos 8 | |
| 16:52:29 | sean-k-mooney | i know ooo hardcodes the fqdn in /etc/hostname | |
| 16:52:38 | yoctozepto | noonedeadpunk: masakari needs fixing nonetheless ;p | |
| 16:52:46 | sean-k-mooney | and i think it also manually sets the [default]/host value | |
| 16:53:12 | sean-k-mooney | nova depends on teh actull hostname not changing | |
| 16:59:41 | sean-k-mooney | yoctozepto: noonedeadpunk this is where we call libvirt if you are interested https://github.com/openstack/nova/blob/master/nova/virt/libvirt/host.py#L976 | |
| 17:00:06 | sean-k-mooney | and that is used here https://github.com/openstack/nova/blob/a7735d5e379c20c59cfb639f9f4d339bcffca2f9/nova/virt/libvirt/driver.py#L8418 | |
| 17:00:47 | yoctozepto | sean-k-mooney: thanks, I just did not realize it's libvirt telling us its name, but thanks to you it's clear for me now | |
| 17:02:03 | sean-k-mooney | ya genericly its the hypervior e.g vmware or hyperv or libvirt that provides this | |
| 17:02:59 | openstackgerrit | Merged openstack/nova stable/train: libvirt: Provide VIR_MIGRATE_PARAM_PERSIST_XML during live migration https://review.opendev.org/747973 | |
| 17:03:15 | openstackgerrit | Merged openstack/nova master: releasenotes: Detail support for server ops with vTPM https://review.opendev.org/748215 | |
| 17:04:26 | noonedeadpunk | sean-k-mooney: just one more stupid question... trying to find host-evacuate api call in https://docs.openstack.org/api-ref/compute/ but don't see for some reason (only server evacuate) | |
| 17:05:03 | yoctozepto | sean-k-mooney: https://docs.openstack.org/api-ref/compute/?expanded=evacuate-server-evacuate-action-detail#evacuate-server-evacuate-action ? | |
| 17:05:10 | yoctozepto | noonedeadpunk: ^ | |
| 17:05:28 | yoctozepto | it's server (instance/vm) that's getting evacuated | |
| 17:06:17 | noonedeadpunk | and how does https://docs.openstack.org/nova/rocky/admin/evacuate.html#evacuate-all-instances work? It gets list of instances and evacuate one by one ? | |
| 17:09:07 | yoctozepto | noonedeadpunk: that's what I would assume, could use checking the code | |
| 17:09:47 | sean-k-mooney | noonedeadpunk: host-evacuate is not an api action | |
| 17:09:55 | sean-k-mooney | noonedeadpunk: its a client command | |
| 17:09:59 | noonedeadpunk | ok, got it | |
| 17:10:00 | noonedeadpunk | yeah | |
| 17:10:24 | sean-k-mooney | noonedeadpunk: http://www.danplanet.com/blog/2016/03/03/evacuate-in-nova-one-command-to-confuse-us-all/ | |
| 17:11:14 | sean-k-mooney | dansmith's blog that is basically mandatory reading on host-evacuate and how it related to every thing else | |
| 17:11:47 | noonedeadpunk | oh, ok, now I know under what conditions our client loose all of their data from ephemeral drives... | |
| 17:11:48 | sean-k-mooney | yoctozepto: host-evacuate does not call teh evacuate api | |
| 17:11:55 | sean-k-mooney | yoctozepto: it does cold migrate | |
| 17:11:55 | noonedeadpunk | when masakari calls instance evacuate | |
| 17:12:33 | sean-k-mooney | actully no it does evacuate | |
| 17:12:39 | sean-k-mooney | i should read the blog more often | |
| 17:13:21 | noonedeadpunk | "The core of the evacuate process in nova is actually rebuild, which in many cases is a destructive operation" | |
| 17:13:31 | noonedeadpunk | so it's not evacuate and you;re right | |
| 17:13:40 | sean-k-mooney | from the blog | |
| 17:13:42 | sean-k-mooney | "The nova host-evacuate command does not translate directly to a server-side operation, but is more of a client-side macro or “meta operation.” When you call this command, you provide a hypervisor hostname, which the client uses to list and trigger evacuate operations on each instance running on that hypervisor. You would use this command post-failure (just like the single-instance evacuate | |
| 17:13:45 | sean-k-mooney | command) to trigger evacuations of all the instances on a failed compute host." | |
| 17:14:10 | sean-k-mooney | noonedeadpunk: no it is evacuate | |
| 17:14:26 | sean-k-mooney | noonedeadpunk: but evacuate does not mean what you think it does | |
| 17:14:57 | noonedeadpunk | yeah | |
| 17:15:20 | noonedeadpunk | so when masakari calls instance evacuate when this instance is not volume based - it get's wiped out | |
| 17:15:21 | sean-k-mooney | evacuate only preseved data if you use bfv or are on shared storage | |
| 17:15:51 | sean-k-mooney | noonedeadpunk: unless nova is using ceph via the rbd imageages type or /var/lib/nova is on nfs | |
| 17:15:53 | noonedeadpunk | and I guess even with rbd drive rebuild is destructove for non-bfv? | |
| 17:16:19 | sean-k-mooney | noonedeadpunk: no its not with ceph we detech its on shared storage | |
| 17:16:19 | noonedeadpunk | hm... | |
| 17:16:46 | noonedeadpunk | hm..... | |
| 17:17:23 | sean-k-mooney | dansmith: its proably in your blog post | |
| 17:17:29 | noonedeadpunk | then I still don't get why I got situations when ephemeral got lost or wiped out during node crush.... | |
| 17:17:38 | noonedeadpunk | but whatever) | |
| 17:17:45 | noonedeadpunk | I know how to disable this :p | |
| 17:18:06 | sean-k-mooney | dansmith: i keep it bookmarked as a reference document | |
| 17:18:20 | dansmith | yeah, I should charge admission | |
| 17:19:16 | sean-k-mooney | noonedeadpunk: ootnote: In the case of volume-backed instances, the root disk of the instance is usually in a common location such as on a SAN device. In this case, the root disk is not destroyed, but any other instance state is recreated (which includes memory, ephemeral disk, swap disk, etc). | |
| 17:19:47 | noonedeadpunk | yeah, so it's re-created from the image? | |
| 17:19:52 | noonedeadpunk | as rebuild do | |
| 17:20:35 | noonedeadpunk | which is destructive and all changes made on vm are gone which is eventually as intended? | |
| 17:20:36 | sean-k-mooney | the root disk shoudl not be but if you have extra ephermeral disk they proably are i dont know how addtional ephemeral disks work with ceph. i now swap is store as a ceph volume | |
| 17:20:43 | yoctozepto | sean-k-mooney, noonedeadpunk: I guess masakari should include a warning in big, red font about the necessity to have HA storage first before running masakari | |
| 17:21:00 | noonedeadpunk | I guess so... | |
| 17:21:05 | yoctozepto | it's obvious to us (well, me at least) but years of practice has proven it's not entirely that obvious to all of users | |
| 17:21:10 | sean-k-mooney | yoctozepto: if masikari automate evacuate yes | |
| 17:21:20 | sean-k-mooney | or keep all your data on a cinder data volume | |
| 17:21:29 | noonedeadpunk | I mean I have ceph for everything.... But still non bfv instances get's wiped out during evacuations... | |
| 17:21:29 | yoctozepto | yeah, that's what I meant | |
| 17:21:59 | noonedeadpunk | ok, whatever) | |
| 17:22:03 | sean-k-mooney | noonedeadpunk: ill check the ceph code quickly but i tought we dedect ceph and preserved it. | |
| 17:22:23 | noonedeadpunk | maybe it's some recent change? Ie T or U? | |
| 17:23:07 | sean-k-mooney | we pass an on_shared_storage flag internllay which should be set to true for ceph and nfs | |
| 17:24:07 | sean-k-mooney | thats set here https://github.com/openstack/nova/blob/20459e3e88cb8382d450c7fdb042e2016d5560c5/nova/api/openstack/compute/evacuate.py#L97 | |
| 17:24:30 | sean-k-mooney | oh you have to pass it as an option | |
| 17:25:07 | noonedeadpunk | sean-k-mooney: yoctozepto seems like another patch to masakari?:) | |
| 17:25:15 | sean-k-mooney | noonedeadpunk: https://docs.openstack.org/nova/latest/reference/api-microversion-history.html#id12 | |
| 17:25:32 | sean-k-mooney | so before 2.14 you had to pas it | |
| 17:25:38 | sean-k-mooney | after that its automatic | |
| 17:26:19 | noonedeadpunk | 2.12 was soooooo long ago.... | |
| 17:27:32 | sean-k-mooney | is that what massikari is using | |
| 17:27:51 | sean-k-mooney | https://github.com/openstack/nova/blob/20459e3e88cb8382d450c7fdb042e2016d5560c5/nova/compute/manager.py#L3504 we now ask the dirver if the insance is on shared storage | |
| 17:28:00 | noonedeadpunk | hm. I think I'm reading it wrong.. But it returns None for 2.14+ ? https://github.com/openstack/nova/blob/20459e3e88cb8382d450c7fdb042e2016d5560c5/nova/api/openstack/compute/evacuate.py#L45 | |
| 17:28:04 | sean-k-mooney | form 2.14 on | |
| 17:28:25 | sean-k-mooney | noonedeadpunk: yes 2.14 on it returns none and we ask the driver | |
| 17:28:31 | noonedeadpunk | ah | |
| 17:29:10 | sean-k-mooney | which does self.image_backend.backend(). | |
| 17:29:13 | sean-k-mooney | is_shared_block_storage() | |
| 17:29:38 | sean-k-mooney | so unless you have differen image_backend on different hosts that should be correct | |
| 17:30:07 | sean-k-mooney | rbd returns true https://github.com/openstack/nova/blob/20459e3e88cb8382d450c7fdb042e2016d5560c5/nova/virt/libvirt/imagebackend.py#L958 | |
| 17:30:45 | sean-k-mooney | so you should see this log message when you evacuate https://github.com/openstack/nova/blob/20459e3e88cb8382d450c7fdb042e2016d5560c5/nova/compute/manager.py#L3514-L3516 | |
| 17:30:58 | sean-k-mooney | and we should keep the disk content | |
| 17:31:30 | noonedeadpunk | looks like this, yes | |
| 17:31:57 | sean-k-mooney | what api microversion is masikari using | |
| 17:33:22 | noonedeadpunk | 2.53 for master... | |