| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-03-20 | |||
| 17:59:27 | sean-k-mooney | eandersson: disk_available_lest will sometime be the same but it can be wildly different depending on your storage backend | |
| 18:00:00 | eandersson | so this is probably due to the rocky upgrade | |
| 18:00:45 | eandersson | we don't use a specified storage backend | |
| 18:00:49 | eandersson | so what ever is default | |
| 18:01:10 | eandersson | brb | |
| 18:01:45 | sean-k-mooney | eandersson: google could me this https://gist.github.com/JCallicoat/43505cab0535057ca4fb | |
| 18:02:14 | sean-k-mooney | i wonder if we have that comment in our code | |
| 18:02:33 | NobodyCam | so dumb question, when creating new project / user it's just the _member_ role that is required to launch instances? | |
| 18:02:55 | sean-k-mooney | NobodyCam: by defualt yes | |
| 18:03:11 | NobodyCam | :) | |
| 18:03:21 | sean-k-mooney | although you have to also have a default quota of instnaces greater then 0 | |
| 18:07:32 | sean-k-mooney | eandersson: what i ment by storage spefic backedn is it gets a little broken when using shared storage e.g. nfs or rbd | |
| 18:08:55 | sean-k-mooney | eandersson: if free_gb is positive and disk_available_lest is negitive it means your worst case over commited for the instance that are shduled to the host exceed that allowed by your over commit raitio | |
| 18:10:12 | klindgren | Question re: live migration + configdrive + iso9960. Instead of scping the file between HV's. Can why can't we just recreate the config drive on the destination host? | |
| 18:10:48 | sean-k-mooney | klindgren: i belive there was a libvirt bug that prevented | |
| 18:10:52 | sean-k-mooney | klindgren: but we could | |
| 18:10:54 | klindgren | Are files that re passed in via the a boot command, not started as part of the vm data? So the config drive would be incomplete? | |
| 18:11:53 | sean-k-mooney | klindgren: i think that depends. i think we store user-data in the db in the instance extra table | |
| 18:12:13 | klindgren | Yea, we changed the code (we have a new enough libvirt) on the scp to look for iso9961 vs iso9960. And the migration works, it just creates a sparsely populated file with no data in it on the dst. | |
| 18:12:20 | sean-k-mooney | klindgren: what files are you spscically refering to | |
| 18:13:11 | klindgren | Nova boot --file /seomdest=blah --file /anotherdest=asdf | |
| 18:13:27 | klindgren | Which gets put in to config drive under content/000, content/0001 | |
| 18:14:17 | sean-k-mooney | klindgren: ya so openstack server create --user-data < my data> is stored in the db i dont think we store teh contets of files passed with --file | |
| 18:14:18 | NobodyCam | another dumb question: is there a trick to allowing the admin to launch service instances: I'm seeing Policy check for os_compute_api:os-hide-server-addresses failed with credentials <blah> when attempting to launch Octavia instances | |
| 18:20:31 | sean-k-mooney | NobodyCam: am the amdin account i think need but admin and _member_ roles | |
| 18:20:43 | sean-k-mooney | but the admin account still had a quota | |
| 18:21:14 | sean-k-mooney | in the case of octavir the octavia service account allso has a quota typeiclay it will be set to -1 to have no limit however | |
| 18:30:47 | openstackgerrit | Merged openstack/nova master: docs: Misc cleanups https://review.openstack.org/644616 | |
| 18:32:21 | mriedem | klindgren: correct personality files (file injection) are not stored in the db so if they are only in the config drive and you rebuild the config drive on the dest, those will be gone | |
| 18:32:37 | mriedem | same issue for shelve/unshelve with a config drive + personality files | |
| 18:33:15 | sean-k-mooney | mriedem: personality files are also deprecated for removal for 2 release or more right? | |
| 18:33:32 | mriedem | correct https://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/deprecate-file-injection.html | |
| 18:33:37 | mriedem | but it's an API so they aren't going to be removed | |
| 18:33:41 | mriedem | just strongly discouraged | |
| 18:34:08 | sean-k-mooney | mriedem: not even with a microverion? | |
| 18:34:17 | mriedem | although, a deployment can simply disable file injection (and that's the default for libvirt) today | |
| 18:34:28 | mriedem | sean-k-mooney: yes we deprecated them by removing them from the API | |
| 18:34:33 | mriedem | but if you use 2.1 then that doesn't matter | |
| 18:35:03 | sean-k-mooney | ah that makes sense. that is what i had assed we would do | |
| 18:35:17 | mriedem | when you assed you make an ass of u and me | |
| 18:35:46 | sean-k-mooney | typos all the typos :) | |
| 18:35:50 | mriedem | this is the libvirt thing https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.inject_partition | |
| 18:35:51 | _pewp_ | [ OpenStack Docs: Configuration Options ] - docs.openstack.org | |
| 18:35:55 | mriedem | disabled by default forever | |
| 18:36:26 | sean-k-mooney | mriedem: i think we reinable it in some of the tempest jobs | |
| 18:36:40 | sean-k-mooney | but i dont rememebr which one | |
| 18:36:44 | sean-k-mooney | anyway its not important | |
| 18:38:20 | mriedem | yes it's the ENABLE_FILE_INJECTION var in devstack | |
| 18:38:25 | mriedem | it's in the base tempest job to test personality files | |
| 18:38:46 | gmann | it is enable in tempest-full | |
| 18:38:58 | sean-k-mooney | mriedem: so related to klindgren's question if we have disabled file injection is there any reason we cant just recreate the config dirve in that case | |
| 18:39:35 | mriedem | idk, not my area of expertise | |
| 18:39:38 | mriedem | mikal could maybe help | |
| 18:39:41 | sean-k-mooney | mriedem: i mean they are not big so im not sure that is really a big issue anyway | |
| 18:39:57 | mriedem | klindgren: what is the problem with transferring the config drive during live migration? | |
| 18:40:10 | mriedem | it becomes read-only or what? | |
| 18:40:31 | klindgren | Allowing ssh between all compute nodes with an ssh key | |
| 18:40:53 | klindgren | Gives an easy vector for an attacker to spread across compute nodes | |
| 18:40:53 | sean-k-mooney | klindgren: i dont think that is required in all cases | |
| 18:41:22 | klindgren | Its required in the configdrive + iso9960 use case | |
| 18:41:30 | sean-k-mooney | if you are using the ssh migration uri for libvrt then it would be | |
| 18:41:59 | sean-k-mooney | but im not sure if you can tunnel the file data over the libvirt connection when using qemus/libvirst native tls feature | |
| 18:42:00 | klindgren | we are using TLS for libvirt but this code does scp's of the config drive | |
| 18:42:07 | sean-k-mooney | kashyap: ^ any idea? | |
| 18:42:13 | klindgren | Because even libvirt 4.x can't migrate it/won't | |
| 18:42:24 | sean-k-mooney | klindgren: oh ok | |
| 18:42:55 | sean-k-mooney | klindgren: any you need config drive because ye cant use metadata service or just dont want too? | |
| 18:43:37 | klindgren | We only use configdrive vs's running metadata, and we have a bunch of stuff from other teams built onto of configdrive | |
| 18:44:08 | sean-k-mooney | oh wait ye are still support file injection too wich needs config drive | |
| 18:44:56 | klindgren | We don't specifically disable file injection, however in our case - no one uses it. | |
| 18:48:07 | klindgren | Basically In: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L8196-L8215 | |
| 18:48:08 | _pewp_ | [ nova/driver.py at master · openstack/nova · GitHub ] - github.com | |
| 18:48:41 | sean-k-mooney | klindgren: ok. at first glance it sound like we may be able to just recreate the config drive in your sepcific case when file inject is diabled. | |
| 18:49:00 | klindgren | If we change iso9660 to iso9661, the live migration works. The confgdrive file is created on the destination hv. However, its created as a correctly size, spare file with no data in it | |
| 18:50:22 | klindgren | sean-k-mooney, We would be willing to do the code for this. Just wanted to make sure that their was some reason why we never did this before and wanted to make sure that we would have some level of support for this. | |
| 18:50:54 | klindgren | Assuming the file injection stuff was probably the blocker before hand. | |
| 18:51:43 | mriedem | klindgren: dansmith and/or jaypipes might remember other reasons | |
| 18:52:25 | dansmith | the configdrive being readonly makes it not migrated by libvirt, IIRC | |
| 18:52:41 | mriedem | figuring out if file injection is disabled on both the source and dest and telling hosts on both sides during live migration, hey, file injection is disabled so if there is a config drive on the instance, just rebuild it on the dest, would probably be pretty straight-forward | |
| 18:52:43 | sean-k-mooney | dansmith: i think tha thas been fixed | |
| 18:53:03 | openstackgerrit | Vladyslav Drok proposed openstack/nova master: libvirt: Do not reraise DiskNotFound exceptions during resize https://review.openstack.org/571410 | |
| 18:53:03 | _pewp_ | [ Gerrit Code Review ] - review.openstack.org | |
| 18:53:32 | sean-k-mooney | mriedem: ya that is what i was thinking too but i dont know if im missing an edge case | |
| 18:53:51 | sean-k-mooney | aslo is _pewp_ a new bot? | |
| 18:54:02 | dansmith | sean-k-mooney: I thought it was actually categorically rejected, but I'm sure I'm wrong | |
| 18:54:20 | mriedem | idk wtf _pewp_ is but it looks like it's hemna_'s | |
| 18:55:44 | sean-k-mooney | dansmith: i was jsut taking this config comment on face value https://github.com/openstack/nova/blob/master/nova/conf/configdrive.py#L17-L33 | |
| 18:55:45 | _pewp_ | [ nova/configdrive.py at master · openstack/nova · GitHub ] - github.com | |
| 18:56:36 | sean-k-mooney | i dont know wat "resolved in libvirt v1.2.17" actully means | |
| 18:56:47 | sean-k-mooney | they coudl have document it as wont fix i guess | |
| 18:57:59 | dansmith | well, that was one reason to have it as vfat, IIRC, | |
| 18:58:30 | dansmith | but I thought we actually still need that for other reasons, like being able to host instances with no iso9660 support, and maybe one of the sub-drivers like virtuozzo that needs a regular filesystem or something | |
| 18:58:32 | sean-k-mooney | ya vfat was a woraround for that issue | |
| 18:58:55 | mriedem | sean-k-mooney: stephenfin deprecated the config_drive_format option so you could follow up with him | |
| 18:59:42 | sean-k-mooney | mriedem: ya i was going to check the release notes for libvirt 1.2.17 | |
| 18:59:55 | mriedem | the commit message referenced https://bugzilla.redhat.com/show_bug.cgi?id=1203032 | |
| 18:59:55 | openstack | bugzilla.redhat.com bug 1203032 in libvirt "Enhancing block/disk migration in libvirt" [Low,Closed: nextrelease] - Assigned to pboldin | |
| 18:59:56 | _pewp_ | [ 1203032 – Enhancing block/disk migration in libvirt ] - bugzilla.redhat.com | |
| 19:00:22 | eandersson | sean-k-mooney, I see | |
| 19:00:42 | eandersson | I think this changed somehow during the upgrade which is why it's weird | |