Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2018-10-29
14:20:43 tobias-urdin and overall there isn't much info about the disabled field on flavors, thought it was deprecated at first but doesn't seem so? openstackclient doesn't even have support for setting it what i can see
14:22:15 mriedem tobias-urdin: there is no REST API support for modifying the disabled field
14:22:40 mriedem flavors, once they are in the API DB, do not use the SoftDeleteMixin in the model so they don't have the deleted/deleted_at columns anymore either
14:23:15 mriedem tobias-urdin: you can delete the flavors and recreate though
14:23:33 mriedem the flavor used to create a server is stored along with the instances record in the db
14:23:44 mriedem so nova knows the info about the flavor used to create the instance even if the original flavor is gone
14:24:54 tobias-urdin mriedem: ah great, then i can probably pull out that info from there if the referenced flavor_id doesn't exist anymore
14:25:28 tobias-urdin is the disable flavor functionality supported and just somewhat implemented? or is it one of those things that will get pulled soon
14:25:36 mriedem starting with microversion 2.47, showing a server will also show it's embedded flavor details
14:26:09 mriedem the flavors.disabled column seems to be an afterthought from rax from several years ago
14:26:20 mriedem was never plumbed into the api to change it, or nova-manage for that matter
14:26:26 BlackDex Hello there
14:26:39 BlackDex I'm having some issues with live-migrate of a large instances
14:26:56 mriedem tobias-urdin: sorrison was having these same types of questions awhile back, and i think i suggested that we could potentially use the flavor update API (PUT) to set/unset the disabled field while changing the flavor
14:26:57 BlackDex it has a large local storage, and takes a long time to transfer
14:27:01 mriedem but i'm not sure how much that would be accepted
14:27:07 BlackDex it looks like it timesout or something
14:27:17 BlackDex can i change this timeout or something?
14:28:02 mriedem BlackDex: https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.live_migration_completion_timeout ?
14:31:00 mriedem BlackDex: you might be interested in https://specs.openstack.org/openstack/nova-specs/specs/stein/approved/live-migration-force-after-timeout.html as well but that's not implemented yet
14:31:56 tobias-urdin mriedem: appreciate the help, i'll look if we can solve our issue taking the flavor data from the instance blob if the flavor doesn't exist
14:32:15 tobias-urdin not related but horizon will show "Flavor not available" as well so need to fix up a patch there as well probably, if >= 2.47 microversion is supported
14:33:34 mriedem when showing the server details in horizon you mean?
14:33:53 mriedem yeah that's true
14:34:32 BlackDex mriedem: Seems to be the value i'm searching for
15:00:48 openstackgerrit Jack Ding proposed openstack/nova-specs master: High Precision Event Timer (HPET) on x86 guests https://review.openstack.org/607989
15:22:34 spatel sean-k-mooney: morning
15:23:01 spatel Look at this http://paste.openstack.org/show/733564/
15:23:18 spatel I found my compute nodes getting reboot itself, look like kernel bug with iommu
15:23:33 spatel My kernel version is 3.10.0-862.14.4.el7.x86_64 #1 SMP
15:35:00 lbragstad jaypipes o/ wondering if you want to go through https://review.openstack.org/#/c/600266/ at some point
15:35:24 lbragstad we (keystone+oslo) want to make sure the interface is sane before merging that approach
15:36:02 lbragstad figured you would have some valuable feedback there regarding the discussions from the PTG
15:36:51 lbragstad cc bnemec ^
15:45:39 gibi mriedem: hi! This weeks nova meeting is an EU timezone one and I'm regularly chair the EU timezone meetings. However 1st of November is a public holiday here so I'm looking for a way to pass this occasion to somebody else. Could you chair the Thursday's meeting?
15:46:35 mriedem gibi: sure
15:46:45 gibi mriedem: thanks
15:46:48 mriedem np
15:47:03 bauzas FWIW, most of EU countries will respect this holiday
15:47:21 bauzas but I guess mriedem won't be surprised to see me off because $holiday
15:49:47 mriedem because .* :)
16:01:23 jaypipes lbragstad: yes, will try to take a look this afternoon. currently stuck working on $work shit in Chef. :(
16:11:12 cdent "When we last saw our hero, he was stuck working in Chef. Today, he's still stuck working in Chef."
16:11:42 jaypipes cdent: tomorrow, he shall be working in Chef.
16:11:59 cdent womp womp
16:16:06 mriedem efried: i guess https://docs.openstack.org/nova/latest/reference/update-provider-tree.html is out of date given reshaper and the allocations kwarg
16:16:12 BlackDex Hello there. Something went wrong with a live-migrate. The instance is running on compute03 now, but openstack/horizon still thinks it is on compute01
16:16:19 BlackDex how can i fix this?
16:16:32 jaypipes BlackDex: please see /topic
16:16:50 BlackDex ah sorry
16:16:56 mriedem BlackDex: likely something failed in post-live migration and compute01 logs should tell you
16:17:08 finucannot gibi: I don't know if you found someone but I can chair it if not, sure
16:17:10 mriedem failed before updating the instance.host/node values to reflect compute03
16:18:50 jaypipes heh
16:21:48 lbragstad jaypipes sounds good - i'll be available this afternoon if you have questions/advice, just ping
16:23:26 gibi stephenfin: mriedem already took it but thanks for the offer
16:25:21 jaypipes lbragstad: will do, thx
16:32:56 openstackgerrit Matt Riedemann proposed openstack/nova master: Provide allocation_ratio/reserved amounts from update_provider_tree() https://review.openstack.org/613126
16:32:57 mriedem bauzas: efried: jaypipes: cdent: i think this is much more sane now ^
16:33:21 cdent mriedem: huzzah, on my list for today
16:36:32 mnaser do we have someone who's a libvirt expert around here?
16:36:43 mnaser i'm fairly certain i found a bug that breaks live migration because of libvirt
16:36:55 mnaser but i need a bit of guidance in nailing down the root cause :\
16:37:20 mnaser or heck, could be a nova bug if it's nova that generates the xml at the destination libvirt
16:37:32 mnaser but afaik that's not how it works
16:37:33 mdbooth_390 mnaser: Shoot. I might know something.
16:37:42 mdbooth_390 mnaser: Or kashyap might.
16:38:02 mnaser mdbooth_390: i have a live migration that is failing with "qemu-kvm: get_pci_config_device: Bad config data: i=0x10 read: 61 device: 1 cmask: ff wmask: c0 w1cmask:0"
16:38:34 mnaser upon doing research, it looks like that issue would come up if you change host_mtu, so i realized that the VMs that fail to migrate don't have host_mtu defined in the source, but for some reason, it's defined in the destination
16:39:00 mnaser so cat /var/log/libvirt/qemu/instance-being-migrated.log shows the new process being created (the one waiting for memory to be streamed in) with host_mtu=1500 defined
16:39:15 mnaser and then the migration fails because there's something something that becomes of a different size that causes that issue
16:39:48 efried mriedem: Re upt doc, yes, agree it should get updated. Guess that was missed when we did the docstrings.
16:40:11 mnaser so ideally: vms shouldn't have host_mtu in the qemu command, so the libvirt xml shouldn't have that... and i'm not sure how we're getting to that state
16:40:35 openstackgerrit Chris Friesen proposed openstack/nova-specs master: Add support for emulated virtual TPM https://review.openstack.org/571111
16:40:55 mdbooth_390 mnaser: Different libvirt versions source and dest?
16:40:59 mnaser mdbooth_390: yes
16:41:27 mnaser mdbooth_390: also, new instances booted in new libvirt contain host_mtu=1500
16:41:41 mnaser which means newly booted instances post-upgrade are live-migrateable, but pre-upgrade are not
16:41:52 openstack bugzilla.redhat.com bug 1449346 in libvirt "Addition of host_mtu=XXXX changes PCI ioport size & addresses for virtio-net device" [Unspecified,Closed: errata] - Assigned to laine
16:41:52 mdbooth_390 mnaser: Any chance it's this: https://bugzilla.redhat.com/show_bug.cgi?id=1449346 ?
16:42:18 mnaser mdbooth_390: i actaully saw this one, the revert is included in my local branch *however* the symptoms of the failure is exactly that (and that's what made me look for host_mtu)
16:42:33 mnaser so both my src and dest already have that patch
16:43:28 mnaser my question is: are xml files generated during a live migration by nova?
16:43:46 mdbooth_390 mnaser: No, they aren't.
16:44:05 mdbooth_390 They're tweaked by Nova, though.
16:44:08 mnaser ok so that eliminates nova possibly sending an mtu when it starts up
16:44:10 mnaser Aaaaah
16:44:24 mdbooth_390 But I don't think we set host_mtu anywhere
16:44:31 mnaser you do
16:44:34 mnaser one sec
16:44:44 mnaser mdbooth_390: https://github.com/openstack/nova/commit/f02b3800051234ecc14f3117d5987b1a8ef75877
16:44:45 cfriesen mnaser: we do generate xml files during live migration
16:45:02 mnaser so this ignores the fact that the source might *not* have host_mtu set
16:45:03 cfriesen the source generates the xml and passes it to libvirt as part of the migration
16:45:04 mdbooth_390 cfriesen: Nah, we do it post.
16:45:21 mdbooth_390 cfriesen: IIRC the xml sent to the dest is the running xml.
16:45:29 cfriesen mdbooth_390: we save the xml post, but we *generate* the xml and pass it to libvirt as part of the live migration call
16:45:36 mdbooth_390 cfriesen: So not generated by nova during the LM.
16:45:43 tomtom001 Hello, I ran into an issue in queens where if I shutdown CPU-A and try to evacuate the evacuation fails, then I delete the VM that I tried to evacuate. When I startup CPU-A nova-cmopute service fails to start because it can't find remote instance (the VM). What data is nova trying to use to determine what instances it needs to look for in it's startup process?
16:46:11 mnaser right, well if you look at the commit above, it does set the host_mtu stuff

Earlier   Later