| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-10-29 | |||
| 13:47:02 | bauzas | so I'll take popcorn and carefully listen | |
| 13:47:13 | bauzas | jaypipes_: nope, Jim | |
| 13:48:32 | jaypipes_ | bauzas: I'll tl;dr it for you: "execs make out like bandits. IBM and RH have such amazing synergistic realities that it will enable the bottom line to expand into unforeseen markets and mesh amazing benches of copesetic technology." | |
| 13:48:49 | bauzas | heh | |
| 13:51:02 | jangutter | I find it an incredibly sad day: AIX-bashing will no longer be possible on the Nova chat :-( | |
| 13:51:28 | jaypipes_ | jangutter: lol | |
| 13:51:43 | edleafe | jangutter: it never went out of style, even among us IBMers | |
| 13:54:43 | bauzas | on the good news, Europe is closer to the US for 2 weeks | |
| 13:54:54 | bauzas | we shifted yesterday | |
| 13:59:43 | fried_rice | n-sch meeting now in #openstack-meeting-alt | |
| 14:16:05 | SteelyDan | jangutter: I bashed AIX more when I was at IBM than ever :) | |
| 14:18:59 | tobias-urdin | struggling how we should be able to update flavor specs, the ideal way would be to use teh `disabled` field on flavors and just create new ones since that keeps all rows in db and relationships still work, but there is a unique constraint on flavor names so cant use the same names | |
| 14:20:09 | tobias-urdin | like updating disk and the flavor name is small.2x4 i cannot change the name, would have been good if the constraint was on the name, disabled, deleted combination (what i can see deleted field is deprecated though) | |
| 14:20:43 | tobias-urdin | and overall there isn't much info about the disabled field on flavors, thought it was deprecated at first but doesn't seem so? openstackclient doesn't even have support for setting it what i can see | |
| 14:22:15 | mriedem | tobias-urdin: there is no REST API support for modifying the disabled field | |
| 14:22:40 | mriedem | flavors, once they are in the API DB, do not use the SoftDeleteMixin in the model so they don't have the deleted/deleted_at columns anymore either | |
| 14:23:15 | mriedem | tobias-urdin: you can delete the flavors and recreate though | |
| 14:23:33 | mriedem | the flavor used to create a server is stored along with the instances record in the db | |
| 14:23:44 | mriedem | so nova knows the info about the flavor used to create the instance even if the original flavor is gone | |
| 14:24:54 | tobias-urdin | mriedem: ah great, then i can probably pull out that info from there if the referenced flavor_id doesn't exist anymore | |
| 14:25:28 | tobias-urdin | is the disable flavor functionality supported and just somewhat implemented? or is it one of those things that will get pulled soon | |
| 14:25:36 | mriedem | starting with microversion 2.47, showing a server will also show it's embedded flavor details | |
| 14:26:09 | mriedem | the flavors.disabled column seems to be an afterthought from rax from several years ago | |
| 14:26:20 | mriedem | was never plumbed into the api to change it, or nova-manage for that matter | |
| 14:26:26 | BlackDex | Hello there | |
| 14:26:39 | BlackDex | I'm having some issues with live-migrate of a large instances | |
| 14:26:56 | mriedem | tobias-urdin: sorrison was having these same types of questions awhile back, and i think i suggested that we could potentially use the flavor update API (PUT) to set/unset the disabled field while changing the flavor | |
| 14:26:57 | BlackDex | it has a large local storage, and takes a long time to transfer | |
| 14:27:01 | mriedem | but i'm not sure how much that would be accepted | |
| 14:27:07 | BlackDex | it looks like it timesout or something | |
| 14:27:17 | BlackDex | can i change this timeout or something? | |
| 14:28:02 | mriedem | BlackDex: https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.live_migration_completion_timeout ? | |
| 14:31:00 | mriedem | BlackDex: you might be interested in https://specs.openstack.org/openstack/nova-specs/specs/stein/approved/live-migration-force-after-timeout.html as well but that's not implemented yet | |
| 14:31:56 | tobias-urdin | mriedem: appreciate the help, i'll look if we can solve our issue taking the flavor data from the instance blob if the flavor doesn't exist | |
| 14:32:15 | tobias-urdin | not related but horizon will show "Flavor not available" as well so need to fix up a patch there as well probably, if >= 2.47 microversion is supported | |
| 14:33:34 | mriedem | when showing the server details in horizon you mean? | |
| 14:33:53 | mriedem | yeah that's true | |
| 14:34:32 | BlackDex | mriedem: Seems to be the value i'm searching for | |
| 15:00:48 | openstackgerrit | Jack Ding proposed openstack/nova-specs master: High Precision Event Timer (HPET) on x86 guests https://review.openstack.org/607989 | |
| 15:22:34 | spatel | sean-k-mooney: morning | |
| 15:23:01 | spatel | Look at this http://paste.openstack.org/show/733564/ | |
| 15:23:18 | spatel | I found my compute nodes getting reboot itself, look like kernel bug with iommu | |
| 15:23:33 | spatel | My kernel version is 3.10.0-862.14.4.el7.x86_64 #1 SMP | |
| 15:35:00 | lbragstad | jaypipes o/ wondering if you want to go through https://review.openstack.org/#/c/600266/ at some point | |
| 15:35:24 | lbragstad | we (keystone+oslo) want to make sure the interface is sane before merging that approach | |
| 15:36:02 | lbragstad | figured you would have some valuable feedback there regarding the discussions from the PTG | |
| 15:36:51 | lbragstad | cc bnemec ^ | |
| 15:45:39 | gibi | mriedem: hi! This weeks nova meeting is an EU timezone one and I'm regularly chair the EU timezone meetings. However 1st of November is a public holiday here so I'm looking for a way to pass this occasion to somebody else. Could you chair the Thursday's meeting? | |
| 15:46:35 | mriedem | gibi: sure | |
| 15:46:45 | gibi | mriedem: thanks | |
| 15:46:48 | mriedem | np | |
| 15:47:03 | bauzas | FWIW, most of EU countries will respect this holiday | |
| 15:47:21 | bauzas | but I guess mriedem won't be surprised to see me off because $holiday | |
| 15:49:47 | mriedem | because .* :) | |
| 16:01:23 | jaypipes | lbragstad: yes, will try to take a look this afternoon. currently stuck working on $work shit in Chef. :( | |
| 16:11:12 | cdent | "When we last saw our hero, he was stuck working in Chef. Today, he's still stuck working in Chef." | |
| 16:11:42 | jaypipes | cdent: tomorrow, he shall be working in Chef. | |
| 16:11:59 | cdent | womp womp | |
| 16:16:06 | mriedem | efried: i guess https://docs.openstack.org/nova/latest/reference/update-provider-tree.html is out of date given reshaper and the allocations kwarg | |
| 16:16:12 | BlackDex | Hello there. Something went wrong with a live-migrate. The instance is running on compute03 now, but openstack/horizon still thinks it is on compute01 | |
| 16:16:19 | BlackDex | how can i fix this? | |
| 16:16:32 | jaypipes | BlackDex: please see /topic | |
| 16:16:50 | BlackDex | ah sorry | |
| 16:16:56 | mriedem | BlackDex: likely something failed in post-live migration and compute01 logs should tell you | |
| 16:17:08 | finucannot | gibi: I don't know if you found someone but I can chair it if not, sure | |
| 16:17:10 | mriedem | failed before updating the instance.host/node values to reflect compute03 | |
| 16:18:50 | jaypipes | heh | |
| 16:21:48 | lbragstad | jaypipes sounds good - i'll be available this afternoon if you have questions/advice, just ping | |
| 16:23:26 | gibi | stephenfin: mriedem already took it but thanks for the offer | |
| 16:25:21 | jaypipes | lbragstad: will do, thx | |
| 16:32:56 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Provide allocation_ratio/reserved amounts from update_provider_tree() https://review.openstack.org/613126 | |
| 16:32:57 | mriedem | bauzas: efried: jaypipes: cdent: i think this is much more sane now ^ | |
| 16:33:21 | cdent | mriedem: huzzah, on my list for today | |
| 16:36:32 | mnaser | do we have someone who's a libvirt expert around here? | |
| 16:36:43 | mnaser | i'm fairly certain i found a bug that breaks live migration because of libvirt | |
| 16:36:55 | mnaser | but i need a bit of guidance in nailing down the root cause :\ | |
| 16:37:20 | mnaser | or heck, could be a nova bug if it's nova that generates the xml at the destination libvirt | |
| 16:37:32 | mnaser | but afaik that's not how it works | |
| 16:37:33 | mdbooth_390 | mnaser: Shoot. I might know something. | |
| 16:37:42 | mdbooth_390 | mnaser: Or kashyap might. | |
| 16:38:02 | mnaser | mdbooth_390: i have a live migration that is failing with "qemu-kvm: get_pci_config_device: Bad config data: i=0x10 read: 61 device: 1 cmask: ff wmask: c0 w1cmask:0" | |
| 16:38:34 | mnaser | upon doing research, it looks like that issue would come up if you change host_mtu, so i realized that the VMs that fail to migrate don't have host_mtu defined in the source, but for some reason, it's defined in the destination | |
| 16:39:00 | mnaser | so cat /var/log/libvirt/qemu/instance-being-migrated.log shows the new process being created (the one waiting for memory to be streamed in) with host_mtu=1500 defined | |
| 16:39:15 | mnaser | and then the migration fails because there's something something that becomes of a different size that causes that issue | |
| 16:39:48 | efried | mriedem: Re upt doc, yes, agree it should get updated. Guess that was missed when we did the docstrings. | |
| 16:40:11 | mnaser | so ideally: vms shouldn't have host_mtu in the qemu command, so the libvirt xml shouldn't have that... and i'm not sure how we're getting to that state | |
| 16:40:35 | openstackgerrit | Chris Friesen proposed openstack/nova-specs master: Add support for emulated virtual TPM https://review.openstack.org/571111 | |
| 16:40:55 | mdbooth_390 | mnaser: Different libvirt versions source and dest? | |
| 16:40:59 | mnaser | mdbooth_390: yes | |
| 16:41:27 | mnaser | mdbooth_390: also, new instances booted in new libvirt contain host_mtu=1500 | |
| 16:41:41 | mnaser | which means newly booted instances post-upgrade are live-migrateable, but pre-upgrade are not | |
| 16:41:52 | mdbooth_390 | mnaser: Any chance it's this: https://bugzilla.redhat.com/show_bug.cgi?id=1449346 ? | |
| 16:41:52 | openstack | bugzilla.redhat.com bug 1449346 in libvirt "Addition of host_mtu=XXXX changes PCI ioport size & addresses for virtio-net device" [Unspecified,Closed: errata] - Assigned to laine | |
| 16:42:18 | mnaser | mdbooth_390: i actaully saw this one, the revert is included in my local branch *however* the symptoms of the failure is exactly that (and that's what made me look for host_mtu) | |
| 16:42:33 | mnaser | so both my src and dest already have that patch | |
| 16:43:28 | mnaser | my question is: are xml files generated during a live migration by nova? | |
| 16:43:46 | mdbooth_390 | mnaser: No, they aren't. | |
| 16:44:05 | mdbooth_390 | They're tweaked by Nova, though. | |
| 16:44:08 | mnaser | ok so that eliminates nova possibly sending an mtu when it starts up | |
| 16:44:10 | mnaser | Aaaaah | |