| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-10-28 | |||
| 09:28:59 | kashyap | Also, have you got the QEMU command-line of the instance log in question? /me goes searching | |
| 09:31:17 | kashyap | Hm, which of the instances is it here - https://api.us-east.open-edge.io:8080/swift/v1/AUTH_e02c11e4e2c24efc98022353c88ab506/zuul_opendev_logs_30d/759831/4/gate/nova-grenade-multinode/30d8eb1/logs/libvirt/qemu/index.html | |
| 09:31:33 | lyarwood | instance-0000001e | |
| 09:31:46 | kashyap | Okay, I looked at the "right one", then | |
| 09:32:05 | kashyap | (Also, the "-incoming defer" confirms it's the dest host) | |
| 09:32:57 | kashyap | lyarwood: It's also in the instance log, as you surely must know - 4d564e22-8ba4-48fb-ac93-27bea660fd77 | |
| 09:34:39 | lyarwood | kashyap: right my point was that the log should use the UUID as the filename, IOW we should use it as the domain name | |
| 09:34:54 | lyarwood | kashyap: but I think there's an upgrade issue with that so we've never tried | |
| 09:34:56 | kashyap | Ah, guessed as much; consistency across the board | |
| 09:35:33 | kashyap | lyarwood: Source host logs are under subnode-2/ dir, right? | |
| 09:35:42 | kashyap | Yep | |
| 09:44:58 | kashyap | lyarwood: So, I collated the useful bits from the logs here, to check w/ the QEMU/libvirt folks - https://kashyapc.fedorapeople.org/LP-1901739/ | |
| 09:47:00 | lyarwood | ack thanks | |
| 09:48:43 | kashyap | lyarwood: So, is this an apt descrption: back-n-forth live migration (w/o shared storage) of a paused instance causes this missing block job data? | |
| 09:50:28 | lyarwood | frickler: any idea how your last recheck of https://review.opendev.org/#/c/759831/ has ended up with it in both the check and gate queues at the same time? | |
| 09:53:31 | frickler | lyarwood: I manually enqueued it into the gate directly | |
| 09:53:38 | lyarwood | frickler: ah my bad | |
| 09:54:10 | frickler | lyarwood: I did the recheck because I didn't see the earlier ones, only the -2 from gate | |
| 10:08:39 | kashyap | lyarwood: BTW, ignore my earlier wondering about the use-case for back-n-forth LM of a paused guest. I just thought of one plausible case: | |
| 10:08:52 | kashyap | You migrate a paused a instance; it succeeds. But then you suddenly realize: "oh, I actually need to do some maint work on this dest, so I need to migrate this VM _back_ to the source again" | |
| 10:18:44 | lyarwood | frickler: ack no issues, I was just confused how it ended up in both queues at the same time | |
| 10:29:02 | gibi | frickler: I have no objection force merging https://review.opendev.org/759831 | |
| 10:40:37 | openstackgerrit | Merged openstack/nova master: libvirt: Only ask tpool.Proxy to autowrap vir* classes https://review.opendev.org/759831 | |
| 10:41:27 | frickler | gibi: ^^ done | |
| 10:46:42 | lyarwood | many thanks | |
| 10:51:15 | gibi | frickler: thank you | |
| 10:52:49 | lyarwood | stephenfin: https://review.opendev.org/#/c/758971/ updated btw | |
| 10:53:15 | lyarwood | gibi: https://review.opendev.org/#/q/topic:bug/1899649 would you mind taking a look at this if you have time ahead of the ptg | |
| 10:56:29 | stephenfin | lyarwood: +2; thanks for that | |
| 11:01:02 | gibi | lyarwood: sure, looking | |
| 11:05:06 | openstackgerrit | Stephen Finucane proposed openstack/nova master: tox: Stop linting as part of docs target https://review.opendev.org/735176 | |
| 11:10:01 | stephenfin | lyarwood: Think you could take a gawk at https://review.opendev.org/#/q/topic:bug/1898272+(status:open+OR+status:merged) too? | |
| 11:10:48 | lyarwood | stephenfin: sure | |
| 11:25:07 | gibi | lyarwood: I left a question in https://review.opendev.org/#/c/757894/2/nova/compute/manager.py@a2233 | |
| 11:27:20 | kashyap | gibi: [Maybe I'll get kicked off this channel for sayin it] Speaking of caffeine, I switched to decaf-only after 10:00 AM (for my afternoon coffee. | |
| 11:28:05 | kashyap | gibi: Because of a certain sleep scientist; check this 2-min msg from him: https://www.youtube.com/watch?v=KGfdR7TSJo4 | |
| 11:28:29 | gibi | kashyap: we are inclusive, no kicks due to decaf consumption :) | |
| 11:28:37 | kashyap | Haha | |
| 11:28:39 | kashyap | gibi: "If you have a cup of coffee at noon, then a quarter of that caffeine is still in your brain at _midnight_." | |
| 11:28:46 | kashyap | But --^ that was startling to me | |
| 11:29:23 | kashyap | (That said, PTG demands more caffeine) | |
| 11:31:38 | gibi | now I'm wondering about 5 o'clock tea too | |
| 11:32:14 | gibi | for me tea has a lower but more elongated effect | |
| 11:32:27 | kashyap | Yeah, I've been decaf in the afternoon for 4 days in a row. And my sleep is steadily improving already | |
| 11:33:31 | kashyap | gibi: BTW, his book (https://www.penguin.co.uk/books/295/295665/why-we-sleep/9780141983769.html) that is referenced earlier is excellent. (I've also suggested it to Eric Fried in the past, he said he thoroughly enojyed it, FWIW) | |
| 11:33:45 | lyarwood | gibi: ack thanks looking | |
| 11:36:25 | gibi | kashyap: thanks | |
| 11:57:36 | openstackgerrit | Stephen Finucane proposed openstack/nova master: db: Clean up base migration https://review.opendev.org/758155 | |
| 11:57:36 | openstackgerrit | Stephen Finucane proposed openstack/nova master: db: Merge in unique constraints https://review.opendev.org/758156 | |
| 11:57:37 | openstackgerrit | Stephen Finucane proposed openstack/nova master: db: Drop "remove" of index https://review.opendev.org/758157 | |
| 11:57:37 | openstackgerrit | Stephen Finucane proposed openstack/nova master: db: Compact Icehouse database migrations https://review.opendev.org/758394 | |
| 11:57:38 | openstackgerrit | Stephen Finucane proposed openstack/nova master: db: Compact Juno database migration https://review.opendev.org/758395 | |
| 11:57:38 | openstackgerrit | Stephen Finucane proposed openstack/nova master: db: Compact Kilo database migrations https://review.opendev.org/758396 | |
| 11:57:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: db: Compact Liberty database migrations https://review.opendev.org/758397 | |
| 11:57:39 | openstackgerrit | Stephen Finucane proposed openstack/nova master: db: Compact Mitaka database migrations https://review.opendev.org/758398 | |
| 11:57:40 | openstackgerrit | Stephen Finucane proposed openstack/nova master: db: Compact Newton database migrations https://review.opendev.org/759085 | |
| 11:57:40 | openstackgerrit | Stephen Finucane proposed openstack/nova master: db: Compact Ocata database migrations https://review.opendev.org/759086 | |
| 11:57:41 | openstackgerrit | Stephen Finucane proposed openstack/nova master: db: Compact Pike database migrations https://review.opendev.org/759087 | |
| 11:57:41 | openstackgerrit | Stephen Finucane proposed openstack/nova master: db: Compact Queens database migrations https://review.opendev.org/759088 | |
| 11:57:42 | openstackgerrit | Stephen Finucane proposed openstack/nova master: db: Compact Rocky database migrations https://review.opendev.org/759089 | |
| 11:57:42 | openstackgerrit | Stephen Finucane proposed openstack/nova master: db: Compact Stein database migrations https://review.opendev.org/759090 | |
| 11:57:43 | openstackgerrit | Stephen Finucane proposed openstack/nova master: db: Compact Train database migrations https://review.opendev.org/759091 | |
| 11:59:11 | stephenfin | gibi: probably want to get this in sooner rather than later https://review.opendev.org/#/c/754396/ | |
| 12:03:05 | gibi | stephenfin: good point | |
| 12:06:24 | kashyap | stephenfin: Hey, for later, you might want to bookmark this talk from SUSE, ongoing at KVM Forum, on vCPU topology, pinning, etc - https://kvmforum2020.sched.com/event/eE1y/virtual-topology-for-virtual-machines-friend-or-foe-dario-faggioli-suse | |
| 12:06:40 | kashyap | (Slides are in link above; so you can already skim the content) | |
| 12:07:36 | gibi | stephenfin: can we skip adding db migration placeholders in the future if no new migration is added during W? now we have a big chunk of placeholders after the last db migration | |
| 12:31:36 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: WIP: add ovo-mypy-plugin to type hinting o.vos https://review.opendev.org/758851 | |
| 12:56:10 | gibi | nova PTG starts in 10 minutes with nova-cyborg corss project session | |
| 12:56:16 | gibi | nova PTG starts in 4 minutes with nova-cyborg corss project session | |
| 12:57:28 | lyarwood | I'm going to be attending the cinder session this afternoon FYI, I don't think I'm of any use in the cross projects today but ping me if something comes up that I can help with | |
| 12:57:36 | gibi | lyarwood: ck | |
| 12:57:40 | gibi | lyarwood: ack | |
| 13:12:22 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: DNM: testing copy image with non admin policy https://review.opendev.org/760128 | |
| 13:46:16 | bauzas | wow, this was violent | |
| 13:46:28 | kashyap | What was violent? | |
| 13:46:28 | bauzas | gibi: my laptop crashed but I'm rejoining | |
| 14:55:52 | lyarwood | zoom-- | |
| 14:56:49 | lyarwood | haha finally get it to work and it's time for a break, typical. | |
| 15:16:20 | melwitt | lyarwood: I saw, the gate is super angry :( | |
| 15:19:43 | lyarwood | melwitt: yup indeed thanks for checking things overnight btw | |
| 15:19:49 | lyarwood | melwitt: it's merged now finally | |
| 15:21:37 | melwitt | lyarwood: yeah, I saw it got force merged if I understood correctly. and yeah, want to get a handle on the gate bugs and figure out some root causes. things have been crazy lately | |
| 15:56:53 | lyarwood | bauzas: mute :) | |
| 15:57:13 | bauzas | shit | |
| 15:57:22 | lyarwood | np | |
| 15:57:38 | bauzas | if anyone knows whether zoom supports space tabs for muting/unmuting automatically, I'm your man | |
| 16:11:39 | bauzas | F32-- | |
| 16:11:46 | bauzas | killed my zoom and a couple of apps | |
| 16:26:00 | gibi | #nova now Deprecate os-hypervisors API | |
| 16:28:58 | openstackgerrit | Merged openstack/nova master: Add placeholder migrations for Victoria backports https://review.opendev.org/754396 | |
| 16:44:10 | bauzas | gibi: wrong channel ^ ;) | |
| 16:44:44 | gibi | :D | |
| 16:44:50 | melwitt | gibi, elod, lyarwood: been meaning to ask for awhile ... would appreciate your opinion about these two open stable-only changes whether you think it's appropriate given the oldness of queens https://review.opendev.org/#/q/topic:bug/1731668 | |
| 16:45:02 | melwitt | tl;dr is there was a bug in some transitionary code that is triggered by lots of parallel requests to placement (example: running Rally). impact (if I read the code correctly) is instances go into ERROR state for failure to create placement allocations | |
| 16:45:17 | gibi | melwitt: I have to get back to these tomorrow morning | |
| 16:45:27 | melwitt | np thanks | |
| 16:50:29 | elod | melwitt: sure, I'll try to understand the whole case and figure out whether it fits to be backported or not. but I'm a bit lagging behind myself, so I won't promise anything right now o:) | |
| 16:50:43 | elod | and thanks for the tl;dr :] | |
| 16:52:02 | melwitt | elod: np. yeah it's just a bug, if you try to create a bunch of instances in parallel in queens, you can hit this bug and cause lots of instances to fall into ERROR state as they fail during the scheduling step | |