| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-11-17 | |||
| 10:30:00 | lyarwood | if we support shelved it would be that's more awkward yeah | |
| 10:30:40 | sean-k-mooney[m] | well the other issue is | |
| 10:31:17 | sean-k-mooney[m] | as a normal user you cant tell if file backed memory is used | |
| 10:31:37 | sean-k-mooney[m] | so you dont know if it will work | |
| 10:32:23 | lyarwood | Yeah it's awkward for end users, admins would need file backed host aggregates for this to work I guess | |
| 10:32:37 | lyarwood | but without the image property stuff this is the best we can do in the short term tbh | |
| 10:32:56 | lyarwood | so it's either deliver something this cycle or back it up behind a pile of other work | |
| 10:33:00 | sean-k-mooney[m] | yes but the vm when it was booted did not request “must be able to attached shares” in any way | |
| 10:33:38 | sean-k-mooney[m] | well lets just say use file backed memory or hugepages | |
| 10:34:00 | sean-k-mooney[m] | and i can look at creating the new imge/flavor extra spec in a sperate spec | |
| 10:34:20 | sean-k-mooney[m] | i think it would be a good addtion outside of this feature | |
| 10:35:08 | sean-k-mooney[m] | lyarwood: hugepages also solve the requirement for vhost-user and are user requestable today via flavor or image | |
| 10:36:31 | sean-k-mooney[m] | kashyap by the way do you recall i mention that file backed memory seamed to be not allocating memory form the file | |
| 10:36:48 | kashyap | sean-k-mooney[m]: Very vaguely :) | |
| 10:37:01 | kashyap | sean-k-mooney[m]: Can you refresh my memory, please? Is there a ticket/bug for this? | |
| 10:37:04 | sean-k-mooney[m] | kashyap i have been wondering the last day or two could that be related to tb-cache or something similar | |
| 10:37:21 | sean-k-mooney[m] | no i just deployed it at home to test it | |
| 10:37:40 | kashyap | Interesting. Can you share your guest XML + QEMU command-line to see if I can reproduce it | |
| 10:37:43 | sean-k-mooney[m] | then booted vms and could not over subscibe my ram with OOM | |
| 10:38:05 | sean-k-mooney[m] | well ill have to repoduce it my self in a test envionent | |
| 10:38:23 | sean-k-mooney[m] | but if i do i can share it with you | |
| 10:38:50 | sean-k-mooney[m] | lyarwood: are you setting up a deployment with file backed memory for your manila dev? | |
| 10:39:02 | lyarwood | yeah I plan to | |
| 10:39:18 | sean-k-mooney[m] | ok can you try to reverify the behavior | |
| 10:39:49 | lyarwood | yeah sure | |
| 10:40:01 | kashyap | lyarwood: Hope you're hale and hearty now | |
| 10:40:14 | kashyap | sean-k-mooney[m]: Yeah, that behaviour does sound like tb-cache thing | |
| 10:40:30 | gibi | can I get a second core on this bugfix (bauzas and sean-k-mooney[m] are already positive on it) https://review.opendev.org/c/openstack/nova/+/813419 ? | |
| 10:40:30 | lyarwood | kashyap: yup back to normal now thanks | |
| 10:40:50 | lyarwood | gibi: queued | |
| 10:41:00 | gibi | lyarwood: thanks! I'm glad you are back! | |
| 10:41:54 | sean-k-mooney[m] | basiclly i just tried to boot 6 8G vms on a host with 48G of ram and the 6th one triggered OOM | |
| 10:46:10 | sean-k-mooney[m] | gibi: my +1 dissapeared at some point but its back on it. the config help text is now better then some of our dedicated docs :) | |
| 10:46:44 | gibi | sean-k-mooney[m]: thanks. bauzas pushed me to have proper config doc and even config value validation | |
| 10:47:25 | sean-k-mooney[m] | i proably would have skipped the validation because its easy to miss updating that if we add a new type | |
| 10:47:44 | sean-k-mooney[m] | but we will proably rememeber | |
| 10:48:20 | sean-k-mooney[m] | on the other hand i am seeing a lot of issue related to edgecases with the network vif plugged events | |
| 10:48:56 | sean-k-mooney[m] | which makes me think we need a systematic soluntion to this problem sooner rather then later | |
| 10:49:46 | sean-k-mooney[m] | i might see if i can revie the work to pass the driver form neutron to nova this cycle | |
| 10:50:49 | sean-k-mooney[m] | but without neutron telling us when the event is sent i feel like we will continue to have wack a mole issues | |
| 11:11:18 | gibi | lyarwood: sean-k-mooney[m]: feel free to add me as a reviewer | |
| 12:28:00 | lyarwood | sean-k-mooney: https://review.opendev.org/c/openstack/nova/+/811716 - can you also hit this again today please? | |
| 13:06:21 | gibi | lyarwood: I getting pretty confident that the kernel panic on stable/victoria in nova-live-migration job happens because we are live migrating a guest that is not booted fully up yet. When I added 30 sec sleep before the live migration then the problem dissapeared (5/5 run green) | |
| 13:07:05 | gibi | lyarwood: from the console log I see that that without the sleep tempest would trigger live migration even 10 seconds before the guest fully booted | |
| 13:07:12 | gibi | you can see the run results here https://review.opendev.org/c/openstack/nova/+/817564 | |
| 13:07:18 | lyarwood | gibi: kk I was sure I tested my PINGABLE/SSHABLE series against it last week and it still failed | |
| 13:07:26 | gibi | so I think your idea to wait for pingable is a good direction | |
| 13:07:44 | gibi | hm, interesting | |
| 13:07:46 | lyarwood | let me look again | |
| 13:08:10 | lyarwood | https://review.opendev.org/c/openstack/nova/+/817636 | |
| 13:08:34 | lyarwood | I want to say that was against https://review.opendev.org/c/openstack/tempest/+/817635/2 | |
| 13:08:47 | lyarwood | I'm just cleaning the series up again now and can retest | |
| 13:10:36 | gibi | lyarwood: I don't see kernel panic in the runs of https://review.opendev.org/c/openstack/nova/+/817636, but there are other errors. Let's re-test it and see where we are | |
| 13:30:13 | opendevreview | Lee Yarwood proposed openstack/nova stable/victoria: DNM - Testing volume detach failures https://review.opendev.org/c/openstack/nova/+/817636 | |
| 13:42:28 | sean-k-mooney | lyarwood: yes will do | |
| 13:43:06 | lyarwood | thanks | |
| 13:48:04 | sean-k-mooney | ya ok im +1 on that ill get to your spec ater the call im on is over | |
| 14:20:30 | opendevreview | Artom Lifshitz proposed openstack/nova master: DNM: Test token expiration during live migration https://review.opendev.org/c/openstack/nova/+/817778 | |
| 14:28:45 | opendevreview | Artom Lifshitz proposed openstack/nova master: DNM: Test token expiration during live migration https://review.opendev.org/c/openstack/nova/+/817778 | |
| 14:52:19 | bauzas | folks, in case you don't know, the OpenInfra keynote is starting in 8 mins | |
| 15:52:38 | gmann | gibi: thanks | |
| 15:52:54 | gibi | gmann: sorry I had no time to go back and properly review that today | |
| 15:52:55 | gibi | :/ | |
| 15:53:13 | gmann | no worry. | |
| 16:13:35 | whoami-rajat | lyarwood, around? | |
| 16:13:50 | lyarwood | whoami-rajat: hey yeah | |
| 16:13:53 | lyarwood | on a call but can chat | |
| 16:13:58 | whoami-rajat | hey | |
| 16:14:00 | whoami-rajat | ack | |
| 16:14:44 | whoami-rajat | so i don't have any issue with your suggestion of keeping the tried and tested way of nova doing the attachment update, but the team agreed on other flow so don't want to go back and forth | |
| 16:15:55 | lyarwood | yeah appreciate that, I wasn't at PTG so wasn't part of the discussions | |
| 16:16:19 | lyarwood | but as someone maintaining this area more than most I'm still against passing the connector around like this | |
| 16:16:34 | lyarwood | it also keeps the cinder implementation straight forward etc so it's a win win in my view | |
| 16:17:21 | lyarwood | if other nova-specs-cores are against this then they can speak out in the spec | |
| 16:18:58 | whoami-rajat | ack, makes sense to me as the code becomes easier to maintain and debug that way, not sure how much optimization that one less API call does | |
| 16:19:45 | whoami-rajat | i tried discussing the same with other cores in yesterday's nova meeting but we went out of time | |
| 16:20:10 | whoami-rajat | so i will update the spec and see if people are against it and we require further discussion on it | |
| 16:22:05 | whoami-rajat | thanks lyarwood for your inputs | |
| 16:22:28 | lyarwood | Awesome thanks and yeah agree, it's avoiding a single c-api call from n-cpu but c-vol will still have the do the same work so it's a tiny optimisation | |
| 16:44:46 | opendevreview | Artom Lifshitz proposed openstack/nova master: DNM: Test token expiration during live migration https://review.opendev.org/c/openstack/nova/+/817778 | |
| 17:39:20 | opendevreview | Artom Lifshitz proposed openstack/nova master: DNM: Test token expiration during live migration https://review.opendev.org/c/openstack/nova/+/817778 | |
| 17:40:07 | lyarwood | gibi: https://42950ae1f17575ae9a7c-fe6c968e98fdbd85f0f135fdbc9bd3ed.ssl.cf2.rackcdn.com/817636/2/check/nova-live-migration/66b064b/testr_results.html - so that worked pretty well aside from the SG group DELETE blowing up but I can fix that up | |
| 17:40:45 | lyarwood | gibi: had to wait 32 seconds for sshd to start in the instance | |
| 17:41:08 | gibi | that seems align with the logs I collected. The kernel needed more than 10 seconds to boot up | |
| 17:41:26 | gibi | unfortunately there is no timestamps in the cloud init part | |
| 17:41:51 | gibi | but dhcp definetly needs seconds to finish | |
| 17:50:22 | sean-k-mooney | lyarwood: that being waiting for ssh/ping to work? or soemthing else | |
| 17:50:34 | lyarwood | yeah ssh | |
| 17:51:43 | sean-k-mooney | cores can restore patches that are abandoned that belong to others right | |
| 17:52:28 | sean-k-mooney | i messaged the autor via gerrit but if they dont respond in a few days i might ask ye to unabandong a chagne | |
| 18:01:41 | melwitt | sean-k-mooney: yeah cores can restore patch. which patch is it? I can do it | |
| 18:02:06 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/741529 | |
| 18:02:44 | melwitt | done | |
| 18:02:52 | sean-k-mooney | thanks | |
| 18:03:25 | sean-k-mooney | since it was there idea i want to give creit where credit is due and reuse there patch rather then start a new one | |
| 18:03:51 | melwitt | makes sense ++ | |
| 19:36:08 | opendevreview | Lee Yarwood proposed openstack/nova-specs master: Repropose Add libvirt support for flavor and image defined ephemeral encryption https://review.opendev.org/c/openstack/nova-specs/+/810868 | |
| #openstack-nova - 2021-11-18 | |||
| 07:00:09 | opendevreview | Yi Yang proposed openstack/nova master: Support legacy tap interface for OVS DPDK https://review.opendev.org/c/openstack/nova/+/818337 | |
| 09:53:09 | opendevreview | Lee Yarwood proposed openstack/nova master: docs: Update libvirt distro support matrix for Xena https://review.opendev.org/c/openstack/nova/+/818357 | |
| 10:05:31 | opendevreview | Tobias Urdin proposed openstack/nova master: libvirt: Add announce-self post live-migration workaround https://review.opendev.org/c/openstack/nova/+/741529 | |