| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-03-09 | |||
| 22:07:14 | sean-k-mooney | basically if i dont need db change i try to jsut checkout the patch i need and restart it | |
| 22:07:22 | sean-k-mooney | if i get error i restack | |
| 22:07:39 | sean-k-mooney | it works more times then it proably should | |
| 22:09:42 | dansmith | it *should* work for most things, so .. I'd be concerned if it didn't ;) | |
| 22:10:25 | sean-k-mooney | i sometime get bitten by the compute service verion if i change to a different series that i want to test | |
| 22:10:41 | sean-k-mooney | i know i can fix that but i never do | |
| 22:10:47 | sean-k-mooney | i just restack | |
| #openstack-nova - 2021-03-10 | |||
| 02:33:39 | openstackgerrit | Jinsheng Zhang proposed openstack/nova-specs master: Add nova support ironic instance port group network metadata spec https://review.opendev.org/c/openstack/nova-specs/+/779644 | |
| 07:54:45 | openstackgerrit | Merged openstack/nova stable/ussuri: Default user_id when not specified in check_num_instances_quota https://review.opendev.org/c/openstack/nova/+/777217 | |
| 07:55:03 | openstackgerrit | Merged openstack/nova stable/ussuri: Add regression test for bug 1914777 https://review.opendev.org/c/openstack/nova/+/777218 | |
| 07:55:05 | openstack | bug 1914777 in OpenStack Compute (nova) victoria "Possible race condition between n-cpu and n-api when deleting a building instance" [High,In progress] https://launchpad.net/bugs/1914777 - Assigned to melanie witt (melwitt) | |
| 08:02:00 | openstackgerrit | Merged openstack/nova stable/ussuri: Handle instance = None in _local_delete_cleanup https://review.opendev.org/c/openstack/nova/+/777219 | |
| 08:09:00 | gibi | good morning | |
| 08:27:14 | yonglihe | good morning | |
| 08:40:44 | openstackgerrit | Yongli He proposed openstack/nova master: Smartnic support - cyborg drive https://review.opendev.org/c/openstack/nova/+/771362 | |
| 08:40:46 | openstackgerrit | Yongli He proposed openstack/nova master: smartnic support - new vnic type https://review.opendev.org/c/openstack/nova/+/771363 | |
| 08:40:48 | openstackgerrit | Yongli He proposed openstack/nova master: smartnic support https://review.opendev.org/c/openstack/nova/+/758944 | |
| 08:47:49 | openstackgerrit | Adit Sarfaty proposed openstack/nova master: Retry on vmware create_vm when it fails https://review.opendev.org/c/openstack/nova/+/764586 | |
| 09:20:56 | bauzas | good morning | |
| 09:25:00 | yonglihe | sean-k-mooney: refer to patch comments, really hope that address those concerns. for first patch commit message changed thanks. | |
| 09:31:13 | gibi | bauzas: jobs are runnig this morning as far as I see | |
| 09:36:56 | bauzas | gibi: I haven't seen jobs running in any sbg 1 to 4 dcs | |
| 09:37:16 | bauzas | but /me gives hugs to OVH folks around | |
| 09:37:45 | bauzas | my backups are impacted, but meh ;) | |
| 10:15:11 | openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Remove dead error handling code https://review.opendev.org/c/openstack/nova/+/779704 | |
| 10:19:49 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP zuul: Replace grenade and nova-grenade-multinode with grenade-multinode https://review.opendev.org/c/openstack/nova/+/778885 | |
| 10:32:40 | stephenfin | kashyap: Can you take a look at https://review.opendev.org/c/openstack/nova/+/779304/ today, please? | |
| 10:36:06 | stephenfin | lyarwood: Seen this before? https://zuul.opendev.org/t/openstack/build/a19ff22927a045df9f8982d7ef678238/log/controller/logs/screen-n-cpu.txt#16367 | |
| 10:36:41 | stephenfin | WARNING: Failed to get udev device handler for device /dev/sda1.\n /dev/sda15: stat failed: No such file or directory\n Path /dev/sda15 no longer valid for device(8,15)\n /dev/sda15: stat failed: No such file or directory\n Path /dev/sda15 no longer valid for device(8,15)\n ... | |
| 10:38:03 | stephenfin | artom: Look what I see in some build logs -> This host appears to have multiple sockets per NUMA node. The `socket` PCI NUMA affinity will not be supported. | |
| 10:38:10 | stephenfin | artom: from https://zuul.opendev.org/t/openstack/build/a19ff22927a045df9f8982d7ef678238/log/controller/logs/screen-n-cpu.txt | |
| 10:42:36 | lyarwood | stephenfin: yeah ./me finds the bug | |
| 10:43:01 | lyarwood | stephenfin: https://bugs.launchpad.net/cinder/+bug/1901783 | |
| 10:43:03 | openstack | Launchpad bug 1901783 in Cinder "volume delete fails because cinder-rootwrap lvs fails with exit code 139" [Undecided,Triaged] | |
| 10:44:09 | stephenfin | oh, fun. This one is happening in nova (via privsep) | |
| 10:44:23 | lyarwood | oh really? | |
| 10:44:32 | lyarwood | sorry I didn't actually click through | |
| 10:44:37 | stephenfin | https://a59f59e90e7507f14599-2cecd17ff9ea5f4d02c12f6d5fc5aedd.ssl.cf1.rackcdn.com/776681/7/check/nova-lvm/a19ff22/controller/logs/screen-n-cpu.txt | |
| 10:44:37 | stephenfin | all good | |
| 10:44:44 | lyarwood | ah right it's the nova-lvm job | |
| 10:44:44 | stephenfin | search for 8405c8d1-a8ed-4aed-8bfd-972ac6ab9c43 | |
| 10:44:51 | stephenfin | (the instance) | |
| 10:44:51 | lyarwood | that makes sense | |
| 10:45:02 | stephenfin | yup, agreed | |
| 10:47:18 | kashyap | stephenfin: Mornin, I have it open; will definitely look today. (Just finishing something more time-constrained.) | |
| 10:47:45 | stephenfin | kashyap: Cool. Just remember feature freeze is tomorrow | |
| 10:47:52 | stephenfin | so this is also pretty time constrained | |
| 10:48:00 | kashyap | stephenfin: Oh, yeah. Darn | |
| 10:49:45 | kashyap | stephenfin: I see that you've read the messy/complicated situation here: https://bugzilla.redhat.com/show_bug.cgi?id=1929357#c5 | |
| 10:49:47 | openstack | bugzilla.redhat.com bug 1929357 in libvirt "UEFI: Provide a way how to configure different combinations of secure boot enabled/disabled and keys enrolled/not enrolled" [Medium,New] - Assigned to phrdina | |
| 10:49:59 | stephenfin | yup | |
| 10:50:23 | kashyap | Sigh, so we have to do custom-parsing still | |
| 10:50:25 | stephenfin | the libvirt auto-configure feature isn't ready for primetime so I've side-stepped it and gone straight to the (QEMU) source | |
| 10:50:54 | stephenfin | Yes, but I think what we've done is definitely better than the previous hardcoded list | |
| 10:51:06 | stephenfin | so it's an improvement, just not as much of an improvement as we'd hoped for | |
| 10:51:15 | kashyap | stephenfin: What do you mean "straight to the source"? You mean using directly the firmware descriptor files shipped by OVMF/EDK2? | |
| 10:51:21 | stephenfin | yes | |
| 10:51:26 | kashyap | stephenfin: Agree, on removing the hard-coded list | |
| 10:52:33 | kashyap | stephenfin: A detail: each distro (depending on the distro) ships them as part of either EDK2 or OVMF package. I did it for Fedora; and filed bugs for Debian and Ubuntu: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=932269 | |
| 10:52:34 | openstack | Debian bug 932269 in ovmf "Ship the firmware "descriptor files" as part of the 'ovmf' package" [Normal,Fixed] | |
| 10:52:54 | stephenfin | I checked. Ubuntu 20.04 has them, as does Debian now | |
| 10:53:04 | kashyap | Excellent; by now they should | |
| 10:53:06 | stephenfin | And I found references to them for SUSE | |
| 10:53:17 | stephenfin | All in the standard locations too, thankfully | |
| 10:53:34 | kashyap | stephenfin: Yes, I've worked w/ SUSE folks too in the past to get it worked in | |
| 10:58:40 | stephenfin | bauzas: Friendly reminder that the UEFI secure boot series is all green now, save for errant gate failures, and ready for your attention | |
| 11:00:36 | artom | stephenfin, is that... bad? | |
| 11:01:08 | stephenfin | It's unexpected, no? | |
| 11:01:33 | artom | Well, from the nodepool VMs, where we would expect everything to be in one node and socket, yeah | |
| 11:01:36 | stephenfin | Though I guess the gate jobs are VMs and therefore not really representative of real hardware | |
| 11:03:01 | artom | stephenfin, well, see here: https://zuul.opendev.org/t/openstack/build/a19ff22927a045df9f8982d7ef678238/log/controller/logs/screen-n-cpu.txt#892-901 | |
| 11:03:25 | artom | It put each CPU in its own socket | |
| 11:03:36 | artom | So my expectation is clearly wrong :) | |
| 11:04:28 | stephenfin | \o/ | |
| 11:30:43 | bauzas | stephenfin: yup, I was looking at other series, but I'll review them again this afternoon | |
| 11:40:37 | yonglihe | sean-k-mooney: I try to work out your concern and that bug is about LM/resize, that's out of scope. refer to comments, there are some idea about that. | |
| 11:43:33 | yonglihe | sean-k-mooney: I did not dig into that bug, just say: It's seems that's data out of sync somehow, change how to store the data won't help? (fix me) | |
| 11:54:36 | alex_xu | sean-k-mooney1: yonglihe I looked those two bugs, hope I understand that correctly https://review.opendev.org/c/openstack/nova/+/771363/11/nova/objects/network_request.py#42 | |
| 12:07:36 | sean-k-mooney1 | artom: nova default to creating 1 socket per vcpu | |
| 12:08:11 | sean-k-mooney1 | so in the gate vms you will have 8 socket and 1 numa node generated implcitly by qemu/libvirt | |
| 12:08:19 | sean-k-mooney1 | even for non numa vms | |
| 12:09:16 | sean-k-mooney1 | stephenfin: by the way passing the function form the driver.py to host.py goes away in a later patch where i actully move the fucntions too host.py | |
| 12:09:29 | stephenfin | sean-k-mooney1: could you do that first? | |
| 12:09:31 | sean-k-mooney1 | we dont actully use the function that take the callback until that patch | |
| 12:09:56 | sean-k-mooney1 | maybe but it will be a bit of a pain to split back out the patches | |
| 12:10:24 | sean-k-mooney1 | i can do that it more a question of time | |
| 12:11:09 | sean-k-mooney1 | artom: would you be ok removing the numa toplogy object form the pci tracker and adding in a numa to socket map instead | |
| 12:12:11 | sean-k-mooney1 | although if we have to account for the many socket to 1 numa node of the gate i guess we need to retink that abit | |
| 12:12:42 | sean-k-mooney1 | in the gate all pci devices will be connected to the first socket | |
| 12:12:50 | sean-k-mooney1 | we only have 1 pcie root | |
| 12:13:05 | sean-k-mooney1 | and there is also only 1 numa node by default | |
| 12:13:53 | sean-k-mooney1 | a hack would be to have that numa to socket dict be int->list[int] and for you to just use the first socket form the list | |
| 12:14:08 | sean-k-mooney1 | we did discuss that at one point | |
| 12:17:32 | sean-k-mooney | alex_xu: its the same bug just upstream and downstream version | |
| 12:17:46 | sean-k-mooney | alex_xu: the bug is not related to claiming the pci devics | |
| 12:18:14 | sean-k-mooney | alex_xu: we correctly claim the pci device the issue is we dont update the neutron port profile correctrly as part of unshelve | |
| 12:18:34 | sean-k-mooney | alex_xu: and we generate the xml with the pci address stored in the profile | |
| 12:18:48 | sean-k-mooney | so we use the pci address form its previous host | |
| 12:19:00 | sean-k-mooney | so the vm we are unsleving is the one using the incorrect device | |
| 12:19:31 | alex_xu | sean-k-mooney: yes, so for cyborg, we should update the port with new arq uuid | |