| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2021-02-11 | |||
| 10:12:48 | gibi | ? | |
| 10:13:08 | kashyap | gibi: Indeed, that's what I mean. You make a good point there, no questions | |
| 10:13:43 | kashyap | gibi: I mean, we keep the unit tests that I included in there, for the minimal coverage there is | |
| 10:14:39 | kashyap | gibi: Also, please take a 4 minutes to read this context I wrote for today's meeting at the bottom: https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting | |
| 10:15:20 | gibi | kashyap: OK, let's expedite https://review.opendev.org/c/openstack/nova/+/762330/ after your fix lands | |
| 10:15:22 | kashyap | That should give you context on _why_ I'm doing what I'm doing. And why I'm trying to keep it short and small - to be backportable | |
| 10:16:46 | kashyap | gibi: Yep; thank you. | |
| 10:16:54 | gibi | I need to recover your topic on the agenda, it seems stephenfin overwrote it :) | |
| 10:17:14 | kashyap | Oh no, what have you done, stephenfin :D | |
| 10:17:19 | stephenfin | whoops | |
| 10:17:47 | kashyap | No problem :) I can re-add it -- assuming gibi is already not doing it | |
| 10:17:55 | gibi | I will fix it | |
| 10:18:11 | kashyap | I spent half hour writing clear sentences to reduce cognitive load, and stephenfin, you erased it :D | |
| 10:18:28 | gibi | this is partially my fault as I did not clean up the agenda after the last meeting | |
| 10:18:49 | kashyap | No problem; Wiki maint is a chore | |
| 10:21:20 | gibi | kashyap: fixed it, read it. It looks OK to me. If no objection on the meeting then I will approve the bp | |
| 10:22:01 | kashyap | gibi: Understood; we can talk there. | |
| 10:22:06 | gibi | coo | |
| 10:22:07 | gibi | l | |
| 10:22:48 | kashyap | Thanks! /me goes to fix his env. meanwhile to test the damn patch for real | |
| 10:33:27 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: DNM: trace migration tests to see what steps take too long https://review.opendev.org/c/openstack/nova/+/775094 | |
| 10:48:47 | kashyap | Isn't this strange? I have these services enabled for my DevStack run: | |
| 10:48:48 | kashyap | ENABLED_SERVICES=g-api,g-reg,key,n-api,n-cpu,n-sch,n-cond,mysql,rabbit,dstat,quantum,q-svc,q-agt,q-dhcp,q-l3,q-meta | |
| 10:49:12 | kashyap | During stack.sh, I see this fly by: "+./stack.sh:echo_summary:416 echo -e Configuring Horizon" | |
| 10:59:27 | kashyap | gibi (or anyone): Have you ever seen this? After 18 minutes of stack.sh, it fails o me like this: -- http://paste.openstack.org/show/802558/ | |
| 11:05:29 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Record the machine_type of instances in system_metadata https://review.opendev.org/c/openstack/nova/+/767533 | |
| 11:05:30 | openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-manage: Add machine_type update command https://review.opendev.org/c/openstack/nova/+/774896 | |
| 11:05:30 | openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-manage: Add machine_type get command https://review.opendev.org/c/openstack/nova/+/769548 | |
| 11:05:31 | openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-status: Add hw_machine_type check for libvirt instances https://review.opendev.org/c/openstack/nova/+/770643 | |
| 11:05:31 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP nova-manage: Add machine_type list_unset command https://review.opendev.org/c/openstack/nova/+/774897 | |
| 11:05:32 | openstackgerrit | Lee Yarwood proposed openstack/nova master: docs: Add admin docs for configuring and updating machine types https://review.opendev.org/c/openstack/nova/+/774899 | |
| 11:05:32 | openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Add a config update workflow test for [libvirt]hw_machine_type https://review.opendev.org/c/openstack/nova/+/774898 | |
| 11:07:13 | lyarwood | kashyap: Didn't find service registered by hostname after 60 seconds | |
| 11:07:30 | kashyap | lyarwood: Yeah; figured out - actually it's a CPU model prob | |
| 11:07:40 | kashyap | I gave a model that is marked as "no" by libvirt | |
| 11:07:56 | lyarwood | kashyap: so n-cpu didn't start? | |
| 11:07:56 | kashyap | Thanks for looking :) sean-k-mooney gave the hint, while we're on a call | |
| 11:08:03 | lyarwood | k | |
| 11:08:33 | kashyap | lyarwood: Saw the exception in journalctl -u devstack@n-cpu | |
| 11:11:20 | gibi | sorry I was away | |
| 11:11:24 | gibi | but I see you solved it | |
| 11:12:06 | kashyap | Yes; no problem. I never count on instant responses :) | |
| 11:56:36 | gibi | sean-k-mooney: hi! here is a bug report https://bugs.launchpad.net/nova/+bug/1915255 based on the git log https://review.opendev.org/c/openstack/nova/+/739131 seems suspicious causing it. Could you please take a look? | |
| 11:56:37 | openstack | Launchpad bug 1915255 in OpenStack Compute (nova) "[Victoria] nova-compute won't start on aarch64 - raises PciDeviceNotFoundById" [Undecided,New] | |
| 11:57:43 | sean-k-mooney | am yes i can take a look | |
| 12:00:00 | sean-k-mooney | so the function that is failing is not one i added but one im started calling | |
| 12:00:28 | sean-k-mooney | that exception i think shoudl alreay be caught but maybe not in the current code path | |
| 12:00:57 | sean-k-mooney | its failing to look up the parent netdev name for badnwith based schduling | |
| 12:02:01 | sean-k-mooney | we could rewrite that function to call libvirt instead of sysfs possibly | |
| 12:14:36 | gibi | hm, then I will take a look too | |
| 12:15:16 | gibi | lyarwood: hi! Could this be another manifestation of the detach retry bug I'm trying to fix? https://bugs.launchpad.net/nova/+bug/1914664 | |
| 12:15:17 | openstack | Launchpad bug 1914664 in OpenStack Compute (nova) "QEMU monitor read failure in ServerStableDeviceRescueTest" [Undecided,New] | |
| 12:18:01 | sean-k-mooney | gibi: apparently the Cavium ThunderX does not always have a PF assinged to each VF | |
| 12:18:16 | sean-k-mooney | from https://bugs.launchpad.net/charm-nova-compute/+bug/1771662 | |
| 12:18:17 | sean-k-mooney | ibvirt has had the assumption that every VF (virtual function) will | |
| 12:18:17 | openstack | Launchpad bug 1771662 in libvirt (Ubuntu Disco) "[bionic] libvirtError: Node device not found: no node device with matching name" [Undecided,Fix released] | |
| 12:18:19 | sean-k-mooney | have a PF (physical function) assigned, but that does not hold true on | |
| 12:18:21 | sean-k-mooney | some special Hardware like the Cavium ThunderX | |
| 12:18:21 | gibi | sean-k-mooney: how that is possible? | |
| 12:18:40 | sean-k-mooney | i am not sure maybe they mean netdev | |
| 12:19:02 | sean-k-mooney | its possible for the PF to no actully work as a nic and only provide VFs | |
| 12:19:34 | gibi | if there is no parent PF then the whole qos stuff is just breaks down as we modell bw inventory on the PF | |
| 12:19:43 | sean-k-mooney | ya | |
| 12:20:01 | sean-k-mooney | so in any case we need to allow the compute agent to start | |
| 12:20:08 | sean-k-mooney | so we need to deal with the excpeiton | |
| 12:20:26 | sean-k-mooney | but i think this is jus ta case of this is strange hardware | |
| 12:20:34 | gibi | yes, we have to allow the compute to start but then this interface will not be ever selected for a port that has qos request | |
| 12:20:43 | sean-k-mooney | yep | |
| 12:20:53 | gibi | thanks for the triage on this bug | |
| 12:22:05 | sean-k-mooney | so there are two ways we can adress this it looks like the old logic just logged. | |
| 12:22:12 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/739131/12/nova/pci/utils.py | |
| 12:22:28 | sean-k-mooney | so we can do that | |
| 12:22:47 | sean-k-mooney | or we can also maybe stop using sysfs and use libvirt | |
| 12:22:55 | sean-k-mooney | or both in seperate patchs | |
| 12:23:09 | lyarwood | gibi: looking | |
| 12:23:18 | gibi | lyarwood: thanks | |
| 12:23:26 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Uncap PrettyTable https://review.opendev.org/c/openstack/nova/+/775142 | |
| 12:23:39 | gibi | sean-k-mooney: as a fix I would just log now | |
| 12:24:14 | sean-k-mooney | oh actully | |
| 12:24:27 | sean-k-mooney | this code is common its not allowed to talk to libvirt | |
| 12:24:28 | gibi | sean-k-mooney: as far as I understand your patch moved away from libvirt for certain reasons | |
| 12:24:34 | sean-k-mooney | that is why we are using sysfs | |
| 12:25:13 | sean-k-mooney | ya libvirt can get out os sync | |
| 12:25:28 | sean-k-mooney | for nodedev names | |
| 12:25:51 | sean-k-mooney | so ya using libvirt here would not be right | |
| 12:26:22 | openstackgerrit | Stephen Finucane proposed openstack/python-novaclient master: requirements: Remove simplejson https://review.opendev.org/c/openstack/python-novaclient/+/775144 | |
| 12:26:22 | openstackgerrit | Stephen Finucane proposed openstack/python-novaclient master: Uncap PrettyTable https://review.opendev.org/c/openstack/python-novaclient/+/775143 | |
| 12:26:41 | sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/739131/12/nova/virt/libvirt/driver.py#b6960 | |
| 12:26:52 | sean-k-mooney | that might be the issue | |
| 12:28:08 | sean-k-mooney | ya so my note | |
| 12:28:10 | sean-k-mooney | # NOTE(sean-k-mooney): if the VF is a parent of a netdev | |
| 12:28:13 | sean-k-mooney | # the PF should also have a netdev. | |
| 12:28:20 | sean-k-mooney | i guess is not true for this hardware | |
| 12:28:36 | sean-k-mooney | so we jus tneed to readd that try except | |
| 12:29:50 | sean-k-mooney | ill triage it now | |
| 12:31:10 | gibi | thanks | |
| 12:36:21 | lyarwood | gibi / dansmith ; https://bugs.launchpad.net/nova/+bug/1914664 - the test failure looks like a volume quota issue in that job, we might want to track that in another bug while we look at the unrelated detach device trace | |
| 12:36:22 | openstack | Launchpad bug 1914664 in OpenStack Compute (nova) "QEMU monitor read failure in ServerStableDeviceRescueTest" [Undecided,New] | |
| 12:45:27 | openstackgerrit | Stephen Finucane proposed openstack/python-novaclient master: Add support for microversion v2.88 https://review.opendev.org/c/openstack/python-novaclient/+/770573 | |
| 12:47:56 | stephenfin | dansmith: Now that most of the migration compaction series has merged for the main DB, could I ask that you look at the API DB equivalent? I employed the same testing strategy so it should all be very familiar https://review.opendev.org/q/topic:%2522bp/compact-db-migrations-wallaby%2522+status:open+NOT+label:workflow%253E%253D1 | |