Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-07-22
13:58:11 gibi artom: thanks. I will check if I can check for vlan
13:58:46 gibi artom: thanks for pointing me to the code that was supposed to clean up the metadata I missed that orignally
13:59:02 artom gibi, it was well hidden. Blame mriedem ;)
13:59:19 gibi :)
13:59:31 artom (For real, it was his idea - performance-wise, it's the good choice. Code-understandability-wise, clearly not so great ;)
14:01:50 gibi artom: as far as I see if I simply remove the tag in vif conditional then the code still don't do any expensive things, jut iterate through instance.device_metadata to find the metadata by vif.address
14:02:18 artom gibi, maybe the easiest way to do it is to do a reverse lookup kind of thing: just loop through the device metadata regardless, and if the mac matches the vif mac, delete that entry.
14:02:27 artom gibi, hah, jiinx
14:02:49 gibi that the baseline code does today
14:02:52 artom IOW, as you said, remove the conditional
14:02:58 gibi https://opendev.org/openstack/nova/src/branch/master/nova/network/neutron.py#L1788
14:03:02 gibi Ok
14:06:24 artom Which reminds me, I really need to get https://bugs.launchpad.net/tempest/+bug/1775947 figured out
14:06:24 openstack Launchpad bug 1775947 in tempest "tempest.api.compute.servers.test_device_tagging.TaggedAttachmentsTest failing" [Medium,Confirmed]
14:06:26 artom And fixed
14:21:36 openstackgerrit Merged openstack/nova master: libvirt: Remove workaround for really old QEMU https://review.opendev.org/740335
14:34:37 openstackgerrit Artom Lifshitz proposed openstack/nova master: DNM: extra logging for tagged device detach https://review.opendev.org/742450
14:39:02 openstackgerrit Balazs Gibizer proposed openstack/nova master: Delete non tag related device metadata during detach https://review.opendev.org/742451
14:42:41 openstackgerrit Balazs Gibizer proposed openstack/nova master: Delete non tag related device metadata during detach https://review.opendev.org/741436
16:03:10 openstackgerrit Balazs Gibizer proposed openstack/nova master: [WIP] Support SRIOV interface attach and detach https://review.opendev.org/740995
16:09:56 openstackgerrit Merged openstack/nova master: hardware: create 'mixed' instance for realtime CPUs https://review.opendev.org/728485
18:22:59 openstackgerrit Gonéri Le Bouder proposed openstack/nova master: QEMU/KVM: accept vmxnet3 NIC https://review.opendev.org/738695
18:41:20 openstackgerrit Gonéri Le Bouder proposed openstack/nova master: QEMU/KVM: accept vmxnet3 NIC https://review.opendev.org/738695
18:48:45 artom Random thought - someone should probably remove efried and mriedem from https://review.opendev.org/#/admin/groups/25,members, no?
18:48:53 artom Nothing against you fellas, obviously :)
19:00:32 mriedem i'm removed myseflf
19:00:38 mriedem *myself even
19:01:14 mriedem and https://review.opendev.org/#/admin/groups/1936,members
19:07:58 artom mriedem, RIP in pieces :(
19:08:21 artom I was randomly linking that page to someone who needed core review
19:08:34 artom This isn't a coup, is what I'm saying :)
19:21:03 mriedem yup, np
20:22:57 jralbert I've raised this question in #openstack and so far received no response; anybody here have any advice? I've just upgraded from Rocky to Stein, and Placement seems to have gotten very confused about resources in-use vs available. A number of compute nodes with running instances which are known to 'openstack server show' are reporting 0 running_vms
20:22:57 jralbert in 'nova hypervisor-stats', and have no entries in the allocations table in the nova_api database. Running nova-manage placement heal_allocations for affected instances returns an error that "Compute host <compute_node> could not be found", but all the compute nodes are visible in nova-manage cell_v2 list_hosts. Has anyone encountered this? Any
20:22:58 jralbert suggestions on how to troubleshoot it further?
20:39:15 melwitt jralbert: it sounds like your nova.instances records instance.host and instance.node aren't matching any records in nova.compute_nodes 'host' and 'hypervisor_hostname', as for why you're getting Compute host could not be found
20:40:30 melwitt did you replace any nova-compute services during the upgrade with different hostnames or something like that?
20:41:36 jralbert melwitt: Thanks, I'll have a look at those tables. We didn't change any hostnames during the upgrade, although because it was being orchestrated by OSA there may have been steps taken we weren't fully aware of
20:43:49 melwitt ok. I can't tell you how things got into that state but just for the sake of understanding why heal_allocations won't work: it will look up the compute node for an instance by querying for the instance.host and instance.node in the nova.compute_nodes table and if it finds no match, you'll get the Compute host could not be found error
20:45:32 melwitt I'm not aware of any change from rocky to stein that could cause something like this. are you running with integrated placement (in nova) or did you split out to running extracted placement I wonder?
20:46:38 jralbert I'm not 100% sure what integrated/extracted placement means, but I can find out - we're relying mostly on OSA defaults with some overrides, but that's not one I'm aware of.
20:46:44 melwitt if you've moved to extracted placement, there are additional steps you need to follow for an upgrade, linked from here: https://docs.openstack.org/releasenotes/nova/stein.html#prelude
20:47:02 jralbert Interestingly, there is a bit of a hint in the tables: http://paste.openstack.org/show/796222/
20:47:05 melwitt https://docs.openstack.org/placement/latest/admin/upgrade-to-stein.html
20:47:29 jralbert The compute nodes get '.openstack.local' hostnames in instances, but a genuine FQDN in compute_nodes
20:48:33 melwitt the reason I ask about extracted placement is because you said your nova_api.allocations table is empty. did you already move to a separate placement db? do you have placement.allocations records?
20:49:13 jralbert Sorry, the allocations table isn't completely empty - it's just missing *some* instances on *some* compute nodes
20:49:30 melwitt ok, I think that difference in the nodename is causing the failure to lookup the compute nodes for the heal_allocations command
20:49:46 jralbert we became aware of this because our instance/vcpu metrics suddenly dropped by about half during the upgrade, but all the instances were still running that had been
20:50:19 melwitt this is where I see the lookup by host and node for heal_allocations https://github.com/openstack/nova/blob/stable/stein/nova/cmd/manage.py#L1829
20:50:42 jralbert So would I safely be able to update those definitions directly in the database, or is there a better-bounded API/CLI approach?
20:52:32 jralbert I guess I would have hoped that compute nodes would be identified by uuids since names (especially hostnames) often produce collisions like this, but I'm guessing there's a legacy reason why this is the way it is.
20:55:09 melwitt yeah, they are not linked by uuid. I don't know the history behind it. probably something like since you can delete/create services via the API, if we didn't link by name, you'd lose the association for all your instances even if the hostname is the same (service create makes a new uuid)
20:56:21 jralbert Yep, I can imagine. We too are victims of our own legacy here: our compute nodes always had shortnames rather than FQDNs, and I'm pretty sure that's how we got burned here
20:56:27 melwitt I think you'd have to update manually in the database for this (please backup first). I'm looking to see where the hypervisor_hostname comes from when the compute_nodes record is initially created, to see how it happened and whether it will happen again if you were to delete/create the nova-compute service later on
20:57:53 jralbert I actually think I can maybe do this without futzing with the database, come to think of it: if I let nova-compute discover the node's FQDN and register itself as a "new" compute-service, it'll probably also create a new compute node record with FQDN, and ideally trigger a new resource allocation update. I'll try this with one node to see how it
20:57:54 jralbert goes - otherwise, I'll update in the database
20:58:05 jralbert Thank you very much for taking the time to help me with this today, I really appreciate it
20:58:28 melwitt yeah, that would be the ideal way. fingers crossed that it'll match what your instances already have stored in their host/node
20:58:44 jralbert To be discovered!
22:11:51 openstackgerrit Ghanshyam Mann proposed openstack/nova master: DNM: testing tempest copy-image test https://review.opendev.org/742547
22:44:31 efried artom: It's not out of the question that I'll still review something here or there. But yeah, soon.
22:45:59 artom efried, still addicted, eh? I thought you quit cold-turkey ;)
22:46:15 artom But ack, you're an adult, you can do the needful when it's time
22:46:18 efried I've quit almost nothing in my life cold turkey
22:46:18 artom ;)
22:47:26 efried With the possible exception of running.
22:47:45 efried I actually had an orthopedic surgeon tell me I was too white to run marathons.
22:47:56 artom Hah, you told me that once
22:48:10 artom I remember laughing like an idiot at the most recent Denver PTG
22:49:23 efried orite, because you were limping, s'how that conversation started. All recovered by now I assume (like three years later)
22:50:12 artom Yeah, back to running :)
22:50:21 artom No full marathons, but I've done a couple of halfs
22:51:09 efried nice. I had a half where all the stars aligned, averaged 7:30/mi, reverse split. My best run ever.
22:51:22 efried (which tells you all you need to know about my running talent)
#openstack-nova - 2020-07-23
00:05:01 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add test coverage of networks policies https://review.opendev.org/742558
00:11:28 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Introduce scope_types in networks policy https://review.opendev.org/742561
00:31:48 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add new default roles in networks policies https://review.opendev.org/742566
00:32:45 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add test coverage of networks policies https://review.opendev.org/742558
00:32:53 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Introduce scope_types in networks policy https://review.opendev.org/742561
00:35:38 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add new default roles in networks policies https://review.opendev.org/742566
00:36:06 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add new default roles in networks policies https://review.opendev.org/742566
00:46:24 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Pass the actual target in networks policy https://review.opendev.org/742568
00:54:59 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Pass the actual target in FIP policy https://review.opendev.org/742570
00:55:16 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Pass the actual target in networks policy https://review.opendev.org/742568
01:50:13 openstackgerrit Merged openstack/nova master: libvirt: Remove two-level loop https://review.opendev.org/741920
02:07:30 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.text_type (2/2) https://review.opendev.org/728117
05:28:24 openstackgerrit Merged openstack/nova master: metadata: export the vCPU IDs that are pinning on the host CPUs https://review.opendev.org/688936
05:28:31 openstackgerrit Merged openstack/nova master: release note: add new features introduces by use-pcpu-and-vcpu-in-one-instance https://review.opendev.org/736997
05:52:51 openstackgerrit Merged openstack/nova master: objects: Update keypairs when saving an instance https://review.opendev.org/683043
06:36:41 openstackgerrit Marek Lyčka proposed openstack/nova-specs master: Enable REST quiescing https://review.opendev.org/702810
06:57:48 openstackgerrit Brin Zhang proposed openstack/nova master: Delete ARQs by UUID if Cyborg ARQ bind fails. https://review.opendev.org/716186
07:13:53 gibi good morning
07:15:58 brinzhang_ gibi: good moning^
07:16:42 brinzhang_ gibi: I was update the https://review.opendev.org/#/c/716186/16/nova/exception.py patch, but as your comment, after I update, taht will be an error, you can see in http://paste.openstack.org/show/796234/
07:17:10 brinzhang_ do you know why? the code looks good
07:22:29 gibi brinzhang_: will check in 20 minutes (on a call right now)
07:23:13 brinzhang_ gibi: thanks, I missed a parameter in __init__, will update
07:30:56 openstackgerrit Brin Zhang proposed openstack/nova master: Delete ARQs by UUID if Cyborg ARQ bind fails. https://review.opendev.org/716186
07:30:57 brinzhang_ gibi: you can check all the update in the latest patch now.

Earlier   Later