Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-07-22
20:47:02 jralbert Interestingly, there is a bit of a hint in the tables: http://paste.openstack.org/show/796222/
20:47:05 melwitt https://docs.openstack.org/placement/latest/admin/upgrade-to-stein.html
20:47:29 jralbert The compute nodes get '.openstack.local' hostnames in instances, but a genuine FQDN in compute_nodes
20:48:33 melwitt the reason I ask about extracted placement is because you said your nova_api.allocations table is empty. did you already move to a separate placement db? do you have placement.allocations records?
20:49:13 jralbert Sorry, the allocations table isn't completely empty - it's just missing *some* instances on *some* compute nodes
20:49:30 melwitt ok, I think that difference in the nodename is causing the failure to lookup the compute nodes for the heal_allocations command
20:49:46 jralbert we became aware of this because our instance/vcpu metrics suddenly dropped by about half during the upgrade, but all the instances were still running that had been
20:50:19 melwitt this is where I see the lookup by host and node for heal_allocations https://github.com/openstack/nova/blob/stable/stein/nova/cmd/manage.py#L1829
20:50:42 jralbert So would I safely be able to update those definitions directly in the database, or is there a better-bounded API/CLI approach?
20:52:32 jralbert I guess I would have hoped that compute nodes would be identified by uuids since names (especially hostnames) often produce collisions like this, but I'm guessing there's a legacy reason why this is the way it is.
20:55:09 melwitt yeah, they are not linked by uuid. I don't know the history behind it. probably something like since you can delete/create services via the API, if we didn't link by name, you'd lose the association for all your instances even if the hostname is the same (service create makes a new uuid)
20:56:21 jralbert Yep, I can imagine. We too are victims of our own legacy here: our compute nodes always had shortnames rather than FQDNs, and I'm pretty sure that's how we got burned here
20:56:27 melwitt I think you'd have to update manually in the database for this (please backup first). I'm looking to see where the hypervisor_hostname comes from when the compute_nodes record is initially created, to see how it happened and whether it will happen again if you were to delete/create the nova-compute service later on
20:57:53 jralbert I actually think I can maybe do this without futzing with the database, come to think of it: if I let nova-compute discover the node's FQDN and register itself as a "new" compute-service, it'll probably also create a new compute node record with FQDN, and ideally trigger a new resource allocation update. I'll try this with one node to see how it
20:57:54 jralbert goes - otherwise, I'll update in the database
20:58:05 jralbert Thank you very much for taking the time to help me with this today, I really appreciate it
20:58:28 melwitt yeah, that would be the ideal way. fingers crossed that it'll match what your instances already have stored in their host/node
20:58:44 jralbert To be discovered!
22:11:51 openstackgerrit Ghanshyam Mann proposed openstack/nova master: DNM: testing tempest copy-image test https://review.opendev.org/742547
22:44:31 efried artom: It's not out of the question that I'll still review something here or there. But yeah, soon.
22:45:59 artom efried, still addicted, eh? I thought you quit cold-turkey ;)
22:46:15 artom But ack, you're an adult, you can do the needful when it's time
22:46:18 efried I've quit almost nothing in my life cold turkey
22:46:18 artom ;)
22:47:26 efried With the possible exception of running.
22:47:45 efried I actually had an orthopedic surgeon tell me I was too white to run marathons.
22:47:56 artom Hah, you told me that once
22:48:10 artom I remember laughing like an idiot at the most recent Denver PTG
22:49:23 efried orite, because you were limping, s'how that conversation started. All recovered by now I assume (like three years later)
22:50:12 artom Yeah, back to running :)
22:50:21 artom No full marathons, but I've done a couple of halfs
22:51:09 efried nice. I had a half where all the stars aligned, averaged 7:30/mi, reverse split. My best run ever.
22:51:22 efried (which tells you all you need to know about my running talent)
#openstack-nova - 2020-07-23
00:05:01 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add test coverage of networks policies https://review.opendev.org/742558
00:11:28 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Introduce scope_types in networks policy https://review.opendev.org/742561
00:31:48 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add new default roles in networks policies https://review.opendev.org/742566
00:32:45 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add test coverage of networks policies https://review.opendev.org/742558
00:32:53 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Introduce scope_types in networks policy https://review.opendev.org/742561
00:35:38 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add new default roles in networks policies https://review.opendev.org/742566
00:36:06 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Add new default roles in networks policies https://review.opendev.org/742566
00:46:24 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Pass the actual target in networks policy https://review.opendev.org/742568
00:54:59 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Pass the actual target in FIP policy https://review.opendev.org/742570
00:55:16 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Pass the actual target in networks policy https://review.opendev.org/742568
01:50:13 openstackgerrit Merged openstack/nova master: libvirt: Remove two-level loop https://review.opendev.org/741920
02:07:30 openstackgerrit Takashi Natsume proposed openstack/nova master: Remove six.text_type (2/2) https://review.opendev.org/728117
05:28:24 openstackgerrit Merged openstack/nova master: metadata: export the vCPU IDs that are pinning on the host CPUs https://review.opendev.org/688936
05:28:31 openstackgerrit Merged openstack/nova master: release note: add new features introduces by use-pcpu-and-vcpu-in-one-instance https://review.opendev.org/736997
05:52:51 openstackgerrit Merged openstack/nova master: objects: Update keypairs when saving an instance https://review.opendev.org/683043
06:36:41 openstackgerrit Marek Lyčka proposed openstack/nova-specs master: Enable REST quiescing https://review.opendev.org/702810
06:57:48 openstackgerrit Brin Zhang proposed openstack/nova master: Delete ARQs by UUID if Cyborg ARQ bind fails. https://review.opendev.org/716186
07:13:53 gibi good morning
07:15:58 brinzhang_ gibi: good moning^
07:16:42 brinzhang_ gibi: I was update the https://review.opendev.org/#/c/716186/16/nova/exception.py patch, but as your comment, after I update, taht will be an error, you can see in http://paste.openstack.org/show/796234/
07:17:10 brinzhang_ do you know why? the code looks good
07:22:29 gibi brinzhang_: will check in 20 minutes (on a call right now)
07:23:13 brinzhang_ gibi: thanks, I missed a parameter in __init__, will update
07:30:56 openstackgerrit Brin Zhang proposed openstack/nova master: Delete ARQs by UUID if Cyborg ARQ bind fails. https://review.opendev.org/716186
07:30:57 brinzhang_ gibi: you can check all the update in the latest patch now.
08:10:20 kashyap lyarwood: I see the VIR_ERR_DEVICE_MISSING is merged; sorry, yetesrday I had the draft ACK sitting but didn't hit 'submit'. I see it's taken care by three cores :)
08:29:14 gibi brinzhang_: thanks, I'm +2
08:29:34 brinzhang_ gibi: coolthanks
08:33:56 gibi stephenfin: I put https://review.opendev.org/#/q/topic:bp/add-emulated-virtual-tpm to the empty runway slot for the next two weeks
08:34:08 gibi stephenfin: is that OK for you?
08:35:10 gibi stephenfin: I will be on PTO next week but I hope others will check
08:35:27 gibi and I can pick that up the week after
08:45:58 gibi dansmith: Do I see it correclty that the work in https://blueprints.launchpad.net/nova/+spec/rbd-glance-multistore has been completed?
09:18:15 aarents Hi nova,
09:18:44 aarents gibi, when it will be possible, can you advise on last comments of https://review.opendev.org/#/c/696084/3 as my change conflicts with yours here: https://review.opendev.org/#/c/739246/1 as lyarwood mentioned
09:22:58 gibi aarents: looking...
09:34:56 gibi aarents: go ahead with your patch my patch can be the last in the chain
09:38:06 aarents gibi: ok thanks
09:39:51 stephenfin gibi: Sure. Could I ask you to grab the base patch in the series again before you go? You reviewed it before and it's just a rebase
09:46:15 openstackgerrit Stephen Finucane proposed openstack/nova stable/ussuri: objects: Update keypairs when saving an instance https://review.opendev.org/742631
09:47:51 jkulik Do you see any chance of implementing some kind of sharding on the nova-compute level? With the vmwareapi driver managing a cluster of hypervisors, there are hundreds of VMs behind every nova-compute node and at times this leads to tasks queuing up a lot.
09:48:53 bauzas gibi: stephenfin: fwiw, I'll enter the Phantom Zone starting end of next week for the next 3 weeks
09:49:18 bauzas gibi: so when you are back, I'll be off (IIUC your vacation time)
09:49:19 jkulik It would be nice, if we could spawn multiple nova-compute processes per cluster and I was thinking of letting them work on VMs based on the UUID so they don't interfere so easily.
09:50:05 openstackgerrit Merged openstack/nova master: compute: Do not allow rescue attempts using volume snapshot images https://review.opendev.org/729261
09:50:23 stephenfin bauzas: What's the chances of having you work the vTPM stuff with lyarwood? I can handle the 'provider.yaml' stuff (and anything else) since I didn't write it, but not vTPM
09:50:47 bauzas stephenfin: by 'work', you mean 'reviews' ?
09:50:55 stephenfin Correct
09:50:57 bauzas I surely could for next week
09:51:17 bauzas that being said, I'm not an expert, but I guess this is not a drama.
09:51:25 stephenfin \o/
09:51:57 stephenfin It's not complicated. libvirt does all the work for us. Very much just wiring up
10:05:22 gibi stephenfin: sure, I will re-review the bottom of vtpm today
10:06:17 gibi bauzas: ack. have a nice vacation
10:08:49 openstackgerrit Jiri Suchomel proposed openstack/nova master: Add ability to download Glance images into the libvirt image cache via RBD https://review.opendev.org/574301
10:10:23 jsuchome Hi gibi, thanks for the review, I've just updated those whitespaces ^
10:13:56 gibi jsuchome: cool. I plugged my +2 back
10:14:01 gibi thanks
10:34:08 openstackgerrit Merged openstack/nova stable/stein: libvirt: Mark e1000e VIF as supported https://review.opendev.org/742390
10:47:54 openstackgerrit Merged openstack/nova stable/pike: libvirt: Do not reraise DiskNotFound exceptions during resize https://review.opendev.org/742181
10:48:00 openstackgerrit Merged openstack/nova master: test: add some notes and new tests for mixed instance https://review.opendev.org/741395
11:17:10 gibi stephenfin: there is small test thing is https://review.opendev.org/#/c/739210/5/nova/tests/unit/objects/test_image_meta.py@424 if you have time to respin the I will re-review, but it is also OK to fix it in a FUP
11:25:00 lyarwood stephenfin: FWIW I have the series open as well
12:07:05 openstackgerrit Stephen Finucane proposed openstack/nova master: trivial: Test object backporting against correct version https://review.opendev.org/742650
12:07:05 openstackgerrit Stephen Finucane proposed openstack/nova master: scheduler: Default request group to None https://review.opendev.org/742651
12:07:12 stephenfin lyarwood, gibi: ^
12:07:57 stephenfin The latter one is in merge conflict for now, but I'm hoping bauzas can review https://review.opendev.org/#/c/739210/ today (since he understands this stuff) and I can rebase it then :)

Earlier   Later