Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-08-17
13:05:42 opendevreview Stephen Finucane proposed openstack/nova master: tests: Enable SADeprecationWarning warnings https://review.opendev.org/c/openstack/nova/+/804708
13:05:47 stephenfin gibi_: ^
13:07:02 opendevreview Lee Yarwood proposed openstack/nova master: api: Introduce microversion 2.89 adding attachment_id to responses https://review.opendev.org/c/openstack/nova/+/804275
13:09:15 gibi stephenfin: ack
13:10:52 gibi stephenfin: thanks, now I'm +2 all the way
13:53:17 opendevreview Stephen Finucane proposed openstack/nova master: tests: Enable SQLAlchemy 2.0 deprecation warnings https://review.opendev.org/c/openstack/nova/+/804709
13:53:18 opendevreview Stephen Finucane proposed openstack/nova master: Replace use of Engine.scalar(), Engine.execute() https://review.opendev.org/c/openstack/nova/+/804878
13:55:29 opendevreview Elod Illes proposed openstack/nova stable/rocky: [stable-only] Fix lower-constraints job https://review.opendev.org/c/openstack/nova/+/769910
14:00:34 opendevreview Alexandre arents proposed openstack/nova master: libvirt: Abort live-migration job when monitoring fails https://review.opendev.org/c/openstack/nova/+/764435
14:28:08 opendevreview Elod Illes proposed openstack/nova stable/rocky: [stable-only] Fix lower-constraints job https://review.opendev.org/c/openstack/nova/+/769910
14:50:29 gibi melwitt: hi! I left feedback in https://review.opendev.org/c/openstack/nova/+/713301 the most concerning for me is the dependency on an oslo.limit patch as non client libraries are going to feature freeze this week
15:49:10 ganso melwitt, gibi hi! if you have a few minutes could please take a look at this 1-liner fix https://review.opendev.org/c/openstack/nova/+/804303 ? Thanks in advance
15:50:32 gibi ganso: looking
15:52:40 spatel sean-k-mooney hey! i am upgrading minor version of victoria and during upgrade at this step i hit this issue - https://paste.opendev.org/show/808150/
15:54:48 gibi FYI, nova meeting starts in 5 minutes here in the channel
15:56:13 gibi ganso: does the 1 vcpu + multiqueue case works for other than vif_type=tap?
15:57:33 ganso gibi that code path is only reached when vif_type=tap. If using openvswitch, it wasn't impacted by the previous patch (that introduced the regression), neither this. I am not sure if the same problem happens with ovs, but since the original problem didn't, I believe this one also doesn't
15:59:48 gibi ganso: ok, let me try with ovs
15:59:55 sean-k-mooney vif_type=tap is not currently used with ovs
16:00:03 sean-k-mooney it was added tempoery and then removed
16:00:18 sean-k-mooney i belive the only thing that uses vif_type=tap today is calico
16:00:28 gibi sean-k-mooney: yepp the bug mentions calico
16:00:35 gibi sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1939604
16:00:41 melwitt gibi: ah, right. it's not a hard dep, so we can untie it. it's nice-to-have since it will cache limit for a repeated try N, N-1, N-2, etc for a multi create
16:01:02 sean-k-mooney gibi: ah ok ill look at it after the meeting
16:01:23 gibi melwitt: OK, it is easier then. lets see if the oslo patch lands before the deadline and if not then just remove the depends-on
16:01:33 gibi but now lets have a meeting
16:01:38 melwitt gibi: ++ thanks
16:01:38 opendevmeet The meeting name has been set to 'nova'
16:01:38 opendevmeet Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:01:38 opendevmeet Meeting started Tue Aug 17 16:01:38 2021 UTC and is due to finish in 60 minutes. The chair is gibi. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:01:38 gibi #startmeeting nova
16:01:55 gibi sean-k-mooney: thanks
16:02:45 gibi #topic Bugs (stuck/critical)
16:02:52 gibi no critical bug open
16:02:56 gibi #link 15 new untriaged bugs (+4 since the last meeting): #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New
16:03:22 gibi is there any specific bug to discuss today?
16:04:16 gibi I see ganso has one in the open discussion, lets bring that up here
16:04:24 gibi (ganso): bug "Compute node deletes itself if rebooted without DNS": https://bugs.launchpad.net/nova/+bug/1939920
16:04:29 gibi was this a design choice? acceptable solutions discussion
16:04:33 gibi EOM
16:04:37 ganso gibi thanks
16:05:00 ganso so, IMO this is a critical bug, and after reading the code and the way it works it kinda feels like a design choice
16:05:29 ganso because seems like it was intentionally implemented for it to scan for "orphan compute nodes" and delete them, clear the allocations and RP, etc
16:05:42 gibi yes that was intentional
16:05:51 ganso but it is producing this effect which is very undesirable
16:06:33 ganso as I suggested in the bug, a possible solution I see if to compare the host field in the nova.compute_nodes table
16:06:33 gibi so in your infra the compute host can change hostname and that causing the issue
16:06:43 ganso if it is the same, then we would skip this
16:06:57 ganso gibi: it is not that it "can" change the hostname. But it happens due to external reasons
16:07:09 ganso like, lack of connectivity when it boots, a DNS outage, etc
16:07:26 melwitt it will recover in that it will create a new compute node etc. the main thing that is "unique" is the hostname, that's what's stored in the instance.host and a whole lot of other places. so changing the name you break all the associations and in reality you essentially have a new/different service and compute node
16:08:39 ganso melwitt: right, so the instance.host captures the entire FQDN, and the FQDN is what is changing, therefore when that changes, running instances are no longer identifiable as running in that node
16:08:39 melwitt if the associations were done using UUID it would be a different story. but unfortunately it is what it is and would take a large work to change it IMHO
16:08:58 melwitt right
16:09:26 ganso melwitt: so that is another side-effect of that FQDN changing problem, but I am not proposing changing that. I am just proposing to skip this "deletion" step if the compute_nodes.host field does not change
16:09:33 ganso if will avoid part of the issues
16:10:00 ganso melwitt it will not avoid the issue you described, but 1 issue is better than 2 I think
16:10:24 dansmith I'm missing the distinction I think
16:11:22 gibi but compute_nodes.host comes from the DB isn't it? so it won't ever change
16:11:24 sean-k-mooney ganso: right so nova does not support compute hosts changing hostname today
16:11:52 ganso gibi: doesn't it derive from the FQDN it reads from the system?
16:11:57 sean-k-mooney so if it is changing for external reasons that is not expected to work out of the box
16:12:34 dansmith sean-k-mooney: ++
16:12:42 ganso sean-k-mooney: right, but I'm not proposing that it does support, but just stop doing what it is doing today. That thing about orphan compute nodes isn't supposed to address changing hostnames either
16:13:05 gibi ganso: when the ComputeNode is created then yes, it is coming from the hostname reported by libvirt, but never changes after
16:13:35 sean-k-mooney ganso: even if we did not clean up the orpah compute nodes teh instnace.host is used to make rpc calls to the host that the instance is on
16:14:00 sean-k-mooney so unless you hardcode the chost paramter in the nova.conf so it does not change
16:14:06 ganso back to normal once the FQDN is fixed and the service is restarted
16:14:06 ganso dansmith: when FQDN changes from "host.domain" to "host.domain1" or just "host" it causes the compute node to delete itself from the DB, clear allocations, RP, etc, and the new name will not match the instances.host field as melwitt mentioned. Out of all those consequences, I'd suggest skipping the compute node deletion, because this is an error state, to avoid deleting up all allocations and RPs, so the node can more easily go
16:14:12 sean-k-mooney that will still break
16:14:32 sean-k-mooney ganso: the compute service will not do that by default
16:14:41 sean-k-mooney the compute service will auto register
16:14:44 ganso sean-k-mooney: yes, that will still be broken, as it is today, no need to fix that right now
16:14:45 sean-k-mooney bvut it wont auto delete
16:15:17 ganso sean-k-mooney: well it does, it thinks there was an orphan and deletes it
16:15:27 sean-k-mooney ganso: what deletes it
16:15:33 sean-k-mooney i think i missed that
16:15:39 ganso sean-k-mooney: https://github.com/openstack/nova/blob/b0099aa8a28a79f46cfc79708dcd95f07c1e685f/nova/compute/manager.py#L9997
16:16:08 sean-k-mooney is this a clustered hypervior
16:16:24 ganso sean-k-mooney: "host.domain" changes to "host", so it deletes "host.domain" from the compute nodes table and creates a new one, as if the node was brand new
16:16:36 sean-k-mooney e.g. ironic or hyperv or something like vmware
16:16:52 dansmith ganso: because that's a hostname change
16:16:57 ganso sean-k-mooney: no, it is just a regular compute node with a libvirt compute service
16:17:05 dansmith ganso: arrange for that not not happen, that's the solution, IMHO
16:17:22 ganso dansmith: unfortunately it is beyond control
16:17:37 sean-k-mooney well for the libvirt driver that is entirly unsupported
16:17:51 ganso my proposal is to leave it in an error state to prevent it from deleting allocations and RP
16:17:51 sean-k-mooney the other way to fix this is to make sure your cannonical hostname is not the fqdn
16:18:42 sean-k-mooney e.g. in /etc/host set <ip> <short hostname> <fqdn>
16:19:09 dansmith sean-k-mooney: or /etc/domainname, but yeah, totally fixable, IMHO
16:19:33 ganso sean-k-mooney: hmm I see, that would override the one currently being provided by the domain provider
16:19:33 sean-k-mooney im still configuse how nodenames is a list in this case
16:20:07 sean-k-mooney or rather how when the the fqdn changes we are actully geting anything back from the db
16:20:14 sean-k-mooney i was expecting it to not match anything
16:20:26 dansmith can't you set the hostname nova uses in the config anyway? to hard-code it per host so it doesn't change, I thought we had that
16:20:44 sean-k-mooney unless you have something linke host1.<domain1> host1.<doamin2>
16:20:53 ganso dansmith: looking in the code now
16:21:06 sean-k-mooney dansmith: you can set the hostname used by the compute service
16:21:09 dansmith we might not want to hold up the meeting to discuss this to completion

Earlier   Later