Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2021-11-23
14:45:16 sean-k-mooney mnaser: that is anyting other then delete including allowing the instance to continue running :)
14:45:26 bauzas gibi: looks like, yes
14:45:39 mnaser sean-k-mooney: but that means more $$$ am i rite? =ap
14:46:09 gibi mnaser: yepp, I agree, but I think this behavior needs to be configurable
14:46:40 gibi mnaser: to handle the case when notification only just good to have
14:46:43 gibi but not must have
14:47:06 opendevreview Artom Lifshitz proposed openstack/nova master: DNM: Explode libvirt builds to trigger tempest error code https://review.opendev.org/c/openstack/nova/+/818932
14:47:16 sean-k-mooney mnaser: :) i think the correct behavior if notification cant be sent is either to just log an error and drop them or continue to have the service work but set its healtch check to degraded
14:47:34 sean-k-mooney the only other option i see would be for the service to terminate
14:48:01 opendevreview Artom Lifshitz proposed openstack/nova master: DNM: Explode libvirt builds to trigger tempest error code https://review.opendev.org/c/openstack/nova/+/818932
14:48:20 sean-k-mooney mnaser: if we are going to stop some operation from workign i dont think we shoudl be reportign the srvice as up in our current status filed
14:48:37 sean-k-mooney but that is based on a heatbeat which might still be working
14:48:44 sean-k-mooney so that not really something we can change
14:49:40 mnaser sean-k-mooney: historically i rather a failure and consistent behaviour than a degraded 'pass' that leaves a bit of a mess
14:49:45 mnaser (see: ignore cinder volume deletes)
14:49:59 sean-k-mooney well notifocation are generally considerd besteffort
14:50:09 sean-k-mooney its not a mission cirtical part of the cloud
14:50:57 sean-k-mooney so we could make the service exit as an opt in behavior but in generally i think logging an error and reportign a degrated health status then allowing something external to decied what to do i think is better
14:51:15 mnaser i guess notifications being critical will dpeend on who you ask ;p
14:51:45 sean-k-mooney form a nova point of view i dont think they ever have had any guarentees
14:52:32 sean-k-mooney i know people do consume them for billing and other uses but the in ablity to send a notificaotn shoudl not have any impact on a user operation by default
14:53:59 sean-k-mooney mnaser: as a customer if i tried to reboot a vm and it failed because fo a notrifcaiton issue i wold expect a 500 or other error code to be logged showing its an infra issue so i can file a ticket with my cloud provider for downtime
14:54:34 mnaser sean-k-mooney: that's true, but i guess since we don't have any buffering right now, it means notifications can be lost .. and in theory, they'll get a 500 anyways now if the notifications are down =P
14:54:53 sean-k-mooney mnaser:basicly in cloud where notificaiotn are treated as critical it would be a cloud outage from a customer point of ivew
14:55:17 sean-k-mooney mnaser: hehe i guess they will get the 500 now ture
14:55:32 sean-k-mooney although only if you dont set a retry limit right
14:55:40 mnaser nope, even if you set retry=0 it fails :(
14:55:51 mnaser see what gibi mentioned above, it gets stuck on trying to connect and that has no retries
14:55:51 sean-k-mooney oh ok then that is a bug
14:55:56 mnaser it loops forever on trying to connect
14:56:04 sean-k-mooney ah right
14:56:27 sean-k-mooney because its the amqp connection that is down its not an issue with sendign to a queue or exchange
14:56:32 mnaser i captured GMR and updated info here - https://bugs.launchpad.net/nova/+bug/1917645
14:57:00 sean-k-mooney mnaser: well that proably should be converted to an oslo messaging bug at this point
14:57:22 mnaser sean-k-mooney: good point actually
14:57:47 sean-k-mooney nova might be able to work around this but eventually we will want to adress this there i suspect
15:01:11 opendevreview Dmitrii Shcherbakov proposed openstack/os-traits master: Add a trait for remote_managed port-capable nodes https://review.opendev.org/c/openstack/os-traits/+/818514
15:03:42 gibi sean-k-mooney: sure oslo.messaging lacks a configuration possibility to stop retrying the connect, but I don't think it is purelya bug
15:03:58 sean-k-mooney pro tip dont put a folder with 2412 sub directories in your test path if you want test discovery to happen before the heat death of the universe
15:04:00 gibi sean-k-mooney: on nova side it is sure a bug if the instances stuck in some tranisitional state due to hanging
15:04:45 sean-k-mooney gibi: yes but i dont think im sold that we should intoduction config driven api behavior
15:04:53 gibi sean-k-mooney: not in nova no
15:05:00 gibi sean-k-mooney: we need oslo.messaging configurability
15:05:12 sean-k-mooney right so we cant block opertaion because we failed to send a notification
15:05:35 sean-k-mooney gibi: well not i think not at all nova need to proceed even if the notificaiton cant be sent
15:05:55 sean-k-mooney we can log an error ro report it via a healthcheck or similar
15:06:11 gibi sean-k-mooney: I don't agree. If notifications are used for billing then I can imagine that the operator want the instance boot to fail if the notification bus is down
15:06:28 sean-k-mooney gibi: its a grey area i guess
15:06:35 gibi sure, don't stuck, but fail cleanly
15:06:53 sean-k-mooney to me unless we can report a 500 to the user im not sure we should block the operation
15:07:22 bauzas gibi: sean-k-mooney: thoughts abotu https://bugs.launchpad.net/nova/+bug/1951623 be a nova or neutron issue ?
15:08:04 sean-k-mooney ..... both
15:08:05 bauzas nova wouldn't know whether the port is disabled
15:08:10 gibi bauzas: nova, as nova waits unconditionally
15:08:15 sean-k-mooney nova can check that
15:08:15 bauzas yeah
15:08:16 gibi bauzas: nova can query the port sate
15:08:18 gibi state
15:08:32 sean-k-mooney nova because this is not checked today
15:08:40 sean-k-mooney and neutron because this change depending on the backend
15:08:53 bauzas gibi: mmm, that means that we would call neutron to get the port state
15:09:15 sean-k-mooney bauzas: i think we have that in the network info cache but yes we have a check for this in some of the code already
15:09:39 bauzas sean-k-mooney: ok, then it's a nova bug if we already call it
15:09:50 sean-k-mooney we just dont in that part of the code
15:10:19 sean-k-mooney but part of me is wondering if we would be better off just not using those event until we actully define a contract with neutorn on when they will be sent
15:10:59 gibi sean-k-mooney: but thing might never happen based on past experience ^^ :)
15:11:07 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L7221-L7228
15:11:21 bauzas gibi: sean-k-mooney: I just triaged the bug https://bugs.launchpad.net/nova/+bug/1951623
15:11:28 sean-k-mooney so we check if its active here but we dont for the migration events
15:12:31 sean-k-mooney https://github.com/openstack/nova/blob/master/nova/network/model.py#L488-L508 or here https://github.com/openstack/nova/blob/master/nova/network/model.py#L567-L586
15:12:59 sean-k-mooney bauzas: gibi so we just need to update the model fucntion to skip disbaled ports
15:13:15 bauzas ack
15:13:22 bauzas bug confirmed anyway
15:13:37 sean-k-mooney the issue is that in some cases we will get event if the port is disabled
15:13:49 sean-k-mooney which we will now be ignorign but thats is likely ok
15:13:53 gibi yeah
15:20:29 bauzas elodilles: I hadn't time to update the meeting agenda, but add your points if you want
15:20:50 bauzas reminder : nova meeting in 40 mins here in #openstack-nova
15:23:38 elodilles bauzas: adding it now then
15:23:47 DK4 when creating instances they are all stuck in creating state. ive combed thrtough logs and did not find anything special. any advice on debugging that?
15:25:49 elodilles bauzas: done, just added a short info update, quickly
15:26:17 elodilles bauzas: you can edit now the page if you want
15:31:06 sean-k-mooney bauzas: by the way unless youhave actully repoduced the bug or root caused it you should be setting it to triaged not confirmed
15:32:12 sean-k-mooney we treat them more or less the same so i guess it does not matter but confrimed imples we coudl repoduces it triage does not
15:37:20 opendevreview Dan Smith proposed openstack/nova master: Revert project-specific APIs for servers https://review.opendev.org/c/openstack/nova/+/816206
15:51:06 bauzas last reminder: nova meeting starts in 10 mins here at #openstack-nova IRC chan
15:51:40 bauzas sean-k-mooney: yes and no, we could put in Triaged if we don't know the direction
15:51:53 bauzas sean-k-mooney: we don't ask bug triagers to test all the issues
15:52:00 bauzas (even if Launchpad says it)
15:52:15 bauzas at least that's the census we have
15:52:24 sean-k-mooney yes i know which is why i am saying confirmed is a larger statement
15:53:40 sean-k-mooney triaged just means we think its valid and have some understading of why it happens, confrimed implies you ahve actully tried to repoduce it an have confirmed the behavior is as reported in the bug
15:53:48 sean-k-mooney or at least that is how i treat it
16:00:15 bauzas #startmeeting nova
16:00:15 opendevmeet Meeting started Tue Nov 23 16:00:15 2021 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot.
16:00:15 opendevmeet Useful Commands: #action #agreed #help #info #idea #link #topic #startvote.
16:00:15 opendevmeet The meeting name has been set to 'nova'
16:00:25 gibi o/
16:00:29 elodilles o/
16:00:32 bauzas #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting

Earlier   Later