Earlier  
Later
New Search
Posted Nick Remark
#openstack-nova - 2020-02-27
16:52:11 gibi efried ++
16:52:11 dansmith Sundar: "tests passing" isn't a bar to meet, "tests validating something useful" is
16:52:12 efried there should be some good examples
16:52:15 dansmith yep, what efried said
16:52:48 Sundar efried: Thanks, will look for that. If you happen to remember the(rough) name of the fixture, off the top of your head, I'll start with that.
16:53:00 dansmith Sundar: what specifically are you stuck on? there's lots of feedback to work on without much response from you yet that I can see.. is there something specific that you're blocked on?
16:53:46 Sundar dansmith: Not sure that any of my changes will actually work in the real world, without a multinode env. I tried to get a 2nd server but it will take time, if at all.
16:53:53 efried Sundar: HostNameWeigherFixture I think
16:53:58 gibi Sundar: HostNameWeigherFixture makes the host selection peridctable
16:54:11 dansmith Sundar: functional tests man :)
16:54:12 Sundar efried, gibi: Thanks, got it
16:54:50 dansmith Sundar: if you have reasonable functional tests that don't mock out insane levels of things and validate reasonable conditions, then any real world failure we find later is understandable
16:55:53 dansmith Sundar: test_servers.py has a bunch of reschedule-specific tests
17:00:36 Sundar dansmith: Ok, let's just focus on functional tests, then. BTW, https://review.opendev.org/gitweb?p=openstack/nova.git;f=nova/conductor/manager.py;hb=refs/changes/44/631244/63#l602 is specific to the resheduling path AFAICS. is there any other path, like rebuild/resize/whatever, that invokes this?
17:01:46 dansmith Sundar: only if one of those things needs a reschedule, AFAIK
17:04:14 Sundar Ok. BTW, I will be travelling over the next couple of days. I will respond with the comments I accepted by today, and will post the rest in a few days.
17:04:36 dansmith okay
17:05:55 Sundar dansmith: In https://review.opendev.org/#/c/631244/63/nova/conductor/manager.py@848, is there any resource to be cleaned up in the exception path?
17:07:29 dansmith Sundar: seems like there's network and block devices to clean up, but I'm not sure why the error paths above aren't doing that
17:07:37 dansmith Sundar: should you be deleting arqs?
17:07:55 dansmith or are you assuming those get deleted when the error'd instance gets deleted?
17:08:19 dansmith Sundar: there is a _cleanup_after_reschedule_fails() method in there...
17:08:23 sean-k-mooney gibi: ack that is fine. i likely will not be ready until then anyway
17:09:02 dansmith er, cleanup_when_reschedule_fails
17:09:10 dansmith which cleans up allocated network resources at least
17:09:25 Sundar Well, not in this patch for sure, because delete ARQ functionality comes later. However, we should do that in the delete patch because Create ARQs may succeed but bind may throw an exception potentially. Then, Cyborg should be informed to delete the created ARQs.
17:09:54 dansmith Sundar: okay your existing code won't even leave the instance in error state, which is bad.. but that cleanup method will do that for you
17:10:44 Sundar dansmith: Yes, I see it. Good. Thanks.
17:11:39 Sundar dansmith: I could tie the delete ARQs to _cleanup_when_reschedule_fails(). I'll check to see what other impact it has.
17:12:18 dansmith Sundar: if deleting the instance in un-scheduled state will clean those up, then it's probably okay to punt to that.. the only problem might be if the ARQs are holding some resource for the dead instance
17:14:47 Sundar dansmith: If the bind succeeds, then ARQs are holding real resources. So, if some other task fails during rescheduling, then Cyborg should be told to delete the ARQs. I think the changes we already have should take care of that.
17:17:27 dansmith Sundar: right but the bind is right before the call back to compute, so if that fails, no bind was done, and if it has failed before that, we're already unbound from the original host, so not holding resource... is that right?
17:21:02 Sundar dansmith: bind is async. The call here just kicks off the process, and the actual bind may succeed or fail after that. If the bind fails, either Cyborg sends a failure notification or Nova gets a timeout (in corner cases). That is already handled. if the bind succeeds, Cyborg has allocated and prepared resources. But something else in Nova may
17:21:02 Sundar fail. It is this second path that I am not terribly sure about.
17:21:36 dansmith I think you're missing my point
17:21:56 dansmith or maybe you're talking about the reschedule racing with the *original* bind?
17:24:51 Sundar dansmith: I am talking of the bind during the rescheduling in https://review.opendev.org/#/c/631244/63/nova/conductor/manager.py@842 . Yes, if there is a failure in _create_and_bind_arqs( ) or earlier, no resources are allocated.
17:25:33 dansmith Sundar: so you're concerned about what happens if the bind during reschedule fails and not knowing whether things are actually bound or not?
17:25:48 dansmith if so that's... pretty scary
17:25:51 Sundar I was talking of the scenario where _create_and_bind_arqs() succeeds, Nova goes ahead with the reschedule, CYborg prepares devices in the new host, and then things go south.
17:26:00 Sundar Things go south in Nova
17:26:20 Sundar So, every failure path in reschedule must call Cyborg to delete ARQs
17:26:29 dansmith depends on where it goes south, but there's not much we can do about it
17:26:45 dansmith try to delete the arqs when we can in error paths,
17:27:02 dansmith but as long as the instance delete will clean things up gracefully, that's the safety net
17:27:49 dansmith after your bind in the reschedule path, the rest of it is identical to the regular build process
17:27:50 Sundar I am trying to see if I am missing any error paths. AFAICS, we haven't missed any in the conductor. In the compute manager, if the reschedule takes the same code paths as the priginal schedule, we should be good.
17:27:58 dansmith right
17:28:18 Sundar Ok, then we are good
17:28:25 dansmith if your bind in reschedule in conductor succeeds, then it's a regular build after that
17:45:13 openstackgerrit Merged openstack/os-traits master: Remove use of six https://review.opendev.org/710274
17:56:37 melwitt mgoddard: do you know if there's any workaround for this bug you're working on? https://bugs.launchpad.net/nova/+bug/1853009
17:56:37 openstack Launchpad bug 1853009 in OpenStack Compute (nova) "Ironic node rebalance race can lead to missing compute nodes in DB" [Undecided,In progress] - Assigned to Mark Goddard (mgoddard)
17:59:32 mgoddard melwitt: don't restart nova-compute? :/
17:59:45 mgoddard well that's more how not to hit it
18:00:01 mgoddard let me check my notes
18:00:43 mgoddard melwitt: also when you say I'm working on it, mostly I'm just waiting for reviews
18:01:38 mgoddard Nothing of much help in my notes
18:02:14 mgoddard Restarting nova-compute one by one might help, but it might also create new problems
18:02:38 mgoddard I think the approach we took was to apply those patches (backported to rocky)
18:02:44 mgoddard rather than workaround
18:03:38 melwitt mgoddard: gotcha ... thanks. I was looking through the lp bug trying to figure whether there's a workaround and wondered if you might already know
18:04:30 mgoddard sometimes you can do some DB hacking to change the compute node IDs. I can't remember if it works in this case
18:08:41 melwitt mgoddard: thanks. definitely will review your patches, I was trying to work out whether there's any workaround possible to stop the bleeding in the immediate term while the patches get reviewed
18:09:10 sean-k-mooney mgoddard: we have a customer that is seeing similar behavior when they do a ooo scale out.
18:10:08 mgoddard if it's a problem there might be something that could be worked out. It should be possible to reproduce with a multinode control plane and a few ironic nodes (possibly fake)
18:10:46 mgoddard keep restarting nova computes, and eventually the DB and placement will go astray
18:10:53 mgoddard anyways, need to head off now
18:10:55 mgoddard good luck!
18:11:05 melwitt thanks mgoddard
18:56:13 CeeMac What would cause a block device mapping failure when deploying a new instance? I see the volume creating fine in cinder and on the backend iscsi storage (zadara) but there seems to be something preventing the volume mounting to the vm. Just trti g to piece the interconnecting pieces together ready for another round of troubleshooting tomorrow
18:57:49 CeeMac This is on a new compute/storage node I added today. I've been through the configs again and nothing obvious jumps out at me as being missed. Running rocky btw.
19:03:47 melwitt CeeMac: there's likely error logging about it in nova-compute logs
19:08:07 CeeMac melwitt: yeah, I couldn't see anything specific beyond it timing out after x seconds etc
19:08:35 CeeMac I'll take another run at it and double check the logs in the morning
19:09:40 melwitt CeeMac: ok. I'd turn on debug logging (debug=True in nova.conf for nova-compute) and find which command times out. I'm guessing it will be something os-brick is trying to do. and then go from there
19:10:28 CeeMac The other weird thing I noticed was when I tried a live migration os-brick wasn't able to see the volume as there were no active iscsi sessions. Except I can see the server connecting to the storage and the volume getting created on the back end.
19:10:45 CeeMac Does os-brick log somewhere specific?
19:10:57 melwitt it logs in nova-compute but only at level DEBUG
19:11:04 CeeMac Right
19:11:13 CeeMac I'll check that first then, thanks
19:11:48 melwitt yeah, once you find what command is timing out or returning something unexpected, you can start trying to piece it together with things that might be happening with the backend
19:14:21 CeeMac melwitt: I'm hoping it'll be something obvious, it's a while since I last added a host and I've not made any changes to the config files that I can recall
19:17:22 melwitt I hope so too
19:52:23 openstackgerrit Merged openstack/os-traits master: Switch to hacking 2.x https://review.opendev.org/710275
20:45:30 efried Nova meeting in ~15 minutes in #openstack-meeting
21:25:33 david_4 Hi, I am new to Nova, I wanted add a scheduler filter and want to know how to test filters in nova-scheduler. I have an openstack env running with 3 head nodes and 3 work node, I am not using devstack.
21:34:07 david_4 any help please !!
22:40:19 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Fix H702 pep8 error with latest hacking https://review.opendev.org/710344
22:45:11 efried gmann: Is that the only thing you want in that patch? ^
22:45:15 efried if so, I'll fast approve
23:01:33 mriedem if david_4 ever comes back, point him at something like https://github.com/openstack/nova/blob/master/nova/tests/functional/test_json_filter.py as an example of how to test scheduler filters in functional tests - easier than getting a real environment setup (assuming he can navigate how to run functional tests)
23:02:08 mriedem smell ya later!
23:31:29 gmann efried: yeah, that is only error for now.
23:38:57 gmann efried: wait, few more seems. let me update.
#openstack-nova - 2020-02-28
00:20:37 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Fix pep8 error with latest hacking https://review.opendev.org/710344
01:02:53 openstackgerrit Merged openstack/nova master: Avoid allocation leak when deleting instance stuck in BUILD https://review.opendev.org/702368
03:23:43 openstackgerrit Ghanshyam Mann proposed openstack/nova master: Fix H702 pep8 error with latest hacking https://review.opendev.org/710344
03:27:34 gmann efried: restored to PS1. let's fix that for now. I will check on hacking job later.
07:58:47 openstackgerrit Johannes Kulik proposed openstack/nova master: Validate id as integer for os-aggregates https://review.opendev.org/710214

Earlier   Later