| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-27 | |||
| 16:34:56 | openstackgerrit | Merged openstack/os-traits master: setup.cfg: Remove unused 'build_sphinx' configuration https://review.opendev.org/704535 | |
| 16:40:46 | gibi | sean-k-mooney: re: provider config. I have to take tomorrow off so I will look at your update earliest on Monday | |
| 16:48:04 | Sundar | dansmith, gibi, sean-k-mooney, efried: On the rescheduling front, I don't have a multi-node env to test it. It is not going to be easy for me to get the lab resources, even without FPGAs, to set that up. Plus, I have other things I need to focus on. Any ideas on bringing this to a timely closure? | |
| 16:48:27 | dansmith | Sundar: functional tests? | |
| 16:48:42 | efried | Sundar: Also, sean-k-mooney set up a multinode CI job. | |
| 16:49:21 | dansmith | multinode ci might be hard to trigger a reschedule though | |
| 16:49:54 | dansmith | Sundar: tangential to this part, did you see my comment about moving the event to the front of the queue, below the -2'd ksa patch? | |
| 16:50:52 | Sundar | dansmith: Yes, I saw that. I have moved it before the ksa patch in my local env. The next commit should have it. | |
| 16:50:59 | dansmith | Sundar: cool | |
| 16:51:14 | Sundar | dansmith: With functional tests, I can check if the logs show rescheduling messages, and that the tests pass. Is there anything more you are looking for? | |
| 16:51:49 | efried | You can do better than that to prove a reschedule occurred: use the fixture that makes the initial host predictable and prove that it lands on a different one. | |
| 16:52:11 | gibi | efried ++ | |
| 16:52:11 | dansmith | Sundar: "tests passing" isn't a bar to meet, "tests validating something useful" is | |
| 16:52:12 | efried | there should be some good examples | |
| 16:52:15 | dansmith | yep, what efried said | |
| 16:52:48 | Sundar | efried: Thanks, will look for that. If you happen to remember the(rough) name of the fixture, off the top of your head, I'll start with that. | |
| 16:53:00 | dansmith | Sundar: what specifically are you stuck on? there's lots of feedback to work on without much response from you yet that I can see.. is there something specific that you're blocked on? | |
| 16:53:46 | Sundar | dansmith: Not sure that any of my changes will actually work in the real world, without a multinode env. I tried to get a 2nd server but it will take time, if at all. | |
| 16:53:53 | efried | Sundar: HostNameWeigherFixture I think | |
| 16:53:58 | gibi | Sundar: HostNameWeigherFixture makes the host selection peridctable | |
| 16:54:11 | dansmith | Sundar: functional tests man :) | |
| 16:54:12 | Sundar | efried, gibi: Thanks, got it | |
| 16:54:50 | dansmith | Sundar: if you have reasonable functional tests that don't mock out insane levels of things and validate reasonable conditions, then any real world failure we find later is understandable | |
| 16:55:53 | dansmith | Sundar: test_servers.py has a bunch of reschedule-specific tests | |
| 17:00:36 | Sundar | dansmith: Ok, let's just focus on functional tests, then. BTW, https://review.opendev.org/gitweb?p=openstack/nova.git;f=nova/conductor/manager.py;hb=refs/changes/44/631244/63#l602 is specific to the resheduling path AFAICS. is there any other path, like rebuild/resize/whatever, that invokes this? | |
| 17:01:46 | dansmith | Sundar: only if one of those things needs a reschedule, AFAIK | |
| 17:04:14 | Sundar | Ok. BTW, I will be travelling over the next couple of days. I will respond with the comments I accepted by today, and will post the rest in a few days. | |
| 17:04:36 | dansmith | okay | |
| 17:05:55 | Sundar | dansmith: In https://review.opendev.org/#/c/631244/63/nova/conductor/manager.py@848, is there any resource to be cleaned up in the exception path? | |
| 17:07:29 | dansmith | Sundar: seems like there's network and block devices to clean up, but I'm not sure why the error paths above aren't doing that | |
| 17:07:37 | dansmith | Sundar: should you be deleting arqs? | |
| 17:07:55 | dansmith | or are you assuming those get deleted when the error'd instance gets deleted? | |
| 17:08:19 | dansmith | Sundar: there is a _cleanup_after_reschedule_fails() method in there... | |
| 17:08:23 | sean-k-mooney | gibi: ack that is fine. i likely will not be ready until then anyway | |
| 17:09:02 | dansmith | er, cleanup_when_reschedule_fails | |
| 17:09:10 | dansmith | which cleans up allocated network resources at least | |
| 17:09:25 | Sundar | Well, not in this patch for sure, because delete ARQ functionality comes later. However, we should do that in the delete patch because Create ARQs may succeed but bind may throw an exception potentially. Then, Cyborg should be informed to delete the created ARQs. | |
| 17:09:54 | dansmith | Sundar: okay your existing code won't even leave the instance in error state, which is bad.. but that cleanup method will do that for you | |
| 17:10:44 | Sundar | dansmith: Yes, I see it. Good. Thanks. | |
| 17:11:39 | Sundar | dansmith: I could tie the delete ARQs to _cleanup_when_reschedule_fails(). I'll check to see what other impact it has. | |
| 17:12:18 | dansmith | Sundar: if deleting the instance in un-scheduled state will clean those up, then it's probably okay to punt to that.. the only problem might be if the ARQs are holding some resource for the dead instance | |
| 17:14:47 | Sundar | dansmith: If the bind succeeds, then ARQs are holding real resources. So, if some other task fails during rescheduling, then Cyborg should be told to delete the ARQs. I think the changes we already have should take care of that. | |
| 17:17:27 | dansmith | Sundar: right but the bind is right before the call back to compute, so if that fails, no bind was done, and if it has failed before that, we're already unbound from the original host, so not holding resource... is that right? | |
| 17:21:02 | Sundar | dansmith: bind is async. The call here just kicks off the process, and the actual bind may succeed or fail after that. If the bind fails, either Cyborg sends a failure notification or Nova gets a timeout (in corner cases). That is already handled. if the bind succeeds, Cyborg has allocated and prepared resources. But something else in Nova may | |
| 17:21:02 | Sundar | fail. It is this second path that I am not terribly sure about. | |
| 17:21:36 | dansmith | I think you're missing my point | |
| 17:21:56 | dansmith | or maybe you're talking about the reschedule racing with the *original* bind? | |
| 17:24:51 | Sundar | dansmith: I am talking of the bind during the rescheduling in https://review.opendev.org/#/c/631244/63/nova/conductor/manager.py@842 . Yes, if there is a failure in _create_and_bind_arqs( ) or earlier, no resources are allocated. | |
| 17:25:33 | dansmith | Sundar: so you're concerned about what happens if the bind during reschedule fails and not knowing whether things are actually bound or not? | |
| 17:25:48 | dansmith | if so that's... pretty scary | |
| 17:25:51 | Sundar | I was talking of the scenario where _create_and_bind_arqs() succeeds, Nova goes ahead with the reschedule, CYborg prepares devices in the new host, and then things go south. | |
| 17:26:00 | Sundar | Things go south in Nova | |
| 17:26:20 | Sundar | So, every failure path in reschedule must call Cyborg to delete ARQs | |
| 17:26:29 | dansmith | depends on where it goes south, but there's not much we can do about it | |
| 17:26:45 | dansmith | try to delete the arqs when we can in error paths, | |
| 17:27:02 | dansmith | but as long as the instance delete will clean things up gracefully, that's the safety net | |
| 17:27:49 | dansmith | after your bind in the reschedule path, the rest of it is identical to the regular build process | |
| 17:27:50 | Sundar | I am trying to see if I am missing any error paths. AFAICS, we haven't missed any in the conductor. In the compute manager, if the reschedule takes the same code paths as the priginal schedule, we should be good. | |
| 17:27:58 | dansmith | right | |
| 17:28:18 | Sundar | Ok, then we are good | |
| 17:28:25 | dansmith | if your bind in reschedule in conductor succeeds, then it's a regular build after that | |
| 17:45:13 | openstackgerrit | Merged openstack/os-traits master: Remove use of six https://review.opendev.org/710274 | |
| 17:56:37 | melwitt | mgoddard: do you know if there's any workaround for this bug you're working on? https://bugs.launchpad.net/nova/+bug/1853009 | |
| 17:56:37 | openstack | Launchpad bug 1853009 in OpenStack Compute (nova) "Ironic node rebalance race can lead to missing compute nodes in DB" [Undecided,In progress] - Assigned to Mark Goddard (mgoddard) | |
| 17:59:32 | mgoddard | melwitt: don't restart nova-compute? :/ | |
| 17:59:45 | mgoddard | well that's more how not to hit it | |
| 18:00:01 | mgoddard | let me check my notes | |
| 18:00:43 | mgoddard | melwitt: also when you say I'm working on it, mostly I'm just waiting for reviews | |
| 18:01:38 | mgoddard | Nothing of much help in my notes | |
| 18:02:14 | mgoddard | Restarting nova-compute one by one might help, but it might also create new problems | |
| 18:02:38 | mgoddard | I think the approach we took was to apply those patches (backported to rocky) | |
| 18:02:44 | mgoddard | rather than workaround | |
| 18:03:38 | melwitt | mgoddard: gotcha ... thanks. I was looking through the lp bug trying to figure whether there's a workaround and wondered if you might already know | |
| 18:04:30 | mgoddard | sometimes you can do some DB hacking to change the compute node IDs. I can't remember if it works in this case | |
| 18:08:41 | melwitt | mgoddard: thanks. definitely will review your patches, I was trying to work out whether there's any workaround possible to stop the bleeding in the immediate term while the patches get reviewed | |
| 18:09:10 | sean-k-mooney | mgoddard: we have a customer that is seeing similar behavior when they do a ooo scale out. | |
| 18:10:08 | mgoddard | if it's a problem there might be something that could be worked out. It should be possible to reproduce with a multinode control plane and a few ironic nodes (possibly fake) | |
| 18:10:46 | mgoddard | keep restarting nova computes, and eventually the DB and placement will go astray | |
| 18:10:53 | mgoddard | anyways, need to head off now | |
| 18:10:55 | mgoddard | good luck! | |
| 18:11:05 | melwitt | thanks mgoddard | |
| 18:56:13 | CeeMac | What would cause a block device mapping failure when deploying a new instance? I see the volume creating fine in cinder and on the backend iscsi storage (zadara) but there seems to be something preventing the volume mounting to the vm. Just trti g to piece the interconnecting pieces together ready for another round of troubleshooting tomorrow | |
| 18:57:49 | CeeMac | This is on a new compute/storage node I added today. I've been through the configs again and nothing obvious jumps out at me as being missed. Running rocky btw. | |
| 19:03:47 | melwitt | CeeMac: there's likely error logging about it in nova-compute logs | |
| 19:08:07 | CeeMac | melwitt: yeah, I couldn't see anything specific beyond it timing out after x seconds etc | |
| 19:08:35 | CeeMac | I'll take another run at it and double check the logs in the morning | |
| 19:09:40 | melwitt | CeeMac: ok. I'd turn on debug logging (debug=True in nova.conf for nova-compute) and find which command times out. I'm guessing it will be something os-brick is trying to do. and then go from there | |
| 19:10:28 | CeeMac | The other weird thing I noticed was when I tried a live migration os-brick wasn't able to see the volume as there were no active iscsi sessions. Except I can see the server connecting to the storage and the volume getting created on the back end. | |
| 19:10:45 | CeeMac | Does os-brick log somewhere specific? | |
| 19:10:57 | melwitt | it logs in nova-compute but only at level DEBUG | |
| 19:11:04 | CeeMac | Right | |
| 19:11:13 | CeeMac | I'll check that first then, thanks | |
| 19:11:48 | melwitt | yeah, once you find what command is timing out or returning something unexpected, you can start trying to piece it together with things that might be happening with the backend | |
| 19:14:21 | CeeMac | melwitt: I'm hoping it'll be something obvious, it's a while since I last added a host and I've not made any changes to the config files that I can recall | |
| 19:17:22 | melwitt | I hope so too | |
| 19:52:23 | openstackgerrit | Merged openstack/os-traits master: Switch to hacking 2.x https://review.opendev.org/710275 | |
| 20:45:30 | efried | Nova meeting in ~15 minutes in #openstack-meeting | |
| 21:25:33 | david_4 | Hi, I am new to Nova, I wanted add a scheduler filter and want to know how to test filters in nova-scheduler. I have an openstack env running with 3 head nodes and 3 work node, I am not using devstack. | |
| 21:34:07 | david_4 | any help please !! | |
| 22:40:19 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Fix H702 pep8 error with latest hacking https://review.opendev.org/710344 | |