| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-27 | |||
| 17:05:55 | Sundar | dansmith: In https://review.opendev.org/#/c/631244/63/nova/conductor/manager.py@848, is there any resource to be cleaned up in the exception path? | |
| 17:07:29 | dansmith | Sundar: seems like there's network and block devices to clean up, but I'm not sure why the error paths above aren't doing that | |
| 17:07:37 | dansmith | Sundar: should you be deleting arqs? | |
| 17:07:55 | dansmith | or are you assuming those get deleted when the error'd instance gets deleted? | |
| 17:08:19 | dansmith | Sundar: there is a _cleanup_after_reschedule_fails() method in there... | |
| 17:08:23 | sean-k-mooney | gibi: ack that is fine. i likely will not be ready until then anyway | |
| 17:09:02 | dansmith | er, cleanup_when_reschedule_fails | |
| 17:09:10 | dansmith | which cleans up allocated network resources at least | |
| 17:09:25 | Sundar | Well, not in this patch for sure, because delete ARQ functionality comes later. However, we should do that in the delete patch because Create ARQs may succeed but bind may throw an exception potentially. Then, Cyborg should be informed to delete the created ARQs. | |
| 17:09:54 | dansmith | Sundar: okay your existing code won't even leave the instance in error state, which is bad.. but that cleanup method will do that for you | |
| 17:10:44 | Sundar | dansmith: Yes, I see it. Good. Thanks. | |
| 17:11:39 | Sundar | dansmith: I could tie the delete ARQs to _cleanup_when_reschedule_fails(). I'll check to see what other impact it has. | |
| 17:12:18 | dansmith | Sundar: if deleting the instance in un-scheduled state will clean those up, then it's probably okay to punt to that.. the only problem might be if the ARQs are holding some resource for the dead instance | |
| 17:14:47 | Sundar | dansmith: If the bind succeeds, then ARQs are holding real resources. So, if some other task fails during rescheduling, then Cyborg should be told to delete the ARQs. I think the changes we already have should take care of that. | |
| 17:17:27 | dansmith | Sundar: right but the bind is right before the call back to compute, so if that fails, no bind was done, and if it has failed before that, we're already unbound from the original host, so not holding resource... is that right? | |
| 17:21:02 | Sundar | fail. It is this second path that I am not terribly sure about. | |
| 17:21:02 | Sundar | dansmith: bind is async. The call here just kicks off the process, and the actual bind may succeed or fail after that. If the bind fails, either Cyborg sends a failure notification or Nova gets a timeout (in corner cases). That is already handled. if the bind succeeds, Cyborg has allocated and prepared resources. But something else in Nova may | |
| 17:21:36 | dansmith | I think you're missing my point | |
| 17:21:56 | dansmith | or maybe you're talking about the reschedule racing with the *original* bind? | |
| 17:24:51 | Sundar | dansmith: I am talking of the bind during the rescheduling in https://review.opendev.org/#/c/631244/63/nova/conductor/manager.py@842 . Yes, if there is a failure in _create_and_bind_arqs( ) or earlier, no resources are allocated. | |
| 17:25:33 | dansmith | Sundar: so you're concerned about what happens if the bind during reschedule fails and not knowing whether things are actually bound or not? | |
| 17:25:48 | dansmith | if so that's... pretty scary | |
| 17:25:51 | Sundar | I was talking of the scenario where _create_and_bind_arqs() succeeds, Nova goes ahead with the reschedule, CYborg prepares devices in the new host, and then things go south. | |
| 17:26:00 | Sundar | Things go south in Nova | |
| 17:26:20 | Sundar | So, every failure path in reschedule must call Cyborg to delete ARQs | |
| 17:26:29 | dansmith | depends on where it goes south, but there's not much we can do about it | |
| 17:26:45 | dansmith | try to delete the arqs when we can in error paths, | |
| 17:27:02 | dansmith | but as long as the instance delete will clean things up gracefully, that's the safety net | |
| 17:27:49 | dansmith | after your bind in the reschedule path, the rest of it is identical to the regular build process | |
| 17:27:50 | Sundar | I am trying to see if I am missing any error paths. AFAICS, we haven't missed any in the conductor. In the compute manager, if the reschedule takes the same code paths as the priginal schedule, we should be good. | |
| 17:27:58 | dansmith | right | |
| 17:28:18 | Sundar | Ok, then we are good | |
| 17:28:25 | dansmith | if your bind in reschedule in conductor succeeds, then it's a regular build after that | |
| 17:45:13 | openstackgerrit | Merged openstack/os-traits master: Remove use of six https://review.opendev.org/710274 | |
| 17:56:37 | openstack | Launchpad bug 1853009 in OpenStack Compute (nova) "Ironic node rebalance race can lead to missing compute nodes in DB" [Undecided,In progress] - Assigned to Mark Goddard (mgoddard) | |
| 17:56:37 | melwitt | mgoddard: do you know if there's any workaround for this bug you're working on? https://bugs.launchpad.net/nova/+bug/1853009 | |
| 17:59:32 | mgoddard | melwitt: don't restart nova-compute? :/ | |
| 17:59:45 | mgoddard | well that's more how not to hit it | |
| 18:00:01 | mgoddard | let me check my notes | |
| 18:00:43 | mgoddard | melwitt: also when you say I'm working on it, mostly I'm just waiting for reviews | |
| 18:01:38 | mgoddard | Nothing of much help in my notes | |
| 18:02:14 | mgoddard | Restarting nova-compute one by one might help, but it might also create new problems | |
| 18:02:38 | mgoddard | I think the approach we took was to apply those patches (backported to rocky) | |
| 18:02:44 | mgoddard | rather than workaround | |
| 18:03:38 | melwitt | mgoddard: gotcha ... thanks. I was looking through the lp bug trying to figure whether there's a workaround and wondered if you might already know | |
| 18:04:30 | mgoddard | sometimes you can do some DB hacking to change the compute node IDs. I can't remember if it works in this case | |
| 18:08:41 | melwitt | mgoddard: thanks. definitely will review your patches, I was trying to work out whether there's any workaround possible to stop the bleeding in the immediate term while the patches get reviewed | |
| 18:09:10 | sean-k-mooney | mgoddard: we have a customer that is seeing similar behavior when they do a ooo scale out. | |
| 18:10:08 | mgoddard | if it's a problem there might be something that could be worked out. It should be possible to reproduce with a multinode control plane and a few ironic nodes (possibly fake) | |
| 18:10:46 | mgoddard | keep restarting nova computes, and eventually the DB and placement will go astray | |
| 18:10:53 | mgoddard | anyways, need to head off now | |
| 18:10:55 | mgoddard | good luck! | |
| 18:11:05 | melwitt | thanks mgoddard | |
| 18:56:13 | CeeMac | What would cause a block device mapping failure when deploying a new instance? I see the volume creating fine in cinder and on the backend iscsi storage (zadara) but there seems to be something preventing the volume mounting to the vm. Just trti g to piece the interconnecting pieces together ready for another round of troubleshooting tomorrow | |
| 18:57:49 | CeeMac | This is on a new compute/storage node I added today. I've been through the configs again and nothing obvious jumps out at me as being missed. Running rocky btw. | |
| 19:03:47 | melwitt | CeeMac: there's likely error logging about it in nova-compute logs | |
| 19:08:07 | CeeMac | melwitt: yeah, I couldn't see anything specific beyond it timing out after x seconds etc | |
| 19:08:35 | CeeMac | I'll take another run at it and double check the logs in the morning | |
| 19:09:40 | melwitt | CeeMac: ok. I'd turn on debug logging (debug=True in nova.conf for nova-compute) and find which command times out. I'm guessing it will be something os-brick is trying to do. and then go from there | |
| 19:10:28 | CeeMac | The other weird thing I noticed was when I tried a live migration os-brick wasn't able to see the volume as there were no active iscsi sessions. Except I can see the server connecting to the storage and the volume getting created on the back end. | |
| 19:10:45 | CeeMac | Does os-brick log somewhere specific? | |
| 19:10:57 | melwitt | it logs in nova-compute but only at level DEBUG | |
| 19:11:04 | CeeMac | Right | |
| 19:11:13 | CeeMac | I'll check that first then, thanks | |
| 19:11:48 | melwitt | yeah, once you find what command is timing out or returning something unexpected, you can start trying to piece it together with things that might be happening with the backend | |
| 19:14:21 | CeeMac | melwitt: I'm hoping it'll be something obvious, it's a while since I last added a host and I've not made any changes to the config files that I can recall | |
| 19:17:22 | melwitt | I hope so too | |
| 19:52:23 | openstackgerrit | Merged openstack/os-traits master: Switch to hacking 2.x https://review.opendev.org/710275 | |
| 20:45:30 | efried | Nova meeting in ~15 minutes in #openstack-meeting | |
| 21:25:33 | david_4 | Hi, I am new to Nova, I wanted add a scheduler filter and want to know how to test filters in nova-scheduler. I have an openstack env running with 3 head nodes and 3 work node, I am not using devstack. | |
| 21:34:07 | david_4 | any help please !! | |
| 22:40:19 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Fix H702 pep8 error with latest hacking https://review.opendev.org/710344 | |
| 22:45:11 | efried | gmann: Is that the only thing you want in that patch? ^ | |
| 22:45:15 | efried | if so, I'll fast approve | |
| 23:01:33 | mriedem | if david_4 ever comes back, point him at something like https://github.com/openstack/nova/blob/master/nova/tests/functional/test_json_filter.py as an example of how to test scheduler filters in functional tests - easier than getting a real environment setup (assuming he can navigate how to run functional tests) | |
| 23:02:08 | mriedem | smell ya later! | |
| 23:31:29 | gmann | efried: yeah, that is only error for now. | |
| 23:38:57 | gmann | efried: wait, few more seems. let me update. | |
| #openstack-nova - 2020-02-28 | |||
| 00:20:37 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Fix pep8 error with latest hacking https://review.opendev.org/710344 | |
| 01:02:53 | openstackgerrit | Merged openstack/nova master: Avoid allocation leak when deleting instance stuck in BUILD https://review.opendev.org/702368 | |
| 03:23:43 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Fix H702 pep8 error with latest hacking https://review.opendev.org/710344 | |
| 03:27:34 | gmann | efried: restored to PS1. let's fix that for now. I will check on hacking job later. | |
| 07:58:47 | openstackgerrit | Johannes Kulik proposed openstack/nova master: Validate id as integer for os-aggregates https://review.opendev.org/710214 | |
| 08:36:31 | openstackgerrit | Brin Zhang proposed openstack/nova master: Remove fatal=False from os-instance-actions show API https://review.opendev.org/710411 | |
| 11:55:29 | openstackgerrit | Mark Goddard proposed openstack/nova master: Fix inactive session error in compute node creation https://review.opendev.org/695189 | |
| 12:22:30 | openstackgerrit | sean mooney proposed openstack/nova master: [DNM] testing removal of cyborg client singleton https://review.opendev.org/710443 | |
| 13:22:30 | brinzhang | stephenfin: For os-instance-action fatal=False in context.can(), https://review.opendev.org/#/c/707777/2/nova/tests/unit/policies/test_instance_actions.py, I give a solution, can you review it if you are free? | |
| 13:23:16 | brinzhang | stephenfin: This patch done of that issue: https://review.opendev.org/#/c/710411/ | |
| 13:41:44 | stephenfin | brinzhang: looking | |
| 14:02:23 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add test coverage of existing os-instance-actions policies https://review.opendev.org/707777 | |
| 14:09:05 | brinzhang | stephenfin:I also update the test coverage patch https://review.opendev.org/#/c/707777/3/nova/tests/unit/policies/test_instance_actions.py@117, and I also cannot pass the test case, that cannot get what I want to get result, and I dont know why :(, error in my local is http://paste.openstack.org/show/790109/ | |
| 14:10:07 | brinzhang | stephenfin: Hope you can help, regarding this use case, thanks | |
| 14:31:03 | CeeMac | I'm having a major issues with block device mappings failing on new instance creation since I added 2 new compute/storage nodes yesterday | |
| 14:32:34 | CeeMac | i've enabled debug in the nova conf but that doesn't really add any extra information unfortunately, beyond waiting for block_device_map_created | |
| 14:33:31 | CeeMac | is it os-brick that creates the block device map? As there are no os-brick specific errors in the log. The volume is definitely being created, I can see this on the backend and I can see it under volumes, so this seems linked specifically to the block device mapping procress | |
| 14:33:42 | CeeMac | feeling a little lost on how best to proceed with troubleshooting | |
| 14:35:19 | CeeMac | actually, I tell a lie. The volumes are in an error state too, although I do see them getting created on the backend storage. | |
| 15:14:01 | openstackgerrit | Sérgio Filipe Marques Manso proposed openstack/nova master: test https://review.opendev.org/710466 | |
| 15:14:01 | openstackgerrit | Sérgio Filipe Marques Manso proposed openstack/nova master: test https://review.opendev.org/710466 | |
| 15:24:55 | efried | lyarwood: any chance you can help out CeeMac? (See logs from ~1h ago) | |