| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-27 | |||
| 17:27:50 | Sundar | I am trying to see if I am missing any error paths. AFAICS, we haven't missed any in the conductor. In the compute manager, if the reschedule takes the same code paths as the priginal schedule, we should be good. | |
| 17:27:58 | dansmith | right | |
| 17:28:18 | Sundar | Ok, then we are good | |
| 17:28:25 | dansmith | if your bind in reschedule in conductor succeeds, then it's a regular build after that | |
| 17:45:13 | openstackgerrit | Merged openstack/os-traits master: Remove use of six https://review.opendev.org/710274 | |
| 17:56:37 | openstack | Launchpad bug 1853009 in OpenStack Compute (nova) "Ironic node rebalance race can lead to missing compute nodes in DB" [Undecided,In progress] - Assigned to Mark Goddard (mgoddard) | |
| 17:56:37 | melwitt | mgoddard: do you know if there's any workaround for this bug you're working on? https://bugs.launchpad.net/nova/+bug/1853009 | |
| 17:59:32 | mgoddard | melwitt: don't restart nova-compute? :/ | |
| 17:59:45 | mgoddard | well that's more how not to hit it | |
| 18:00:01 | mgoddard | let me check my notes | |
| 18:00:43 | mgoddard | melwitt: also when you say I'm working on it, mostly I'm just waiting for reviews | |
| 18:01:38 | mgoddard | Nothing of much help in my notes | |
| 18:02:14 | mgoddard | Restarting nova-compute one by one might help, but it might also create new problems | |
| 18:02:38 | mgoddard | I think the approach we took was to apply those patches (backported to rocky) | |
| 18:02:44 | mgoddard | rather than workaround | |
| 18:03:38 | melwitt | mgoddard: gotcha ... thanks. I was looking through the lp bug trying to figure whether there's a workaround and wondered if you might already know | |
| 18:04:30 | mgoddard | sometimes you can do some DB hacking to change the compute node IDs. I can't remember if it works in this case | |
| 18:08:41 | melwitt | mgoddard: thanks. definitely will review your patches, I was trying to work out whether there's any workaround possible to stop the bleeding in the immediate term while the patches get reviewed | |
| 18:09:10 | sean-k-mooney | mgoddard: we have a customer that is seeing similar behavior when they do a ooo scale out. | |
| 18:10:08 | mgoddard | if it's a problem there might be something that could be worked out. It should be possible to reproduce with a multinode control plane and a few ironic nodes (possibly fake) | |
| 18:10:46 | mgoddard | keep restarting nova computes, and eventually the DB and placement will go astray | |
| 18:10:53 | mgoddard | anyways, need to head off now | |
| 18:10:55 | mgoddard | good luck! | |
| 18:11:05 | melwitt | thanks mgoddard | |
| 18:56:13 | CeeMac | What would cause a block device mapping failure when deploying a new instance? I see the volume creating fine in cinder and on the backend iscsi storage (zadara) but there seems to be something preventing the volume mounting to the vm. Just trti g to piece the interconnecting pieces together ready for another round of troubleshooting tomorrow | |
| 18:57:49 | CeeMac | This is on a new compute/storage node I added today. I've been through the configs again and nothing obvious jumps out at me as being missed. Running rocky btw. | |
| 19:03:47 | melwitt | CeeMac: there's likely error logging about it in nova-compute logs | |
| 19:08:07 | CeeMac | melwitt: yeah, I couldn't see anything specific beyond it timing out after x seconds etc | |
| 19:08:35 | CeeMac | I'll take another run at it and double check the logs in the morning | |
| 19:09:40 | melwitt | CeeMac: ok. I'd turn on debug logging (debug=True in nova.conf for nova-compute) and find which command times out. I'm guessing it will be something os-brick is trying to do. and then go from there | |
| 19:10:28 | CeeMac | The other weird thing I noticed was when I tried a live migration os-brick wasn't able to see the volume as there were no active iscsi sessions. Except I can see the server connecting to the storage and the volume getting created on the back end. | |
| 19:10:45 | CeeMac | Does os-brick log somewhere specific? | |
| 19:10:57 | melwitt | it logs in nova-compute but only at level DEBUG | |
| 19:11:04 | CeeMac | Right | |
| 19:11:13 | CeeMac | I'll check that first then, thanks | |
| 19:11:48 | melwitt | yeah, once you find what command is timing out or returning something unexpected, you can start trying to piece it together with things that might be happening with the backend | |
| 19:14:21 | CeeMac | melwitt: I'm hoping it'll be something obvious, it's a while since I last added a host and I've not made any changes to the config files that I can recall | |
| 19:17:22 | melwitt | I hope so too | |
| 19:52:23 | openstackgerrit | Merged openstack/os-traits master: Switch to hacking 2.x https://review.opendev.org/710275 | |
| 20:45:30 | efried | Nova meeting in ~15 minutes in #openstack-meeting | |
| 21:25:33 | david_4 | Hi, I am new to Nova, I wanted add a scheduler filter and want to know how to test filters in nova-scheduler. I have an openstack env running with 3 head nodes and 3 work node, I am not using devstack. | |
| 21:34:07 | david_4 | any help please !! | |
| 22:40:19 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Fix H702 pep8 error with latest hacking https://review.opendev.org/710344 | |
| 22:45:11 | efried | gmann: Is that the only thing you want in that patch? ^ | |
| 22:45:15 | efried | if so, I'll fast approve | |
| 23:01:33 | mriedem | if david_4 ever comes back, point him at something like https://github.com/openstack/nova/blob/master/nova/tests/functional/test_json_filter.py as an example of how to test scheduler filters in functional tests - easier than getting a real environment setup (assuming he can navigate how to run functional tests) | |
| 23:02:08 | mriedem | smell ya later! | |
| 23:31:29 | gmann | efried: yeah, that is only error for now. | |
| 23:38:57 | gmann | efried: wait, few more seems. let me update. | |
| #openstack-nova - 2020-02-28 | |||
| 00:20:37 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Fix pep8 error with latest hacking https://review.opendev.org/710344 | |
| 01:02:53 | openstackgerrit | Merged openstack/nova master: Avoid allocation leak when deleting instance stuck in BUILD https://review.opendev.org/702368 | |
| 03:23:43 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Fix H702 pep8 error with latest hacking https://review.opendev.org/710344 | |
| 03:27:34 | gmann | efried: restored to PS1. let's fix that for now. I will check on hacking job later. | |
| 07:58:47 | openstackgerrit | Johannes Kulik proposed openstack/nova master: Validate id as integer for os-aggregates https://review.opendev.org/710214 | |
| 08:36:31 | openstackgerrit | Brin Zhang proposed openstack/nova master: Remove fatal=False from os-instance-actions show API https://review.opendev.org/710411 | |
| 11:55:29 | openstackgerrit | Mark Goddard proposed openstack/nova master: Fix inactive session error in compute node creation https://review.opendev.org/695189 | |
| 12:22:30 | openstackgerrit | sean mooney proposed openstack/nova master: [DNM] testing removal of cyborg client singleton https://review.opendev.org/710443 | |
| 13:22:30 | brinzhang | stephenfin: For os-instance-action fatal=False in context.can(), https://review.opendev.org/#/c/707777/2/nova/tests/unit/policies/test_instance_actions.py, I give a solution, can you review it if you are free? | |
| 13:23:16 | brinzhang | stephenfin: This patch done of that issue: https://review.opendev.org/#/c/710411/ | |
| 13:41:44 | stephenfin | brinzhang: looking | |
| 14:02:23 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add test coverage of existing os-instance-actions policies https://review.opendev.org/707777 | |
| 14:09:05 | brinzhang | stephenfin:I also update the test coverage patch https://review.opendev.org/#/c/707777/3/nova/tests/unit/policies/test_instance_actions.py@117, and I also cannot pass the test case, that cannot get what I want to get result, and I dont know why :(, error in my local is http://paste.openstack.org/show/790109/ | |
| 14:10:07 | brinzhang | stephenfin: Hope you can help, regarding this use case, thanks | |
| 14:31:03 | CeeMac | I'm having a major issues with block device mappings failing on new instance creation since I added 2 new compute/storage nodes yesterday | |
| 14:32:34 | CeeMac | i've enabled debug in the nova conf but that doesn't really add any extra information unfortunately, beyond waiting for block_device_map_created | |
| 14:33:31 | CeeMac | is it os-brick that creates the block device map? As there are no os-brick specific errors in the log. The volume is definitely being created, I can see this on the backend and I can see it under volumes, so this seems linked specifically to the block device mapping procress | |
| 14:33:42 | CeeMac | feeling a little lost on how best to proceed with troubleshooting | |
| 14:35:19 | CeeMac | actually, I tell a lie. The volumes are in an error state too, although I do see them getting created on the backend storage. | |
| 15:14:01 | openstackgerrit | Sérgio Filipe Marques Manso proposed openstack/nova master: test https://review.opendev.org/710466 | |
| 15:14:01 | openstackgerrit | Sérgio Filipe Marques Manso proposed openstack/nova master: test https://review.opendev.org/710466 | |
| 15:24:55 | efried | lyarwood: any chance you can help out CeeMac? (See logs from ~1h ago) | |
| 15:34:10 | CeeMac | efried: I think I may be making progress, and I may have missed a driver install out of my build process. Just working on that now. I've disabled cinder-volume on the new nodes and the rest of the stack is working again | |
| 15:35:08 | efried | Cool. (I don't have the chops to help you out -- or really even understand what you're saying :P -- but lyarwood is our resident storage-in-nova expert.) | |
| 15:40:39 | lyarwood | efried: just on a call now, will take a look once this is over | |
| 15:41:18 | efried | lyarwood: Thanks (I think CeeMac is okay for the moment, but I'll step aside) | |
| 15:49:38 | CeeMac | thanks for the intro/forward efried :) lyarwood I'm just rebooting the nodes after updating drivers for the storage interfaces, will let you know if that's fixed things once they're back up and I've retested | |
| 16:04:18 | lyarwood | CeeMac: ack np, let me know if you're still having issues after the compute reboots. | |
| 16:54:24 | CeeMac | lyarwood: we're all good thanks, was definitely me not documenting a vital step in the build process! Thanks anyway though :) | |
| 17:01:33 | lyarwood | CeeMac: haha no issues | |
| 17:08:45 | spatel | sean-k-mooney: good afternoon... | |
| 17:09:10 | spatel | I have quick question related CPUTopology with KVM. | |
| 17:09:32 | spatel | This is my Physical compute machine - http://paste.openstack.org/show/790120/ | |
| 17:09:50 | spatel | This is my Virtual machine running on that compute - http://paste.openstack.org/show/790121/ | |
| 17:10:53 | spatel | If you see physical machine sharing L1 cache but in Virtual machine each PU has own cache? | |
| 17:11:41 | spatel | I have compare this behavior with AWS and i am seeing they are correct and their output is very true like Physical | |
| 18:12:44 | spatel | How do i pass "<cache mode='passthrough'/>" to kvm | |
| 18:30:06 | melwitt | spatel: I don't know, but this is the closest doc I could find https://docs.openstack.org/nova/latest/admin/configuration/hypervisor-kvm.html#host-pass-through | |
| 18:31:25 | spatel | I already have host-pass-through on all my guest VM but its missing cache mode, that is why my guest VM showing wrong cache presentation | |
| 18:31:39 | melwitt | ah ok | |
| 18:31:56 | melwitt | only other thing I see is this but there's no 'passthrough' https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.disk_cachemodes | |
| 18:32:08 | spatel | This is what i have | |
| 18:32:38 | spatel | if you are running openstack then can you check your CPU Topology? | |
| 18:33:57 | melwitt | sorry, you'll probably need sean-k-mooney again for this one | |
| 18:34:37 | spatel | yes.. just trying to understand what is wrong with my KVM, i have check with other cloud provide who running KVM they have correct CPU Topo | |
| 18:34:45 | spatel | that means something is missing myside.. | |
| 20:08:06 | openstackgerrit | sean mooney proposed openstack/nova master: Provider Config File: YAML file loading and schema validation https://review.opendev.org/673341 | |
| 20:08:07 | openstackgerrit | sean mooney proposed openstack/nova master: Provider Config File: Function to further validate and retrieve configs https://review.opendev.org/676029 | |
| 21:13:45 | openstackgerrit | Merged openstack/nova master: Fix H702 pep8 error with latest hacking https://review.opendev.org/710344 | |
| #openstack-nova - 2020-02-29 | |||
| 00:27:18 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add test coverage of existing os-instance-actions policies https://review.opendev.org/707777 | |
| #openstack-nova - 2020-03-02 | |||
| 02:01:07 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Define Cyborg ARQ binding notification event. https://review.opendev.org/692707 | |