| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2020-02-27 | |||
| 18:03:38 | melwitt | mgoddard: gotcha ... thanks. I was looking through the lp bug trying to figure whether there's a workaround and wondered if you might already know | |
| 18:04:30 | mgoddard | sometimes you can do some DB hacking to change the compute node IDs. I can't remember if it works in this case | |
| 18:08:41 | melwitt | mgoddard: thanks. definitely will review your patches, I was trying to work out whether there's any workaround possible to stop the bleeding in the immediate term while the patches get reviewed | |
| 18:09:10 | sean-k-mooney | mgoddard: we have a customer that is seeing similar behavior when they do a ooo scale out. | |
| 18:10:08 | mgoddard | if it's a problem there might be something that could be worked out. It should be possible to reproduce with a multinode control plane and a few ironic nodes (possibly fake) | |
| 18:10:46 | mgoddard | keep restarting nova computes, and eventually the DB and placement will go astray | |
| 18:10:53 | mgoddard | anyways, need to head off now | |
| 18:10:55 | mgoddard | good luck! | |
| 18:11:05 | melwitt | thanks mgoddard | |
| 18:56:13 | CeeMac | What would cause a block device mapping failure when deploying a new instance? I see the volume creating fine in cinder and on the backend iscsi storage (zadara) but there seems to be something preventing the volume mounting to the vm. Just trti g to piece the interconnecting pieces together ready for another round of troubleshooting tomorrow | |
| 18:57:49 | CeeMac | This is on a new compute/storage node I added today. I've been through the configs again and nothing obvious jumps out at me as being missed. Running rocky btw. | |
| 19:03:47 | melwitt | CeeMac: there's likely error logging about it in nova-compute logs | |
| 19:08:07 | CeeMac | melwitt: yeah, I couldn't see anything specific beyond it timing out after x seconds etc | |
| 19:08:35 | CeeMac | I'll take another run at it and double check the logs in the morning | |
| 19:09:40 | melwitt | CeeMac: ok. I'd turn on debug logging (debug=True in nova.conf for nova-compute) and find which command times out. I'm guessing it will be something os-brick is trying to do. and then go from there | |
| 19:10:28 | CeeMac | The other weird thing I noticed was when I tried a live migration os-brick wasn't able to see the volume as there were no active iscsi sessions. Except I can see the server connecting to the storage and the volume getting created on the back end. | |
| 19:10:45 | CeeMac | Does os-brick log somewhere specific? | |
| 19:10:57 | melwitt | it logs in nova-compute but only at level DEBUG | |
| 19:11:04 | CeeMac | Right | |
| 19:11:13 | CeeMac | I'll check that first then, thanks | |
| 19:11:48 | melwitt | yeah, once you find what command is timing out or returning something unexpected, you can start trying to piece it together with things that might be happening with the backend | |
| 19:14:21 | CeeMac | melwitt: I'm hoping it'll be something obvious, it's a while since I last added a host and I've not made any changes to the config files that I can recall | |
| 19:17:22 | melwitt | I hope so too | |
| 19:52:23 | openstackgerrit | Merged openstack/os-traits master: Switch to hacking 2.x https://review.opendev.org/710275 | |
| 20:45:30 | efried | Nova meeting in ~15 minutes in #openstack-meeting | |
| 21:25:33 | david_4 | Hi, I am new to Nova, I wanted add a scheduler filter and want to know how to test filters in nova-scheduler. I have an openstack env running with 3 head nodes and 3 work node, I am not using devstack. | |
| 21:34:07 | david_4 | any help please !! | |
| 22:40:19 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Fix H702 pep8 error with latest hacking https://review.opendev.org/710344 | |
| 22:45:11 | efried | gmann: Is that the only thing you want in that patch? ^ | |
| 22:45:15 | efried | if so, I'll fast approve | |
| 23:01:33 | mriedem | if david_4 ever comes back, point him at something like https://github.com/openstack/nova/blob/master/nova/tests/functional/test_json_filter.py as an example of how to test scheduler filters in functional tests - easier than getting a real environment setup (assuming he can navigate how to run functional tests) | |
| 23:02:08 | mriedem | smell ya later! | |
| 23:31:29 | gmann | efried: yeah, that is only error for now. | |
| 23:38:57 | gmann | efried: wait, few more seems. let me update. | |
| #openstack-nova - 2020-02-28 | |||
| 00:20:37 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Fix pep8 error with latest hacking https://review.opendev.org/710344 | |
| 01:02:53 | openstackgerrit | Merged openstack/nova master: Avoid allocation leak when deleting instance stuck in BUILD https://review.opendev.org/702368 | |
| 03:23:43 | openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Fix H702 pep8 error with latest hacking https://review.opendev.org/710344 | |
| 03:27:34 | gmann | efried: restored to PS1. let's fix that for now. I will check on hacking job later. | |
| 07:58:47 | openstackgerrit | Johannes Kulik proposed openstack/nova master: Validate id as integer for os-aggregates https://review.opendev.org/710214 | |
| 08:36:31 | openstackgerrit | Brin Zhang proposed openstack/nova master: Remove fatal=False from os-instance-actions show API https://review.opendev.org/710411 | |
| 11:55:29 | openstackgerrit | Mark Goddard proposed openstack/nova master: Fix inactive session error in compute node creation https://review.opendev.org/695189 | |
| 12:22:30 | openstackgerrit | sean mooney proposed openstack/nova master: [DNM] testing removal of cyborg client singleton https://review.opendev.org/710443 | |
| 13:22:30 | brinzhang | stephenfin: For os-instance-action fatal=False in context.can(), https://review.opendev.org/#/c/707777/2/nova/tests/unit/policies/test_instance_actions.py, I give a solution, can you review it if you are free? | |
| 13:23:16 | brinzhang | stephenfin: This patch done of that issue: https://review.opendev.org/#/c/710411/ | |
| 13:41:44 | stephenfin | brinzhang: looking | |
| 14:02:23 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add test coverage of existing os-instance-actions policies https://review.opendev.org/707777 | |
| 14:09:05 | brinzhang | stephenfin:I also update the test coverage patch https://review.opendev.org/#/c/707777/3/nova/tests/unit/policies/test_instance_actions.py@117, and I also cannot pass the test case, that cannot get what I want to get result, and I dont know why :(, error in my local is http://paste.openstack.org/show/790109/ | |
| 14:10:07 | brinzhang | stephenfin: Hope you can help, regarding this use case, thanks | |
| 14:31:03 | CeeMac | I'm having a major issues with block device mappings failing on new instance creation since I added 2 new compute/storage nodes yesterday | |
| 14:32:34 | CeeMac | i've enabled debug in the nova conf but that doesn't really add any extra information unfortunately, beyond waiting for block_device_map_created | |
| 14:33:31 | CeeMac | is it os-brick that creates the block device map? As there are no os-brick specific errors in the log. The volume is definitely being created, I can see this on the backend and I can see it under volumes, so this seems linked specifically to the block device mapping procress | |
| 14:33:42 | CeeMac | feeling a little lost on how best to proceed with troubleshooting | |
| 14:35:19 | CeeMac | actually, I tell a lie. The volumes are in an error state too, although I do see them getting created on the backend storage. | |
| 15:14:01 | openstackgerrit | Sérgio Filipe Marques Manso proposed openstack/nova master: test https://review.opendev.org/710466 | |
| 15:14:01 | openstackgerrit | Sérgio Filipe Marques Manso proposed openstack/nova master: test https://review.opendev.org/710466 | |
| 15:24:55 | efried | lyarwood: any chance you can help out CeeMac? (See logs from ~1h ago) | |
| 15:34:10 | CeeMac | efried: I think I may be making progress, and I may have missed a driver install out of my build process. Just working on that now. I've disabled cinder-volume on the new nodes and the rest of the stack is working again | |
| 15:35:08 | efried | Cool. (I don't have the chops to help you out -- or really even understand what you're saying :P -- but lyarwood is our resident storage-in-nova expert.) | |
| 15:40:39 | lyarwood | efried: just on a call now, will take a look once this is over | |
| 15:41:18 | efried | lyarwood: Thanks (I think CeeMac is okay for the moment, but I'll step aside) | |
| 15:49:38 | CeeMac | thanks for the intro/forward efried :) lyarwood I'm just rebooting the nodes after updating drivers for the storage interfaces, will let you know if that's fixed things once they're back up and I've retested | |
| 16:04:18 | lyarwood | CeeMac: ack np, let me know if you're still having issues after the compute reboots. | |
| 16:54:24 | CeeMac | lyarwood: we're all good thanks, was definitely me not documenting a vital step in the build process! Thanks anyway though :) | |
| 17:01:33 | lyarwood | CeeMac: haha no issues | |
| 17:08:45 | spatel | sean-k-mooney: good afternoon... | |
| 17:09:10 | spatel | I have quick question related CPUTopology with KVM. | |
| 17:09:32 | spatel | This is my Physical compute machine - http://paste.openstack.org/show/790120/ | |
| 17:09:50 | spatel | This is my Virtual machine running on that compute - http://paste.openstack.org/show/790121/ | |
| 17:10:53 | spatel | If you see physical machine sharing L1 cache but in Virtual machine each PU has own cache? | |
| 17:11:41 | spatel | I have compare this behavior with AWS and i am seeing they are correct and their output is very true like Physical | |
| 18:12:44 | spatel | How do i pass "<cache mode='passthrough'/>" to kvm | |
| 18:30:06 | melwitt | spatel: I don't know, but this is the closest doc I could find https://docs.openstack.org/nova/latest/admin/configuration/hypervisor-kvm.html#host-pass-through | |
| 18:31:25 | spatel | I already have host-pass-through on all my guest VM but its missing cache mode, that is why my guest VM showing wrong cache presentation | |
| 18:31:39 | melwitt | ah ok | |
| 18:31:56 | melwitt | only other thing I see is this but there's no 'passthrough' https://docs.openstack.org/nova/latest/configuration/config.html#libvirt.disk_cachemodes | |
| 18:32:08 | spatel | This is what i have | |
| 18:32:38 | spatel | if you are running openstack then can you check your CPU Topology? | |
| 18:33:57 | melwitt | sorry, you'll probably need sean-k-mooney again for this one | |
| 18:34:37 | spatel | yes.. just trying to understand what is wrong with my KVM, i have check with other cloud provide who running KVM they have correct CPU Topo | |
| 18:34:45 | spatel | that means something is missing myside.. | |
| 20:08:06 | openstackgerrit | sean mooney proposed openstack/nova master: Provider Config File: YAML file loading and schema validation https://review.opendev.org/673341 | |
| 20:08:07 | openstackgerrit | sean mooney proposed openstack/nova master: Provider Config File: Function to further validate and retrieve configs https://review.opendev.org/676029 | |
| 21:13:45 | openstackgerrit | Merged openstack/nova master: Fix H702 pep8 error with latest hacking https://review.opendev.org/710344 | |
| #openstack-nova - 2020-02-29 | |||
| 00:27:18 | openstackgerrit | Brin Zhang proposed openstack/nova master: Add test coverage of existing os-instance-actions policies https://review.opendev.org/707777 | |
| #openstack-nova - 2020-03-02 | |||
| 02:01:07 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Define Cyborg ARQ binding notification event. https://review.opendev.org/692707 | |
| 02:01:08 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: ksa auth conf and client for Cyborg access https://review.opendev.org/631242 | |
| 02:01:09 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Pass accelerator requests to each virt driver from compute manager. https://review.opendev.org/698581 | |
| 02:01:09 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Create and bind Cyborg ARQs. https://review.opendev.org/631244 | |
| 02:01:09 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Add Cyborg device profile groups to request spec. https://review.opendev.org/631243 | |
| 02:01:10 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Delete ARQs for an instance when the instance is deleted. https://review.opendev.org/673735 | |
| 02:01:10 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Compose accelerator PCI devices into domain XML in libvirt driver. https://review.opendev.org/631245 | |
| 02:01:11 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Enable start/stop of instances with accelerators. https://review.opendev.org/699553 | |
| 02:01:11 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Enable hard/soft reboot with accelerators. https://review.opendev.org/697940 | |
| 02:01:12 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Bump compute rpcapi version and reduce Cyborg calls. https://review.opendev.org/704227 | |
| 02:01:12 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Enable and use COMPUTE_ACCELERATORS trait. https://review.opendev.org/699554 | |
| 02:01:13 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Add cyborg tempest job. https://review.opendev.org/670999 | |
| 02:01:13 | openstackgerrit | Sundar Nadathur proposed openstack/nova master: Block unsupported instance operations with accelerators. https://review.opendev.org/674726 | |
| 03:28:49 | openstackgerrit | Brin Zhang proposed openstack/nova master: WIP: Support re-configure the delete_on_termination in server https://review.opendev.org/693828 | |
| 08:52:41 | lyarwood | https://review.opendev.org/#/c/708745/ - looking for some final core review on this to unblock the use of Libvirt >= v6.0.0 if any non-RH cores have time today. | |
| 09:42:20 | openstackgerrit | Lee Yarwood proposed openstack/nova master: WIP nova-next: Start testing the q35 machine type https://review.opendev.org/708701 | |