| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-09-23 | |||
| 13:26:51 | mriedem | sean-k-mooney: yes it does | |
| 13:27:05 | mriedem | osc by default (1) requires you specify a host and (2) defaults to 2.1 | |
| 13:27:08 | mriedem | which by default forces the host | |
| 13:27:19 | sean-k-mooney | :( | |
| 13:27:31 | sean-k-mooney | ok i know 1 but not 2 | |
| 13:27:36 | mriedem | that is deprecated in osc 4.0 with something i added in train | |
| 13:27:55 | mriedem | https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/server.html#server-migrate | |
| 13:28:01 | mriedem | --live <hostname> is deprecated | |
| 13:28:15 | mriedem | but for rhosp customers using queens that's going to be an open issue for a long time | |
| 13:28:30 | sean-k-mooney | unless we backport it | |
| 13:28:45 | bauzas | anyway, nevermind | |
| 13:28:45 | sean-k-mooney | that said i dont know if we can do that | |
| 13:28:48 | stephenfin | we'll backport it, I imagine | |
| 13:29:03 | mriedem | we == red hat | |
| 13:29:10 | sean-k-mooney | well we cant backprot the api change downstream | |
| 13:29:19 | mriedem | the api change is in queens, that's not your problem | |
| 13:29:23 | mriedem | it's the client side tooling | |
| 13:29:28 | stephenfin | well you were talking about rhosp, so yes, clearly Red Hat | |
| 13:29:33 | sean-k-mooney | oh ok | |
| 13:30:20 | sean-k-mooney | i guess we should add it to our backlog to consider. the question is can we do it without breaking people | |
| 13:30:28 | sean-k-mooney | if we cant we cant backport | |
| 13:31:07 | sean-k-mooney | that said just adding --live-migration shoudl be fine | |
| 13:31:25 | stephenfin | efried, gibi, (dansmith): Interesting issue here https://review.opendev.org/#/c/663102/11/nova/tests/functional/libvirt/test_numa_servers.py@1060 | |
| 13:32:44 | stephenfin | efried, gibi, (dansmith): As the comment suggests, that test is working by creating two hosts, one of which doesn't have the second node to which a given physnet is associated https://review.opendev.org/#/c/663102/11/nova/tests/functional/libvirt/test_numa_servers.py@855 | |
| 13:33:20 | stephenfin | and we're doing that because it's not possible to have different configuration for different compute "services" in functional tests | |
| 13:33:52 | stephenfin | however, there's a check that prevents us doing just this (configured a physnet for a NUMA node that doesn't exist) that the test simply wasn't triggering | |
| 13:34:57 | stephenfin | because it's only hit as part of the calculation of the host NUMA topology object, which happens in the 'update_available_resource' periodic task, which our test wouldn't normally have time to trigger | |
| 13:35:31 | efried | not having looked yet, just reacting to ^, we have a way to trigger that periodic | |
| 13:35:39 | efried | but it's also triggered as part of instance creation fwiw | |
| 13:36:10 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Error out interrupted builds https://review.opendev.org/666857 | |
| 13:36:16 | stephenfin | it seems to be triggered when we query against the placement fixture https://review.opendev.org/#/c/663102/11/nova/tests/functional/libvirt/test_numa_servers.py@82 | |
| 13:36:50 | stephenfin | and that's the issue - by adding that query, we end up in a situation when the compute service for that falls over | |
| 13:37:15 | stephenfin | aaactually, maybe it was falling over already and we just never thought to check for that | |
| 13:37:20 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Pull up compute node queries to init_host https://review.opendev.org/682680 | |
| 13:37:48 | stephenfin | which is the real reason we're getting NoValidHost - not because the compute service can't support the request but because it never actually started | |
| 13:38:51 | sean-k-mooney | stephenfin: you could check that by looking at the comptue service logs | |
| 13:39:21 | stephenfin | that's what I'm doing as we speak | |
| 13:39:22 | stephenfin | :) | |
| 13:40:24 | stephenfin | yup, it never even started \o/ | |
| 13:40:40 | sean-k-mooney | i dont know the specific incantation to do that in the functional test but i have seen mriedem and gibi asserting log messages are emmited in the functional test form different agent in the past | |
| 13:41:01 | mriedem | self.assertIn('whatever', self.stdlog.logger.output) | |
| 13:41:15 | gibi | sean-k-mooney: you can access the logs in from the logger fixture | |
| 13:41:34 | gibi | sean-k-mooney: self.stdlog.logger.output | |
| 13:41:40 | gibi | sean-k-mooney: but it is not per service | |
| 13:41:50 | sean-k-mooney | cool ya its the self.stdlog.logger.output bit that i had seen but not used personally in the past | |
| 13:46:41 | gibi | mriedem: do you think this qualifies as a rc candidate https://bugs.launchpad.net/nova/+bug/1844993 ? | |
| 13:46:42 | openstack | Launchpad bug 1844993 in OpenStack Compute (nova) "migrate a server with qos port with compute RPC pinned to 5.1 fails and leaves the qos port in an inconsistent state" [Undecided,In progress] - Assigned to Balazs Gibizer (balazs-gibizer) | |
| 13:47:13 | openstackgerrit | Stephen Finucane proposed openstack/nova master: Remove 'test_cold_migrate_with_physnet_fails' test https://review.opendev.org/683961 | |
| 13:47:24 | stephenfin | gibi, sean-k-mooney: ^ (mriedem too, maybe) | |
| 13:47:37 | gibi | stephenfin: looking | |
| 13:47:48 | efried | gibi: was that a regression in train? | |
| 13:48:04 | efried | (the qos one) | |
| 13:48:11 | gibi | efried: not, it is a bug in the bandwith + migration code we merged to Train | |
| 13:48:20 | mriedem | gibi: technically no because it was a regression in stein | |
| 13:48:32 | mriedem | gibi: but it would be good to get it fixed in train regardless | |
| 13:49:23 | mriedem | gibi: oh i guess that's a side effect of the other bug right? | |
| 13:50:21 | sean-k-mooney | stephenfin: so the fact we cant confiugre different compute nodes with different config has come up afew times recently | |
| 13:50:42 | sean-k-mooney | we could remove that test but should we file a bug to add support for that in the future | |
| 13:50:51 | gibi | mriedem: your migrate bug is visible if the rpc is pinned to 5.0 | |
| 13:51:01 | gibi | mriedem: the qos migrate bug is visible even if the rpc is pinned to 5.1 | |
| 13:51:20 | sean-k-mooney | stephenfin: i think there has to be a way to use mocking to make the CONF non global | |
| 13:51:59 | sean-k-mooney | and in effect then allow us to have different confics in different agents althoug set_flags likely wont work in that context | |
| 13:52:33 | mriedem | gibi: right so my bug, introduced in stein, is a problem if computes are pinned to rocky (5.0) and yours if computes are pinned to stein (5.1) | |
| 13:52:47 | mriedem | gibi: then yeah it's probably worth tagging for rc1 | |
| 13:53:03 | gibi | mriedem: OK, then we are on the same page | |
| 13:55:01 | gibi | mriedem: I will respin the fix based on your rpc api method suggestion. I haven't seen the live migration way yet | |
| 13:56:34 | mriedem | it's just a simple abstraction so the caller doesn't need to know the router.client or version internals | |
| 13:57:21 | gibi | stephenfin: theortically you can go and mock things via self.compute['test_compute1'].manager.driver and that will be service and driver selective | |
| 14:07:27 | openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reject migration with QoS port from conductor if RPC pinned https://review.opendev.org/683948 | |
| 14:25:24 | mriedem | oh god schedule_and_build_instances is too big, have to write 100 LOC test case just to test one new line of code | |
| 14:43:05 | kashyap | stephenfin: Hey, about? | |
| 14:43:21 | kashyap | stephenfin: When you are -- | |
| 14:43:30 | stephenfin | on a call but yeah | |
| 14:43:50 | kashyap | stephenfin: How much of this is possible _today_ in Nova? https://kashyapc.fedorapeople.org/NUMA-pinning.txt | |
| 14:43:58 | kashyap | stephenfin: Sure, respond when you can | |
| 14:44:45 | stephenfin | 1 and 2 | |
| 14:44:50 | kashyap | stephenfin: Context is a QEMU dev was asking about it. | |
| 14:44:59 | stephenfin | we don't provide a way to configure IOThreads | |
| 14:45:04 | kashyap | Yeah, I see from the code that we don't do anything for IOThreads | |
| 14:45:13 | efried | mriedem: what would it take to split tempest-slow-py3 into two parts? | |
| 14:45:16 | stephenfin | and I don't know what a PXB device is so let's assume not for that too | |
| 14:46:51 | kashyap | stephenfin: For that, `grep` for 'pci-expander-bus' here: https://libvirt.org/formatdomain.html | |
| 14:46:56 | mriedem | efried: why? | |
| 14:47:21 | efried | mriedem: That would make a run take ~1.5h rather than ~2.5h | |
| 14:47:29 | efried | that last hour is always just waiting for tempest-slow-py3 to finish. | |
| 14:47:40 | openstackgerrit | Merged openstack/python-novaclient master: Update master for stable/train https://review.opendev.org/683627 | |
| 14:47:42 | kashyap | stephenfin: So the suggested IOThreads 'formula' for management tools is, _assuming_ we know the required info here: | |
| 14:47:50 | mriedem | efried: what do you mean the last hour is waiting for the job to finish? | |
| 14:47:54 | mriedem | for tempest to finish? | |
| 14:47:56 | mriedem | or to collect logs? | |
| 14:48:06 | kashyap | stephenfin: num_iothreads = min(num_devices, num_vcpus, num_host_cpus) | |
| 14:48:18 | kashyap | Thanks for the answer so far | |
| 14:49:24 | efried | mriedem: I mean that for a given patch, all the other jobs take <=1.5h to complete, but tempest-slow-py3 takes ~2.5h. Trying to reduce the wait time for a single patch to make it through CI. | |
| 14:50:26 | openstackgerrit | Matt Riedemann proposed openstack/nova master: Sanity check instance mapping during scheduling https://review.opendev.org/683730 | |
| 14:50:58 | mriedem | efried: well, there are a few options. one is to go the way of tempest-integrated-compute and make a compute-specific job that only runs compute tests we care about (sort of like nova-next) but only runs slow tests (which tempest-integrated-compute does not), | |
| 14:51:12 | mriedem | the other major difference is tempest-slow* is a multinode job where nova-next and tempest-integrated-compute are single node | |
| 14:51:26 | mriedem | the multinode + slow combo is important to run some tests that we don't otherwise run | |
| 14:51:45 | mriedem | one idea is just drop tempest-slow-py3 from nova's job list and make nova-next multi-node | |
| 14:51:56 | mriedem | nova-next runs compute api and scenario tests, including slow | |
| 14:52:04 | sean-k-mooney | one factor for multinode jobs is it only starts stacking the second node after the first completes | |