| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2018-07-10 | |||
| 10:20:04 | cdent | neha_alhat_: that's right, there is no "for use in code" placement client. This is because making your own is very straightforward. Have a look at the code around https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L270 for some ideas | |
| 10:20:08 | mdbooth | stephenfin: i.e. it runs unit tests which don't run in py27 | |
| 10:20:26 | stephenfin | I don't think so https://github.com/openstack/nova/blob/master/tox.ini#L249 | |
| 10:20:29 | gibi | stephenfin: I think you see this https://bugs.launchpad.net/nova/+bug/1779711 | |
| 10:20:29 | openstack | Launchpad bug 1779711 in OpenStack Compute (nova) "test_pre_live_migration_volume_backed* intermittently fails serialized json compare" [Critical,Fix released] - Assigned to Matt Riedemann (mriedem) | |
| 10:20:45 | stephenfin | gibi: Yup, that looks like the one and the same | |
| 10:21:14 | mdbooth | gibi: Whoa... | |
| 10:21:28 | mdbooth | gibi: Ah... dict ordering | |
| 10:21:44 | gibi | yeah, good old dict ordering | |
| 10:21:50 | stephenfin | gibi: Not the first thing I've been caught out by ordering in recent days :) | |
| 10:21:54 | stephenfin | *first time | |
| 10:23:02 | gibi | python 3.7 adds stable dict key ordering https://docs.python.org/3/tutorial/datastructures.html#dictionaries | |
| 10:23:03 | mdbooth | Aren't all dicts ordered in the newest anguine offering? | |
| 10:23:07 | stephenfin | fwiw, Python 3.6+ dicts are insertion ordered, which is rather nice | |
| 10:23:10 | mdbooth | gibi: Snap ;) | |
| 10:23:16 | stephenfin | mdbooth: double snap | |
| 10:23:19 | gibi | :D | |
| 10:24:12 | stephenfin | I'm sure whoever is working on nova in 10 years or whenever we can drop Python < 3.6 support will be very happy | |
| 10:31:44 | kashyap | (Maybe there won't be any.) | |
| 10:35:16 | neha_alhat_ | cdent: yes, will check. Thanks | |
| 11:13:22 | cdent | johnthetubaguy: you still happy with https://review.openstack.org/#/c/543262/ ? | |
| 11:34:12 | flwang1 | i'm going to ask again, can anybody help understand how to set the local_gb for hypervisor? thanks | |
| 11:43:22 | jroll | flwang1: that's set by nova-compute when it updates its resources. it looks at the size of the block device that instances are on | |
| 11:43:53 | flwang1 | jroll: thanks, then my question is how can i change it? | |
| 11:44:39 | jroll | change local_gb? I don't believe you can. why do you want to? | |
| 11:45:38 | flwang1 | after installed devstack, i found the local_gb is too small because it's using my root /, though i have almost 800GB at /home | |
| 11:45:49 | jroll | ah | |
| 11:45:53 | flwang1 | but my / only has 30GB | |
| 11:46:06 | jroll | flwang1: there's a config option for where the instance data lives, you can change that | |
| 11:46:52 | flwang1 | #instances_path = /opt/stack/data/nova/instances | |
| 11:46:52 | flwang1 | #state_path = /opt/stack/data/nova | |
| 11:46:52 | flwang1 | ? | |
| 11:47:13 | jroll | yes, instances_path | |
| 11:47:34 | flwang1 | i changed them both and doesn't work | |
| 11:48:02 | jroll | did you restart nova-compute (and give it time to update resources)? | |
| 11:48:29 | flwang1 | yes, i did. i restarted all nova services | |
| 11:49:27 | jroll | hm, that's odd. I'm not sure if there's something else that needs to be done, hopefully someone else can help, sorry | |
| 11:50:53 | flwang1 | jroll: no worries, thank you for the help | |
| 11:51:01 | jroll | :) | |
| 11:52:32 | efried | Anyone happen to know where it's documented how/where we, as developers, are supposed to use translation macros in code? | |
| 11:53:39 | flwang1 | jroll: i think that's the issue, after restart the nova-compute, it's still trying to talk to the old path | |
| 11:53:42 | flwang1 | not sure why | |
| 11:54:29 | jroll | yeah, that would be my guess | |
| 11:54:49 | jroll | efried: this is a good start: https://docs.openstack.org/oslo.i18n/latest/user/guidelines.html | |
| 11:54:59 | efried | Thanks jroll | |
| 11:55:00 | jroll | primarily "All text messages the user sees via exceptions or API calls should be marked for translation" | |
| 11:55:14 | efried | jroll: Yeah, I need something even more basic than that, for karimull. | |
| 11:55:41 | openstackgerrit | Yikun Jiang (Kero) proposed openstack/nova master: Fix all invalid obj_make_compatible test case https://review.openstack.org/574240 | |
| 11:55:47 | jroll | I'm not sure how to make that more basic :/ | |
| 11:56:52 | efried | jroll: See https://review.openstack.org/#/c/569498/10..11/nova/conductor/manager.py | |
| 11:59:22 | efried | I.e. need to describe what "marked for translation" means. I think that doc works. | |
| 11:59:37 | jroll | efried: yeah, I don't have anything better | |
| 12:00:06 | efried | karimull: After you've looked at the doc, let me know if you'd like some more background. | |
| 12:06:20 | sahid | stephenfin: thanks for your eye on this. I just replied | |
| 12:07:56 | openstackgerrit | Merged openstack/nova master: Handle compare in test_pre_live_migration_volume_backed* directly https://review.openstack.org/580702 | |
| 12:18:51 | mago_ | Hi team, I'm working on GPU Passtrough on OpenStack. I would like to limit the number of GPUs available to a user / project. There is no quotas for Passthrough resources and I was wondering why ? I was googling around but found no where a similar question. | |
| 12:22:37 | mago_ | Side question, as quotas are not only for Nova, but also for the Network and Block Storage services, is there a better place to ask my question ? | |
| 12:23:30 | openstackgerrit | Yikun Jiang (Kero) proposed openstack/python-novaclient master: Microversion 2.64 - Use new format policy in server group https://review.openstack.org/578261 | |
| 12:24:36 | efried | mago_: You mean like project-wide? Cause I would imagine there probably wouldn't be enough GPUs on a single host for quotas to make much sense. | |
| 12:24:49 | efried | mago_: but are you talking about physical or virtual GPUs? | |
| 12:26:55 | jaypipes | johnthetubaguy: https://review.openstack.org/#/c/543262/ could use your +2 restoration. | |
| 12:27:38 | mago_ | efried, well you could imagine a cluster with nodes with 4 to 8 GPU cards, if you have let's say 10 nodes, you can have up to 80 GPUs available. I think a quota would make sense. | |
| 12:27:44 | jaypipes | mago_: there is no quota class for either virtual or physical GPUs, no. | |
| 12:28:22 | mago_ | efried, I talk about GPU cards, passed to VMs using passthrough (also called Direct Device Assignment) | |
| 12:29:24 | efried | mago_: Yup, got it. I defer to jaypipes here, perhaps he can explain why there are no quotas. | |
| 12:30:08 | jaypipes | mago_: I can't explain *why* there are no quotas for GPUs. or at least, I can't explain that without some seriously opinionated rant about how our PCI manager system was created. | |
| 12:31:13 | mago_ | jaypipes, do you know why, I mean is it technically not possible because of the way quotas are implemented, or could it be imagined ? for GPUs, or for passthrough resources in general | |
| 12:32:12 | jaypipes | mago_: quotas were not considered by the creators of the pci management module in nova. | |
| 12:33:07 | jaypipes | mago_: when and if we move the management of PCI resources to the placement service, we will be able to use quotas for those PCI resource classes, but until then, it's not going to happen. | |
| 12:35:14 | mago_ | jaypipes, ok, understood. It does not seem to be an easy task, but do you know if this is already addressed / under discussion ? | |
| 12:38:13 | mdbooth | stephenfin gibi: Do we have something like a fixture which perturbs dicts? Running nova.tests.unit.virt.libvirt.test_driver.LibvirtConnTestCase.test_pre_live_migration_volume_backed_encrypted fails for me literally every other time when running only that test. | |
| 12:38:20 | mdbooth | It's weird. | |
| 12:39:22 | mdbooth | I know dict order isn't guaranteed, but all things being equal I'd expect it to be consistent in practise unless something is deliberately making it inconsistent. | |
| 12:39:37 | gibi | mdbooth: I don't think we have. I think the order depends on the PYTHONHASHSEED | |
| 12:39:38 | jaypipes | mago_: we've been planning on moving the queries for calculation of resource usage out of the nova-api service and towards the placement service for a while (https://review.openstack.org/#/c/509042/) but until we are tracking all the different types of resources using the placement service consistently, such an effort is only ever going to yield a small benefit. PCI devices, NUMA topology and dedicated CPUs, and certain networking resources | |
| 12:39:38 | jaypipes | like SR-IOV VFs are some of the resource types we do *not* currently track using the placement service. our work on nested resource providers, granular request groups, and the "reshaper" effort are paving the way to supporting these types of resources in the placement service. I predict that we'll have most of that work done by the end of the "T" release series, at which point the quotas-in-placement work can become a reality. | |
| 12:39:39 | efried | mdbooth: There's an open bug and proposed fix for this. | |
| 12:39:50 | mdbooth | efried: Yeah, we discussed it earlier. | |
| 12:40:11 | efried | mdbooth: Okay, upon rereading I figured that's not what you were getting at :) | |
| 12:40:27 | mdbooth | But even knowing the bug, I'm still surprised by the behaviour in practise. | |
| 12:40:56 | efried | mdbooth: You mean that there's some apparent predictability to the unpredictability of the hash ordering? | |
| 12:41:11 | mdbooth | I mean, I'd expect potentially different ordering if we were creating objects in a different order, or running things in a different order, or doing *anything* different. | |
| 12:41:38 | mdbooth | But in 2 consecutive absolutely identical test runs I'm surprised to see a difference in behaviour unless we're intentionally perturbing it somehow. | |
| 12:41:44 | gibi | mdbooth: try the same PYTHONHASHSEED | |
| 12:41:54 | mago_ | jaypipes, efried thank you very much for the explanations. | |
| 12:43:08 | gibi | mdbooth: I think if PYTHONHASHSEED=0 then you will get the same order every time | |
| 12:43:19 | gibi | mdbooth: but by default PYTHONHASHSEED=random | |
| 12:44:06 | mdbooth | gibi: PYTHONHASHSEED=1 ./.tox/lower-constraints/bin/python -m testtools.run nova.tests.unit.virt.libvirt.test_driver.LibvirtConnTestCase.test_pre_live_migration_volume_backed_encrypted | |
| 12:44:13 | mdbooth | Still inconsistent | |
| 12:44:22 | mdbooth | Meh, I was just curious | |
| 12:44:30 | mdbooth | This doesn't actually matter in the slightest :) | |
| 12:44:48 | gibi | :) | |
| 12:46:00 | jaypipes | mago_: np. sorry to not have great news! | |
| 12:53:30 | openstackgerrit | Surya Seetharaman proposed openstack/nova master: [WIP/POC] Updating queued_for_delete from instance_destroy() https://review.openstack.org/566813 | |
| 12:53:35 | jangutter | jaypipes: for SR-IOV VF's there's a rough a proxy: Neutron Ports can under quota.... but it's very very very leaky at best. | |
| 12:54:22 | jangutter | jaypipes: very difficult to see how that model would translate to GPU, though. | |
| 12:57:38 | efried | cdent, jaypipes: +A on https://review.openstack.org/#/c/543262/ -- great stuff. | |
| 12:58:13 | efried | johnthetubaguy: FYI ^ | |
| 13:03:27 | cdent | efried: On self.compute thing, it's been long enoug since I made that change that I can't remember the details :( | |
| 13:03:49 | efried | no worries; gate-bound in any case. If it passes zuul, it's good enough for me. | |