| Posted | Nick | Remark | |
|---|---|---|---|
| #openstack-nova - 2019-04-11 | |||
| 14:57:31 | sean-k-mooney | stephenfin: i share dansmith view that we shoudl enable inplace upgrades to this new way of doing things if we can do that i will be happy with this | |
| 14:58:30 | sean-k-mooney | stephenfin: that requrie either paralle implementions and a config to opt in to new beahvior or no change to exising configs | |
| 15:01:14 | bauzas | mriedem: sec, was otp | |
| 15:02:07 | mriedem | bauzas: somewhat related but you might have something to add to my reply here https://review.openstack.org/#/c/649534/5/nova/objects/request_spec.py@606 | |
| 15:02:08 | bauzas | uh, and now in meeting actually :( | |
| 15:02:18 | mriedem | bauzas: not high priority | |
| 15:07:31 | mriedem | stephenfin: can we not have a py3 unit test for https://review.openstack.org/#/c/650235/ ? | |
| 15:08:07 | stephenfin | mriedem: Not really, no. It's an environment thing | |
| 15:08:27 | mriedem | but can't we control the environment in a test? | |
| 15:09:21 | mriedem | btw zigo made the same change in osc https://review.openstack.org/#/c/541609/ | |
| 15:09:24 | stephenfin | Yes? No? I honestly don't know. We'd be monkeypatching Python internals, I suspect | |
| 15:09:44 | mriedem | idk about that, | |
| 15:09:48 | mriedem | nova's tox.ini sets this: | |
| 15:10:00 | mriedem | LC_ALL=en_US.utf-8 | |
| 15:10:36 | stephenfin | right, so the Python process is correctly configured in that case | |
| 15:10:59 | stephenfin | I think we'd have to reload the interpreter to misconfigure things | |
| 15:11:01 | sean-k-mooney | is this realted to the gat using the LC_ALL=C again | |
| 15:11:05 | stephenfin | yup | |
| 15:11:22 | mriedem | stephenfin: takashi also asked that something is documented about this which could probably go here https://docs.openstack.org/python-novaclient/latest/cli/nova.html#nova-console-log | |
| 15:11:54 | stephenfin | mriedem: I don't think OSC is an issue because of the follow up patch to the one you linked https://review.openstack.org/#/c/554698/1 | |
| 15:12:25 | mriedem | ah ok | |
| 15:12:40 | stephenfin | Alas, we don't use cliff in novaclient | |
| 15:12:53 | stephenfin | As for docs, it's on my todo list and I'll try drag something out by the end of the week | |
| 15:15:05 | stephenfin | mriedem: There's an alternative approach we can take that doesn't require changing environment configuration, but I don't know if we want to do it as it's a huge hack https://review.openstack.org/#/c/583535/ | |
| 15:15:14 | stephenfin | I'd nearly rather suggest people use Python 3 if they're encountering these kinds of Unicode issues | |
| 15:18:17 | mriedem | i'll see if i can add a unit test in novaclient | |
| 15:30:26 | melwitt | mriedem: thanks, will take a look. I'm in the middle of updating everything, will be able to push the updates soon today I think | |
| 15:34:34 | melwitt | soon™ | |
| 15:42:45 | lyarwood | melwitt: https://review.openstack.org/#/c/611974/ - finally got to this btw, LGTM after playing around with it locally. | |
| 15:43:08 | melwitt | lyarwood: just saw that, thanks so much | |
| 16:06:14 | openstackgerrit | Matt Riedemann proposed openstack/python-novaclient master: Add test for console-log and docs for bug 1746534 https://review.openstack.org/651827 | |
| 16:06:16 | openstack | bug 1746534 in python-novaclient "encoding error when doing console-log" [High,Fix released] https://launchpad.net/bugs/1746534 - Assigned to Thomas Goirand (thomas-goirand) | |
| 16:06:20 | mriedem | stephenfin: see how ^ grabs you | |
| 16:06:27 | mriedem | i couldn't reproduce the original bug in the unit test | |
| 16:08:53 | stephenfin | mriedem: Yup, that looks good to me. Good find with those click docs | |
| 16:13:10 | mriedem | need a stable core to hit these https://review.openstack.org/#/q/topic:bug/1821824+branch:stable/stein | |
| 16:15:20 | lyarwood | mriedem: have the branch open to review now btw | |
| 16:54:56 | openstackgerrit | Merged openstack/nova master: devstack: Remove 'tempest-dsvm-tempest-xen-rc' https://review.openstack.org/650018 | |
| 17:14:07 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif master: Remove IP proxy methods https://review.openstack.org/643115 | |
| 17:14:13 | openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/os-vif master: Refactor functional base test classes https://review.openstack.org/643101 | |
| 17:14:33 | lyarwood | mriedem: https://review.openstack.org/#/q/status:open+topic:bug/1803961 - Would you mind taking a look at this again when you have time. I'm going to suggest that we land and backport this over the competing cinder fix for the time being. | |
| 17:17:47 | openstackgerrit | Merged openstack/nova master: trivial: Remove dead nova.db functions https://review.openstack.org/649570 | |
| 17:48:46 | mriedem | lyarwood: couldn't get jgriffith or another cinder person to look at https://review.openstack.org/#/c/637224/ ? | |
| 17:51:53 | lyarwood | mriedem: I did a while ago and they pointed me towards https://review.openstack.org/#/c/638995/ but that has now stalled and I'm thinking it might just be easier to fix and backport in Nova given the cinder change is touching lots of different backends. | |
| 17:52:23 | lyarwood | mriedem: I can ask again to get confirmation that using migration_status is okay with them as a workaround until ^ lands. | |
| 18:22:20 | mriedem | melwitt: thinking out loud about quota and cross-cell resize and your placement change, when a user resizes a server today, placement will track vcpu/ram usage against both the source and dest node, but the /limits API will only show vcpu/ram usage for the new flavor since that is counted from the instance right? | |
| 18:22:57 | mriedem | so i think if i'm resizing from a flavor with vcpu=2 to vcpu=4, usage from placement for that project will say a total of 6, but the compute limits API would say 4 | |
| 18:23:49 | mriedem | i'm not necessarily saying that's wrong, but is that accurate? | |
| 18:24:41 | melwitt | yeah, it's definitely not going to say 6, but I don't remember if it will say 2 or 4 before the resize is confirmed | |
| 18:24:57 | mriedem | placement would say 6 | |
| 18:25:14 | melwitt | yeah | |
| 18:25:53 | melwitt | looking at the code to see when the new flavor is saved to the Instance object vcpus and memory_mb attributes | |
| 18:25:59 | mriedem | once the server is resized the limits api would say 4 https://github.com/openstack/nova/blob/03322bb517925a9f5a04ebdb41c3fd31e7962440/nova/objects/instance.py#L1535 | |
| 18:26:07 | melwitt | those attributes are what's counted today | |
| 18:26:38 | mriedem | it's finish_resize on the dest host https://github.com/openstack/nova/blob/03322bb517925a9f5a04ebdb41c3fd31e7962440/nova/compute/manager.py#L4632 | |
| 18:26:49 | mriedem | same method that changes the instance status to VERIFY_RESIZE | |
| 18:27:44 | melwitt | ah ok | |
| 18:28:34 | mriedem | maybe this is part of why we were talking about adding consumer types to placement so nova could ask for usage for 'instances' to filter out the usage tracked by the migration record holding the old flavor usage | |
| 18:30:22 | mriedem | anyway, it's a thing for cross-cell resize because while the instance is in VERIFY_RESIZE status we'll have the instance in both the source and target dbs and if we're counting from the dbs we don't want to double count the instance, so i need to filter out the hidden one, | |
| 18:30:30 | mriedem | but that got me thinking about how vcpus/ram will be counted | |
| 18:31:15 | melwitt | yeah. food for thought | |
| 18:31:30 | mriedem | i'm assuming no one wants to think about that or eat that food though | |
| 18:31:37 | melwitt | of course not | |
| 18:33:13 | openstackgerrit | Merged openstack/nova stable/rocky: doc: Fix openstack CLI command https://review.openstack.org/648425 | |
| 18:33:21 | openstackgerrit | Merged openstack/nova stable/rocky: doc: Capitalize keystone domain name https://review.openstack.org/650601 | |
| 18:33:40 | melwitt | this was brought up before in earlier discussions about counting from placement I think, and IIRC some (dansmith?) thought if placement is consuming 6 resources at that point in time, it makes sense for quota usage counting to reflect that as well | |
| 18:34:13 | dansmith | it depends on the resource and the direction | |
| 18:34:27 | melwitt | when do the old allocations go away from placement? VERIFY_RESIZE or after CONFIRM_RESIZE? | |
| 18:34:28 | dansmith | ideally we would only consume max(old_cpus, new_cpus) from quota | |
| 18:34:36 | dansmith | (and placement | |
| 18:34:41 | dansmith | because that's all we need to revert | |
| 18:34:51 | dansmith | but potentially we need to claim sum(old_disk, new_disk) depending | |
| 18:35:03 | melwitt | I see | |
| 18:35:04 | dansmith | and always both if we're not on the same node of course | |
| 18:35:49 | mriedem | right it's a mess and resize to same host doesn't help https://bugs.launchpad.net/nova/+bug/1790204 | |
| 18:35:51 | openstack | Launchpad bug 1790204 in OpenStack Compute (nova) "Allocations are "doubled up" on same host resize even though there is only 1 server on the host" [High,Triaged] | |
| 18:36:14 | mriedem | melwitt: the old allocations go away on confirm | |
| 18:36:24 | melwitt | ack | |
| 18:36:55 | mriedem | it's probably fair to say that our internal resource tracking and quota usage reporting during resize have just never aligned | |
| 18:37:13 | mriedem | the resource tracker would always report usage for the old flavor on the source node while resized to save room for a revert, | |
| 18:37:20 | mriedem | but the quota usage wouldn't reflect that | |
| 18:37:47 | mriedem | i don't know if the old pre-counting reservations stuff held some quota during a resize or not | |
| 18:38:17 | mriedem | i.e. did we hold a reservation on the target host and then /limits + reserved would report 6 in this scenario rather than 4, idk | |
| 18:38:22 | mriedem | no one would probably notice | |
| 18:44:23 | melwitt | mriedem: doesn't look like it. from this, you would fail quota check if you didn't have room to revert https://github.com/openstack/nova/blob/stable/ocata/nova/compute/api.py#L3201 | |
| 18:48:10 | mriedem | that code is all black magic to me, i'd have to setup an ocata devstack to see what happens in the API | |
| 18:48:30 | mriedem | definitely not a priority, was just thinking about it while i write a functional test for this for my cross-cell series | |
| 18:48:32 | melwitt | it does reserve for the new flavor here though at the beginning of the resize https://github.com/openstack/nova/blob/stable/ocata/nova/compute/api.py#L3351 | |
| 18:50:05 | melwitt | if it's an upsize | |
| 18:53:31 | melwitt | so from what I can tell, it will consume quota usage for the new flavor as soon as the resize starts | |
| 18:53:57 | melwitt | only for an upsize | |
| 18:54:21 | melwitt | otherwise it will consume for the old flavor still | |
| 18:55:22 | mriedem | ok so pre-pike the RT would report usage for both the old and new flavor and /limits API would show usage for the old and new flavor (assuming upsize), and starting in pike we'll only report usage for the new flavor | |
| 18:55:28 | mriedem | which could also be a downsize | |
| 18:55:44 | mriedem | maybe i drop vcpu but bump disk or something | |
| 19:00:37 | melwitt | no, I think the limits API would show the usage for only the new flavor because it would be upsize_delta + current usage (old flavor) | |
| 19:00:43 | melwitt | pre-pike | |
| 19:01:34 | melwitt | sorry I wasn't clear that it reserves the delta between old and new | |
| 19:21:11 | mriedem | incoming | |
| 19:21:47 | openstackgerrit | Merged openstack/nova stable/rocky: Add functional regression test for bug 1669054 https://review.openstack.org/649325 | |